Let’s just get it out of the way at the start of this blog: COVID-19 has remade retail and has forced a rapid shift toward digital customer service interactions. And now the holidays are nigh, and e-commerce seems to be the order of the day (or the “plague year”). To help with this rush of online selling, many of you will be relying on the assistance of chatbots.

The Standard Metrics For Chatbot Performance Are Good, But . . .

You’ve designed your chatbot to help prospective customers identify the best gifts or configure the best solution. Therefore, your natural inclination is to judge the success of the chatbot through the prism of revenue generation. This makes sense — after all, you built the bot to help sell stuff and support customers who buy stuff from you, so you are probably already measuring some standard commerce metrics for chatbots such as conversion (which Forrester defines as “the rate of interactions focused on commerce that result in a sale through the chatbot”). Also, since many retailers use chatbots to help customers figure out what to buy but not for the actual selling, you may be looking at revenue influenced by the chatbot.

You’re also likely looking at metrics that center on the overall performance of the chatbot technology, such as intent recognition. It’s hard to get a customer excited about your amazingly tasty Nova-style lox shipped overnight when the chatbot misunderstood the customer and offers up information about incredibly secure padlocks. (C’mon, there must be some retailer out there that sells both cold-smoked fish and protection devices that use a shackle to secure items.) Additionally, you might also examine other statistics that will help you continuously improve the chatbot’s salesmanship, such as coverage, which allows you to see the topics the chatbot is trained to respond to as a percentage of all questions that come into the bot.

. . . Two Key Metrics Will Help You Future-Proof And Be More Strategic

These are all sensible starting points . . . but if you want the chatbot to be useful after the holidays are over, they can’t be the only things you measure. Forrester’s research points you to other areas you should be looking at:

  • Experience metrics. Here you’ll be looking not so much at how well the chatbot completes tasks but rather how users feel about their interactions with the chatbot. Some of these metrics are explicit measures of experience such as customer satisfaction, either derived from a survey or from text analytics run on the logs of conversations with the chatbot. But there are also metrics that try to peek around the corner to get a view of how these chatbots make consumers feel — for example, measuring active users, which measures how many users return to use the chatbot a second (or hopefully a third and a fourth) time in a given time period. The assumption underlying this metric is simple: If consumers hate the chatbot experience, they will do whatever it takes to avoid it in the future. If they stick with the chatbot, consumers must be finding some value in the experience.
  • Anticipatory customer service metrics. While most of the headlines around delays in mail deliveries in the United States have been focused on political implications, slower-than-expected postal service will also impact consumer behavior for holiday shopping. You should be monitoring and measuring the questions about customer service issues just waiting to happen. Customers may ask “What are realistic delivery dates?” or “Do you guarantee delivery by [insert date here]?” If you started your chatbot development before these issues arose, you may only spot these questions if you are examining confusion triggers, the utterances that cause the chatbot to ask for clarification. Or you might only figure out what is happening because you experience a rise in the rate of escalation to a human based on explicit customer requests.

Looking at these two categories of chatbot measurement will help you extend the useful life of your chatbot well beyond the holiday season. For more research that can help, see our reports on the right metrics to use and the best ways to scale your chatbot. Happy holiday selling!