01 - Motivation
At any company with significant customer support, the staff handling these cases often face the difficult task of keeping customers satisfied while also finding solutions to problems quickly. One technology client was in search of such a solution to helping their customer-facing staff search through records quickly and with as little effort as possible.
02 - Challenges
There were two main challenges with this client. The first was a natural language processing challenge. Many of the client's products were referred to by multiple names in the case files the customer support agents and they did not have a lot of staff that could spare time to help us. We would have to find a way to extract these files with minimal annotation.
Secondly, we had to determine an easy interface that wouldn't prove problematic or present even more challenges in the workflow. The database look up method the staff used up to that point took a different screen and was a cumbersome interface. Again, we had minimal staff to work with, so we would have to hope our user research would be adequate, and we would supplement with other users in a similar space we could interview.
03 - Data modelling
Given that we had to extract as many entities from the data as we could given limited annotation, we were lucky to draw on the work of Chris Ré's team at Stanford to help with building models by extracting a set of basic patterns we could identify, extract, verify with annotation, and then iterate quickly on using Snorkel.
Within twelve weeks, we build a prototype model with Snorkel with an acceptable precision rate. We felt that the key metric to meet was that of a low false positive rate: it was one thing to simply not have a result, but we didn't want for a staff member to get frustrated by getting bad search results repeatedly, or to give incorrect information to a customer.
03 - Interface
Sometimes, less really is more. The interface the customer support staff used at the time had more than a dozen parameters, which simply increased the time they spent searching for the cases they needed to examine rather than getting to the information they needed quickly.
We decided to try a chatbot as a new solution to the problem - while the parameters could be included in a conversation with a bot, the bot could simply do one search based on what information it was given and able to extract from the inquiry. It would run in a separate application on the customer support staff's machine rather than in a browser. The staff used the browser for the accessing the customer's information, and having a separate application gave a slight cognitive difference that the staff we tested with found appealing.
03 - Result
Our client chose to move in another direction with the work we did with them rather than pursue the VirtualEngineer project. Instead, we used the work as a prototype to show other clients the potential for working with intelligent agents as a path forward for a search application in a work space.