Customer service and generative AI: a uniquely transformational moment for the industry

In recent months, ChatGPT and its successor GPT-4 have set the world of business and technology abuzz by demonstrating an ability to compose human-like responses to a wide range of queries and prompts.

This has given rise to widespread speculation about how generative AI – a collective term for artificial intelligence tools that generate text, code or other media – could disrupt everything from web search to marketing, content writing to the legal profession.

While most of these predictions have yet to be borne out, one industry is already seeing a transformative impact: customer service.

“There is now the possibility of driving a real transformational change in how support is delivered,” says Declan Ivory, VP Customer Support at software company Intercom, which specialises in business messaging and customer service. “[Before the advent of ChatGPT], there hadn’t been major transformation in relation to how support is delivered for quite some time.”

I spoke to Ivory about the ways that Intercom has been using generative AI in its customer support toolset, the results the company has seen so far, and how Intercom works to mitigate the technology’s shortcomings – as well as how he predicts the advent of generative AI will change customer service as a whole.

What can ‘AI in the inbox’ do?

From the launch of ChatGPT on 30^th November 2022, Intercom’s team took just eight weeks to announce the integration of new generative AI-powered features into its range of customer support tools for businesses. To begin with, the new features were not designed to be external-facing – i.e., customers would not be interfacing directly with a GPT-powered chatbot – but rather to make life easier for support agents behind the scenes.

“We’ve used the technology to power a set of what we call ‘AI in the Inbox’ features, and they are primarily focused on support agent productivity,” Ivory explains. “It’s a tough job – there’s a lot of things that support agents have to handle – and anything we can do to alleviate that part of the role … is always appreciated.”

Among the features that Intercom has introduced are a conversation summarisation tool, which facilitates handovers between support agents by automatically condensing the exchange with a customer into key bullet points. “It takes a lot of time and effort [when] taking over a case to try and understand what has happened to date, what’s the status, where exactly are we … The value of being able to summarise what has gone on in that case, or conversation, and make it available to someone is really appreciated by the teams,” says Ivory.

Generative AI can also accomplish things like adjusting the tone of a response to suit the customer – Ivory notes that some customers like an informal interaction, while others prefer more formality – or rephrasing sentences to vary the wording of a response. Another feature, Expand, can take a short note written by a support agent and expand it into a fully-fledged reply. Intercom’s team has been using all of these tools internally behind the scenes, as well as making them available to clients in a private beta, which enables them to refine the features based on feedback.

“Well over 100 clients are using the beta features from an early stage,” Ivory says. “They’re using terms like ‘game-changer’. … Our customers are really excited about it – they’re saying [that there’s] the potential to make the role of the support agent a lot better, and that has been our initial focus in terms of where we’re applying this technology.”

“I don’t see the [QA] challenge as any different… to managing human responses”

While Ivory says it is difficult to quantify the exact impact of using these tools due to the varying complexity among businesses, he estimates that the summarisation tool alone may save 10-15 minutes with each use. I ask whether generative AI has proven to be accurate so far in reflecting the content of the conversations it summarises.

“In general, yes – [is it] a hundred percent, no; I don’t think anything is ever one hundred percent,” says Ivory. “The team have been quite impressed with the ability of the tool to pick up all the key points in the conversation and reflect them back.

“It is the exception where it doesn’t pick up something, as opposed to the rule … The tool is probably as good as a human in terms of picking out all the key points and making sure that they’re summarised.”

Similarly, when detailing how Intercom guards against the potential for ‘hallucinations’ (when a generative AI confidently states erroneous information), Ivory points out that human agents are also not infallible. “If [humans] are looking at a body of information, sometimes they can misinterpret it; sometimes they can give a bad response to a customer,” Ivory says.

Humans make mistakes; technology makes mistakes. This is not a new world.

Quality assurance (QA) is a concern for any company offering customer support, and Intercom approaches the issue of AI-powered customer service in the same way. “You have to have a QA process around – what responses is the automated technology providing to the customer? How do you train it to make sure that it gives the right answers? I don’t see the challenge as any different to the challenge around managing human responses,” says Ivory.

“Humans make mistakes; technology makes mistakes – this is not a new world. It’s about how you put the right bounds around the technology to mitigate anything that may not work exactly as you want it.”

A useful framing is to move from “quality assuring the agent”– i.e., quality assuring the actions of an individual support agent, which is how organisations typically approach QA in customer service – to paying attention to the overall quality of the customer experience, regardless of how it’s delivered. “That experience can be delivered by a bot, or it could be delivered by a human … you’re making sure that it’s meeting the standards that you’ve set for your industry and your business around what is a high-quality experience for the customer.”

“One of the most exciting times to be involved in the customer service industry”

Ivory has been working in customer support for “many, many decades”, but he sees this as a uniquely transformational moment for the industry.

“There’s been lots of almost point applications of some of this technology – Natural Language Processing has been around for a while, you can do sentiment analysis, you can use AI to determine trends – but not pervasively, not at scale, not in terms of an overarching view of how you can change support,” he explains.

“What I think ChatGPT really sparked in people was [realising that] the technology had matured to the stage that you could start thinking about wide applicability across the life cycle of support. There’s the support agent piece, but obviously when you expose that technology to customers, there’s a whole set of other opportunities in terms of how you transform the support experience.”

…we’re starting to measure performance primarily based on resolution rates. So far, they are exceeding our expectations.

While Intercom initially held back from incorporating generative AI into any customer-facing tools, citing concerns about the risks presented by hallucinations, the company has since announced that it is trialling Fin, a chatbot powered by GPT-4. Fergal Reid, Director of Machine Learning at Intercom, explained over email that when testing GPT-4, the team “felt that the technology had crossed the threshold where it could be used in front of customers.”

He went on, “It’s still early days for Fin, but we’re starting to measure performance primarily based on resolution rates. So far, they are exceeding our expectations. … We’re also continuing to pay attention to Fin’s ability to communicate in a human-like way and adhere to the guardrails we put in place to avoid misleading information and hallucinations.”

“In Intercom, we want the experience to be conversational, very natural,” says Ivory. “And I think it can be sometimes difficult for customers to feel – if they’re engaging with chatbots – that it is conversational, that it is contextual and personalised. Generative AI now has the ability to make that engagement really smooth, and can leverage all the data that a support agent has access to in terms of understanding the customer context.”

Ivory notes that the technology is maturing amid heightened customer expectations and economic pressures, making businesses all the more motivated to pull out the stops to retain customers and optimise their costs. “All those things are colliding to say, ‘This is the time to think about applying this technology. This is the time to really transform the support experience.’

“And it’s probably one of the most exciting times to be involved in the customer service industry – because I really think that transformation can be delivered through the advances that we’re now seeing.”

All those things [heightened customer expectations and economic pressures] are colliding…

The impact on support agents

What about the longer-term impact on human support agents? We’re all familiar with the ‘robots taking our jobs’ trope and hysteria, which has varying degrees of likelihood depending on the job in question. However, Ivory is adamant that it is “absolutely not” the end of the support agent role.

“[The role of the support agent is] going to change and pivot in different ways. If you’re using AI and automation to take out repetitive, simple tasks, by default, what’s coming through to your human support team should be more complex tasks, and things that require that level of understanding.

“So on one level, that can enrich the role of the support agent – they have the opportunity to specialise and gain more skills in whatever domain they’re operating in. People want to specialise – they want to be able to become experts in their domain. From that point of view, that’s a positive thing.”

A recent study by Stanford University and Massachusetts Institute of Technology on a Fortune 500 company that implemented a generative AI-based conversational assistant to aid its customer support team, however, found that while the tool improved agent productivity by 14% overall, the gains were largely enjoyed by less-experienced and less-skilled workers, with the most highly-skilled workers only experiencing a slight increase in calls handled per hour, and even seeing a slight decrease in resolution rates and customer satisfaction.

Researchers speculated that this could be due to agents feeling distracted by the generative AI suggestions, which they did not need in order to effectively carry out their jobs. The net effect on lower-skilled workers was to move them much more quickly along the ‘experience curve’, enabling them to improve much more rapidly without the need for as much coaching, potentially because the AI – trained on successful customer service interactions – helps to disseminate tips and tricks already internalised by experienced agents.

Another benefit from introducing the AI assistant was increased levels of employee retention, particularly among newer agents – crucial for an industry with notoriously high rates of turnover. In theory, this could lead to more agents staying in their roles long enough to gain more experience, benefiting the overall experience levels in the organisation. However, the researchers stressed that the study “is not designed to shed light on the aggregate employment or wage effects of generative AI tools.”

Ivory foresees that new roles could emerge in customer support that focus on the seamless integration of AI into the support workflow. “As an example, there’s this concept of a conversation designer: thinking about how you design the conversation or flow with the customer from the time they engage through to the time you issue a resolution to their question.

“I think you have to be very deliberate around how you design that conversation – it doesn’t happen by default. What’s the handoff [from AI] to human support? How do the two interact with each other?”

A much-talked-about new role in the wake of generative AI adoption is that of ‘prompt engineer’, which involves devising prompts that are most likely to get the desired result from a generative AI, and/or devising prompts that can be used to test an AI chatbot and improve its responses. Ivory believes that this skill will also be valuable in ensuring customer queries are interpreted by the chatbot in the right way: “The customer may ask a question in one way, but another customer may ask the same question in a different way. How do you think about all those different ways that the technology could be asked a question, and make sure they’re factored into how you train the technology?”

These new roles would be slightly different to the normal support agent role, but would be “great career advancements” for workers in customer support, says Ivory. “There’ll be more emphasis put on support agents being able to identify the opportunities for continuous improvement, which include automation and application of AI technology.

“They’ll become the people who point the conversation designers, the prompt engineers in the right direction: ‘Here’s where we can make a change in terms of the customer experience and deliver in a better way for the customer, and in a more cost-effective way for the business.’ And I think organisations will tap into the knowledge of their support teams for that.”