8 questions or concerns about what to expect on the day
In example (5), the user asks a question about the MRI scan (l. 1). The
user reacts to the bot’s repair initiation (l. 2-3) by accommodating
through choosing one of the bot’s prompts (l. 4). Whilst the bot’s
answer may not be an altogether satisfactory response to the question,
it concludes this sequence and moves the conversation on.
In contrast, users who use the strategies with a lower success ratio
tend to have to work harder to have their intents understood and ensure
progression:
Example (6)
1 Please select a date and time for your appointment 18:00 19:00
None
2 7pm
3 Sorry, but I could not reschedule the appointment. Please try
again a bit later, or try re-booking
4 with different date/time. If however you are still facing issues
I can connect you with someone
5 from the clinic to help you out, just type - speak to someone.
Thanks, Asa
6 I meant 19:00
7 Did you mean? google *calendar* link for my scheduled
appointment *faq* - frequently asked
8 questions restart *None* of these
9 None of these
10 Apologies, bear with me. I’m still learning. Can you rephrase?
11 Can I reschedule
12 No worries. Please let me know the number of the appointment
you would like to reschedule: 1.
13 Screening on Thu 09 Jun ’22 at 19:00
In example (6), the trouble source is the user’s input representing
their desired appointment time (l. 2). The bot’s response is relevant,
but nevertheless initiates user repair by indicating that the desired
appointment time is not available and by providing a number of options
for progressing the conversation to the user (l. 3-5). The user
completes repair by rephrasing their desired appointment time (l. 6).
The user’s repair is wordier than their previous version, with ‘I mean’
representing a personal mitigator akin to a negative politeness marker.
After another comprehension request by the bot (l. 7-8), the user
changes strategy and uses accommodation strategies by recycling one of
the bot prompts (l. 9). When this strategy is also unsuccessful, the
user’s next self-repair repeats the originally rescheduling request,
this time framing the request as conditionally indirect request (‘Can
I…’) (l. 10). This third attempt at user self-repair is leads to
a relevant response but forces the user to restart the rescheduling
conversation.
Discussion
Summary of results
The analysis presented in the previous section has showcased, through a
combination of qualitative and quantitative description of user repair
turn and subsequent bot follow-on turns, how users in a task-oriented
chatbot use repair to work through sequences in which their intent is
misinterpreted by a bot. In the subsequent discussion section, we will
summarize and discuss these insights and discuss them in relation to
developing communicative competence for conversational AI.
The study has shown that the main purpose of user self-repair is to
address issues in which the bot does not understand users’ intent or
misunderstands it. Users have a wide array of strategies available to do
self-repair, some of which are akin to face-to-face social interaction
and some of which are perhaps less so, e.g., changing choices originally
made.
The study has also shown that the self-repair strategy users deploy most
frequently – rephrasing – is actually one of the least successful
strategies, leading in approximately a half of all instances to
irrelevant bot answers (false positive) or explicit repair initiations
(false negative answers). On the contrary, restating the purpose has a
high success ration, but is only very infrequently deployed by users. In
addition, ‘accommodation’ – using the bot’s own prompts – and making
different choices also has a more than 50% success ratio.
This strongly suggests that users transfer strategies from face to face
spoken interaction into their conversations with Asa (see Nass & Moon,
2000). For example, we have shown that some users not only rephrase
their original turns, but also enhance them with additional politeness
markers. This mirrors strategies previously observed in face to
face-interaction. For example, Kasper (2006) observed that, in
subsequent requests, interviewers in an oral proficiency use the
politeness marker ‘please’ to frame an overall more direct request. Plug
(2014) observed that, in self-initiated self-repair, speakers engage in
“prosodic marking” through higher pitch, and higher speaking tempo,
and Hauser (2019) observed ‘upgraded’ self-repeated gestures in Japanese
interaction. Some of the self-repair strategies observed in this study
mirror these patterns.
Whether ‘upgraded’ politeness is a matter of applying behaviours from
face-to-face interaction, or is a result of other demands, remains to be
further investigated by further research. In doing so, the notion of
‘pragmatic transfer’ (Kasper, 1992) which traditionally has been used to
describe the transfer of L1 pragmatic strategies into an L2 and has
informed plethora of research in intercultural, cross-cultural and
interlanguage pragmatic since could be deployed to inform research on
human-AI interaction, too.
The analysis has also revealed further insights which may have
implications for users’ perceived rapport with the bot. As shown,
‘accommodation’ – defined here as convergence to the bot’s prompts in
self-repair is one of the most frequently used, and one of the more
successful strategies, though success is not necessarily guaranteed.
Yet, such ‘upwards convergence’ towards interactional partners in
superordinate positions (Giles & Ogay, 2007) may have implications for
users’ perceptions of rapport with the bot for two reasons. Firstly, the
conversational repair mechanisms observed here are ‘role-defining’
(Liebscher & Daily-O’Cain 2003) in that they define users’ role as that
of a respondent to rather than an initiator of interaction. Secondly,
having to converge to receive a relevant answer in the first place, and
such convergence actually often NOT leading to a relevant answer (see
table 4), has the potential for implications for users’ perceptions of
rapport with the bot (Spencer-Oatey, 2008). However, the exact rapport
implications in user-bot interaction need to be the subject of further,
more detailed investigations.
On the other hand, together with ‘restate purpose’, accommodation is the
strategy that has the highest potential success rate. Yet, user
awareness of this strategy and their readiness to use it, may depend on
users’ overall experience with chatbots and their general knowledge of
the workings of AI (Luger & Sellen, 2016). It may also depend on users’
overall orientations to these interactions: To what extent do they
perceive them as ‘relational’ (relationship-oriented), or as
‘transactional’ (task-oriented) (Koester, 2004)? To what extent do they
perceive of them as having human characteristics (anthropomorphism)
(Hermann, 2022)?
Implications for AI skills
As discussed above, we purport here that, in order to engage with
AI-driven chatbots efficiently, users need to acquire specific skills
that do not necessarily mirror those for spoken interaction. A range of
models of communicative competence have previously been used to describe
the competences required for social interaction. For example, Canale &
Swain (1984) describe communicative competence as consisting of four
components: grammatical competence (words and rules), sociolinguistic
competence (appropriateness), discourse competence (cohesion and
coherence) and strategic competence (appropriate use of communicative
strategies). This analysis of self-repair in user-bot interaction shows
that for effective engagement in these interactions three of these
competences are particularly important:
- Sociolinguistic competence: Users need to be able to be aware of the
sociolinguistic environment they encounter when they interact with the
bot, what this means for how they manage rapport and, consequently,
how they use language. This analysis has revealed that some users are
conflicted with respect to the sociolinguistic environment they are
working in: they may accommodate to bot by using its prompts or
restate the purpose of the interaction after an extended repair
period, but can also be seen to addressing Asa by name and ask
questions with markers of negative politeness which mitigate the force
of the request/question. Hence they are conflicted in their
orientation to the bot as a machine or as human-like (Waytz et al.,
2010), betraying some confusion about the rules of the interactional
game.
- Discourse competence: Users need to be able to assess how the way they
use language contributes to a coherent whole in user-bot interactions,
and whether different ways of using language may lead to different
outcomes. For example, how does the bot process and respond to a human
request to form an effective question-answer pair? As the analysis of
the user-bot interactions here have shown, it is useful for users to
know which repair strategies are most likely to lead to the bot
understanding their intent and trigger a relevant response, to then
form a coherent adjacency pair.
- Strategic competence: Strategic competence is at the core of
conversational repair as it describes the skills required for
misunderstanding, and for preventing misunderstanding. As our analysis
has shown, some repair strategies are inherently more successful than
others, and the criteria for success are different to regular face to
face interaction. The analysis has also shown that conversational
flexibility may be the most important criterion for strategic
competence. Users of conversational AI need to be able to recognise
when a strategy is not successful, and then dig deep into their
conversational arsenal to identify and then deploy alternative
strategies.
In summary, in the same way as pragmatic competence is not necessarily
developed alongside other aspects of communicative competence (Kasper &
Roever, 2005), communication skills for conversational AI are not innate
to all users (Luger & Sellen, 2016), suggesting that there may be value
in teaching these skills explicitly and helping users understand the
sociolinguistic environment they are operating in. An approach to
teaching skills for effective engagement in conversational AI would
benefit from including all of the elements previously described for the
development of communicative competence (e.g., Jones & Stubbe, 2004 for
professional communication; Dippold, 2015 for classroom interaction),
such as awareness raising, experiential learning, reflection. Indeed
Howard’s (2012) instructional paradigm for teaching CMC skills includes
all these elements, and Weisz et al.’s (2019) experimental account of
teaching strategies for successful human-agent interaction does so, too.
Their intervention included a phase on raising users’ empathy with the
bot to highlight the difficulties for dealing with user input. In this
study, the instructional intervention led users to report that they had
developed better strategies (e.g., use simple language, specify intent
precisely). Users also developed their understanding of the algorithmic
thinking process and learned to disambiguate its capabilities from human
capabilities. The study is, to my knowledge, the only one currently that
reports on an attempt to explicitly teach conversational AI skills. More
studies which focus on AI skills ‘in the wild’, using authentic rather
than simulated interactions, would supplement this work.
Limitations
This study has a number of limitations. Firstly, the data analysed here
were gathered as part of a methodological pilot project with a bot
product that was in a minimal viable product stage of its development.
The bot responses here therefore do not necessarily reflect its full
capability for dealing with user intents and initiating repair.
Finally, the data set this paper is based on is a relatively small
corpus of user-bot interactions. Further research with a larger data set
is necessary to further substantiate the patterns reported here. This
should, ideally, also include gathering data in the wild rather than a
simulated environment. When more is at stage, users’ repair strategies
may be different, e.g., for how many turns they persist in attempting
repair or what strategies they choose.
Nevertheless, the analysis has provided insights which follow-up
research can now further evaluate. For instance, further research could
further investigate whether and how speakers transfer pragmatic
strategies from spoken interaction into bot interactions. Secondly,
further research can ask how users’ repair strategies relate to their
expectations for rapport and their ideologies of language use.
Retrospective interviews asking in detail about users’ decision-making
processes when interacting with chatbots have the potential to be able
to tap into these questions.
In addition, it is of course also necessary to widen our gaze beyond
repair. Informal observations during my analysis suggest that user
success in transactional bot interactions also depends on other factors.
Many of the trouble sources for instances were longer, discursive
sentences. An interesting follow-up question is therefore to what extent
recognition and user progress through the interaction is contingent on
the form in which initial inputs are made, e.g. whether and which
politeness strategies are used.
In this research, this paper investigated repair strategies in a chatbot
whose main purpose was ‘transactional’ and which fulfilled tasks such as
booking and rescheduling. Further research needs to investigate the
users’ repair strategies in interactions with bots with a more social
orientation, e.g., bots which provide mental health support or
companionship.
Conclusion
Through the example of conversational repair, this paper has shown that
some users are able to navigate the communication challenges posed in
this environment very well, while some struggle working towards the
transactional goals that the bot is meant to help them achieve. Talking
with a chatbot requires a distinct skills-set, and these skills are
likely to be even more complex in chatbots which use voice interaction.
Hence, the future of human-bot interaction is not just about developing
bots that are more like people: they won’t be. It is about upskilling
people to interact with bots.
Further studies on the pragmatics of social interaction with chatbots
will help build the “bigger picture” of what these skills are and need
to be followed up with studies investigating how these skills can best
be taught. Developing users’ knowledge about the workings of AI will
ensure that all AI users have equitable access to the everyday services
bots support.