8 questions or concerns about what to expect on the day
In example (5), the user asks a question about the MRI scan (l. 1). The user reacts to the bot’s repair initiation (l. 2-3) by accommodating through choosing one of the bot’s prompts (l. 4). Whilst the bot’s answer may not be an altogether satisfactory response to the question, it concludes this sequence and moves the conversation on.
In contrast, users who use the strategies with a lower success ratio tend to have to work harder to have their intents understood and ensure progression:
Example (6)
1 Please select a date and time for your appointment 18:00 19:00 None
2 7pm
3 Sorry, but I could not reschedule the appointment. Please try again a bit later, or try re-booking
4 with different date/time. If however you are still facing issues I can connect you with someone
5 from the clinic to help you out, just type - speak to someone. Thanks, Asa
6 I meant 19:00
7 Did you mean? google *calendar* link for my scheduled appointment *faq* - frequently asked
8 questions restart *None* of these
9 None of these
10 Apologies, bear with me. I’m still learning. Can you rephrase?
11 Can I reschedule
12 No worries. Please let me know the number of the appointment you would like to reschedule: 1.
13 Screening on Thu 09 Jun ’22 at 19:00
In example (6), the trouble source is the user’s input representing their desired appointment time (l. 2). The bot’s response is relevant, but nevertheless initiates user repair by indicating that the desired appointment time is not available and by providing a number of options for progressing the conversation to the user (l. 3-5). The user completes repair by rephrasing their desired appointment time (l. 6). The user’s repair is wordier than their previous version, with ‘I mean’ representing a personal mitigator akin to a negative politeness marker. After another comprehension request by the bot (l. 7-8), the user changes strategy and uses accommodation strategies by recycling one of the bot prompts (l. 9). When this strategy is also unsuccessful, the user’s next self-repair repeats the originally rescheduling request, this time framing the request as conditionally indirect request (‘Can I…’) (l. 10). This third attempt at user self-repair is leads to a relevant response but forces the user to restart the rescheduling conversation.
Discussion
Summary of results
The analysis presented in the previous section has showcased, through a combination of qualitative and quantitative description of user repair turn and subsequent bot follow-on turns, how users in a task-oriented chatbot use repair to work through sequences in which their intent is misinterpreted by a bot. In the subsequent discussion section, we will summarize and discuss these insights and discuss them in relation to developing communicative competence for conversational AI.
The study has shown that the main purpose of user self-repair is to address issues in which the bot does not understand users’ intent or misunderstands it. Users have a wide array of strategies available to do self-repair, some of which are akin to face-to-face social interaction and some of which are perhaps less so, e.g., changing choices originally made.
The study has also shown that the self-repair strategy users deploy most frequently – rephrasing – is actually one of the least successful strategies, leading in approximately a half of all instances to irrelevant bot answers (false positive) or explicit repair initiations (false negative answers). On the contrary, restating the purpose has a high success ration, but is only very infrequently deployed by users. In addition, ‘accommodation’ – using the bot’s own prompts – and making different choices also has a more than 50% success ratio.
This strongly suggests that users transfer strategies from face to face spoken interaction into their conversations with Asa (see Nass & Moon, 2000). For example, we have shown that some users not only rephrase their original turns, but also enhance them with additional politeness markers. This mirrors strategies previously observed in face to face-interaction. For example, Kasper (2006) observed that, in subsequent requests, interviewers in an oral proficiency use the politeness marker ‘please’ to frame an overall more direct request. Plug (2014) observed that, in self-initiated self-repair, speakers engage in “prosodic marking” through higher pitch, and higher speaking tempo, and Hauser (2019) observed ‘upgraded’ self-repeated gestures in Japanese interaction. Some of the self-repair strategies observed in this study mirror these patterns.
Whether ‘upgraded’ politeness is a matter of applying behaviours from face-to-face interaction, or is a result of other demands, remains to be further investigated by further research. In doing so, the notion of ‘pragmatic transfer’ (Kasper, 1992) which traditionally has been used to describe the transfer of L1 pragmatic strategies into an L2 and has informed plethora of research in intercultural, cross-cultural and interlanguage pragmatic since could be deployed to inform research on human-AI interaction, too.
The analysis has also revealed further insights which may have implications for users’ perceived rapport with the bot. As shown, ‘accommodation’ – defined here as convergence to the bot’s prompts in self-repair is one of the most frequently used, and one of the more successful strategies, though success is not necessarily guaranteed. Yet, such ‘upwards convergence’ towards interactional partners in superordinate positions (Giles & Ogay, 2007) may have implications for users’ perceptions of rapport with the bot for two reasons. Firstly, the conversational repair mechanisms observed here are ‘role-defining’ (Liebscher & Daily-O’Cain 2003) in that they define users’ role as that of a respondent to rather than an initiator of interaction. Secondly, having to converge to receive a relevant answer in the first place, and such convergence actually often NOT leading to a relevant answer (see table 4), has the potential for implications for users’ perceptions of rapport with the bot (Spencer-Oatey, 2008). However, the exact rapport implications in user-bot interaction need to be the subject of further, more detailed investigations.
On the other hand, together with ‘restate purpose’, accommodation is the strategy that has the highest potential success rate. Yet, user awareness of this strategy and their readiness to use it, may depend on users’ overall experience with chatbots and their general knowledge of the workings of AI (Luger & Sellen, 2016). It may also depend on users’ overall orientations to these interactions: To what extent do they perceive them as ‘relational’ (relationship-oriented), or as ‘transactional’ (task-oriented) (Koester, 2004)? To what extent do they perceive of them as having human characteristics (anthropomorphism) (Hermann, 2022)?
Implications for AI skills
As discussed above, we purport here that, in order to engage with AI-driven chatbots efficiently, users need to acquire specific skills that do not necessarily mirror those for spoken interaction. A range of models of communicative competence have previously been used to describe the competences required for social interaction. For example, Canale & Swain (1984) describe communicative competence as consisting of four components: grammatical competence (words and rules), sociolinguistic competence (appropriateness), discourse competence (cohesion and coherence) and strategic competence (appropriate use of communicative strategies). This analysis of self-repair in user-bot interaction shows that for effective engagement in these interactions three of these competences are particularly important:
In summary, in the same way as pragmatic competence is not necessarily developed alongside other aspects of communicative competence (Kasper & Roever, 2005), communication skills for conversational AI are not innate to all users (Luger & Sellen, 2016), suggesting that there may be value in teaching these skills explicitly and helping users understand the sociolinguistic environment they are operating in. An approach to teaching skills for effective engagement in conversational AI would benefit from including all of the elements previously described for the development of communicative competence (e.g., Jones & Stubbe, 2004 for professional communication; Dippold, 2015 for classroom interaction), such as awareness raising, experiential learning, reflection. Indeed Howard’s (2012) instructional paradigm for teaching CMC skills includes all these elements, and Weisz et al.’s (2019) experimental account of teaching strategies for successful human-agent interaction does so, too. Their intervention included a phase on raising users’ empathy with the bot to highlight the difficulties for dealing with user input. In this study, the instructional intervention led users to report that they had developed better strategies (e.g., use simple language, specify intent precisely). Users also developed their understanding of the algorithmic thinking process and learned to disambiguate its capabilities from human capabilities. The study is, to my knowledge, the only one currently that reports on an attempt to explicitly teach conversational AI skills. More studies which focus on AI skills ‘in the wild’, using authentic rather than simulated interactions, would supplement this work.
Limitations
This study has a number of limitations. Firstly, the data analysed here were gathered as part of a methodological pilot project with a bot product that was in a minimal viable product stage of its development. The bot responses here therefore do not necessarily reflect its full capability for dealing with user intents and initiating repair.
Finally, the data set this paper is based on is a relatively small corpus of user-bot interactions. Further research with a larger data set is necessary to further substantiate the patterns reported here. This should, ideally, also include gathering data in the wild rather than a simulated environment. When more is at stage, users’ repair strategies may be different, e.g., for how many turns they persist in attempting repair or what strategies they choose.
Nevertheless, the analysis has provided insights which follow-up research can now further evaluate. For instance, further research could further investigate whether and how speakers transfer pragmatic strategies from spoken interaction into bot interactions. Secondly, further research can ask how users’ repair strategies relate to their expectations for rapport and their ideologies of language use. Retrospective interviews asking in detail about users’ decision-making processes when interacting with chatbots have the potential to be able to tap into these questions.
In addition, it is of course also necessary to widen our gaze beyond repair. Informal observations during my analysis suggest that user success in transactional bot interactions also depends on other factors. Many of the trouble sources for instances were longer, discursive sentences. An interesting follow-up question is therefore to what extent recognition and user progress through the interaction is contingent on the form in which initial inputs are made, e.g. whether and which politeness strategies are used.
In this research, this paper investigated repair strategies in a chatbot whose main purpose was ‘transactional’ and which fulfilled tasks such as booking and rescheduling. Further research needs to investigate the users’ repair strategies in interactions with bots with a more social orientation, e.g., bots which provide mental health support or companionship.
Conclusion
Through the example of conversational repair, this paper has shown that some users are able to navigate the communication challenges posed in this environment very well, while some struggle working towards the transactional goals that the bot is meant to help them achieve. Talking with a chatbot requires a distinct skills-set, and these skills are likely to be even more complex in chatbots which use voice interaction. Hence, the future of human-bot interaction is not just about developing bots that are more like people: they won’t be. It is about upskilling people to interact with bots.
Further studies on the pragmatics of social interaction with chatbots will help build the “bigger picture” of what these skills are and need to be followed up with studies investigating how these skills can best be taught. Developing users’ knowledge about the workings of AI will ensure that all AI users have equitable access to the everyday services bots support.