8 questions or concerns about what to expect on the day
In example (5), the user asks a question about the MRI scan (l. 1). The user reacts to the bot’s repair initiation (l. 2-3) by accommodating through choosing one of the bot’s prompts (l. 4). Whilst the bot’s answer may not be an altogether satisfactory response to the question, it concludes this sequence and moves the conversation on.
In contrast, users who use the strategies with a lower success ratio tend to have to work harder to have their intents understood and ensure progression:
Example (6)
1 Please select a date and time for your appointment 18:00 19:00 None
2 7pm
3 Sorry, but I could not reschedule the appointment. Please try again a bit later, or try re-booking
4 with different date/time. If however you are still facing issues I can connect you with someone
5 from the clinic to help you out, just type - speak to someone. Thanks, Asa
6 I meant 19:00
7 Did you mean? google *calendar* link for my scheduled appointment *faq* - frequently asked
8 questions restart *None* of these
9 None of these
10 Apologies, bear with me. I’m still learning. Can you rephrase?
11 Can I reschedule
12 No worries. Please let me know the number of the appointment you would like to reschedule: 1.
13 Screening on Thu 09 Jun ’22 at 19:00
In example (6), the trouble source is the user’s input representing their desired appointment time (l. 2). The bot’s response is relevant, but nevertheless initiates user repair by indicating that the desired appointment time is not available and providing a number of options for how the appointment can still be rescheduled (l. 3-5). The user completes repair by rephrasing their desired appointment time (l. 6). The user’s repair is wordier than their previous version, with ‘I mean’ representing a personal mitigator akin to a negative politeness marker. After another comprehension request by the bot (l. 7-8), the user changes strategy and uses accommodation strategies by recycling one of the bot prompts (l. 9). When this strategy is also unsuccessful, the user’s next self-repair repeats the originally rescheduling request, this time framing the request as conditionally indirect request (‘Can I…’) (l. 10). This third attempt at user self-repair is leads to a relevant response but forces the user to restart the rescheduling conversation.
Discussion
Summary of results
The analysis presented in the previous section has showcased, through a combination of qualitative and quantitative description of user repair turn and subsequent bot follow-on turns, how users in a task-oriented chatbot use repair to work through sequences in which their intent is misinterpreted by a bot. In the subsequent discussion section, we will summarize and discuss these insights and discuss them in relation to developing communicative competence for AI.
The study has shown that the strategies the majority of all instances of repair – and thus the focus of the study – are other-initiated self-repairs of repairables in user turns. Hence, the main purpose of repair is to address issues in which the bot does not understand users’ intent – leading to false negative responses – or misunderstands it – leading to false positive answers. The study has also shown that the strategy users deploy most frequently – rephrasing – Is actually one of the least successful one, leading in approximately a half of all instances to false negative answers (other repair initiations by the bot) or false positive answers (irrelevant bot responses). On the contrary, restating the purpose has a high success ration, but is only very infrequently deployed by users. In addition, ‘accommodation’ – using the bot’s own prompts – and making different choices also has a more than 50% success ratio.
This strongly suggests that users transfer strategies from face to face spoken interaction into their conversations with Asa. For example, we have shown that some users not only rephrase their original turns, but also enhance them with additional politeness markers. This mirrors strategies previously observed in face to face-interaction. For example, Kasper (2006) observed speakers that, in subsequent requests, interviewers in an oral proficiency exam treat conventionally indirect request frames ad redundant, instead using the politeness marker ‘please’ to frame an overall more direct request. Plug (2014) observed that, in self-initiated self-repair, speakers engage in “prosodic marking” through higher pitch, and higher speaking tempo, and Hauser (2019) observed ‘upgraded’ self-repeated gestures in Japanese interaction. Whilst the self-repair patterns in this paper contradict rather than confer with confer with Kasper’s (2006) observations – for instance, in some instances speakers formulate ‘repaired’ questions in a conventionally indirect manner – they nevertheless support an overall pattern of people applying learned social behaviours to their interactions (see Nass & Moon, 2000).
Whether ‘upgraded’ politeness is a matter of applying behaviours from face-to-face interaction, or is a result of other demands, remains to be further investigated by further research. In doing so, the notion of ‘pragmatic transfer’ (Kasper, 1992) which traditionally has been used to describe the transfer of L1 pragmatic strategies into an L2 and has informed plethora of research in intercultural, cross-cultural and interlanguage pragmatic since could be deployed to inform research on human-AI interaction, too.
The analysis has also revealed further insights which may have implications for users’ perceived rapport with the bot. As shown, ‘accommodation’ – defined here as convergence to the bot’s prompts in self-repair is one of the most frequently used, and one of the more successful strategies, though success is not necessarily guaranteed. Yet, such ‘upwards convergence’ towards interactional partners in superordinate positions (Giles & Ogay, 2007) may have implications for users’ perceptions of rapport with the bot for two reasons. Firstly, the conversational repair mechanisms observed here ‘role-defining’ (Liebscher & Daily-O’Cain 2003) in that they define users’ role as that of a respondent to rather than an initiator of interaction. Secondly, having to converge to receive a relevant answer in the first place, and such convergence actually often NOT leading to a relevant answer (see table 4), has the potential for implications for users’ perceptions of rapport with the bot (Spencer-Oatey, 2008). However, the exact rapport implications in user-bot interaction need to be the subject of further, more detailed investigations.
On the other hand, together with ‘restate purpose’, accommodation is the strategy that has the highest potential success rate. Yet, user awareness of this strategy and their readiness to use it, may depend on users’ overall experience with chatbots and their general knowledge of the workings of AI (Luger & Sellen, 2016). It may also depend on users’ overall orientations to these interactions: To what extent do they perceive them as ‘relational’ (relationship-oriented), or as ‘transactional’ (task-oriented) (Koester, 2004)? To what extent do they perceive of them as having human characteristics (anthropomorphism) (Hermann, 2022)?
Implications for AI skills
As discussed above, we purport here that, in order to engage with AI-driven chatbots efficiently, users need to acquire specific skills that don’t necessarily mirror those for spoken interaction. A range of models of communicative competence have previously been used to describe the competences required for social interaction. For example, Canale & Swain (1984) describe communicative competence as consisting of four components: grammatical competence (words and rules), sociolinguistic competence (appropriateness), discourse competence (cohesion and coherence) and strategic competence (appropriate use of communicative strategies). This analysis of self-repair in user-bot interaction shows that for effective engagement in these interactions three of these competences are particularly important:
In summary, in the same way as pragmatic competence is not necessarily developed alongside other aspects of communicative competence (Kasper & Roever, 2005), communication skills for conversational AI are not innate to all users (Luger & Sellen, 2016).
This suggests that there may be value in teaching these skills explicitly. An approach to teaching skills for effective engagement in conversational AI would benefit from including all of the elements previously described for the development of communicative competence (e.g., Jones & Stubbe, 2004 for professional communication; Dippold, 2015 for classroom interaction), such as awareness raising, experiential learning, reflection. Indeed Howard’s (2012) instructional paradigm for teaching CMC skills includes all these elements, and Weisz’ (2019) experimental account of teaching strategies for successful human-agent interaction does so, too. Their intervention included a phase on raising users’ empathy with the bot to highlight the difficulties for dealing with user input. In this study, the instructional intervention led users to report that they had developed better strategies (e.g., use simple language, specify intent precisely). Users also developed their understanding of the algorithmic thinking process and learned to disambiguate its capabilities from human capabilities. This study is, to my knowledge, the only one currently that reports on an attempt to explicitly teach conversational AI skills. More studies which focus on AI skills ‘in the wild’ rather than an experimental setting would supplement this work.
AI design implications
The findings of this paper also have implications for AI design and development. For instance, in ordinary face to face interaction other-repair can be initiated by partially repeating the trouble source, often accompanied by a question word (Schegloff et al., 1977). In English as a lingua franca research, raising the explicitness of talk – described by Mauranen (2007) as ‘explicitness strategy’ – has consistently been observed to be an interactional strategy between speakers of different varieties of English. These also include, according to Mauranen (2006), asking specific questions, e.g., after lack of comprehension of lexical items, repetition of problematic items in order to elicit some form of explanation. In contrast, Asa the bot initiates repair by simply indicating lack of understanding without pinpointing what was misunderstood (see, for instance, example 1). Raising the explicitness of other-initiated repair may lessen the ambiguity of repair-initiations for users. However, implications for face and rapport need further consideration as “explicitly acknowledging a mistake lowers the likability and perceived intelligence of the agent and may add friction to the interaction as the user is obliged to respond to the initiation” (Ashktorab et al., 2019, p. 3).
In addition, bot development could benefit by using user self-repair in more meaningful ways to further develop the bot’s understanding for understanding conversational intent. For example, self-initiated self-repair by users of English as a lingua franca has been described as a strategy to prevent misunderstanding by raising the explicitness of talk (Kaur, 2012). Of course, user self-repair is rarely in the open – in the data reported on above, there were only ten instances in total (see table 4, no response for illustration) – occurring more commonly when users edit their turns before they press enter. I propose that algorithms which record and learn from user self-editing might have the potential to provide insights into users’ intents. The same also applies to user self-repair after other-initiation: our data have shown sometimes extensively long sequences in which users ‘repair’ the intent multiple times. I propose that the accumulated information from these self-repairs could be used to build a more complete picture of user’ intent rather than interpreting just one user turn at a time. This proposal is supported by Li (2020) who purports that “given how hard it is (even for humans) to correctly detect the meaning of a ‘broken’ message by reading ether half in isolation, we recommend that chatbot developers consider having their agents process multiple messages at a time, rather than responding so quickly” (p. 9). This would also be able to better account for the co-construction and joint mentalizing that is common to human-human interaction (Kopp & Krämer, 2021), a concept that has yet to be applied to the development of bot interactions.
Finally, the insights gained through this project also suggest that, as users are unsure about the sociolinguistic environment they are operating in, the workings of AI-driven discourse should be made clear to users. As discussed above, Weisz et al (2019) used this to teach experimental participants about the workings of machine learning algorithms to interpret intent. In practice, a short explanation at the outset of an interaction could make explicit to users the ‘rules of the game’ that appear rather opaque to many. This includes, for example, how to use the bot’s prompts, how to ‘reset’ a conversation (as in example 3). Making the rules of the game explicit to users could prevent misunderstanding before it occurs.
Finally, conversational AI systems could also be designed to recognise users’ orientations towards the discourse from participants’ own input. For example, direct forms of address directed at the bot are potentially indicative of a more anthropomorphic orientation by which users orient to the agent as humanlike (Waytz et al., 2010). Such an orientation might imbue the users with more difficulties in adapting the styles of interaction which lead to satisfactory international outcomes, but are less akin to face-to-face interaction (e.g., restating purpose, accommodation). Using user input to come to conclusions about their anthropomorphic or dehumanised orientations towards the bot will allow for conversations to be designed in a way to recognise users who might more support in their interactions than others.
Limitations
This study has a number of limitations. Firstly, the data analysed here were gathered as part of a methodological pilot project with a bot product that was in a ‘minimal viable product’ stage of its development. The bot responses here therefore do not necessarily reflect its full capability for dealing with user intents and initiating repair.
Finally, the data set this paper is based on is a relatively small corpus of user-bot interactions. Further research with a larger data set is necessary to further substantiate the patterns reported here. This should, ideally, also include gathering data ‘in the wild’ rather than a simulated environment. When more is at stage, users’ repair strategies may be different, e.g., for how many turns they persist in attempting repair or what strategies they choose.
Nevertheless, the analysis has provided insights which follow-up research can now further evaluate. For instance, further research could further investigate whether and how speakers transfer pragmatic strategies from spoken interaction into bot interactions. Secondly, further research can ask how users’ repair strategies relate to their expectations for rapport and their ideologies of language use. Retrospective interviews asking in detail about users’ decision-making processes when interacting with chatbots have the potential to be able to tap into these questions.
In addition, it is of course also necessary to widen our gaze beyond repair. Informal observations during my analysis suggest that user success in transactional bot interactions also depends on other factors. Many of the repairables for instances were longer, discursive sentences. Are alternative formulations less likely to require repair in the first place? What do users make request (e.g., their use of politeness strategies), and how is further progress (e.g., in a booking process) contingent on its form?
In this research, this paper investigated repair strategies in a chatbot whose main purpose was ‘transactional’, and which fulfilled tasks such as booking and rescheduling. Further research needs to investigate the users’ repair strategies in interactions with bots with a more social orientation, e.g., bots which provide mental health support or companionship.
Conclusion
Through the example of conversational repair, this paper has shown that some users are able to navigate the communication challenges posed in this environment very well, while some struggle working towards the transactional goals that the bot is meant to help them achieve. Talking with a chatbot requires a distinct skills-set, and these skills are likely to be even more complex in chatbots which use voice interaction. Further studies on the pragmatics of social interaction with chatbots will help build the “bigger picture” of what these skills are and need to be followed up with studies investigating how these skills can best be taught. Developing users’ knowledge about the workings of AI will ensure that all AI users have equitable access to the everyday services bots support.