What a conversation asks of a machine

Listen to two people in conversation. While one talks, the other says "mm" and "ehe" now and then, laughs at the right moment, and comes in with a question the moment the first one finishes. Occasionally they both start at once, and one of them stops, usually with barely a hitch. Little of it is ever taught explicitly. Much of it is hard for a machine.

The study of conversation found long ago that talk is organised so that, overwhelmingly, one person speaks at a time, and that most changes from one speaker to the next happen with little or no gap and little or no overlap [1]. In a study of ten languages from around the world, people usually answered a yes-or-no question within a fraction of a second [2]. People manage this turn after turn, without thinking about it.

The person listening is not silent, either. The "mm", "yes" and "ehe" that people say while someone else is talking were named the back channel in 1970: a way of showing you are following without taking the turn [3]. Someone who never does it can seem not to be listening at all.

And sometimes both people talk at once. Overlap is not a breakdown of conversation but part of it, and people have orderly ways of resolving it, usually within a moment: one speaker drops out, sometimes after a brief contest in which both speak louder or repeat themselves [4]. Being able to cut in, and to be cut in on, is part of what makes talk feel like talk.

Many voice systems have been built differently. A common approach, described in a 2021 review, is to wait until you have been quiet for a set length of time, and only then start to reply [5]. Wait too briefly, and the system cuts in when you pause to think. Wait too long, and answers arrive after an awkward gap [5]. And a system that waits for silence cannot say "mm" while you are still speaking.

A machine that converses has to do what those two people do: listen and speak at the same time, tell when you have finished and when you are only pausing, show that it is following without taking the floor, and give way when you cut in. That is what a full-duplex model is for, and it is what we are building at Pan-African AI Lab, starting with the languages of South Africa.

None of this is fully solved, in any language.

References

1. Sacks, H., Schegloff, E. A., and Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. Language 50(4), 696–735.
2. Stivers, T., Enfield, N. J., Brown, P., Englert, C., Hayashi, M., Heinemann, T., Hoymann, G., Rossano, F., de Ruiter, J. P., Yoon, K.-E., and Levinson, S. C. (2009). Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences 106(26), 10587–10592.
3. Yngve, V. H. (1970). On getting a word in edgewise. Papers from the Sixth Regional Meeting of the Chicago Linguistic Society, 567–578.
4. Schegloff, E. A. (2000). Overlapping talk and the organization of turn-taking for conversation. Language in Society 29(1), 1–63.
5. Skantze, G. (2021). Turn-taking in conversational systems and human-robot interaction: a review. Computer Speech & Language 67, 101178.