In this paper, we argue for the importance of conducting replication studies over various schools and countries when addressing topics about learning and instruction and propose educational technology to be a tool for this endeavor. We present an example of a cross-cultural replication study that makes use of educational technology in the form of a digital game-based automated feedback system. The study addresses feedback related behavior in 11-15-year-old students in US and Swedish classrooms, investigating students' choices to seek confirmatory (i.e., positive) or critical (i.e., negative) feedback, as well as their subsequent choices to revise their work based on this feedback. Comparisons of the data collected at several schools in the US and Sweden showed similar patterns of relationships among students' feedback-seeking behavior, their tendency to revise their work, and their learning outcomes in and outside the assessment environment. Overall, the findings revealed that this assessment approach seems to be generalizable from a North American to a European population. However, the findings showed both a significant difference between Sweden and the US regarding the preference for critical feedback and between different schools within each country. Thus, it is possible that the difference between countries reflects school differences rather than cultural differences.

In this paper we explore gender, perceived humanlikeness, animacy, intelligence and likeability of a female and male Furhat robot, and test for uncanny valley effects. We found no gender differences for neither user nor Furhats in perceived humanlikness or likeability, but female users rated the Furhats higher on animacy and intelligence. The study showed a tendency to an uncanny valley effect, but somewhat surprisingly the Furhat robots were overall not perceived as humanlike.

Does a teachable agent influence the uptake or neglect of critical constructive feedback and learning within a digital environment? 285 middle-school students engaged with a history learning game in a 2x2 study design. One dimension was inclusion of a teachable agent. Orthogonal was whether critical constructive feedback was presented automatically or only when students chose. Analyses showed that a teachable agent positively affected students responses to feedback and mitigated feedback neglect; the results were especially strong for lower-achieving students. Additionally, presence of a teachable agent improved post-test performance for students overall, and this effect was mediated by lower feedback neglect.

The field of Intelligent Virtual Agents (IVAs) has evolved immensely with respect to design and development of agents over the years. This has brought the following questions: What processes and methods are used to design virtual agents, and in particular, to what extent and how are users involved in the design process of virtual agents? In this paper, we review papers from the conference “Intelligent Virtual Agents” for the last five years (2015 - 2019) to shed light on these questions. The review included 308 short and long papers, with a focus on the interactive aspects of design. The review showed that only 14% of 308 papers explicitly mentioned/referred to user participation during the design of an IVA. User involvement is classified into two categories: one-time and iterative. The few studies that mention design only report on the use of standard Human-Computer Interaction (HCI) design methods to a rather limited degree.

Studies of mental state attribution to robots usually rely on verbal measures. However, verbal measures are sensitive to peoples rationalizations, and the outcomes of such measures are not always reflected in a persons behavior. In light of these limitations, we present the first steps toward developing an alternative, non-verbal measure of belief attribution to robots. We report preliminary findings from a comparative study indicating that the two types of measures (verbal vs. non-verbal) are not always consistent. Notably, the divergence between the two measures was larger when the task of inferring the robots belief was more difficult.

In many types of human-robot interactions, people must track the beliefs of robots based on uncertain estimates of robots perceptual and cognitive capabilities. Did the robot see what happened and did it understand what it saw? In this paper, we present preliminary experimental evidence that people estimating what a humanoid robot knows or believes about the environment anthropocentrically assume it to have human-like perceptual and cognitive capabilities. However, our results also suggest that people are able to adjust their incorrect assumptions based on observations of the robot.

Peoples mental models of robots affect their predictions of robot behavior in interactions. The present study highlights some of the uncertainties that enter specifically into peoples considerations about the minds and behavior of robots by exploring how people fare in the standard "Sally-Anne" false-belief task from developmental psychology when the protagonist is a robot.

How should a pedagogical agent in educational software be designed to support student learning? This question is complex seeing as there are many types of pedagogical agents and design features, and the effect on different student groups can vary. In this paper we explore the effects of designing a pedagogical agents self-efficacy in order to see what effects this has on students interaction with it. We have analyzed chat logs from an educational math game incorporating an agent, which acts as a digital tutee. The tutee expresses high or low self-efficacy through feedback given in the chat. This has been performed in relation to the students own self-efficacy. Our previous results indicated that it is more beneficial to design a digital tutee with low self-efficacy than one with high self-efficacy. In this paper, these results are further explored and explained in terms of an increase in the protege effect and a reverse role modelling effect, whereby the students encourage digital tutees with low self-efficacy. However, there are indications of potential drawbacks that should be further investigated. Some students expressed frustration with the digital tutee with low self-efficacy. A future direction could be to look at more adaptive agents that change their self-efficacy over time as they learn.

This study examines the effects of teachable agents’ expressed self-efficacy on students. A total of 166 students, 10- to 11-years-old, used a teachable agent-based math game focusing on the base-ten number system. By means of data logging and questionnaires, the study compared the effects of high vs. low agent self-efficacy on the students’ in-game performance, their own math self-efficacy, and their attitude towards their agent. The study further explored the effects of matching vs. mismatching between student and agent with respect to self-efficacy. Overall, students who interacted with an agent with low self-efficacy performed better than students interacting with an agent with high self-efficacy. This was especially apparent for students who had reported low self-efficacy themselves, who performed on par with students with high self-efficacy when interacting with a digital tutee with low self-efficacy. Furthermore, students with low self-efficacy significantly increased their self-efficacy in the matched condition, i.e. when instructing a teachable agent with low self-efficacy. They also increased their self-efficacy when instructing a teachable agent with high self-efficacy, but to a smaller extent and not significantly. For students with high self-efficacy, a potential corresponding effect on a self-efficacy change due to matching may be hidden behind a ceiling effect. As a preliminary conclusion, on the basis of the results of this study, we propose that teachable agents should preferably be designed to have low self-efficacy.

The reported study investigated three gender-related effects on the rated persuasiveness of a speech given by a humanoid robot: (1) the female or male gendered voice and visual appearance of the robot, (2) the female or male gender of the participant, and (3) the interaction between robot gender and participant gender. The study employed a measure of persuasiveness based on the Aristotelian modes of persuasion: ethos, pathos and logos. In contrast to previous studies on gender bias toward intelligent virtual agents and robots, the gender of the robot did not influence the rated persuasiveness of the speech, and female participants rated the speech as more persuasive than men overall.

In a study with 108 10- to 12-year-olds who used a digital educational game targeting history, we addressed the phenomenon of perseverance, that is, the tendency to stick with a task even when it is challenging. The educational game was designed to make all students encounter tasks they did not succeed to solve, at which point they were offered a set of choices corresponding to perseverance and non-perseverance. Methods used were behavioral log data, post-questionnaires, and an in-game questionnaire conducted by a game character, who asked the students about the reason for their choice. Overall, we found no differences between high and low-perseverance students as to their experiences of effort, difficulty, and learning, and neither in their self-reported motives for persevering – when doing so. With respect to performance, however, high-persevering students solved significantly more tasks at higher difficulty levels. Comparing high-perseverance students who tended to take a break directly after a failed test – before they continued with the same task – with those who did not take a break, we found no significant differences, indicating that taking a break is not detrimental to learning and perseverance.

Learning in educational games is often associated with some form of competition. We investigated how students responded to winning or losing in an educational math game, with respect to playing with or without a Teachable Agent (TA). Students could choose between game modes in which the TA took a more passive or active role, or let the TA play a game entirely on its own. Based on the data logs from 3983 games played by 163 students (age 10-11), we analyzed data on students persistence, challenge-seeking and performance during gameplay. Results indicated that students showed greater persistence when playing together with the TA, by more often repeating a lost game with the TA, than a lost game after playing alone. Students challenge-seeking, by increasing the difficulty level, was greater following a win than following a loss, especially after the TA won on its own. Students gameplay performance was unaffected by their TA winning or losing but was, unexpectedly, slightly worse following a win by the student alone. We conclude that engaging a TA can make students respond more productively to both winning and losing, depending on the particular role the TA takes in the game. These results may inform more specific hypotheses as to the differential effects of competing and collaborating in novel, AI-supported social constellations, such as with TAs, on students motivation and ego-involvement in educational games.

People rely on shared folk-psychological theories when judging behavior. These theories guide peoples social interactions and therefore need to be taken into consideration in the design of robots and other autonomous systems expected to interact socially with people. It is, however, not yet clear to what degree the mechanisms that underlie peoples judgments of robot behavior overlap or differ from the case of human or animal behavior. To explore this issue, participants (N = 90) were exposed to images and verbal descriptions of eight different behaviors exhibited either by a person or a humanoid robot. Participants were asked to rate the intentionality, controllability and desirability of the behaviors, and to judge the plausibility of seven different types of explanations derived from a recently proposed psychological model of lay causal explanation of human behavior. Results indicate: substantially similar judgments of human and robot behavior, both in terms of (1a) ascriptions of intentionality/controllability/desirability and in terms of (1b) plausibility judgments of behavior explanations; (2a) high level of agreement in judgments of robot behavior -(2b) slightly lower but still largely similar to agreement over human behaviors; (3) systematic differences in judgments concerning the plausibility of goals and dispositions as explanations of human vs. humanoid behavior. Taken together, these results suggest that peoples intentional stance toward the robot was in this case very similar to their stance toward the human.

The present study used a questionnaire-based method for investigating people's interpretations of behavior exhibited by a person and a humanoid robot, respectively. Participants were given images and verbal descriptions of different behaviors and were asked to judge the plausibility of seven causal explanation types. Results indicate that human and robot behavior are explained similarly, but with some significant differences, and with less agreement in the robot case.

This paper explores how students perceive the gender of a visually androgynous teachable agent, and if and how the perceived gender relates to the perceived personality traits of the agent. It is shown that the students perception of the agents gender was independent of their own gender. There were few significant differences in the perceived personality traits in relation to perceived gender. However, when looking at the perceived degree of gender, the traits of the agent were more positively rated by those students who perceived it as moderately gendered as compared to androgynous or strongly gendered.

We present results from a study of truck drivers experience of using two different interfaces; spoken interaction and visual manual interaction, to perform secondary tasks while driving. The instruments used to measure their experience are based on three popular questionnaires, measuring different aspects of usability and cognitive load: SASSI, SUS and DALI. Our results show that the speech interface is preferred both regarding usability and cognitive demand.

Previous work indicates that physical robots elicit more favorable social responses than virtual agents. These effects have been attributed to the physical embodiment. However, a recent meta-analysis by Li [1] suggests that the benefits of robots are due to physical presence rather than physical embodiment. To further explore the importance of presence we conducted a pilot study investigating the relationship between physical and social presence. The results suggest that social presence of an artificial agent is important for interaction with people, and that the extent to which it is perceived as socially present might be unaffected by whether it is physically or virtually present.

The current study investigated if and how students’ behaviour when using a teachable agent-based educational software were related to their working memory capacity. Thirty Swedish students aged 11–12, participated in the study. Results showed that differences in behaviour such as time spent on an off-task activity, time spent on interactive dialogues, and the number of tests that students let their TA take, were associated with differences in working memory capacity. 

This paper explores how users ascribe gender to a visually androgynous teachable agent, and if and how the ascribed gender can influence the perceived personality characteristics of the agent. Previous studies have shown positive effects of using agents with more neutral or androgynous appearances, for instance, a more gender neutral agent evoked more positive attitudes on females than did a more stereotypical female agent [1] and androgynous agents were less abused than female agents [2]. Another study showed that even though an agent was visually androgynous, the user typically ascribed a gender to it [3].

This paper explores the potentials of a new type of pedagogical agent – a Challenger Teachable Agent. The aim of such a pedagogical agent is to increase engagement and motivation, and challenge students into deeper learning and metacognitive reasoning. It is based on the successful implementation of the Learning by Teaching approach in Teachable Agents, and in addition it draws on previous work that has shown the potential of resistance or challenge as means to improve learning. In this paper we discuss how these two bases can be combined and realized through new types of behaviours in a Teachable Agent

This chapter presents the first steps towards a new type of pedagogical agent – a Challenger Teachable Agent, CTA. The overall aim of introducing a CTA is to increase engagement and motivation and challenge students into deeper learning and metacognitive reasoning. The paper discusses desired design features of such an agent on the basis of related work and results from a study where 11-year old students interacted with a first version of a CTA in the framework of an educational software for history. The focus is on how students respond when the CTA disagrees and questions their suggestions, and how groups of students, differing in response behavior and in self-efficacy, experience the CTA.

We report a study on student’s attitudes to a visually androgynous in comparison to a male and a female Teachable Agent (TA). Results were that overall the androgynous agent was preferred over the female and male agents. A visually androgynous agent does not embody categorical gender attributes. At the same time it does not have to be genderless but instead represent both maleness and femaleness so that students can chose for themselves. Androgyny, in this sense, is potentially a way to have femaleness and maleness represented, with corresponding educational benefits such as role modelling and identification, without risking negative reinforcement of gender stereotypes

This study explored if the visual gender representations (androgynous, male, or female) of an embodied agent would influence students’ perceptions of their agent and their attitudes toward the agent as their conversational partner. The study also explored if students’ gender and cultural background would interact with the agent’s visual gender to influence their perceptions and attitudes. Participants were 208 early-teen students sampled from US and South Korea. The results revealed that student gender was a significant factor for influencing students' perceptions and attitudes and that the students showed positive attitudes toward an androgynous agent more than toward a gendered agent (either male or female).

Previous studies have shown that female ECAs are more likely to be abused than male agents, which may cement gender stereotypes. In the study reported in this paper a visually androgynous ECA in the form of a teachable agent in an educational math game was compared with a female and male agent. The results confirm that female agents are more prone to be verbally abused than male agents, but also show that the visually androgynous agent was less abused than the female although more than the male agent. A surprising finding was that very few students asked the visually androgynous agent whether it was a boy or a girl. These results suggest that androgyny may be a way to keep both genders represented, which is especially important in pedagogical settings, simultaneously lowering the abusive behavior and perhaps most important, loosen the connection between gender and abuse.

This chapter discusses design challenges encountered when developing a conversational pedagogical agent. By tracing the historical roots of pedagogical agents in Intelligent Tutoring Systems (ITS), we discern central developments in creating an agent that is both knowledgeable and fosters a social relationship with the learner. Main challenges faced when attempting to develop a pedagogical agent of this kind relate to: i) learners’ expectations on the agent’s knowledge and social profile, ii) dealing with learners’ engagement in off-task conversation and iii) managing potential abuse of the agent. We discuss these challenges and possible ways to address them, with reference to an ongoing Research & Development project, and with a focus on the design of a pedagogical agent’s visual embodiment and its conversational capabilities.

The paper discusses the addition of off-task socially oriented conver-sational abilities to an existing “teachable agent” (TA) in an educational game in mathematics. The purpose of this extension is to affect constructs known to promote learning, such as self-efficacy and engagement as well as enhance students’ experiences of interacting with the game. A comparison of students that played the game with the off-task interaction to those who played without it, shows trends that indicate that students who played the game with off-task interaction had a more positive experience of the game, and that they also learnt more, as reflected in the learning outcomes of their TAs.

The paper discusses the motivation for and outcome of the addition of socially oriented so called “off-task” conversational abilities to an existing “teachable agent” (TA) in an educational mathematics game. The purpose of the extension is to affect constructs known to promote learning, such as self-efficacy and engagement, as well as enhancing students’ experiences of the game. A comparison of students that played the educational game with the off-task interaction included to those who played without it, indicate that the former had a more positive experience of the game, and that they also learnt more in the sense of teaching their TA better. The potential for pedagogical interventions in this and similar systems is discussed as well as differences found between high- and low-achievers.

The paper discusses some cultural considerations that we stand before in developing and exploiting an agent based educational software for use by Swedish and American students, age 11-14. The reported cultural challenges arise in software develop­ment, study designs, and decisions on actual pedagogical use in the two cultural settings.

In this paper, we argue for the design of off-task interaction in technology enchanced learning. This implies a rethinking of pedagogy which can substantially contribute to achieving specific learning goals. We describe a work-in-progress and provide concrete examples of how we deal with relevant issues raised by allowing for off-task interactions in an educational math game as an integrated learning system.

We are developing a virtual learning environment, which includes a math game with a teachable agent, the embodiment of the agent, and a social conversation with the agent. In this work we are using various methods for design and system development, with a focus on iterative methods, and high involvement of the user in the process. In this paper we discuss the motivation for and the applications of these methods in the development of the social conversation with the agent.

We describe our work-in-progress of developing an educational game in mathematics for 12-14 year olds, by adding social and conversational abilities to an existing “teachable agent” (TA) in the game. The purpose of this extension is to affect cognitive, emotional and social constructs known to promote learning, such as self-efficacy and engagement, as well as enhancing students’ experiences of interacting with the agent over an extended period of time. Drawing from the EnALI framework, which states practical design guidelines, we discuss specific design challenges and exemplify research considerations as to developing the agent’s visual representation and conversational module. We present some initial findings from using prototype agents with students from the target group. Promising developments seem to reside in pronouncing the agent’s personality traits and expanding its knowledge database, particularly its range of conversational topics. Finally we propose some future studies and research directions.

In this paper we discuss how social conversation with an agent in an educational math game can be used in order to gain pedagogical benefits, for example to increase positive attitudes towards learning math and math self-efficacy. We present the iterative development of the conversational module, architectural considerations, and the type of dialogue phenomenon that support the pedagogical interventions

In this thesis, the design and use of ontologies as domain knowledge sources in information-providing dialogue systems are investigated. The research is divided into two parts, theoretical investigations that have resulted in a requirements specifications on the design of ontologies to be used in information-providing dialogue systems, and empirical work on the development of a framework for use of ontologies in information-providing dialogue systems.

The framework includes three models: A model for ontology-based semantic analysis of questions. A model for ontology-based dialogue management, specifically focus management and clarifications. A model for ontology-based domain knowledge management, specifically transformation of user requests to system oriented concepts used for information retrieval.

In this thesis, it is shown that using ontologies to represent and reason on domain knowledge in dialogue systems has several advantages. A deeper semantic analysis is possible in several modules and a more natural and efficient dialogue can be achieved. Another important aspect is that it facilitates portability; to be able to reuse adapt the dialogue system to new tasks and domains, since the domain-specific knowledge is separated form generic features in the dialogue system architecture. Other advantages are that it reduces the complexity of linguistic produced in various domains.

Next generation question answering systems are challenged on many fronts including but not limited to massive, heterogeneous and sometimes streaming collections, diverse and challenging users, and the need to be sensitive to context, ambiguity, and even deception. This chapter describes new directions in question answering (QA) including enhanced question processing, source selection, document retrieval, answer determination, and answer presentation generation. We consider important directions such as answering questions in context (e.g., previous queries, day or time, the data, the task, location of the interactive device), scenario based QA, event and temporal QA, spatial QA, opinionoid QA, multimodal QA, multilingual QA, user centered and collaborative QA, explanation, interactive QA, QA reuse, and novel architectures for QA. The chapter concludes by outlining a roadmap of the future of question answering, articulating necessary resources for, impediments to, and planned or possible future capabilities.

In this thesis a new concept called domain knowledge management for informationproviding dialogue systems is introduced. Domain knowledge management includes issues related to representation and use of domain knowledge as well as access of background information sources, issues that previously have been incorporated in dialogue management.

The work on domain knowledge management reported in this thesis can be divided in two parts. On a general theoretical level, knowledge sources and models used for dialogue management, including domain knowledge management, are studied and related to the capabilities they support. On a more practical level, domain knowledge management is examined in the contexts of a dialogue system framework and a specific instance of this framework, the ÖTRAF system. In this system domain knowledge management is implemented in a separate module, a Domain Knowledge Manager.

The use of a specialised Domain Knowledge Manager has a number of advantages. The first is that dialogue management becomes more focused as it only has to consider dialogue phenomena, while domain-specific reasoning is handled by the Domain Knowledge Manager. Secondly, porting of a system to new domains is facilitated since domain-related issues are separated out in specialised domain knowledge sources. The third advantage with a separate module for domain knowledge management is that domain knowledge sources can be easily modified, exchanged, and reused.

Dialogue systems utilise a variety of knowledge sources and models. However, there is a confusion concerning the purposes and contributions of specific models and the relationships among them. In this paper we present a study of different dialogue systems and the knowledge sources and models they use. The models are characterised in terms of what knowledge they contain and the roles of various models and the relations between them are discussed. Implications for development of dialogue systems are also presented.

Dialogue systems utilise a variety of knowledge sources and models. However, there is a confusion concerning the purposes and contributions of specific models and the relationships among them. In this paper we present a study of different dialogue systems and the knowledge sources and models they use. The models are characterised in terms of what knowledge they contain and the roles of various models and the relations between them are discussed. Implications for development of dialogue systems are also presented.