The vCare patient User Interface (UI) is realized as an Android application running on a tablet device. The UI supports different representation and interaction modalities, including the human-like avatar, the vCare Virtual Coach (VC).

The VC is a ubiquitous part of the UI that assists and guides the patient through their rehabilitation program. The goal of the human-like UI representation is to engage and motivate the patient to adhere to their rehabilitation plan and doctor’s advices in their home environment. The great benefit and advantage, compared to non-avatar-based interfaces, is that they can use human-like interaction styles such as verbal and non-verbal communication expressed by a recognizable human-like Avatar. Therefore, the Avatar conveys information verbally using real-time state of the art Text-To-Speech (TTS) technology along with predefined gestures, motions, and emotion expressions. Conversely, patients can respond and interact with the Avatar verbally via speech input, which is recognized using state of the art speech recognition technology, enabling more natural and human-like communication patterns to create a livelier user experience.

Design Workshop

A design workshop was conducted with technical and clinical project partners. The goal of this workshop was to define different aspects of the VC and the representation by an Avatar, like general characteristics, qualities, faults, and the interaction with patients. For this, two team exercises were performed. The first exercise was to draw/sketch the VC and to define its key character features: age, sex, weight/size, qualities, faults, and the name. The second exercise was to describe a typical day from the VC’s point of view, i.e., how to interact with the patient. The collected data during the creative workshop was qualitatively analysed as follows.

Figure 1: Design workshop Avatar and scenery drawings/sketches

General characteristics of the Virtual Coach

One outcome of the workshop was that the VC should have an age between 30 and 40: the idea is that the VC should show both, to be an expert in the rehabilitation sector and an expert in the digital world. The VC is mainly seen as a female. This aspect is related to caring behaviours and feelings. Furthermore, behind the aspect of the VC, users should recognise both a physician, which is an expert in diseases and rehabilitation programmes and a “gym” trainer, which is an expert in physical activity and healthy behaviours, such as sport or diet. Figure 1 shows Avatar and scenery drawings/sketches produced at the design workshop.

Figure 1: Design workshop Avatar and scenery drawings/sketches

Qualities, faults, and interaction with patients

Other aspects such as the qualities of the VC, its faults and how the interaction with the patients can look like were also discussed in that workshop. Qualities were categorized in relational skills and professionalism, with associated attributes mentioned like patience, friendly, open-smile and punctual, precise or authoritativeness.

Faults were used to better define the VC’s characteristics and to try to balance all the aspects, avoiding the risk of negative perceptions/evaluations by users. The VC shouldn’t seem to be perceived as being too proud, too strict or directive and shouldn’t exhibit bad human characteristics like greed.

With regards to a typical interaction between the VC and the patient, it was found that both “push” and “pull” mechanisms can be implemented, meaning that an interaction can be started by either party. When activities are planned, an interaction should consist of “Before Activities”, “During Activities” and “After Activities”. In “Before Activities”, the VC could send alerts according to the patient’s agenda, to remind the him or her to follow his or her plans. “During Activities”, such as motor exercises, the VC should support the performance of the patient by giving feedback, inciting, giving suggestions etc. “After Activities”, the VC should give positive feedback when activities are completed by the patient, e.g., “Bravo!”, “Well done!”.

The outcomes of the design workshop provided valuable input for the design of the Avatar look and feel. The general characteristics were used as guidelines regarding the age of the Avatar but also for the outfit and general appearance.

Technical realization

The affective Avatar is implemented following AIT’s Avatar Generation Toolchain (AGT). Figure 2 illustrates the overall architecture of the AGT.

Figure 2: Overall Architecture of the AGT.

The AGT consists of six main components (3D modelling, animation, audio processing, rendering, video generation, video delivery) classified in three different processing steps (pre-processing, processing and the optional post-processing). The pre-processing part defines the avatar’s character and appearance as well as the domain and scenario-specific setting including the avatar’s movements. The processing part is responsible for the audio processing and for the run-time animation and rendering. This part is influenced and controlled by internal and external parameters (“controlling parameters”; e.g., the text which has to be spoken, the avatars emotion or motion). The post-processing part is an optional part and just required in situations where a live rendering is either not possible or not applicable (e.g., if the avatar needs to be embedded in existing services or specific Web applications).

In vCare, the avatar component uses the Unity3D game engine for rendering the Avatar in an Android App. The Avatar’s characteristics can be adjusted to the current context to e.g., express emotions by varying facial expressions, varying voice intonations using Speech Synthesis Markup Language (SSML) and different animations for the Avatar’s body posture and gestures.

Evaluation and validation

The Avatar evaluation and validation will be part of vCare’s outcome measurements at the end of the pilot phase using TAM (Technology Acceptance Model) score, SUS score (System Usability Scale) and Qualitative UEQ (User Experience Questionnaire).

Avatar and UI impressions

Figure 3 shows the current male and female characters, resembling the design workshop’s input like the age or the recognition as a physician based on the outfit. Figure 4 shows the current UI implementation with the Avatar in the top left corner, some navigation elements below in the bottom left area and the main information area on the right side.

 

Figure 3: Male and female VC designs

Figure 4: Screenshot of the UI showing the Avatar in the top left corner and patient observations on the right-hand side.