Prioritizing user initiative in AI collaboration.

Designers should design for users to make key decisions in AI collaboration for enhanced initiative and better user experience.

About this paper

The author designed DuetDraw, an AI interface for collaborative drawing, and conducted a user study with 30 participants.

Findings indicate users preferred detailed instructions and enjoyed the interactions despite the AI's low ratings in predictability, controllability, and comprehensibility.

Here are some methods used in this study:

Think-Aloud Grounded Theory

Which part of the paper did the design guideline come from?

“Overall, the participants wanted the AI to provide enough instruction during the tasks. However, at the same time, they did not want the AI to give too many instructions. As seen in the survey results, we also identified that participants preferred Detailed Instruction to Basic Instruction in the qualitative analysis. Participants said Detailed provided a better understanding of the system and made them feel they were communicating and interacting with another intelligent agent. For example, (...)” (Section 5.0: Just Enough Instruction)

Oh, C., Song, J., Choi, J., Kim, S., Lee, S., & Suh, B. (2018). I Lead, You Help but Only with Enough Details. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems.

Inspiration and scope

This paper focused on user characteristics collaborating with AI to enhance user experience in AI collaboration.

You are designing for university TAs, aiming to provide real-time, actionable feedback. Your context and the paper's both emphasize real-time feedback. Enhancing AI collaboration user experience benefits from immediate responses and adaptive suggestions, just as the AI for TAs relies on timely feedback to support assessment and improve teaching.

Also, both contexts focus on user empowerment. Designing AI collaboration should empower users through intuitive interfaces and decision-making support, correspondingly, the AI for TAs should empower them with actionable insights and performance enhancements to boost their effectiveness.

Leveraging these similarities, consider designing an AI system that provides real-time feedback and handles administrative tasks automatically. That way, TAs can focus on improving strategies, enhancing overall educational experience.

Your input

  • What: I'm designing an AI-based system to assess teaching assistants' performance and provide them with real time feedback.
  • Who: Teaching assistants in universities
  • Design stage: Research, Ideation, Evaluation

Understanding users

The following user needs and pain points may apply to your design target as well:

Providing Detailed Instructions

Ensure the AI system delivers comprehensive and detailed feedback to teaching assistants to help them understand their performance and areas for improvement. Detailed feedback can enhance their teaching methods effectively.

User Empowerment Through Initiative

Design the AI system to allow teaching assistants to lead their improvement process. By giving them more control over how they receive feedback and suggesting improvements, it will foster a sense of ownership and engagement.

Design ideas

Consider the following components for your design:

1

Incorporate personalized dashboards that display real-time performance metrics and actionable feedback.

2

Integrate AI-driven tools to automate administrative tasks like grading and participation tracking, freeing TAs to focus on teaching.

3

Design intuitive interfaces with user-friendly controls, such as drag-and-drop elements and visual analytics.

Methods for you

Consider the following method(s) used in this paper for your design work:

Think-Aloud Method

Utilizing the think-aloud method allows designers to gain real-time insights into the user's thought processes and interactions with the AI system. Designers should ensure participants are comfortable speaking out loud and remind them to verbalize their thoughts during the tasks.

Semi-Structured Interviews

Conducting semi-structured interviews helps in understanding nuanced user feedback and experiences post-task, which can be crucial for refining AI interactions. Designers should prepare a flexible interview guide to explore various aspects of user experience while allowing room for unexpected user insights.

Metrics for you

Consider the following metric(s) used in this paper to evaluate your design work:

Predictability

Predictability measures how well users can anticipate the AI system’s actions based on previous interactions. For designers, it helps in reducing user frustration and enhancing user trust in the system. Ensure the AI behavior aligns with user expectations to maintain predictability, especially for inexperienced users.

Control

Control assesses the user's ability to guide the AI's actions and make decisions throughout the process. This metric is critical for designers aiming to provide users an empowered and effective interaction with the AI. Ensure AI feedback mechanisms allow users to influence and adjust the actions of the system easily.

[Figure 1] From this figure, you can gain insight into how collaborative interactions between users and AI can be designed to enhance performance, which is relevant for real-time feedback systems for teaching assistants.