Evaluating algorithm's acceptability and impact.

Designers should evaluate algorithms for accuracy, acceptability to users, and impacts on target performance to ensure comprehensive and effective tool development.

About this paper

The author critiques the prevalent machine learning-based approaches to algorithm design for their lack of user engagement and reliance on historical data, proposing instead a method called Value Sensitive Algorithm Design that incorporates stakeholders' feedback early in the process.

This novel method aims to balance multiple stakeholders' needs and includes an example project on designing intelligent socialization algorithms for WikiProjects in Wikipedia.

Here are some methods used in this study:

Value-Sensitive Design Participatory Design

Which part of the paper did the design guideline come from?

“Results. We now present our answers to our questions. Question 1: What happened to the newcomers if they received invitations from WikiProject organizers? Specifically, do the newcomers participate and contribute? And how is receiving organizers' invitations different from receiving template invitations from the researcher team, or receiving nothing? In Models 1 and 3 (in Table 4 and Figure 3), we compared the within-project contributions of the six newcomer groups (i.e., NQ, Q/XX, Q/OI, (...)” (Section 4.6: Step 5: Evaluate Algorithms' Acceptance, Accuracy, and Impacts)

Zhu, H., Yu, B., Halfaker, A., & Terveen, L. (2018). Value-Sensitive Algorithm Design. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), 1–23.

Inspiration and scope

This paper explores project organizers' and new members' characteristics in designing an algorithmic tool for WikiProjects.

You are designing for global healthcare workers, addressing their needs and pain points. Your context differs from the paper's, which focuses on efficient recruitment within online project groups. At the same time, both aim to enhance engagement with their target groups. The academic paper's focus on algorithmic tools can apply to engaging healthcare workers. Techniques like feedback loops and motivational design are relevant in both.

They differ in that the academic paper is limited to online communities, while yours involves digital tools and physical environments. At the same time, both designs require user adaptation and training. New members need to learn tools for efficient integration, while healthcare workers must adapt to new tools and processes. Effective training and simplicity in design are crucial in both.

Leverage these similarities by designing a framework with continuous feedback mechanisms, motivational techniques, and user-centric training for healthcare workers. Refine tools based on actual user experiences and needs, leading to improved performance and wellbeing for healthcare workers.

Your input

  • What: reflection, change, pain points
  • Who: global healthcare workesr
  • Design stage: Evaluation

Methods for you

Consider the following method(s) used in this paper for your design work:

Survey of Stakeholders' Perspectives

Using surveys to gather insights from stakeholders about their needs, motivations, and opinions can help identify key user values and potential trade-offs. Designers should ensure the survey questions are comprehensive and cover diverse aspects relevant to the target users.

Iterative Feedback and Refinement

Collecting feedback through iterative cycles helps in refining the design to better meet user needs and improve engagement. Designers should use the feedback to make targeted updates and ensure the improvements align with user expectations.

Metrics for you

Consider the following metric(s) used in this paper to evaluate your design work:

Community Acceptance

Evaluating community acceptance helps ensure that the design aligns well with the expectations and values of global healthcare workers, enhancing user engagement. Designers should consider potential barriers and factors unique to target users in different geographical areas.

Algorithm Accuracy

Measuring algorithm accuracy ensures the solutions provided fit the actual needs and objectives in the healthcare context, improving decision-making processes. Designers must balance accuracy while considering real-world complexity and biases in health data.

[Figure 4] From this figure, you can draw insights on evaluating user engagement post-intervention, which is crucial for assessing whether global healthcare workers are effectively using the design solution.