top of page

AI EVALUATION & PRODUCT RESEARCH

KARL A. NEUMANN

You can build anything. The hard part is knowing how people will actually use it. I evaluate products against how real people behave before teams spend months building in the wrong direction.

Previously: led user research for mental health app from 0→100K users, Employee #8 at YC startup from seed→Series B, and now running Desire Path, an independent consultancy

Work

Partnerships and advisory with early-stage and scaling companies.

Conversational AI Evaluation

596f6fab31f3-6639ee3f3030e627d9c2bbcc_EwFuqM1BHSVX-G2lN1OYvhNk9218Q0xHnA3pBulPTiM copy.jpg

In 2021, I evaluated and bias-validation method for a patient-facing chatbot, then used it to test across >700 postpartum patients by race & socioeconomic status. Co-published training methods in BMJ Innovations with UPenn.

Embedded  |  Rapid Research  |  Proptech

Screenshot+2024-11-27+at+5.49.09 PM.webp

Designed the LLM evaluation framework for clinical case note summaries, built around practitioner review panels. Their app Soluna has helped >100K young Americans. Worked on company's global AI ethics committee.

Project-based  |  Strategy  |  HR Tech

IMG_1508_edited.jpg

Product advisory for a platform facilitating court evidence in custody and family law cases. Engagement covers UX evaluation of desktop & iOS app development, QA of key user flows, and strategic direction for branding and product marketing.

Advisory  |  End-to-end  |  Legal Tech 

Embedded as the research and UX arm of a small product team while they scale. I rebuilt onboarding, hired a designer, and reframed the core dashboard. I also cover what a lean team can't staff: competitive intelligence, product marketing, hiring, & UX writing.

Embedded  |  Rapid Research  |  Proptech

I evaluated their platform, candidate assessments, and onboarding to show where the experience had friction. then ran strategy workshops and wrote executive briefs the leadership team now plans their roadmap against.

Project-based  |  Strategy  |  HR Tech

Product advisory for a platform facilitating court evidence in custody and family law cases. Engagement covers UX evaluation of desktop & iOS app development, QA of key user flows, and strategic direction for branding and product marketing.

Advisory  |  End-to-end  |  Legal Tech 

WORK

Case Studies

WORK
link to AI bias case study

Statistical analyses of >700 patients investigated patient engagement, satisfaction, & accuracy related to patients’ race & SES. Answered: "how might Memora Health quantify AI bias?"

Quantitative  |  Evaluative  |  Health Equity 

link to qualitative case study

Semi-structured interviews explored clinicians' & care teams' workflows, expectations, & needs. This telehealth system connected rural ER patients & remote psychiatrists.

Qualitative  |  Generative  |  Lean UX

link to product research case study

Product strategy & 8-week research timeline to design and refine a mobile app for Crohn's patients to monitor key gut biomarkers. Methods include interviews, focus groups, & unmoderated tree testing.

Mixed Methods  |  Service Design

How I Work

Every engagement begins by understanding what you have and what our open questions are

01

Evaluate

Your product, designs, and analytics hold clues about what's working and what isn't. I identify friction, gaps, and assumptions, then we iterate to improve the experience before testing with real people.

02

Test

How I test depends on what stage you are at: stakeholder interviews, usability studies, in-app experiments, or analysis of usage patterns. We find exactly where the experience holds up and where it breaks.

03

Integrate

You get data-driven guidance tied to specific decisions via stakeholder readouts, team workshops, executive summaries, or strategy sessions. Then we move to the next priority or cycle back to step 01 with sharper questions.

Writing

"There is no great writing, only great rewriting."

​

– Justice Louis Brandeis 

Karl's publication in BMJ Innovations

Co-published with UPenn researchers in BMJ Innovations on methodology for evaluating healthcare AI chatbots. Established processes for proactive training and safety validation across patient interactions.

AI Eval  |  Peer-Reviewed Publication  

digital-health-africa_edited.jpg

Established a four-pillar framework for evaluating digital health equity: accessibility, literacy, demographics, and identity. Includes analyses of AI performance across 1,400+ patients in a UPenn postpartum program.

Technical Writing  |  Public Health Tech

WestworldHBO.com_.jpg

Explored ethics of machine intelligence via a popular TV show, arguing that how we treat AI reflects power dynamics rather than principled ethics. Written before AI was a mainstream concern.

AI Ethics  |  Public Writing

WRITING

And now, a poem

Behind the Scenes:

Every artificial object secretly saturated with intent, its subjects in mind, by design

 

Each item that comprises life involved countless meetings to (re)invent, iteratively built, by design

 

Everything engineered to fix problems we are now likely to prevent, taken for granted, by design.

​

June 9th, 2018

by Karl

About

"We can only see a short distance ahead, but we can see plenty there that needs to be done.”

​

– Alan Turing

ABOUT
BB628D46-4DF7-4ADF-BB5F-E7E9EF7AE5A7IMG_0714.jpeg

I run Desire Path Research, an independent UX research and AI evaluation consultancy. I work with founders & product leaders who need to understand how real people will experience what they've built before hiring a full UX org.

 

My background is in research. I began by studying the sense of smell at the University of Chicago, the neuroscience of music and pain psychology at McGill, and then the placebo effect during an internship at the NIH. After graduation, I spent two years in clinical research at a biopharma startup before moving into user research and tech.

In 2019, I started as UX researcher and employee #8 at Memora Health, scaling with the company from Seed through Series B. While there, I co-published research on AI evaluation methodology with UPenn in BMJ Innovations, which shaped how I think about what it means to evaluate language-based systems rigorously. I then consulted with ZS Associates on digital health products for Fortune 500 clients.

Before going independent, I led UX research at Kooth Digital Health, where I worked on their app Soluna. I joined before launch and saw the product scale and help >100K 13- to 25-year-olds with their mental health. I was also part of their global AI ethics committee and designed LLM evaluation frameworks with clinical review panels.

The common thread across all of my work is a focus on whether the experiences we build actually hold up when real people try them, and what rigorous research can tell us about how we can improve their designs.

Outside of work, most of my time revolves around music: listening, producing, and playing instruments. Otherwise, you might find me tending to my plants, making coffee, or somewhere along Lake Michigan.

Let's connect

  • LinkedIn

Thanks for connecting!

Copyright © 2026  | Karl A. Neumann

bottom of page