AI EVALUATION & PRODUCT RESEARCH
KARL A. NEUMANN
You can build anything. The hard part is knowing how people will actually use it. I evaluate products against how real people behave before teams spend months building in the wrong direction.
Previously: led user research for mental health app from 0→100K users, Employee #8 at YC startup from seed→Series B, and now running Desire Path, an independent consultancy
Work
Partnerships and advisory with early-stage and scaling companies.
Conversational AI Evaluation
In 2021, I evaluated and bias-validation method for a patient-facing chatbot, then used it to test across >700 postpartum patients by race & socioeconomic status. Co-published training methods in BMJ Innovations with UPenn.
Embedded | Rapid Research | Proptech
Designed the LLM evaluation framework for clinical case note summaries, built around practitioner review panels. Their app Soluna has helped >100K young Americans. Worked on company's global AI ethics committee.
Project-based | Strategy | HR Tech
Product advisory for a platform facilitating court evidence in custody and family law cases. Engagement covers UX evaluation of desktop & iOS app development, QA of key user flows, and strategic direction for branding and product marketing.
Advisory | End-to-end | Legal Tech
Embedded as the research and UX arm of a small product team while they scale. I rebuilt onboarding, hired a designer, and reframed the core dashboard. I also cover what a lean team can't staff: competitive intelligence, product marketing, hiring, & UX writing.
Embedded | Rapid Research | Proptech
I evaluated their platform, candidate assessments, and onboarding to show where the experience had friction. then ran strategy workshops and wrote executive briefs the leadership team now plans their roadmap against.
Project-based | Strategy | HR Tech
Product advisory for a platform facilitating court evidence in custody and family law cases. Engagement covers UX evaluation of desktop & iOS app development, QA of key user flows, and strategic direction for branding and product marketing.
Advisory | End-to-end | Legal Tech
Case Studies
Statistical analyses of >700 patients investigated patient engagement, satisfaction, & accuracy related to patients’ race & SES. Answered: "how might Memora Health quantify AI bias?"
Quantitative | Evaluative | Health Equity
Semi-structured interviews explored clinicians' & care teams' workflows, expectations, & needs. This telehealth system connected rural ER patients & remote psychiatrists.
Qualitative | Generative | Lean UX
Product strategy & 8-week research timeline to design and refine a mobile app for Crohn's patients to monitor key gut biomarkers. Methods include interviews, focus groups, & unmoderated tree testing.
Mixed Methods | Service Design
How I Work
Every engagement begins by understanding what you have and what our open questions are
01
Evaluate
Your product, designs, and analytics hold clues about what's working and what isn't. I identify friction, gaps, and assumptions, then we iterate to improve the experience before testing with real people.
02
Test
How I test depends on what stage you are at: stakeholder interviews, usability studies, in-app experiments, or analysis of usage patterns. We find exactly where the experience holds up and where it breaks.
03
Integrate
You get data-driven guidance tied to specific decisions via stakeholder readouts, team workshops, executive summaries, or strategy sessions. Then we move to the next priority or cycle back to step 01 with sharper questions.
Writing
"There is no great writing, only great rewriting."
– Justice Louis Brandeis
Co-published with UPenn researchers in BMJ Innovations on methodology for evaluating healthcare AI chatbots. Established processes for proactive training and safety validation across patient interactions.
AI Eval | Peer-Reviewed Publication
Established a four-pillar framework for evaluating digital health equity: accessibility, literacy, demographics, and identity. Includes analyses of AI performance across 1,400+ patients in a UPenn postpartum program.
Technical Writing | Public Health Tech
Explored ethics of machine intelligence via a popular TV show, arguing that how we treat AI reflects power dynamics rather than principled ethics. Written before AI was a mainstream concern.
AI Ethics | Public Writing
And now, a poem
Behind the Scenes:
Every artificial object secretly saturated with intent, its subjects in mind, by design
Each item that comprises life involved countless meetings to (re)invent, iteratively built, by design
Everything engineered to fix problems we are now likely to prevent, taken for granted, by design.
June 9th, 2018
by Karl
About
"We can only see a short distance ahead, but we can see plenty there that needs to be done.”
– Alan Turing

I run Desire Path Research, an independent UX research and AI evaluation consultancy. I work with founders & product leaders who need to understand how real people will experience what they've built before hiring a full UX org.
My background is in research. I began by studying the sense of smell at the University of Chicago, the neuroscience of music and pain psychology at McGill, and then the placebo effect during an internship at the NIH. After graduation, I spent two years in clinical research at a biopharma startup before moving into user research and tech.
In 2019, I started as UX researcher and employee #8 at Memora Health, scaling with the company from Seed through Series B. While there, I co-published research on AI evaluation methodology with UPenn in BMJ Innovations, which shaped how I think about what it means to evaluate language-based systems rigorously. I then consulted with ZS Associates on digital health products for Fortune 500 clients.
Before going independent, I led UX research at Kooth Digital Health, where I worked on their app Soluna. I joined before launch and saw the product scale and help >100K 13- to 25-year-olds with their mental health. I was also part of their global AI ethics committee and designed LLM evaluation frameworks with clinical review panels.
The common thread across all of my work is a focus on whether the experiences we build actually hold up when real people try them, and what rigorous research can tell us about how we can improve their designs.
Outside of work, most of my time revolves around music: listening, producing, and playing instruments. Otherwise, you might find me tending to my plants, making coffee, or somewhere along Lake Michigan.













