PRODUCT, UX, & AI EVALUATION
KARL A. NEUMANN
Will people actually follow the experience you plan for them? I evaluate your product against how real people actually think, decide, & act, so we build based on evidence, not assumptions.
Previously: led UXR for a mental health app from 0→100K active users. Employee #8 at Memora Health from Seed→Series B.
Published on AI evaluation in BMJ Innovations.
Work
"Don't think, but look!"
– Ludwig Wittgenstein
Recent Engagements
AI-Powered
Homeownership Platform
Embedded UX and product consulting with HouseFacts, an early-stage proptech company. Work spans usability, redesigns, onboarding flows, technical hiring, copy/UX writing, competitive analysis, product marketing, & product strategy.
Embedded | Evaluative | Proptech
HR & Organizational
Assessment Consultancy
Collaboration with CTO & board executive. Expert UX evaluation of user onboarding flows, personality assessments, & outcomes readout. Also delivered a cross-functional product strategy workshop in addition to board-ready executive summaries.
Project-based | Strategy | Multicultural
Family Law & Custody Communications Platform
Ongoing product advisory for a platform streamlining communication in custody and family law contexts. Engagement covers UX evaluation of development (iOS app, sharing, import flows), product marketing, quality assurance, & branding strategy
Advisory | End-to-end | Legal Tech
Case Studies
Statistical analyses of >700 patients investigated patient engagement, satisfaction, & accuracy related to patients’ race & SES. Answered: "how might Memora Health quantify AI bias?"
Quantitative | Evaluative | Health Equity
Semi-structured interviews explored clinicians' & care teams' workflows, expectations, & needs. This telehealth system connected rural ER patients & remote psychiatrists.
Qualitative | Generative | Lean UX
This product strategy & 8-week research timeline was made to design and refine a mobile app for CD patients to monitor key gut biomarkers. Methods include interviews, focus groups, & unmoderated tree testing.
Mixed Methods | Service Design
Writing
"There is no great writing, only great rewriting."
– Justice Louis Brandeis
Co-published with UPenn researchers in BMJ Innovations on methodology for evaluating healthcare AI chatbots. Established processes for proactive training and safety validation across patient interactions.
AI Eval | Peer-Reviewed Publication
Established a four-pillar framework for evaluating digital health equity: accessibility, literacy, demographics, and identity. Includes analyses of AI performance across 1,400+ patients in a UPenn postpartum program.
Technical Writing | Public Health Tech
Explored ethics of machine intelligence via a popular tv show, arguing that how we treat AI reflects power dynamics rather than principled ethics. Written before AI was a mainstream concern.
AI Ethics | Public Writing
And now, a poem
Behind the Scenes:
Every artificial object secretly saturated with intent, its subjects in mind, by design
Each item that comprises life involved countless meetings to (re)invent, iteratively built, by design
Everything engineered to fix problems we are now likely to prevent, taken for granted, by design.
June 9th, 2018
by Karl
About
"We can only see a short distance ahead, but we can see plenty there that needs to be done.”
– Alan Turing
About
"We can only see a short distance ahead, but we can see plenty there that needs to be done.”
– Alan Turing
I run Desire Path Research, an independent UX research and AI evaluation consultancy working with founders and product leaders who need to understand how real people experience what they've built.
My backround is in research. I studied the sense of smell at the University of Chicago, the neuroscience of music and pain psychology at McGill, and the placebo effect during an internship at the NIH. After graduating, I spent two years in clinical research at a biopharma startup before moving into user research and tech.
In 2019, I started as UX researcher and employee #8 at Memora Health, scaling with the company from Seed through Series B. While there, I co-published research on AI evaluation methodology with UPenn in BMJ Innovations, which shaped how I think about what it means to evaluate language-based systems rigorously. I then consulted with ZS Associates on digital health products for Fortune 500 clients.
Before going independent, I led UX research at Kooth Digital Health, where I took their product Soluna from launch to >100K active users serving 13- to 25-year-olds' mental health, and designed LLM evaluation frameworks built around expert review panels.
The common thread across all of my experiences: a focus on whether the systems we build actually hold up when real people encounter them, and what rigorous research can tell us when they don't.
Outside of work, most of my time revolves around music: listening, producing, and playing instruments. Otherwise, you might find me tending to my plants, making coffee, or somewhere along Lake Michigan.
