Dr. Llewyn Paine is an AI consultant, product strategist, and innovation leader with nearly two decades of experience working in emerging technology. As the principal of Llewyn Paine Consulting, she specialises in helping product leaders implement evidence-based rigour and responsible AI practices in their UX and design workflows. Her career includes leading initiatives on intelligent agents and physical AI at Microsoft and developing experimental media at Disney. Paine serves as the lead curator for Rosenfeld Media’s Designing with AI conference and frequently speaks at major institutions, including the Library of Congress.
How to AI UXR is supported by Strella, an AI-powered customer research platform that partners with you to build, moderate, and synthesise interviews, allowing you to go from question to actionable insights in just a few hours.
In This Conversation
As the How to AI UXR map shows, research teams are now using AI across every part of the research workflow and beyond. They’re drafting screeners, summarising interviews, shaping reports, coaching non-researchers, automatically routing research requests, building end-to-end research systems—and learning just how much those systems cost to maintain. Velocity is the name of the game, but the important question isn’t whether AI can make research faster; it’s whether research teams can move faster and preserve the judgement, evidence, and methodological care that make research valuable in the first place.
In this episode, Llewyn argues that AI in research should begin with judgement. Rather than treating AI adoption as a race towards more output, she makes the case for returning to the disciplines researchers already know well, including jobs to be done (JTBD), systems design, social science, statistics, and evaluation.
This episode is a useful corrective to the pressure many teams feel to “AI everything.” Llewyn’s view is not that researchers should reject AI or position themselves as blockers to progress; it’s that they should understand the tools well enough to assess risk, ask better questions, and decide where AI supports better outcomes rather than simply producing more material. This conversation will give you a clearer way to assess where AI belongs in your research system, what evidence to ask for, and what to watch for before you trust the work it helps produce.
The How to AI UXR Map
This series builds on the insights shared in the How to AI UXR map, a five-page map that charts key trends, helps you pinpoint your AI maturity level, and offers practical, real-world applications you can adapt to your research systems.
Download the Map
In this episode, we cover:
- Why a practice being popular does not make it good research practice
- The three lenses research teams need when evaluating AI: stakeholder needs, social science, and computer science
- Why teams should begin with the stakeholder’s job to be done before choosing an AI tool
- What an AI harness is, and why the model itself is only one part of the system (Llewyn shares a great Teenage Mutant Ninja Turtles analogy)
- Why Llewyn draws a distinction between AI that increases output and AI that improves outcomes
- Why qualitative synthesis is one of the most tempting and highest-risk uses of AI in research
- The difference between accuracy problems, such as incorrect quotes, and omission problems, where AI misses the most interesting or important insight
- Why operational use cases, including coaching, routing, training, and support, may be among the most valuable applications of AI for research teams
- Why researchers should understand enough about AI to ask for evidence, challenge weak claims, and avoid becoming passive consumers of vendor or influencer promises
- How token costs may push teams towards better system design, clearer context, and more thoughtful use of AI
- Why researchers do not need to carry evaluation work alone, and how they can partner with engineering and others to assess whether tools are doing what they claim
Partway through the episode, as a leader building one of the tools these researchers are exploring and using, Priya Krishnan, the cofounder and COO of Strella, shares her take on the conversation.
Things Referenced
- Jobs to be done (JTBD) is a framework for understanding what stakeholders are trying to accomplish, rather than beginning with a tool or deliverable.
- AI harnesses are the surrounding software infrastructure that gives a model access to tools, context, rules, workflows, memory, and guardrails.
- Evals (evaluations) are structured ways of testing whether an AI system is producing outputs that meet defined criteria.
- Markdown is a lightweight plain-text formatting language that helps humans and machines structure information clearly.
- CSV files are simple spreadsheet-style files often used to move structured data between tools.
- Tokens are the word fragments and other units of text that AI systems process as input and output.
- Designing with AI is a Rosenfeld Media conference that Llewyn helps curate.
- Paul Ford is a technology writer and software builder (best known for Bloomberg’s “What Is Code?”) who argues that AI makes human judgement and accountability more important, not less.
- Oen Michael Hammonds is a UX and AI practitioner who uses “AI speed bump” to describe adding deliberate friction and checks so teams don’t ship unsafe or unreliable AI systems.
- World Usability Day is an annual global event (held on the second Thursday in November) focused on usability and human-centred design through talks and local meetups worldwide.
- Teenage Mutant Ninja Turtles is a pop culture franchise whose villain Krang operates a mechanical body, used here to illustrate an AI “harness” (system) versus the model (brain).
- Pac-Man is a classic 1980 arcade game used as a metaphor for tokens as the “bits” an AI system consumes and produces.
Connect with the Guests
- Llewyn Paine, founder and consultant, Llewyn Paine Consulting
- Priya Krishnan, cofounder and COO of Strella
How to AI UXR is supported by Strella, an AI-powered customer research platform that partners with you to build, moderate, and synthesise interviews, allowing you to go from question to actionable insights in just a few hours.
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit
www.theresearchopsreview.com
