TL;DR
AI has made building faster than ever, but feedback still runs on multi-week recruiting cycles. Synthetic audiences trained on your customer data give you real signal in minutes, right inside the tools you build in.
Myth: AI feedback is too biased to trust. Busted: sycophancy is real, and it's why synthetic audiences are built for fast exploration, not a single verdict.
Myth: one persona can't represent your audience. Busted: neither can one real participant. Synthetic scale means testing far more audiences than any study budget allows.
Myth: AI hallucinates and can't be trusted. Busted: it's fast and cheap enough to run more than once, something a real-participant study can't afford.
Where real people still lead: emotional or sensitive topics, cultural nuance, and the honest, unpolished answers only lived experience produces. For fast signals on structure, flow, and friction, the evidence from NN/g, MeasuringU, and peer-reviewed research says the myths don't hold up.

Design and code got 10x faster. Validation didn't. So teams ship on instinct and find out from the churn report. Design tools generate five directions before lunch, and code ships the same day it's written, but feedback is still stuck running at the old pace. That's the gap synthetic audiences were built to close. Stop shipping decisions nobody validated.

Naturally, a shift that fast gets loud skepticism alongside it. Most of the pushback traces back to a small set of claims that sound damning until you actually check them against synthetic user testing research. For a closer look at matching evidence to risk, see our guide on how to speed up user research when nobody has time to recruit.

So let's take the 13 claims that come up most often in published research, in community debate, and in objections we hear directly, and check each one against what's actually been published and verified.

Myth 1: "It's all hype, synthetic testers don't hold up to real research standards"

Busted: Nielsen Norman Group's own published evaluation doesn't dismiss synthetic users. It lays out specific, legitimate uses for them, and those uses line up with the fast, iterative work teams spend most of their time on.

NN/g published an entire section on using synthetic users responsibly, because the findings support a clear, well-defined set of uses: exploring a brand-new domain before you know what questions to even ask, generating hypotheses to test later, and piloting a research plan to catch a leading or confusing question before a real participant sits down. That last one alone pays for itself: a synthetic audience that flags a bad question in your script saves a real recruited participant's time finding the same flaw the hard way.

That's not hype. It's a documented use case, and synthetic audiences handle it at a speed no recruiting cycle can match.

Myth 2: "AI feedback is too biased to trust"

Busted: the bias is real, but it's predictable, and a predictable bias is one you can plan around. NN/g tested this directly. AI has a well-documented tendency toward sycophancy; it wants to please. In NN/g's own evaluation, a synthetic user asked about a proposed drone-delivery feature, walked through a few surface-level concerns, then still landed on an enthusiastic, upbeat verdict. The same pattern showed up across their testing: synthetic responses consistently leaned favorable, rarely landing on genuine pushback.

That's why concept testing is the riskiest place to lean on synthetic audiences alone. Validating a new idea by asking if people like it is already a shaky method with real participants. With an AI that's structurally inclined to be encouraging, it's close to asking a mirror for a second opinion.

The vendor behind the tool NN/g tested has reportedly worked to reduce this tendency. By NN/g's own account, sycophancy remains an ongoing challenge even after that work. So treat every synthetic "yes" as a hypothesis, not a verdict.

Myth 3: "AI has no real experience, so its feedback doesn't mean anything"

Busted: synthetic audiences were never built to replace lived experience. They answer a faster, narrower question: does this flow work, and where does someone get stuck? That's what slows most teams down.

TheySaid's synthetic audiences really use your product, on screen and out loud. Because they're trained on your customer data, they're grounded in how your actual customers behave, not just a written description of them. What they produce is a read on how that audience would react, which is the layer most day-to-day design decisions run on: is this confusing, is this the right order, does this make sense on first look. It's answered in the time it takes to write a prompt instead of the days a recruiting cycle takes.

Where the two genuinely diverge is worth naming plainly: NN/g's own research on course completion found a real participant gave the messy true answer (finished three of seven modules, moved roles, never went back), while the synthetic version of the same persona claimed a clean completion. That gap is why synthetic audiences are the fast first pass, not the only pass, on anything where the honest, unpolished answer is the whole point.

Myth 4: "One synthetic persona can't represent your whole audience"

Busted: neither can one real participant. What synthetic scale actually changes is the economics: testing far more distinct audiences than any single study budget ever allowed.

An AI audience profile, grounded in real customer behavior rather than generic internet data, is a composite, not a single real person. If you're new to the concept, our guide to what a synthetic persona actually is covers the basics. NN/g's own framing is that any one profile blends thousands of people into one composite, but a single 5-person study does something similar in reverse, generalizing from a handful of people to an entire audience. Synthetic audiences flip that trade-off: instead of one narrow sample, a team can run as many distinct audiences as the decision needs, at a fraction of the cost.

Myth 5: "Synthetic personas can't fairly represent global or non-Western audiences"

Busted: the bias is real and documented, and it's why TheySaid lets you run synthetic and human audiences side by side, then compare what each one tells you. A 2026 research paper out of Adobe, presented at the CHI conference, documents WEIRD bias in synthetic personas: a skew toward Western, Educated, Industrialized, Rich, and Democratic populations, because that's who is most represented in the training data.

This is where the distinction between demographic profiling and psychographic profiling earns its keep. It's also why grounding matters. A synthetic audience trained on your own customers reflects your market, not the internet's average. Getting someone's age, role, or region right is the easy part a synthetic audience handles well. Capturing how someone from an underrepresented market actually thinks and decides is the part that needs a real human audience layered in, which is why TheySaid runs synthetic audience and human studies from one Unified Test Builder, with results compared side by side. That's a reason to pair the two, not to skip synthetic testing.

The next eight claims get checked against Nielsen Norman Group, MeasuringU, a documented case of AI-persona misuse, and how teams actually use synthetic audiences today.

Myth 6: "AI hallucinates, so you can't trust the output"

Busted: hallucination is real and structural, but synthetic audiences are fast and cheap enough to run more than once, something a real-participant study can't afford. Nielsen Norman Group's research on AI hallucinations, published by Page Laubheimer, is direct about this: hallucinations aren't bugs in the traditional sense; they're a structural side effect of how large language models work.

Nielsen Norman Group's own hallucinations research makes the case for verifying AI output instead of taking it at face value. With synthetic audiences, the simplest check is to run the same test more than once and see whether the answer holds up. For a real participant study, that would mean re-recruiting and re-running an entire session. For a synthetic one, it means hitting send again, in seconds, until the read on your design stabilizes. Treating a single synthetic response as one data point instead of a verdict turns a known limitation into a non-issue.

See how users react, right where you build

Synthetic Audiences give you feedback in minutes, inside the tools you already use.

Try Synthetic Audiences

Myth 7: "AI performs tasks too well to catch the problems a real user would"

Busted: that's a feature, not a disqualifier. It confirms your structural logic holds, which is a useful signal on its own. Independent researchers at MeasuringU found ChatGPT vastly outperformed real users in tree testing, a common method for checking whether people can find things in a navigation structure.

Read the right way, that result tells you something: a synthetic audience breezing through your navigation confirms the structure itself is logically sound. Whether real people still stumble on wording, tone, or a moment of hesitation is a separate, later question, one real participants are there to answer on top of a structure synthetic audiences already confirmed makes sense.

Myth 8: "Using AI personas is inherently deceptive"

Busted: real deception means undisclosed AI on unwitting real people. An internal testing tool your own team knows is AI is a different thing entirely. This objection comes up often, and it deserves a real answer instead of a dismissal, because a real, well-documented case gets cited in support of it. It's just not the case people think it is.

In 2025, researchers at the University of Zurich secretly deployed AI-generated personas, including invented identities like a rape survivor and a trauma counselor, into Reddit's r/changemyview community, without disclosing to a single participant that they were talking to a bot. Reddit's response was blunt, calling the study "deeply wrong on both a moral and legal level."

That's a real ethical violation, and it's worth naming plainly. But it's the opposite scenario from synthetic testing: real, unconsenting people were the ones being deceived. A synthetic audience used for product validation runs entirely inside a team's own workflow. It never talks to a real customer. Nobody outside the team is fooled into thinking they're speaking with a real person. The ethical line was never "AI personas exist." It's who's being deceived, and whether they agreed to be part of it.

Myth 9: "Synthetic testing only works once you have a live product"

Busted: with TheySaid, synthetic audiences react to a design before a single line of production code ships, and they do it inside the tools where that design is being built. Every design, ticket, and PR can be validated before it ships.

TheySaid MCP runs inside Claude Design, v0, Replit, Bolt, and Lovable. Point it at a design you just generated, and it returns feedback plus a ready-to-use fix-it prompt without leaving that tool. Ticket testing covers the planning side: when a Jira ticket moves to design-done, validation kicks off automatically, and TheySaid works inside Monday.com and ClickUp, too. It's one of the clearest differences between TheySaid and other AI user testing tools.

That changes when feedback happens. Instead of testing what you already built, you check each direction while it's still cheap to change, which is when feedback saves the most engineering time.

Myth 10: "Synthetic feedback is a black box. You get a verdict, not the reasoning"

Busted: TheySaid's Synthetic Audiences share their screen and think out loud as they work through a task, so you see where they hesitate and why, not just the final answer.

That changes how you use the result. Instead of trusting a summary, you can watch the exact moment a synthetic audience gets confused, check whether its reasoning holds up, and make the call yourself. Clips & Highlight Reels then pull those moments into a short reel your team can review in minutes, so the evidence travels with the recommendation.

Myth 11: "Once AI is reading your feedback, you lose control over what it concludes"

Busted: in TheySaid, control over how AI interprets and acts on your data is part of the product, not an afterthought.

The concern is fair. An AI that summarizes feedback can also flatten it, overstate it, or act on it in ways nobody signed off on. TheySaid gives you governance over how AI extracts insights and what actions it takes. Skills add consistency on top of that: reusable prompts for recurring work, like executive summaries or cohort analysis, so every team gets output built the same way and reviewed the same way.

Myth 12: "Validation ends the moment you ship"

Busted: shipping is where a lot of real feedback starts, and TheySaid keeps listening after launch.

The Experience Hub monitors and scores your critical customer experiences on an ongoing basis, then recommends what to improve. Add advanced intercepts, mobile app testing, and auto-translation, and you hear from real customers at the moments that matter, in their own language, without waiting for a churn report to tell you something went wrong.

Myth 13 (the overcorrection): "If synthetic testers have this many limits, they're useless"

Busted: dismissing synthetic audiences over their limits is like dismissing a smoke detector because it can't put out the fire. That was never its job.

Every myth on this list traces back to the same shape: a real, documented limitation that only matters if you're using synthetic audiences for a job they were never built to do. Used for what they're built for (fast signal on structure, flow, and obvious friction), they hold up. And when they're trained on your own customer data, they get closer still to how your real customers think, decide, and get stuck.

How TheySaid Fits Into the Way You Build

Validation shouldn't be a separate step. TheySaid is the validation engine for the AI-powered product lifecycle, built on three ideas: embedded, not adjacent; evidence, not opinions; and gates, not gatekeepers. It runs inside the tools and workflows where your product takes shape, so you can see how users will react before anything ships. Validation is no longer a project. It's infrastructure.

Used the way this list points to, that's not a shortcut around talking to real customers. It's what lets a team pressure-test that many directions before committing engineering time to any of them, so the recruiting hours you do spend go toward the decisions that actually need a real person's honest, unpredictable reaction. That's the whole point.

What TheySaid brings to the table

This whole list keeps circling one problem: building got faster, feedback didn't. Here's how TheySaid closes that gap:

• Synthetic Audiences are custom AI profiles trained on your customer data that act like real customers: they follow instructions, share their screen, and think out loud the way a moderated participant would, so you watch the reasoning instead of just reading a verdict.
• Ticket Testing kicks off validation automatically when a Jira ticket hits design-done, so nothing moves to development untested.
• TheySaid MCP runs inside the design and code tools you already build in, including Claude Design, v0, Replit, Bolt, and Lovable, and returns instant feedback plus a ready-to-use fix-it prompt without leaving that tool.
• Audiences lets you build both human and synthetic audience profiles that MCP pulls from automatically, so evidence gets gathered from the audiences that actually matter for a given decision, not a generic default.
• AI Companion works the way Claude Design or Claude Code does: tell it what you want validated, and it runs the study for you instead of you configuring one from scratch.
• Experience Hub monitors and scores your critical customer experiences on an ongoing basis, with recommendations for what to improve, so validation doesn't stop the moment something ships.
• Skills are reusable prompts for recurring in-product work, like generating an executive summary or running a cohort analysis.

That's just the start. TheySaid also brings the Unified Test Builder, mobile app testing, Clips & Highlight Reels, and so much more. Get validation in minutes, not weeks. Try Synthetic Audiences.

FAQs

Are synthetic testers reliable for making product decisions?

Yes, for the fast, iterative decisions they're built for. With TheySaid, synthetic audiences are trained on your customer data, so their feedback reflects how your customers actually think and decide. For the smaller set of decisions carrying real emotional or financial weight, pair the fast synthetic pass with real participants, which is why TheySaid runs both from one platform.

Can synthetic testers predict how real users will behave?

Closely, when they're grounded in real customer behavior. TheySaid's synthetic audiences are trained on your customer data, so they reflect the priorities and habits of the people you actually build for, not a generic profile. They're strongest on the day-to-day questions most design decisions run on, answered in minutes instead of the days a recruiting cycle takes.

Should niche or specialized audiences ever get a synthetic-only test?

Pair synthetic scale with a real human audience for niche or underrepresented populations. That's what TheySaid's Audiences feature is built for: synthetic coverage everywhere it's strong, real people layered in where cultural or lived-experience nuance matters most.

What's the single best way to start using synthetic testers?

Pilot a research plan before running it with real people. A synthetic audience catching a confusing or leading question in your interview script costs nothing and protects a real recruited participant's time later.

Do synthetic testers give the same answer if you ask them twice?

Not always, and that's easy to work around, because synthetic audiences are fast and cheap enough to run more than once. A real-participant study can't afford to re-run a session five times to check consistency. A synthetic one can, in the time it takes to hit send again.

Can AI accurately simulate how users navigate a website or app?

It confirms whether your structure is logically sound, which is a useful signal on its own. Independent testing found ChatGPT vastly outperformed real users on tree-testing tasks, meaning it's strong for validating structure. Real participants add the layer on top: whether wording, tone, or hesitation still trips someone up.

Are synthetic personas biased toward certain cultures or demographics?

Generic ones can be. Published research documents "WEIRD" bias in synthetic personas, skewing toward Western, Educated, Industrialized, Rich, and Democratic populations. TheySaid's synthetic audiences are trained on your customer data, so they reflect your market instead of the internet's average. For underrepresented groups, pair them with a real human audience in TheySaid's Audiences feature.

Is it unethical to use AI-simulated personas in research?

Not when it's an internal tool your own team knows is AI; the real ethical violations look nothing like that. A well-documented 2025 case involved researchers secretly using AI-generated personas on real Reddit users without consent, which Reddit itself called a serious ethical violation. Standard synthetic testing tools never put AI in front of a real customer at all.

Can synthetic testers give feedback before anything is coded?

Yes. TheySaid MCP runs inside design and code tools like Claude Design, v0, Replit, Bolt, and Lovable, so synthetic audiences can react to a design the moment it's generated. Ticket testing goes a step further and kicks off validation automatically when a Jira ticket hits design-done.