You share a study link on a Monday, and by Tuesday there are three hundred completed sessions in the data view. It is a good feeling — until the second thought arrives: how many of these are real, and how many of the real ones were actually paying attention? Anyone who has run an unsupervised online study has felt the gap between a full response count and a trustworthy one.
The good news is that this is a well-studied problem, and the tools for handling it are concrete. The less comforting news is that no single check settles it, and some of the most popular checks carry costs of their own. What follows is a practical account of what the methods literature actually supports — written for researchers deciding how much to trust the sessions coming in, and how to plan quality controls before launch rather than firefighting after.
Two different problems that get lumped together
"Bad data" online is really two separate problems, and they call for different defences.
The first is automated or fraudulent participation — bots, scripts, and click-farms that complete your study for the incentive without a human meaningfully doing the task. This is an adversarial problem: someone is actively trying to look like a valid participant.
The second is inattentive responding by real people — a genuine human who is distracted, multitasking, rushing for payment, or satisficing (doing the minimum to get through rather than engaging with each item). No deception is involved; the person is real, but the responses do not reflect the construct you meant to measure.
Conflating the two leads to weak defences, because a check that catches one often misses the other. A question only a human could answer will not stop a careless human; a consistency measure tuned for careless humans will not, on its own, unmask a well-built bot. It helps to keep the two threats labelled and defend against each on purpose.
The bot problem is real, and it is documented
It is easy to dismiss bot worry as paranoia. The evidence says otherwise. In one widely discussed episode on Amazon Mechanical Turk, researchers traced a marked drop in data quality to respondents using virtual private servers — in effect routing their connection through another country's computer — to disguise their location and slip past the platform's geographic filters (Kennedy et al., 2020). The countermeasure they proposed was itself technical: checking each respondent's IP address against databases of known server farms.
The problem is not confined to paid crowdsourcing panels. A health-research team recruiting through an open online survey found that, after a five-step cleaning protocol, they had to discard roughly 62% of their initial responses as bot-generated before they were left with a usable sample (Griffin et al., 2022). Two lessons travel well beyond that specific study: the fraction of junk can be large, and an open, incentivised link is the most exposed thing you can publish. Anything that broadcasts "complete this and get paid" to the open web invites automated traffic.
Real people, not paying attention
Even with every bot filtered out, a second problem remains — and it is often the larger one for a typical cognitive or survey study. In two studies of survey data, Meade and Craig found that careless responding is common enough to distort results, and that it comes in at least two flavours: random responding (answers with no relationship to the items) and non-random responding such as straight-lining down a scale (Meade & Craig, 2012). The two patterns leave different fingerprints, which is why a single detection method rarely catches everything.
How common is inattention? When Oppenheimer and colleagues introduced the instructional manipulation check — an item embedded in a task that quietly instructs participants to respond in an unusual way, so that only those actually reading the instructions answer correctly — failure rates in their samples ran from 14% to 46% (Oppenheimer et al., 2009). A meaningful slice of ordinary, well-meaning participants simply are not reading as closely as the design assumes.
Data quality online is also not static. A four-wave study spanning 2018 documented a measurable drop in Mechanical Turk data quality over a matter of months, with more participants failing validity checks, lower reliability on a standard personality measure, and even failures to reproduce well-established effects (Chmielewski & Kucker, 2020). A source that was clean last year is not guaranteed to be clean this year — quality is something to re-check per study, not assume once.
The usual defences — and where they cost you
The standard toolkit is worth knowing precisely, including the fine print.
Attention checks and instructional manipulation checks ask a question whose correct answer proves the participant read the material — a classic form is an instruction, buried in a long prompt, to ignore the obvious answer and click somewhere specific instead. They genuinely surface inattentive respondents. But screening on them is not free. Excluding participants after the fact can quietly break the random assignment your design depends on, if the people you remove differ systematically across conditions or on the very traits you are measuring. Zhou and Fishbach showed this vividly for dropout: across standard paradigms, 30–50% of participants quit before finishing, and because that attrition differed by condition, analysing only the "good" completers produced conclusions that looked clean but were false (Zhou & Fishbach, 2016). The same logic applies to any post-hoc exclusion rule: the moment you drop people, you are reshaping your sample, and that reshaping is only safe if it is unrelated to condition and outcome. The defensible move is to decide and preregister your exclusion rules before you see the data, and to report results with and without exclusions.
Consistency and response-pattern measures — checking whether answers to similar items agree, flagging long runs of identical responses, or spotting impossibly fast completion — catch careless humans that attention checks miss, and need no extra items bolted onto your study. They are probabilistic, though: a fast, consistent responder is more likely valid, not certainly so.
Simple bot barriers — completion challenges, honeypot fields, and reCAPTCHA-style gates — raise the cost of trivial automation. They are a filter, not a wall, and the more sophisticated the fraud, the less they catch on their own.
No single one of these is decisive. Their value is cumulative, which is the argument for combining a few rather than leaning hard on one.
What paradata is, and what it can tell you
Running underneath all of this is a quieter source of evidence: paradata. The term simply means data about the process of responding rather than the responses themselves — the "how" alongside the "what." In a web study, paradata includes things like how long each screen took, whether the browser tab lost focus mid-task (a sign of switching away), how many times a page was revisited, the input device, and fine-grained timing of clicks and keystrokes. Survey methodologists distinguish server-side paradata (what the server can log, such as timestamps) from client-side paradata (richer signals captured in the participant's browser, such as focus changes and per-item latencies), with the client-side stream offering a more detailed picture of the response process (McClain et al., 2019).
Paradata is powerful precisely because a participant is not curating it. Someone can answer an attention check correctly and still have spent four seconds on a task that takes a careful person two minutes, or have had the tab in the background for most of the session. Those process signals are hard to fake casually and often reveal what the responses alone conceal.
Paradata is also circumstantial. A long completion time might mean careful engagement or a coffee break; a tab-switch might be cheating or a legitimate interruption. Paradata is best read as a signal to help you triage — a way to rank sessions for a closer look and to combine with content checks — not as a verdict that a given participant is fraudulent. Treated as proof, it will wrongly condemn honest, slightly distracted people.
Where Open Lab fits
Open Lab surfaces this kind of process-level evidence for you (see the documentation). As sessions come in, it derives paradata-based quality signals and flags each participant as verified, suspicious, or AI — a triage layer to help you spot likely bots and low-quality sessions without hand-inspecting every record. The framing matters, and it is the same one the literature above recommends: these flags are a signal to help you triage, not a final judgment. A "suspicious" flag is an invitation to look closer and to weigh it alongside your own attention and consistency checks — not an instruction to delete a row.
One practical detail matters before any of this helps: these quality signals are off by default. Collecting the process-level paradata they depend on is something a researcher switches on per study, in the study's Data Quality centre, under the "Bot detection" tab — it does not run silently in the background of every study. That is a deliberate, privacy-first default, not an oversight. Paradata is additional behavioural data about your participants — how long screens took, when a tab lost focus, fine-grained input timing — and a sound principle of data minimisation (collect only what the study actually needs, and what participants have been told about) says you should not gather that extra layer on everyone unless you mean to. Collecting it by default would quietly do exactly that, whether or not the researcher wanted it or disclosed it in their consent form and ethics application. Making it an explicit opt-in puts the choice — and the responsibility to declare it — with the researcher, which also fits Open Lab's broader privacy posture (EU hosting, encryption, and configurable retention). So if you want the verified / suspicious / AI signals for a study, enable Bot detection in its Data Quality centre when you set the study up, and note the paradata collection in your consent and ethics materials.
Two design-time levers matter as much as any after-the-fact flag:
- Pilot before you launch. Using Preview to walk your own study as a participant catches the instruction that is easy to misread, the trial that runs too fast, or the flow that invites skipping — the design flaws that manufacture "inattentive" data in the first place. The cheapest bad session is the one your design never provokes.
- Protect your randomisation and your exclusion plan. Because dropping participants reshapes your sample, decide exclusion rules in advance, keep them symmetric across conditions, and report outcomes with and without them. Building conditions with proper randomisation in the study flow, and treating quality flags as one preregistered input, keeps the comparison you publish honest.
It is worth ending on the encouraging half of the picture, because the point is not that online data is untrustworthy. When online studies are designed and screened with care, classic effects reproduce: one evaluation replicated a battery of cognitive paradigms — Stroop, task switching, Flanker, Simon, Posner cueing, attentional blink, and more — on web participants, while being candid about where browser timing is noisier (Crump et al., 2013). Web-collected data can be excellent. It just has to be earned — with defences planned before launch, several signals combined rather than one trusted, and process-level flags read as triage rather than proof.
The bottom line
Trusting web-collected data is not about finding one perfect filter. It is about (1) separating the two threats — automated fraud and inattentive humans — and defending against each on purpose; (2) combining content checks (attention and consistency items) with process evidence (paradata and quality signals), since each catches what the others miss; (3) deciding exclusion and randomisation rules before the data arrives, so that cleaning does not quietly manufacture the result; and (4) reading every quality flag as a prompt to look closer, not as a verdict. Do that, and a full response count can become a trustworthy one.
