Data Quality Centre

Overview

The Data Quality Centre is a per-study workspace for screening the data you collect: reviewing completion times, detecting duplicate submissions, analysing response patterns, checking for bot-like or inattentive behaviour, and managing which participants are included in your analysis. Its summary line describes its purpose exactly: "Review completion times, detect duplicate submissions, analyse response patterns, and manage exclusions. Exclusions automatically propagate to data exports." The Centre is currently labelled Beta in the interface.

Every signal the Centre shows is a triage aid, not a verdict. Duplicate devices, fast responses, tab-switching and bot-suspicion scores are heuristics that help you decide where to look more closely — they are not proof that a submission is invalid. You always make the final include/exclude decision yourself using the quality flags described below.

This page accompanies the Open Lab blog post "Can you trust web-collected data?" and documents every tab and control in the Centre as it actually behaves.

Opening the Data Quality Centre

Open the Centre from a study's dashboard. The Quality stats card has an Open Data Quality link, and quality alerts (for example "suspicious sessions" or "AI-detected sessions") include a Review link to the same place. The direct URL is https://research.open-lab.online/dashboard/studies/[study-id]/quality.

Plan availability

Bot and AI detection is a data-quality baseline available on every plan, including Free. On all tiers you can enable paradata collection, review the bot-suspicion score, signals and the Verified/Suspicious/AI-detected verdict for every participant, see the summary bar, and flag or exclude individual sessions from the Bot Detection tab. Knowing whether your data is contaminated is never paywalled.

A Basic plan or higher unlocks the deeper analysis and bulk workflow: the full Completion Time, Participation Integrity and Response Patterns tables (on the Free tier these show up to five participants), the exclusion rule builder and the "Exclude all duplicates" action, and the exclusion-log CSV export. On the Free tier a banner in those areas links to plans; the exclusion rule builder cannot be applied on the Free tier.

Summary bar, quality flags & exclusions

A summary bar sits above the tabs and shows five counts for the study: Total, Completed, Flagged, Excluded, and Duplicate devices.

Every tab's participant table has a Flag column with three states you set manually:

None — no action; the participant is included in analysis.
Flagged — marked for attention, but still included. Use this to note something for later review without removing the data.
Excluded — removed from analysis. Setting a participant to Excluded automatically sets includeInAnalysis = false on all of that participant's datasets, so the exclusion propagates to your data exports and analysis views. Changing the flag back to None or Flagged re-includes the data.

When at least one participant is excluded, a Download exclusion log button appears. It exports a CSV of every flagged or excluded participant with their stable ID, status, quality flag, notes, completion time, enrolment and completion timestamps, duplicate-device status and timezone — a record of your quality decisions you can keep alongside your analysis.

The five tabs

The Centre is organised into five tabs. The first two load with the page; the Dropout Funnel, Response Patterns and Bot Detection tabs load their data the first time you open them.

1. Completion Time

Completion time is measured from when a participant enters the study to the last time their progress was recorded, so it includes every component — landing page, consent, tasks and redirect screens. The tab shows summary statistics (Median, Mean, SD, Min, Max, P5 and P95) and a histogram of the completion-time distribution for included participants.

An exclusion rule builder lets you exclude participants by completion time. You can Preview a rule (a dry run that reports how many participants and which codes would be affected) before applying it. Three rule types are available:

Absolute cutoff — exclude anyone faster than a minimum or slower than a maximum number of seconds (leave a bound blank to skip it).
SD-based (Ratcliff-style) — exclude participants whose completion time deviates by more than N × the standard deviation from the mean. The default multiplier is 2.5.
Quantile trim — trim the fastest and/or slowest participants by a percentage from each end.

Applying a rule sets the matching participants to Excluded and records a note that they were auto-excluded by that rule.

2. Participation Integrity

This tab checks whether the same person or device may have taken the study more than once. It surfaces four signals per participant:

IP hash — a one-way hash of the participant's IP address. Identical hashes mean the same network origin. The raw IP address is never stored; only the salted hash is kept, computed when the data is received. Matching hashes can also happen between unrelated people on a shared Wi-Fi network or VPN, so treat them as a hint rather than proof.
Device fingerprint — a short hash of the browser/user-agent, operating system and screen resolution. When two participants share the same fingerprint it is likely the same physical device.
Timezone — the browser-reported timezone (for example Europe/London). A timezone inconsistent with a participant's stated location can indicate a VPN or proxy.
Duplicate device — flagged when a participant's device fingerprint exactly matches another participant in the same study.

Summary tiles show total participants, unique IP hashes, duplicate devices and unique timezones. When duplicate devices are present, an Exclude all duplicates button lets you exclude every participant sharing a device fingerprint in one action. Device fingerprint and timezone data are captured when participants complete the study via its link; participants added another way will show no data in those columns.

3. Dropout Funnel

The Dropout Funnel shows, task by task in order, how many participants reached a task (submitted any data for it) versus how many completed it (submitted a final response). Each step reports the in-task dropout count and rate, and how many participants were lost before reaching that task. An overall summary gives the number enrolled, the number who reached the first task, the number who completed the last task, and total dropouts. This helps you find where in your study people are abandoning it.

4. Response Patterns

This tab looks for careless or non-engaged responding within the task data. Per participant it reports:

Median response time (RT) — the median duration (in milliseconds) between a stimulus appearing and the participant responding. Very low medians suggest the participant was not reading the content.
Fast response rate — the proportion of responses faster than a threshold you set. A participant is flagged as a fast responder when at least 80% of their responses are below the threshold across at least five items.
Straight-lining — detected when 90% or more of a participant's responses are the same value, across at least five items.
Unique responses — the number of distinct response values, which reinforces straight-lining detection when it is very low.

The fast response threshold is configurable (the interface notes typical reading time is roughly 1,500–3,000 ms). A sender filter lets you restrict the analysis to a single component of your study (each event carries a sender field identifying which screen or task node produced it) so you can analyse one instrument in isolation. A participant is marked with a suspicious pattern when they straight-line or when they are a fast responder. If your study uses end-to-end encryption, response-pattern analysis is limited or unavailable, because the content needed to compute these signals is encrypted; the tab tells you when that is the case.

5. Bot Detection

The Bot Detection tab estimates how bot-like or automated each submission looks, using paradata — passive, aggregate counts of a participant's interaction with the task. Because paradata is extra behavioural process data, it is opt-in and off by default. The rest of this section explains why, how to turn it on, and how to read the results.

Why paradata is opt-in (off by default)

Paradata collection is disabled for every study until a researcher deliberately enables it. This is a privacy-by-default choice grounded in data minimisation and informed consent: the extra behavioural process data (how a participant clicks, types, switches tabs and pastes) is only gathered when you have decided you need it, so that you can disclose it in your consent form and ethics application before collecting it. Studies that do not need bot screening never collect this data at all.

How to enable paradata for a study

At the top of the Bot Detection tab is a Paradata collection control with an Enable checkbox. Ticking it turns paradata collection on for that study only. The setting applies to future sessions: once enabled, wait for new participants to complete the study — participants who finished before you enabled it will not have paradata. When no paradata has been collected yet, the tab shows a "Paradata not yet collected" notice.

When enabled, Open Lab silently counts clicks, keystrokes, tab switches and paste events during task execution. No response content is recorded — only aggregate counts per session (for example "42 clicks, 130 keystrokes, 1 tab switch, 0 pastes"). Keystroke values, clipboard contents and the responses themselves are never captured by paradata. Nothing is collected during preview or demo runs of a study.

Reading the bot-suspicion score and signals

Each participant with paradata gets a risk score from 0 to 100, banded as Low (under 30), Medium (30–59) and High (60 or above). The score is the sum of a few simple heuristics based on the counts, and the specific reasons appear as Signals:

No interaction — no clicks and no keystrokes across a session lasting more than about half a minute.
Minimal interaction — fewer than five clicks and keystrokes combined across a session lasting more than a minute.
Frequent tab switching — the participant switched away from the study tab many times (a smaller amount of switching adds to the score without raising a named signal).
Excessive pasting / Paste detected — several paste events, or at least one, during the task.

A participant with paradata and no triggered signals is shown as Clean. The table lists the underlying counts (clicks, keystrokes, tab switches, pastes) alongside the score and signals, and you can flag or exclude any row directly. Remember that these signals are heuristics: tab-switching and pasting can be perfectly legitimate (looking something up, using assistive tools), so the score is a prompt to review, never an automatic exclusion.

Verified, Suspicious & AI-detected verdicts

Alongside the Data Quality Centre, the study dashboard rolls each session up into a single at-a-glance verdict — Verified, Suspicious or AI-detected — shown in the Quality stats card and the recent-sessions list. These verdicts are derived from the same paradata-based bot-suspicion score used in the Bot Detection tab (expressed here on a 0.00–1.00 scale) together with your manual flags:

Verified — the behavioural-anomaly score is low (under 0.30) and you have not flagged the session manually.
Suspicious — the score is moderate (0.30–0.59), or you have marked the participant Flagged or Excluded.
AI-detected — the score is high (0.60 or above).

Two things follow from this. First, these labels are a triage aid, not a judgement: "AI-detected" means the behavioural pattern resembles automated or non-human responding, not that the participant has been proven to be a bot. Second, because the score depends on paradata, the "AI-detected" band can only ever appear for studies where you have enabled paradata; without it, sessions are only ever Verified (by default) or Suspicious (from your own flags). Use these verdicts to decide which sessions to open and examine in the Data Quality Centre, then make the include or exclude call yourself.

What to tell participants (consent & ethics)

Some of the data these tools rely on is collected during a session, so keep your consent form and ethics application in step with the features you use:

If you enable paradata, disclose that you collect passive behavioural process data — aggregate counts of clicks, keystrokes, tab switches and paste events during the tasks — for the purpose of data-quality and bot screening. Because it is off by default, only studies that switch it on need to mention it. Make clear that no keystroke content, clipboard content or response text is captured by paradata.
Integrity signals (a salted, one-way IP hash, a device fingerprint derived from browser/OS/screen, and the browser-reported timezone) are used to detect duplicate or multiple submissions. It is good practice to note in your privacy information that you collect technical device and network characteristics for fraud and duplicate-submission prevention, and that IP addresses are stored only as a hash.
Exclusion decisions should follow a plan you can justify. Because every signal here is a triage aid, decide and ideally pre-register your exclusion criteria (for example a minimum completion time or a duplicate-device rule), and keep the exclusion log as a record of what you removed and why.

For related material see Data Analysis (how included and excluded data flow into exports), Participant Management, and the Encryption concept (which affects response-pattern analysis).