OpenAI's Project Lily: what human review of ChatGPT chats means for work

OpenAI has a program, reported as "Project Lily," where hundreds of contractors read real ChatGPT conversations and rate the model's responses to improve it. That's the core of 404 Media journalist Joseph Cox's September 14, 2026 report, based on leaked internal documents — and it confirms something most of ChatGPT's 900+ million weekly users likely assume isn't happening to their conversations.

None of this is exotic or malicious. It's ordinary RLHF (reinforcement learning from human feedback) work, the same technique used to shape most production chat models. But the gap between what people assume about a "private" AI conversation and what actually happens to it is the real story — and it's a smaller-scale preview of a gap most organizations have with their own AI usage.

Update (October 20, 2026): a class action now cites the reporting

A federal class action filed two days after the 404 Media story builds on it. According to a summary of the complaint published by OpenClassActions, Vredenburgh v. OpenAI OpCo, LLC, No. 3:26-cv-10527, was filed on September 16, 2026 in the U.S. District Court for the Northern District of California. The summary says OpenAI was served on September 21, that its response was due October 13, and that the first case management conference is set for December 18, 2026. I could not confirm from public sources whether OpenAI has filed a response.

These are allegations only. No class has been certified, OpenAI has not been found liable, and there is nothing for users to claim. The summary says the complaint draws nearly all of its factual claims about how the program works from the 404 Media reporting, and that it argues OpenAI's Terms of Use, Privacy Policy and model-training pages never mention a human reader outside flagged-content checks. It also describes the proposed relief: no sending of conversations to outside reviewers without separate opt-in consent, the "Improve the model for everyone" setting off by default, and a warning in the chat window.

TL;DR

What Project Lily actually is

According to the leaked documents Cox reviewed, contractors working on Project Lily get access to complete conversation threads — not isolated, anonymized snippets — and rate how well the model's responses perform. The stated goals are specific: reduce the model's tendency toward anthropomorphization (talking or acting like it has feelings), and decrease sycophancy, where the model over-agrees with a user rather than pushing back.

Sycophancy isn't an abstract quality issue here. The reporting connects it directly to real-world harm: it's been cited in wrongful-death lawsuits alleging that ChatGPT's over-agreeable behavior contributed to user suicides. Reviewing what the model actually said in real conversations is, in a narrow sense, the same fix OpenAI would need to make regardless — you can't tune a model's tendency to over-agree with people without someone reading examples of it doing that.

The tension is that fixing it requires exactly the kind of access most users assume doesn't exist: a human being reading what they actually typed.

How the review work is described

404 Media's piece, which is partly paywalled, says OpenAI is hiring hundreds of contractors who read a large stream of real ChatGPT prompts, sometimes whole conversations, and rate and critique the chatbot's generated replies. Its subhead says leaked internal documents and real prompts were seen by the reporter, and that the chats can include sensitive personal information. One person who works with the prompts told the outlet they did not think users would imagine "some contractor somewhere" analysing their conversations.

The complaint summary adds a more detailed picture of the workflow, attributed there to 404 Media's reporting: contractors recruited through a staffing firm pick a task from a dashboard, are shown a real prompt, write a short summary of what the user seemed to want, read four ChatGPT responses, highlight passages that match or miss the targeted behaviour, score each from one to seven and write a rationale. Treat these workflow details as reported by secondary sources, since the full article is behind a paywall. The same summary says an automated "Privacy Filter" runs before reviewers see conversations, that the filter's own published documentation calls it a redaction aid rather than a guarantee, and that reviewer instructions reportedly tell contractors to escalate tasks containing personal information. I could not verify those filter details independently.

Why this matters beyond one company: the work is a labour pipeline. Model quality depends on people reading real usage, which is why 404 Media notes that Anthropic confirmed it also uses human review to improve its models. The question for users is not whether any human is involved but which settings and plans keep their content out.

What OpenAI says about privacy safeguards

OpenAI told 404 Media that contractors reviewing conversations under Project Lily don't see the usernames attached to them. But the same reporting notes OpenAI's own acknowledgment that sensitive details — health information, financial details, other identifying content — can still slip through the filters meant to screen conversations before a contractor sees them.

That's a meaningful, honest caveat, not a loophole framed as a feature: text-based content filtering on free-form conversation is inherently imperfect. A user doesn't fill out a "sensitive information" field before typing; they just type, and whatever filter runs afterward is working against unstructured prose written by someone who had no reason to think about what a filter would catch.

This is not just an OpenAI story

The 404 Media reporting includes confirmation that Anthropic uses human review of real Claude conversations too, for similar quality and safety purposes. That context matters: this isn't a scandal unique to one company's practices, it's how RLHF-based consumer chat products are built across the industry. Improving a model on real usage requires looking at real usage.

That doesn't make the privacy tradeoff disappear — it just means the honest framing is "this is standard practice you should account for," not "this is one company behaving badly."

What developers are saying

The September 24 Hacker News thread about the lawsuit had a short but pointed discussion. drewfax argued that anyone who assumes AI chats are private should not, since companies will evaluate sessions to improve their models whether or not they disclose it. ticulatedspline wrote that, as they read it, the suit targets a gap in the privacy policy about outsourcing data labelling to third-party contractors, and that there would be no suit if OpenAI employees did the evaluations. binlog quoted the passage in the complaint summary about the help-centre FAQ answering "Do humans view my content?", and rcr-anti asked whether no-training and zero-retention agreements are credible. These are individual commenters' views, not verified facts.

What to actually do about your own ChatGPT use

The practical guidance from the reporting is straightforward:

• Turn off "Improve the model for everyone" in ChatGPT's data controls. OpenAI documents it as the control for whether new conversations are used to train its models; whether it also keeps them from human review is not spelled out in the pages cited above.
• Use temporary chat for sensitive one-offs, and skip the thumbs. Temporary chats are not used to improve models while they remain temporary, and feedback buttons can send the whole conversation for training even after an opt-out.
• Prefer a managed workspace for work. Business, Enterprise and Edu workspaces are excluded from training by default.
• Don't treat a chat window as a diary. Avoid pasting identifying health, financial, or otherwise sensitive personal detail into a consumer AI chat unless you'd be comfortable with a stranger reading it verbatim.
• Assume the same is true of whatever chat tool you use next. Given Anthropic's confirmation, switching products doesn't remove the practice — it just changes who's reading.

What the vendors' own documentation says

The reporting and the settings are different things, so it helps to read what each vendor publishes.

OpenAI. OpenAI's Data controls in ChatGPT page says that when "Improve the model for everyone" is off, your new conversations will not be used to train OpenAI models, and that they can still appear in your chat history. Turning it off does not delete saved chats; deletion is a separate action. The help centre article on how OpenAI handles data in consumer services says OpenAI may use content submitted to ChatGPT to improve model performance, depending on settings. Several details from these pages are easy to miss:

• Temporary chats stay out of history, do not create or update memories and are not used to improve models while they remain temporary, but OpenAI may retain a copy for up to 30 days for safety purposes. Saving a temporary chat turns it into a regular chat that follows your settings.
• Feedback overrides the opt-out. If you give a thumbs up or thumbs down after opting out, the entire conversation tied to that feedback may be used to train models.
• Codex. On a personal ChatGPT plan, the same setting applies to Codex tasks, but Codex has a separate "Include environments" setting that controls whether extra context from your environments can be used for training, and changing the ChatGPT setting does not change it.
• Managed workspaces. By default OpenAI does not use content from ChatGPT Business, Enterprise, Edu or Healthcare workspaces, or the API Platform, to improve its models. Organisations can choose to share data when that option is offered.

One honest limit: these pages describe training use. They do not say, in the text I could read, whether the setting also excludes a conversation from human review of the kind 404 Media describes, and the 404 Media article does not state it in its visible portion. The complaint summary says the help article it quotes ties access to the setting with the words "unless you have opted out", but that is the plaintiffs' reading. Treat the setting as the best available control, not as a guarantee about human review.

Anthropic. Anthropic's privacy centre article Is my data used for model training? covers consumer plans (Free, Pro, Max) and Claude Code on those accounts. It says chats and coding sessions are used to improve models if you choose to allow it, if a conversation is flagged for safety review, or if you otherwise opt in explicitly. It says Incognito chats are not used to improve Claude even if model improvement is enabled, and that thumbs-up or thumbs-down feedback stores the entire related conversation for up to five years, de-linked from your user ID. Raw connector content, including from MCP servers, is excluded unless you copy it into the conversation. For commercial products such as Claude for Work and the API, Anthropic points to separate terms.

The common pattern: the strongest protection comes from the account type (managed or commercial plans), then from settings, then from your own care in what you paste.

The version of this problem organizations actually have

Project Lily is a story about a company reviewing conversations it controls, with a stated purpose and (imperfect) filtering in front of it. Most organizations have a worse version of the same exposure, with none of that structure: employees using AI chat tools and coding agents on personal accounts, with no policy, no review, and no record of what left the building.

That's the pattern behind 2026's DLP telemetry showing employees paste something sensitive into an AI chatbot roughly every three days, mostly on unmanaged personal accounts a company has zero visibility into — not through any single dramatic leak, but through the accumulated, unlogged ordinary use of tools that read exactly like the ones behind Project Lily.

The same blind spot extends past chat windows into what AI coding agents do on a developer's machine: the shell commands they run, the files they touch, the credentials they can reach. If a consumer chatbot with a stated privacy policy and content filters still needs a caveat about what slips through, it's a reasonable baseline assumption that an unmonitored coding agent with filesystem and network access deserves at least as much scrutiny — and today, most organizations have none.

This is the problem Beam is built around: recording what an AI agent actually does — locally, on the device, before anything is a matter of trusting a vendor's own review pipeline — rather than finding out only when something goes wrong. Beam Sentinel, Beam's browser extension, checks text against known-sensitive patterns before it's sent to ChatGPT, Claude, Gemini, Perplexity, Copilot, or Grok, and pauses high-severity matches for an explicit decision instead of letting them go out silently. On the coding-agent side, Beam's local collector keeps a record of an agent's shell commands, file writes, and MCP tool calls, so "what did the agent actually do" has an answer that doesn't depend on asking the agent.

None of that requires trusting Beam with the data either — the checks and the recording happen locally, not in a cloud pipeline reviewed by anyone else's contractors. The goal isn't a different company reading your conversations; it's giving the organization actually responsible for the risk a record it can inspect itself.

Related reading

• How much sensitive data do people share with AI chatbots? The 2026 numbers
• What is AI agent monitoring? Definition, scope, and tooling
• Enterprise data agents are the new access-control problem, not just a new feature
• MCP security: a practical guide
• AI agent security: securing coding assistants
• Codex training setting re-enabled? Claimed vs verified
• Original reporting, via explainx.ai: OpenAI's Project Lily: human review, ChatGPT, and privacy in 2026

Updated October 20, 2026 with the lawsuit and vendor documentation. Facts about Project Lily are as reported by 404 Media (Joseph Cox, September 14, 2026) and summarized by explainx.ai, current as of this post's publish date. Beam Sentinel's and Beam's local collector's capabilities reflect chrome-apps/beam-sentinel and apps/sentinel-collector's READMEs as of September 15, 2026.