Masked, not erased: how broken redaction fuels AI data leaks

Sensitive corporate data is not just slipping into chatbots. It is leaking long before it ever gets there.

Last month, a widely shared jailbreak tricked an AI model into revealing genuine Windows product keys, including one tied to a major bank.

While the viral headlines focused on the “gotcha” moment, the real concern runs deeper: that key, and others like it, were already floating in the digital wild. Somewhere along the line, a document containing those details was handled, shared or “redacted,” but not properly sanitized, before being scraped and exposed.

There are more incidents like this one. They are symptoms of a growing gap in how businesses protect sensitive information, particularly as AI accelerates the speed and unpredictability of data spread.

Unless organizations address the weak link in their information security workflows, they will continue to see their secrets surface in places they never intended.

How leaks like this actually happen

When we hear “data breach,” we picture hackers breaking into servers or phishing passwords. But in cases like these, the most likely culprit is poor document handling.

Corporate documents, such as contracts, legal filings, internal memos and partner reports, contain passwords, encryption keys, system credentials, financial data or personally identifiable information (PII).

These files are often shared between teams, vendors, law firms, regulators and sometimes the public domain. At each step, someone is tasked with redacting or scrubbing sensitive information.

Here is where the process can break down:

• Visual redaction instead of true removal. The most common method is to place a black box over sensitive text in a PDF or Word file. It looks secure, but the text layer underneath is still there. Anyone can highlight, copy and paste it into a new file, exposing what was supposed to stay hidden.
• Metadata exposure. Even when the visible text is cleared, document metadata like revision histories, hidden layers and comments can still contain sensitive details.
• Lack of consistency and oversight. Manual redaction is slow, error-prone and varies by employee or department. Without automated detection or audit trails, it is easy to miss a Social Security number or a password.

The infamous Meta redaction case is a cautionary tale. In its recent antitrust proceedings, Meta’s legal team used flawed PDF redaction that left entire paragraphs recoverable with a simple copy-paste.

The “hidden” text revealed Apple’s internal iMessage metrics, Snap’s TikTok threat assessments and Meta’s own strategic evaluations. This intelligence is worth millions in research and development and legal positioning.

The fallout was immediate. Apple executives publicly questioned whether Meta could be trusted with sensitive data. Snap labeled the handling “egregious.” Google called it a “casual disregard” for competitor confidentiality.

Why AI makes this a crisis, not a nuisance

Before AI became ubiquitous, a redaction misstep might have resulted in a leak to a partner or a file being buried on a public docket. Today, a oversight can snowball and cause significant damage.

Large language models (LLMs) are trained on massive amounts of data, including public-facing documents, scraped sites and archives that may contain improperly sanitized files.

If a product key, password or private detail survives a company’s redaction process, it can become part of a model’s training or be retrieved through clever prompts.

AI is also speeding up the exploitation of human error. Cybercriminals can automate searches across public datasets, forums or model outputs to identify high-value targets like credentials or proprietary information.

AI is not creating the leaks, but it is magnifying their consequences, turning what used to be quiet errors into high-stakes exposures.

What companies should do right now

If your organization handles sensitive data (as most organizations do), it is time to confront these truths:

• Most legacy redaction tools do not actually remove data. They mask, blur or hide, but rarely fully erase.
• Human error is unavoidable. No team can manually review every document line by line without mistakes.
• Regulators and rivals are paying attention. Frameworks like GDPR, HIPAA and the California Privacy Rights Act (CPRA) carry steep fines. Not to mention, public mishandling can do greater reputational damage than legal penalties.

Here is how to get ahead of the problem:

• Audit your document workflows. Map where sensitive information lives, how it is shared and where redaction or anonymization is required. This includes resumes, contracts, background checks, medical records, disciplinary reports, financial data and regulatory submissions.
• Adopt permanent redaction practices. Ensure that once something is redacted, it is not just invisible but truly gone: no recoverable text, no hidden layers, no residual metadata.
• Automate wherever possible. AI and natural language processing (NLP) can reliably detect PII, system credentials, trade secrets and other sensitive data across massive volumes of documents, reducing the risk of oversight.
• Build in accountability. Use systems with audit trails that log who redacted what and when, ensuring compliance and traceability for regulators or legal teams.
• Validate your processes. Test redacted files. Try to recover the data yourself or hire external auditors. If you can retrieve something that should not be there, so can someone else.

Privacy as a competitive advantage

Data privacy isn't another compliance checkbox. The volume and speed of today’s data flows, amplified by AI, mean that every leak has the potential to become public and pose a threat.

It is not enough to react to the next AI jailbreak with outrage. For organizations willing to evolve, this challenge is also an opportunity. Proper redaction is not flashy, but it is the foundation that prevents today’s AI mishaps from ever happening in the first place.

Companies that treat privacy as a core competency, not just a legal requirement, stand to earn deeper trust from customers, partners and regulators.

We've featured the best encryption software.

This article was produced as part of TechRadarPro's Expert Insights channel where we feature the best and brightest minds in the technology industry today. The views expressed here are those of the author and are not necessarily those of TechRadarPro or Future plc. If you are interested in contributing find out more here: https://www.techradar.com/news/submit-your-story-to-techradar-pro