How to redact a PDF properly (and why black boxes fail)

Every few years a court filing, a government report or a company document is published with its sensitive passages blacked out, and within hours someone copies the black bars into a text editor and reads what was underneath. The mistake is always the same. The author drew over the text instead of removing it. This guide explains why that happens, shows the real cases, and sets out a way to redact a PDF so the information is really gone.

Why a black box doesn’t hide anything

A PDF page is not a flat picture. It is a list of drawing instructions: put this text here, draw this image there, fill this shape. The reader follows the list in order and paints the result. Text stays real text, with its characters stored in the file, so you can select it, search it and copy it.

When you draw a black rectangle over a sentence with a drawing tool, a highlighter set to black, or a whiteout feature, you add one more instruction to the list: paint a black rectangle on top. The sentence is still earlier in the list. It is hidden from view only because something opaque sits on it.

That has several consequences, and any of them can undo your redaction:

• Copy and paste. Select the area, copy it and paste it into a text editor. The words come out, because they were never removed.
• Search. The find command matches text beneath boxes, and shows you where it is.
• Deleting the box. In an editor, the rectangle is just an object. Click it, press delete, and the text is back.
• Screen readers and conversion. Assistive technology and “PDF to Word” converters read the text layer, not the picture of it.
• Images. A box over part of a photograph or scan leaves the whole image in the file. Cropping the page view, or lifting the box, reveals it.

Real leaks caused by fake redaction

This is a well-documented failure, and it has happened to serious organisations with serious lawyers. The four cases below are checked against the sources listed at the end of this guide. Three are true redaction failures; the first is a close cousin, where data meant to stay out of a PDF was hidden rather than removed before the file was made.

• Barclays and Lehman Brothers, September 2008. Lawyers for Barclays were turning an Excel spreadsheet of Lehman contracts into a PDF for a US bankruptcy court filing. Rows for 179 contracts that Barclays did not mean to take on had been hidden in the spreadsheet and marked “N”. When a junior associate resized all the rows, the hidden rows reappeared without their markers, and the contracts went into the filing as if they were part of the deal. It was not a black-box redaction, and nobody pulled text out of a PDF. It is a clear example of data that was hidden, not removed, before the file was made. The firm asked the court to correct the record.
• US Army report on the Calipari shooting, 2005. The report into the killing of the Italian intelligence officer Nicola Calipari by US soldiers at a checkpoint near Baghdad was posted online as an unclassified PDF with large sections blacked out. The blacking-out was only drawn on top: copying the document and pasting it into a word processor produced the full text. The file was withdrawn, but copies had already spread.
• Transportation Security Administration, 2009. A 93-page TSA screening Standard Operating Procedures document was posted to the federal contracting site FedBizOpps in March 2009, with sensitive sections redacted. The redaction was easy to defeat, which became public in December 2009. TSA said the version posted was outdated and not the manual officers use at checkpoints. The DHS Inspector General later reported on how it happened.
• Paul Manafort’s lawyers, January 2019. On 8 January 2019, Manafort’s defence team filed a response to the special counsel’s claim that he had breached his plea agreement. Four stretches of black bars hid no deleted text: copying and pasting them into a new document showed what was underneath, including an allegation that he had shared polling data with Konstantin Kilimnik. Journalists, among them the Guardian’s Jon Swaine, spotted it within hours, and a corrected filing followed.

None of these involved hackers. Each time, a person with a reader and a text editor, or a resized spreadsheet row, exposed what had been marked secret. The people who made the documents believed the file showed only what they could see, because on screen it did.

Sources

• Barclays and Lehman: Not Just Numbers, “Hidden spreadsheet rows hit Barclays with toxic Lehman”; The Register, 15 October 2008.
• Calipari report: The Register, 3 May 2005; Antiwar.com, 1 May 2005.
• TSA manual: ABC News, December 2009; DHS Office of Inspector General report.
• Manafort filing: Columbia Journalism Review, January 2019; Above the Law, January 2019.

What real redaction does

A proper redaction tool changes the page’s instructions, not just the picture. For each area you mark, it:

1. Deletes the text that falls under the area, so the characters are no longer in the file.
2. Removes or clips vector graphics that sit there, such as lines and shapes.
3. Clears the image pixels in the area, so the picture is changed, not covered.
4. Draws the visible box where the content used to be, so readers can see something was removed.

It then writes the whole file again, rather than adding changes to the end. Without that last step, a PDF can keep the old version of the page inside itself.

PDFMagpie’s Redact PDF does these steps in your browser. You draw boxes on page previews, or search for a word or pattern and redact every match. After it finishes, the tool reads the new file back and reports whether any of the removed text can still be found.

Where else the information hides

Redacting the visible page is only part of the job. A PDF stores the same facts in several other places, and a careful reader will check them.

A method that holds up

1. Work on a copy. Redaction cannot be undone, so keep the original somewhere safe and redact a duplicate.
2. Flatten forms. If the file has form fields, flatten them first so their values are on the page where a box can reach them.
3. Find every occurrence. Use the search to find each spelling of a name, number or address, then check the shaded matches. A name might appear in headers, footers and captions as well as the body.
4. Draw boxes for what search cannot find. Signatures, logos, photographs and handwritten notes need boxes drawn by hand.
5. Preview. Look at every page with the preview on. A redaction that is two millimetres off can leave half a line of text visible.
6. Apply, then clean the file. Clear properties and, if the file has comments or attachments, run Remove metadata.
7. Test it, as set out below, before it leaves your hands.
8. Protect it if it is still sensitive. A redacted file can still hold personal data. Add a password with Protect PDF if it travels by email.

Testing the result

Never trust a redaction because it looks right. Run three quick checks on the finished file, in a reader other than the one you used to make it:

• Search for it. Press Ctrl+F (Cmd+F on a Mac) and look for a word you removed, and for part of a number. There should be no results.
• Select and copy. Press Ctrl+A to select everything, copy, and paste into a plain text editor. Look through the text for anything that should be gone.
• Extract the text. Drop the file into PDF to text, which pulls out every word the file holds. If a removed name is in the output, it is still in the file.

Scans and images

A scanned page is a picture. There are no characters to delete, so a redaction tool clears the pixels under the box. That works well, with one catch: many scans are run through OCR, which adds an invisible layer of text over the picture so you can search it. If a tool covers the picture but leaves the text layer, the words stay searchable.

You can’t search a scan for a name unless it has been through OCR PDF. Without a text layer, draw boxes by eye, and check each page after you apply.

Mistakes that cause leaks

• Redacting only some occurrences. The surname is blacked out on page one and spelled out in a footer on page eleven.
• Redacting the visible page, not the file. Properties, comments and attachments carry the same facts.
• Using whiteout or a black highlighter. These tools are designed to look right, not to remove anything. Our Edit and fill tool has whiteout and says so plainly; it is for cosmetic fixes.
• Trusting the preview. What the page looks like is not evidence of what the file contains. Always test.
• Uploading the original to redact it. If the unredacted file travels to a website, the information has been disclosed to that site before you redact it.
• Overwriting the only copy. If you realise you redacted the wrong thing, you need the original to start again.