How to detect invisible characters in text
It reads the text codepoint by codepoint and flags every one that renders with no mark. Detection is a property lookup rather than a guess, so the result is exact rather than probable.
Three Unicode General Categories cover almost every invisible character. Cf(Format) holds the zero-width characters and directional marks. Zs (Space Separator) holds the 17 space variants, of which only U+0020 is ordinary.Mn (Mark, non-spacing) holds combining marks that add no width.
Four more characters are invisible without belonging to those categories: U+3164 and U+1160 are classified as letters, and U+2800 and U+FFFC as symbols. The scanner flags them explicitly, because a category match alone would miss them.
Why text contains characters you never typed
Copied text carries the formatting of its source. Every invisible character in a normal document arrived through the clipboard rather than the keyboard.
Four sources account for most of them:
- PDF exports — U+00A0 in place of spaces, to preserve fixed layout
- Spreadsheet cells — U+00A0 from number formatting and imported data
- Rich-text editors and web pages — U+200B inserted for line-break control
- Multilingual text — U+200E and U+200F to correct display order
None of these is a defect in the source. They become a problem only when the text moves somewhere that treats them as data: a code file, a database key, a CSV column or a URL.
What invisible characters break
They break exact matching, and everything built on it. Two strings that look identical stop being equal, and every downstream comparison inherits the failure.
Search and lookup. A find-on-page for a visible word fails when a zero-width character splits it. A database lookup by name returns nothing.
Code. An invisible character inside an identifier produces a symbol that looks correct and does not resolve. The error message names a variable that appears to exist.
Data files. A U+00A0 inside a numeric column makes the value parse as text, so a spreadsheet sum silently skips the row.
Validation. A form field that looks empty passes a required check, because the value has a non-zero length.
How to remove invisible characters from text
To remove invisible characters, delete the Cf category and fold theZs category to a plain space. Deleting both outright destroys the spacing the text needs.
The scanner does exactly this, with one exception it makes on your behalf.
U+200D Zero-Width Joiner is structural inside emoji. It binds a multi-person emoji into a single glyph, so stripping every Cf character splits one family emoji into four separate people. The cleaner keeps U+200D when it sits between two pictographic characters and removes it everywhere else. Turn the option off to strip it unconditionally.
U+200C Zero-Width Non-Joiner carries meaning too. It is orthographically required in Persian, Hindi and Bengali, where removing it changes the word rather than cleaning it. Leave Cf removal off when cleaning text in those scripts.
Detecting and cleaning in code
Match the category, never a list of codepoints. A codepoint list goes stale with every Unicode release; a category match does not.
The classes \p{Cf} and \p{Zs} work in Python via the regex module, and in Java, .NET, PCRE and JavaScript with theu flag. Full property data for every character the scanner reports is on the Unicode invisible characters table.
For text produced by a language model, the character mix is narrower and the right default differs — see the AI invisible text remover.