Invisible characters in copied text

Sometimes a text looks perfect on screen and still behaves strangely. Search does not find a word that is clearly there. A link opens an error page. A code snippet refuses to run. Often the reason is a character you cannot see.

What these characters are

Text is a sequence of code points, and not all of them draw something. The usual suspects:

• Zero-width characters (U+200B zero-width space, U+200C and U+200D joiners, U+FEFF byte order mark). They take no room on screen.
• Bidirectional marks and overrides (U+202A to U+202E and similar). They control the direction of text, which matters for Arabic or Hebrew, and does nothing useful in an English paragraph.
• Soft hyphen (U+00AD). It marks a place where a word may break, and is invisible until a line wraps.
• Odd spaces: the non-breaking space and a family of narrow, wide and thin spaces that look like a normal space but are not.
• Typographic punctuation: curly quotation marks, the long dash, the ellipsis as a single character, bullets and arrows.

The last group is visible, but it is a different character from the plain one on your keyboard, and software often treats it differently.

Where they come from

Mostly from ordinary copying:

• chat windows and web pages, whose layout engines add soft hyphens or zero-width spaces to control line breaks;
• word processors, which replace straight quotes and hyphens with "smart" ones as you type;
• translation tools and PDF extraction, which may insert odd spaces or break words;
• documents passed through several programs, each adding its own habits.

Nobody did anything wrong. That is just how text travels. We make no claim that any particular service puts invisible characters into its output on purpose, and we have no data that would let us say so.

Why they cause real problems

• Search breaks. A word with a hidden character inside is a different string. "Contract" and "con[zero-width]tract" do not match.
• Diffs look strange. Two lines that look identical are reported as different.
• Code and URLs fail. A curly quote in a shell command or a zero-width space in a link is enough to stop it working, and the error message rarely tells you why.
• Plagiarism checkers get confused. Hidden characters and swapped spaces change how a text is split into words, so matches may be missed or reported wrongly.
• Layout glitches. Strange gaps, lines that refuse to wrap, lines that break in the wrong place.

How to find and clean them

AuthorText has two ways to handle this.

First, the check. It highlights watermark-like characters and typography in the text, so you can see where they sit instead of guessing. The detector covers 259 such code points: zero-width characters, soft hyphen, invisible operators, bidirectional overrides, variation selectors, tag characters, private-use characters and a few more.

Second, the Normalize tool ("Clean & normalize"). It removes these characters and turns typographic punctuation into plain ASCII: straight quotes, a simple hyphen, three dots. Compare the result with the original before you use it.

A quick routine:

1. Paste the text into the checker and look at the highlighted places.
2. Decide whether you want everything normalized (usually yes for code, URLs, data) or only some of it.
3. Run the cleaning and compare.

Keep in mind that a long dash in a literary text may be a deliberate choice. Normalizing is right for technical text, and optional for prose.

A different thing: statistical watermarks

There is a second meaning of "watermark" and it is worth keeping apart. Researchers have proposed watermarking language models at the token level: while generating, the model is nudged to prefer certain words in a pattern that a detector holding the key can later test for.

Such a watermark is not made of characters. Nothing is hidden in the text as a code point. It lives in the choice of words, so a character cleaner cannot find it and cannot remove it, and AuthorText does not claim to detect it. Our analysis looks at style, not at a secret key. Invisible characters and statistical watermarks are two separate topics, and the first one does not tell you anything about the second.

Also, a text with hidden characters is not automatically AI-generated, and a text without them is not automatically human. They show how a text was copied, not who wrote it.

What to do

Before you publish, submit or run copied text, look at it once. Check the text to see hidden characters and typography, then use the Normalize tool when plain ASCII is what you need. If a piece of text is under suspicion for other reasons, our short guide to the signs of AI text explains what else to look at, and why none of it is proof.