Text Deduplicator
Remove consecutive or all duplicate words from running text, keeping either only non-repeated words or the first occurrence of each word.
Result:
How to Use the Text Deduplicator
Paste any running text into the box and choose a mode. "Consecutive Duplicates Only" removes a word only when it repeats immediately after itself (like a typo where you accidentally typed "the the"). "All Duplicates" scans the entire text and removes every repeated occurrence of a word anywhere in it, keeping only the first time each word appears. This tool works at the word level, on running text — it does not operate on whole lines.
Example
"the the quick brown brown fox" becomes "the quick brown fox" in Consecutive mode (only the immediately repeated "the" and "brown" are removed). In All Duplicates mode, "the quick brown fox jumps over the lazy dog and the dog barks" becomes "the quick brown fox jumps over lazy dog and barks" — every later "the" and "dog" is removed, keeping only the first occurrence of each.
Common Use Cases
- Cleaning up a stuttered or accidentally doubled word after a typing mistake.
- Tightening repetitive writing by removing every repeated word in a paragraph.
- Cleaning up text pasted from voice-to-text transcription, which often repeats words.
FAQs
- How is this different from the Remove Duplicate Lines tool? Remove Duplicate Lines operates on whole LINES of text — it removes an entire line if it exactly repeats an earlier line. This Text Deduplicator instead operates on individual WORDS within running sentences and paragraphs, which is a completely different level of granularity.
- Is word matching case-sensitive? No — matching is case-insensitive, so "The" and "the" are treated as the same word for deduplication purposes, though the first occurrence's original casing is preserved in the output.
- Does punctuation attached to a word affect matching in All Duplicates mode? Punctuation is ignored when comparing words, so "dog" and "dog," are treated as the same word, but the original punctuation is kept on whichever occurrence is retained.