preprocessing: binarize defaults OFF — measured accuracy win on three…

[schipp0]

… collections (v0.8.8)

Tesseract 5's LSTM engine thresholds internally and reads grayscale better than anything we hand it pre-binarized. Our adaptive window was flipping background paper grain to ink, which Tesseract emitted as stray punctuation ('.', '|', ':') and glued onto real words ('the.', 'of.').

Measured mean WC, adaptive threshold ON -> OFF:

ZeffPapers/MSP00074 (106 pg)  0.828 -> 0.909   review pages 28 -> 6
MSA00005 (11 pg)              0.583 -> 0.781                 7 -> 3
MSA00005 (2 pg)               0.453 -> 0.945                 2 -> 0
MSA00018 (82 pg)              0.803 -> 0.919                25 -> 6

Token-level diffing confirms the words that disappear are exactly those artifacts while real words are gained, so the lower total word count after the change is noise leaving, not text lost.

• PreprocessConfig.do_binarize now False, with the evidence recorded inline.
• CLI takes --binarize/--no-binarize; the old --no-binarize spelling still parses so existing scripts keep running.
• gui._migrate_settings retires a persisted do_binarize=True once. Saved GUI settings outrank dataclass defaults, so a default flip alone would never have reached anyone who had already run the app.
• CLAUDE.md documents the default and, separately, --psm 1 as the fix for collections with rotated scans: it recovers sideways pages and still emits ALTO coordinates in the input frame (verified by overlaying boxes on the untouched scan), but is deliberately NOT the default — it is a wash on unrotated material, costs ~13% wall time, and its OSD manufactures noise on blank pages (a clean empty page became 9 junk words at PC 0.39).

Found by the first production pilot, MSA00018 Cornell Bell papers. Outputs produced before this commit were binarized and score lower than a re-run would; re-OCR before upload, same caveat as the pre-v0.8.3 frame fix.