Image to Text (OCR)

What OCR actually does, and why it fails the way it does

The OCR engine works in stages. The picture is converted to black and white with a threshold that has to guess where the paper ends and the ink begins; the dark shapes are grouped into lines, then into words, then into characters; a neural network - an LSTM, trained on millions of lines of scanned text - reads each character in the context of the ones around it; and a dictionary pass nudges the result towards real words. Nothing in that chain involves understanding. The recognizer does not know what a word means, so it cannot notice that "reciept" is wrong, and it will happily return a sentence that reads plausibly and says something else.

That is also why the failure modes are so predictable. Handwriting throws out the character shapes the model was trained on, so it is mostly out of scope - clean block capitals sometimes work, joined-up writing rarely. A page photographed at an angle gives the line finder a fight. Low contrast, shadows, a phone flash, thin fonts, text over a photograph, and print below about ten pixels tall all cost accuracy in the same way: the letters stop being distinguishable before the network ever sees them.

The confidence number in the results is the recognizer’s own estimate for the whole picture, and it is worth reading: above 90 usually means clean print recognized correctly, the sixties usually means something was hard - and it is a warning to check the text rather than trust it. Nothing here corrects, spell-checks or tidies what came back, because a tool that quietly rewrites your document is worse than one that shows you what it really saw.

The megabytes, and why they come from this site

A recognizer is not a formula, it is a trained model, and a trained model is data. The recognition core that runs in your browser is 2.8 MB (about 1.1 MB over the wire once the server compresses it), and a model is another few megabytes that are already compressed: 2.8 MB for English, 1.6 MB for simplified Chinese. Together with the supporting files that is about 4.2 MB for English, downloaded the first time you read a picture here and never seen again - the browser stores the model locally, so the second picture and the second visit start immediately.

Those files are served from this site rather than from a CDN, which is a deliberate choice. A CDN would be smaller for us and faster for you, and it would also mean a page on this site quietly contacting a third party the moment you use a tool - on a site whose whole promise is that your documents stay yours. So the files are served from this site’s own address, shipped with the page, and you can check the list of requests your browser makes to confirm that every request during a recognition is same-origin.

Which model you get is a choice, and it is the only place in this tool where the size of the download depends on the reader. English alone is the default and 2.8 MB of model; Chinese alone is smaller still, because its model is 1.6 MB; the combined model is twice the data but reads a contract with English names in it without falling back to guesses. A picture in a language the loaded model was not trained on comes back full of words it thinks it can see, because that is what a recognizer does when it is asked to read something it was never taught.

Layout modes, and the first thing to try when it reads nonsense

The OCR engine has to be told roughly what kind of page it is looking at, and the setting is called page segmentation. "Automatic" lets it find blocks and columns itself, which is right for a scanned page or a book. "One block of text" treats the picture as a single paragraph, which is the most accurate mode for a label, a number plate or a screenshot of a message. "A single line" is for a serial number or a receipt header. "Sparse text" hunts for words anywhere, without assuming paragraphs, which suits signs, packaging and screenshots of an interface.

Getting this wrong is the most common reason a perfectly readable picture comes back as gibberish: the letters were read correctly but the line analysis rearranged them. So the order of troubleshooting is: change the mode, then straighten and brighten the picture, then enlarge it. Cropping to just the text helps more than anything else, because it removes the parts the recognizer has to decide to ignore.

What helps accuracy, in the order it helps

The first fix is almost always cropping: the recognizer has to decide which parts are text and which are not, so removing the margins shrinks its job to the words. Straightening and brightening come next, because the line finder is what breaks on a tilted page and the threshold is what breaks on a dark one.

Resolution matters too - a word the eye reads at 12 pixels tall is usually beyond a recognizer, and doubling the page is the difference between silence and a result. None of it guarantees a clean read, because OCR is estimation, but crops, straighten, brighten, enlarge fixes more pictures than any setting on the form.