OCR on real handwriting: what the tutorials leave out

Scanned handwritten page beside the transcribed text output with corrected words highlighted

Every OCR tutorial uses a clean scan of printed text. Real inputs are photographed at an angle, in bad light, on lined paper, in handwriting that varies between the top and bottom of the same page.

The model is rarely the hard part. Getting the image into a state the model can read, and correcting what it gets wrong afterwards, is where the accuracy actually comes from.

Preprocessing earns more than model choice

A three-degree rotation is invisible to you and quietly destructive to line segmentation, because a line detector looking for horizontal bands finds bands that overlap two lines of text. Deskewing first is not a refinement; it is a precondition.

preprocess.pypython

import cv2
import numpy as np


def deskew(image):
    """Rotate so text lines are horizontal.

    Line segmentation looks for horizontal bands of ink, so even a few
    degrees of skew makes one band span two lines of text.
    """
    grey = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    inverted = cv2.bitwise_not(grey)
    coords = np.column_stack(np.where(inverted > 0))
    angle = cv2.minAreaRect(coords)[-1]

    if angle < -45:
        angle = 90 + angle

    h, w = image.shape[:2]
    matrix = cv2.getRotationMatrix2D((w // 2, h // 2), angle, 1.0)
    return cv2.warpAffine(
        image, matrix, (w, h),
        flags=cv2.INTER_CUBIC,
        borderMode=cv2.BORDER_REPLICATE,
    )


def binarise(image):
    """Adaptive, not global. A photo lit from one side has no single
    threshold that works across the whole page."""
    grey = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    grey = cv2.fastNlMeansDenoising(grey, h=10)
    return cv2.adaptiveThreshold(
        grey, 255,
        cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
        cv2.THRESH_BINARY,
        blockSize=31,
        C=10,
    )

Segment to lines, recognise per line

Handwriting recognition models are trained on single lines, not pages. Feeding a whole page produces garbage regardless of how good the model is. Project the ink onto the vertical axis, find the valleys, cut there.

diagram

flowchart LR
    P[Photo] --> D[Deskew]
    D --> B[Adaptive binarise]
    B --> L[Line segmentation]
    L --> R[Per-line recognition]
    R --> C[Lexicon correction]
    C --> O[Text output]
    L -.->|one line per crop| R

The recognition model is one step of six. Five of the six are image handling and text correction.

Lined paper needs one extra step, because the rules themselves are ink as far as the binariser is concerned. A morphological opening with a wide horizontal kernel removes long thin horizontal runs while leaving letter strokes largely intact.

Correct against a lexicon afterwards

Recognition errors are not random. They cluster on visually confusable characters — rn for m, cl for d, 0 for O, 1 for l. A domain lexicon plus edit distance fixes a large share of them, because "pnescription" has exactly one plausible neighbour.

Measure with CER, not accuracy

Word-level accuracy is discouraging and uninformative: one wrong character fails the whole word. Character error rate — edit distance over reference length — tells you whether output is nearly right or nowhere close, and those are very different situations for a human reviewer.

Under 5% CER — comfortable to skim and correct.
5–15% CER — usable with review, and often still faster than typing from scratch.
Over 15% CER — retyping is quicker, and shipping it wastes the reviewer's time.

Be explicit about which band you are in and set expectations accordingly. A transcription tool that says "review this" is trusted. One that presents 20% CER output as finished gets abandoned after the first page.

OCRComputer VisionOpenCVPreprocessingTransformers

All articles