Doctor handwriting is a meme for a reason.
After analyzing 1,100+ handwritten prescriptions, I can confirm: doctor handwriting is legitimately the hardest OCR challenge in healthcare.
Harder than radiology reports (typed 90% of the time).
Harder than pathology labels (mostly printed).
Harder than patient forms (block letters encouraged).
Doctor prescriptions? Cursive. Abbreviated. Context-dependent. Written in 10 seconds while thinking about the next patient.
This is how we built an OCR pipeline that achieves 70-85% accuracy on this chaos.
70-85%
OCR Accuracy
1,100+
Prescriptions Processed
Claude
OCR Engine
3-8s
Processing Time
Why Doctor Handwriting Breaks Traditional OCR
Before we dive into solutions, let's understand why traditional OCR fails spectacularly on medical prescriptions.
Challenge 1: Extreme Abbreviations
Doctors don't write "Paracetamol 500mg". They write:
// What doctor writes:
"PCM 500 BD x 3/7"
// What it means:
"Paracetamol 500mg, twice daily, for 3 days"
// OCR challenge:
- "PCM" could be "PCM", "PGM", "POH", "ROM" (similar shapes)
- "BD" could be "BD", "BID", "RD", "PD"
- "3/7" could be "3/7", "3/1", "5/7", "3/2"
Challenge 2: Context-Dependent Interpretation
The same squiggle means different things based on context:
// Example: The shape "___/"
In dosage context: "1/" (1 tablet)
In frequency context: "TID" (3 times daily)
In duration context: "5/" (5 days)
// Traditional OCR sees: Random line
// Smart OCR needs: Context awareness
Challenge 3: Variable Handwriting Quality
VaidyaAI targets the "Average doctor handwriting" category: 70-85% accuracy on real-world prescriptions.
The OCR Pipeline: 6-Stage Processing
Here's the complete pipeline that takes a prescription image and extracts structured data:
┌──────────────────────────────────────────────────────────────┐
│ STAGE 1: IMAGE UPLOAD & VALIDATION │
│ • Accept image (JPEG, PNG, PDF) │
│ • Max size: 5MB │
│ • Validate format & dimensions │
│ • Store to /uploads directory │
│ Time: ~100ms │
└────────────────────┬─────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ STAGE 2: IMAGE PREPROCESSING │
│ • Resize to max 2000px (preserve aspect ratio) │
│ • Contrast enhancement (adaptive histogram equalization) │
│ • Noise reduction (Gaussian blur) │
│ • Binarization (Otsu's method) │
│ • Deskewing (rotate if tilted) │
│ Time: ~500ms │
└────────────────────┬─────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ STAGE 3: BASE64 ENCODING │
│ • Convert preprocessed image to base64 │
│ • Prepare for Claude API payload │
│ Time: ~50ms │
└────────────────────┬─────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ STAGE 4: CLAUDE API VISION CALL │
│ • Model: claude-3-haiku-20240307 (fast & cheap) │
│ • Input: Base64 image + structured prompt │
│ • Output: JSON with extracted text │
│ Time: 2-5 seconds │
└────────────────────┬─────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ STAGE 5: POST-PROCESSING & VALIDATION │
│ • Parse JSON response │
│ • Validate medicine names against formulary │
│ • Expand abbreviations (PCM → Paracetamol) │
│ • Standardize dosages (1tab → 1 tablet) │
│ • Flag ambiguous entries for review │
│ Time: ~200ms │
└────────────────────┬─────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ STAGE 6: STRUCTURED OUTPUT │
│ • Return prescription as JSON │
│ • Include confidence scores │
│ • Highlight uncertain fields │
│ • Store for audit trail │
│ Time: ~50ms │
└──────────────────────────────────────────────────────────────┘
Total Pipeline Time: 3-8 seconds
Success Rate: 70-85% fully automated
Fallback Rate: 15-30% requires human review
Stage 1: Image Upload & Validation
First, we need to accept and validate the prescription image:
// PHP file upload handling
function handlePrescriptionUpload() {
// Validate file was uploaded
if (!isset($_FILES['prescription_image'])) {
return ['error' => 'No file uploaded'];
}
$file = $_FILES['prescription_image'];
// Validate file size (max 5MB)
if ($file['size'] > 5 * 1024 * 1024) {
return ['error' => 'File too large (max 5MB)'];
}
// Validate file type
$allowedTypes = ['image/jpeg', 'image/png', 'image/jpg', 'application/pdf'];
if (!in_array($file['type'], $allowedTypes)) {
return ['error' => 'Invalid file type'];
}
// Generate unique filename
$filename = uniqid() . '_' . time() . '.' . pathinfo($file['name'], PATHINFO_EXTENSION);
$uploadPath = '/uploads/prescriptions/' . $filename;
// Move uploaded file
if (!move_uploaded_file($file['tmp_name'], $uploadPath)) {
return ['error' => 'Upload failed'];
}
return [
'success' => true,
'filepath' => $uploadPath,
'filename' => $filename
];
}
Stage 2: Image Preprocessing (The Secret Sauce)
Raw prescription images are terrible for OCR. We need to clean them up:
// Image preprocessing with GD library
function preprocessPrescriptionImage($filepath) {
// Load image
$image = imagecreatefromjpeg($filepath);
if (!$image) $image = imagecreatefrompng($filepath);
// Get dimensions
$width = imagesx($image);
$height = imagesy($image);
// Resize if too large (max 2000px on longest side)
$maxDim = 2000;
if ($width > $maxDim || $height > $maxDim) {
$scale = $maxDim / max($width, $height);
$newWidth = round($width * $scale);
$newHeight = round($height * $scale);
$resized = imagecreatetruecolor($newWidth, $newHeight);
imagecopyresampled($resized, $image, 0, 0, 0, 0,
$newWidth, $newHeight, $width, $height);
imagedestroy($image);
$image = $resized;
}
// Enhance contrast (adaptive histogram equalization approximation)
imagefilter($image, IMG_FILTER_CONTRAST, -30);
// Reduce noise (Gaussian blur with low intensity)
imagefilter($image, IMG_FILTER_GAUSSIAN_BLUR);
// Sharpen text edges
imagefilter($image, IMG_FILTER_MEAN_REMOVAL);
// Convert to grayscale
imagefilter($image, IMG_FILTER_GRAYSCALE);
// Save preprocessed image
$preprocessedPath = str_replace('.jpg', '_processed.jpg', $filepath);
imagejpeg($image, $preprocessedPath, 95);
imagedestroy($image);
return $preprocessedPath;
}
Why These Preprocessing Steps?
- Resize: Large images slow down Claude API. 2000px is sweet spot for quality vs speed.
- Contrast enhancement: Makes faint text darker, easier to read.
- Gaussian blur: Removes camera noise, scanner artifacts.
- Mean removal: Sharpens edges, makes text crisper.
- Grayscale: Reduces data size, focuses on text structure.
Result: 15-20% accuracy improvement from preprocessing alone.
Stage 4: Claude API Vision Call (The Core)
This is where the magic happens. Claude's vision model reads the prescription:
// Claude API call with vision
function extractPrescriptionWithClaude($imagePath) {
// Convert image to base64
$imageData = file_get_contents($imagePath);
$base64Image = base64_encode($imageData);
// Detect image type
$imageType = 'image/jpeg';
if (strpos($imagePath, '.png') !== false) $imageType = 'image/png';
// Prepare Claude API request
$apiUrl = 'https://api.anthropic.com/v1/messages';
$apiKey = getenv('ANTHROPIC_API_KEY');
$prompt = "You are a medical prescription OCR system. Extract ALL information from this handwritten prescription.
Return a JSON object with:
{
\"medicines\": [
{
\"name\": \"full medicine name\",
\"dosage\": \"dosage amount\",
\"frequency\": \"how often (e.g., twice daily)\",
\"timing\": \"when to take (e.g., after meals)\",
\"duration\": \"how many days\"
}
],
\"doctor_name\": \"prescribing doctor\",
\"patient_name\": \"patient name\",
\"date\": \"prescription date\",
\"diagnosis\": \"diagnosis if mentioned\",
\"instructions\": \"any special instructions\",
\"confidence\": \"high/medium/low\"
}
Important:
- Expand abbreviations (PCM → Paracetamol, BD → twice daily)
- If unsure, mark confidence as 'low'
- Extract ALL visible medicines
- Include dosage timing (before/after meals)
- Return ONLY valid JSON, no other text";
// Make API call
$ch = curl_init($apiUrl);
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_POST => true,
CURLOPT_HTTPHEADER => [
'Content-Type: application/json',
'x-api-key: ' . $apiKey,
'anthropic-version: 2023-06-01'
],
CURLOPT_POSTFIELDS => json_encode([
'model' => 'claude-3-haiku-20240307',
'max_tokens' => 2000,
'messages' => [
[
'role' => 'user',
'content' => [
[
'type' => 'image',
'source' => [
'type' => 'base64',
'media_type' => $imageType,
'data' => $base64Image
]
],
[
'type' => 'text',
'text' => $prompt
]
]
]
]
]),
CURLOPT_TIMEOUT => 30
]);
$response = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
if ($httpCode !== 200) {
error_log("Claude API error: HTTP $httpCode");
return ['error' => 'OCR failed'];
}
// Parse response
$result = json_decode($response, true);
$extractedText = $result['content'][0]['text'];
// Extract JSON from response (Claude sometimes adds markdown)
if (preg_match('/\{.*\}/s', $extractedText, $matches)) {
$extractedText = $matches[0];
}
$prescriptionData = json_decode($extractedText, true);
return $prescriptionData;
}
Why This Prompt Works
- Explicit JSON structure: Claude knows exactly what format we need
- Abbreviation expansion: "Expand abbreviations" instruction crucial
- Confidence scoring: Lets us filter low-confidence results
- "Return ONLY valid JSON": Prevents Claude from adding explanations
Stage 5: Post-Processing & Validation
Claude's output needs validation against our medicine database:
// Validate and correct extracted medicines
function validatePrescription($prescriptionData, $pdo) {
$validated = $prescriptionData;
foreach ($validated['medicines'] as $index => $medicine) {
// Check if medicine name exists in formulary
$stmt = $pdo->prepare("
SELECT name, generic_name
FROM medicines
WHERE name LIKE ? OR generic_name LIKE ?
LIMIT 1
");
$searchTerm = '%' . $medicine['name'] . '%';
$stmt->execute([$searchTerm, $searchTerm]);
$match = $stmt->fetch();
if ($match) {
// Found exact or close match - use canonical name
$validated['medicines'][$index]['name'] = $match['name'];
$validated['medicines'][$index]['validated'] = true;
} else {
// No match - flag for manual review
$validated['medicines'][$index]['validated'] = false;
$validated['medicines'][$index]['warning'] = 'Medicine not in formulary';
}
// Standardize frequency abbreviations
$frequencyMap = [
'BD' => 'Twice daily',
'TID' => 'Three times daily',
'QID' => 'Four times daily',
'OD' => 'Once daily',
'HS' => 'At bedtime',
'PRN' => 'As needed'
];
$frequency = strtoupper($medicine['frequency']);
if (isset($frequencyMap[$frequency])) {
$validated['medicines'][$index]['frequency'] = $frequencyMap[$frequency];
}
}
return $validated;
}
Real-World Accuracy Breakdown
After processing 1,100+ prescriptions, here's what we learned:
| Prescription Element | Accuracy | Notes |
|---|---|---|
| Medicine names | 85-90% | Best performance - distinct words |
| Dosages | 80-85% | Numbers easier than words |
| Frequency | 70-75% | Abbreviations cause confusion |
| Duration | 65-70% | "3/7" vs "3 days" ambiguity |
| Doctor name | 90-95% | Usually printed or stamped |
| Special instructions | 50-60% | Free text, high variability |
Common Failure Patterns
Top 5 OCR Failures
- Severe cursive: When letters connect, Claude struggles to segment
- Overwriting: Corrections on top of original text confuse the model
- Faded ink: Light pencil or old prescriptions have low contrast
- Background patterns: Prescription pads with logos/watermarks
- Non-standard abbreviations: Doctor-specific shorthand not in training data
Error Handling & Fallback Strategy
When OCR fails, we need graceful degradation:
// Complete OCR workflow with error handling
function processPrescriptionOCR($imagePath, $pdo) {
try {
// Stage 2: Preprocess
$processedPath = preprocessPrescriptionImage($imagePath);
// Stage 4: Extract with Claude
$extracted = extractPrescriptionWithClaude($processedPath);
if (isset($extracted['error'])) {
throw new Exception($extracted['error']);
}
// Stage 5: Validate
$validated = validatePrescription($extracted, $pdo);
// Calculate overall confidence
$validatedCount = count(array_filter(
$validated['medicines'],
fn($m) => $m['validated'] ?? false
));
$totalCount = count($validated['medicines']);
$confidence = $totalCount > 0 ? ($validatedCount / $totalCount) * 100 : 0;
// Determine if human review needed
$needsReview = $confidence < 70 ||
$validated['confidence'] === 'low';
return [
'success' => true,
'data' => $validated,
'confidence' => round($confidence),
'needs_review' => $needsReview,
'review_reason' => $needsReview ?
'Low confidence extraction' : null
];
} catch (Exception $e) {
// Fallback: Return empty structure for manual entry
error_log("OCR failed: " . $e->getMessage());
return [
'success' => false,
'error' => $e->getMessage(),
'fallback_mode' => 'manual_entry',
'image_path' => $imagePath // Show image for manual typing
];
}
}
Performance Optimization
Cost Analysis
💰 OCR Cost Breakdown
- Claude Haiku API: ~₹0.15 per prescription image
- Image preprocessing: Negligible (server CPU)
- Database validation: Negligible (cached)
- Total cost per OCR: ~₹0.15
- Monthly cost (1,000 OCR): ₹150
Compare to: Manual data entry @ ₹5 per prescription = ₹5,000/month
Savings: 97% cost reduction
Speed Optimization
Original pipeline: 8-12 seconds.
Optimized pipeline: 3-8 seconds.
Optimizations made:
- Async processing: Preprocessing happens while user sees loading screen
- Image size limit: 2000px max (vs 4000px originally) = 2x faster
- Haiku model: 3x faster than Sonnet with minimal accuracy loss
- Cached validation: Medicine name lookups use indexed queries
The Human-in-the-Loop Strategy
70-85% accuracy means 15-30% needs review. We handle this intelligently:
✅ Review UI Design
- Side-by-side view: Original image + extracted text
- Highlighted uncertainties: Low-confidence fields marked in yellow
- Quick corrections: Click field to edit inline
- Autocomplete: Medicine names auto-suggest from formulary
- Approval workflow: One-click approve after review
Result: Manual correction takes 30-60 seconds vs 3-5 minutes for full retyping
Future Improvements: The Roadmap
Q1 2026: Fine-tuned Model
- Collect 10,000+ reviewed prescriptions
- Fine-tune Claude on our specific doctor handwriting patterns
- Target: 85-90% accuracy
Q2 2026: Multi-language Support
- Add Hindi, Telugu, Tamil script recognition
- Regional language prescription support
- Target: 70%+ accuracy on regional languages
Q3 2026: Real-time OCR
- Mobile app with live camera OCR
- Instant feedback as doctor writes
- Prevent illegible prescriptions proactively
Key Takeaways: Prescription OCR at Scale
- 70-85% accuracy achievable: On real doctor handwriting using Claude Haiku
- Preprocessing is critical: 15-20% accuracy gain from image enhancement
- Structured prompts work: Explicit JSON schema gets consistent output
- Validation catches errors: Cross-reference with formulary prevents mistakes
- Human-in-the-loop essential: Review UI for 15-30% edge cases
- Cost-effective: ₹0.15 per OCR vs ₹5 for manual entry (97% savings)
- 3-8 second processing: Fast enough for clinical workflows
- Graceful degradation: Fallback to manual entry when OCR fails
Perfect OCR is impossible. 70-85% accuracy with intelligent review UI is better than 0% accuracy from manual retyping. Ship the 80% solution, iterate from there.
The Bottom Line
Doctor handwriting will always be challenging. But 70-85% OCR accuracy means:
- 7-8 out of 10 medicines extracted correctly
- 30-60 seconds review time vs 3-5 minutes full retyping
- 80-90% time savings overall
- 97% cost reduction vs manual data entry
That's good enough to transform workflows. Perfect is the enemy of shipped.
See OCR in Action
Upload a handwritten prescription and watch VaidyaAI extract the data. Free trial, no credit card required.