By Luis Lopez, AI transportation consultant, CEO of Go Hub.io Holdings Corp and subsidiaries, and host of the Freight Guru Podcast
“It’s very accurate” is the most common claim made about AI tools in freight, and the least useful. Accurate at what? Measured how? On whose documents? A tool that reads bills of lading correctly most of the time can still miss the one field that matters, and a fraud flag that catches every bad carrier might also block half of the good ones.
As an AI transportation consultant, the first thing I ask when someone shows me an accuracy number is how it was measured. This article explains, in plain English, the few metrics that actually tell you whether an AI tool is working in freight operations, and how to check them yourself.
Why a Single Accuracy Number Hides the Problem
Imagine a fraud screening tool that reviews carriers before booking. If only a small share of carriers are actually fraudulent, a tool that approves everyone will look highly “accurate” because it is right about most carriers. It is also useless. That is why freight teams need to look at errors by type, not just overall correctness.
Precision, Recall and False Positives in Plain English
False Positives and False Negatives
- False positive: the tool flags something that is fine. A legitimate carrier gets marked as suspicious. A clean invoice gets marked as a mismatch.
- False negative: the tool misses something that is wrong. A fraudulent carrier passes. A wrong amount on an invoice goes through.
Precision
Precision answers: when the tool raises a flag, how often is it right? If a fraud tool flags twenty carriers and only a few are actually problems, precision is low. Your team spends its day clearing false alarms and starts ignoring the flags.
Recall
Recall answers: of all the real problems out there, how many did the tool catch? If there were ten fraudulent carriers and the tool flagged three, recall is low. The tool feels quiet and calm while problems slip through.
The Tradeoff
Precision and recall usually pull against each other. Make the fraud tool more sensitive and it catches more bad actors but also flags more good carriers. Make it stricter and the flags are more trustworthy, but more problems slip by. The right balance depends on the cost of each kind of mistake, which is a business decision, not a technical one. Our article on AI fraud detection and carrier vetting discusses why fraud screening usually leans toward catching more, with human review on the flags.
Field-Level vs Document-Level Accuracy
Document extraction is where accuracy claims get most confusing. There are two very different ways to measure it.
Field-Level Accuracy
Count every field the tool extracted, such as shipper name, consignee, weight, piece count, reference numbers, and dates, and measure how many were correct. This number tends to look high because easy fields pull the average up.
Document-Level Accuracy
Count how many documents came out completely correct, with every required field right. This number is always lower, and it is closer to what operations feels. If one field is wrong on a rate confirmation, someone still has to touch it.
Weight the Fields That Matter
Not all fields are equal. A misspelled street suffix is a nuisance. A wrong container number, amount, or delivery date causes real damage. Track accuracy separately for the critical fields. Our comparison of OCR vs AI document extraction covers the tools themselves; the measurement discipline applies to both.
Sampling and Spot Checks
You do not need a data science team to measure accuracy. You need a consistent sampling habit.
- Pick a random sample. Each week, pull a set of records the AI processed. Random matters: if people only review the ones that look odd, the results will be skewed.
- Check against the source. A person compares the AI’s output with the original document or event and records what was right and wrong, field by field.
- Record error types. Note whether errors were false positives, false negatives, wrong values, or missing values.
- Include the hard cases. Make sure the sample includes difficult inputs such as handwritten notes, poor scans, and unusual customers, not just the clean ones.
- Keep the sample size steady. A consistent sample week to week lets you see trends.
Spot checks also catch problems that metrics miss, like a tool that is technically correct but formats dates in a way your billing system misreads.
Confidence Thresholds
Many AI tools return a confidence score with each answer. Used well, that score routes work:
- High-confidence results flow through automatically.
- Middle-confidence results go to a quick human review.
- Low-confidence results get full manual handling.
The threshold is something you set and adjust, not something the vendor decides for you. Do not assume confidence scores are calibrated. Check whether high-confidence results really are correct more often by including them in your spot checks. If a tool is confidently wrong, that is a serious problem; see our piece on catching AI hallucinations in freight.
Track Accuracy Over Time
Accuracy is not a one-time test. It drifts. A new customer sends documents in a different layout. A carrier changes its invoice template. The model gets updated. Seasonal freight brings unfamiliar commodities. Keep a simple running log:
- Weekly precision and recall for flagging tools.
- Weekly field-level and document-level accuracy for extraction tools, with critical fields shown separately.
- Share of work auto-processed vs sent to review.
- Notes on any model or process change.
When a number drops, the log tells you when it started and what changed around that time.
What Good Enough Means Per Task
There is no universal accuracy target. Good enough depends on the cost of a mistake and on whether a human catches it downstream.
- Email triage and routing: moderate accuracy can be fine, because a misrouted email gets noticed and moved.
- Shipment status updates: errors annoy customers and erode trust, so the bar is higher.
- Billing amounts and invoice matching: mistakes cost money directly, so automation should be limited to very high-confidence cases with review on the rest.
- Fraud and carrier vetting: favor recall and keep a person on every flag and every final approval.
- Compliance-related data: anything feeding regulatory filings needs human verification regardless of what the tool reports.
Before turning a tool on, write down what an acceptable error rate looks like for that task and who reviews the exceptions. Then measure against it. That one habit separates teams that get real value from AI from teams that just have another dashboard.
For more conversations on freight and AI, subscribe to the Freight Guru Podcast.
About the author: Luis Lopez is a Miami-based AI transportation consultant and logistics entrepreneur, the CEO of Go Hub.io Holdings Corp and subsidiaries, and host of the Freight Guru Podcast.
