From Signals to Calibrated Confidence: The Evolution of anyformat's Reliability Framework

In a previous article, we introduced how anyformat computes confidence from model signals such as token log-probabilities, entropy, and probability gaps. That first version focused on building a meaningful signal: a way to estimate when the model hesitated during generation.

Since then, we have evolved our methodology from signal-based confidence to fully calibrated confidence, grounded in labeled data and statistical validation. The goal is no longer just to rank outputs from more certain to less certain, but to ensure that:

A confidence score of 90% means that approximately 90% of those predictions are correct.

This shift, from signal to calibrated probability, is what makes confidence actionable in production systems.

Confidence Is Not Just a Signal. It Is a Probability

Modern LLMs expose internal uncertainty signals: token logprobs, alternative token distributions, entropy, probability gaps. These signals are extremely informative.

But they are not probabilities of correctness.

Logprobs measure linguistic likelihood under the model's internal distribution. They do not directly measure whether an extracted value matches the ground truth of a document.

To bridge that gap, we need calibration. Calibration means:

Among predictions with confidence p, approximately p% should be correct.

Formally, we learn a mapping:

f(logprob signals) ≈ P(Y = 1 | signals)

Where:

• Y = 1 denotes that the prediction is correct.
• The signals come from token-level statistics aggregated over meaningful units.

Calibration is:

• Post-hoc: applied after the model generates outputs.
• Model-agnostic: it does not require retraining the base LLM.
• Supervised: it requires labeled correctness data.

Without labeled data, there is no way to know whether a raw score of 0.7 corresponds to 70%, 40%, or 95% real-world accuracy.

Per-Model Calibration

Not all models express uncertainty in the same way.

The logprob distributions of GPT-family models differ significantly from those of Gemini-family models. A logprob of -0.1 in one model does not mean the same thing as -0.1 in another.

For this reason:

• We calibrate confidence separately per model.
• A 90% confidence score means the same thing whether the extraction was produced by a GPT-family or a Gemini-family model.
• Switching models does not silently change your operational risk profile.

Calibration is always model-specific.

Per-Task Calibration: Parsing vs Extraction

We also calibrate confidence independently for our two main tasks:

1. Document parsing
2. Structured value extraction (from a parsed document)

These tasks differ in granularity and in what correctness means.

Parsing: confidence at span level

For parsing, our calibration unit is a span, which can be a table cell, a meaningful text line, and other similar units.

We deliberately chose spans because they represent the business-relevant semantic units of a document. Downstream extraction operates over these units. Calibrating at token level would not reflect how the data is actually consumed.

Extraction: confidence at value level

For extraction, the unit is a value instance: a single leaf in the output JSON, such as invoice.total or vendor.address.city.

This is the smallest unit where correctness matters for business logic. Confidence must reflect whether that specific value is correct, not whether individual tokens were likely.

Ground Truth: Why Labeled Data Is Necessary

To calibrate our system, we used a diverse labeled benchmark dataset covering a wide range of real-world document types such as forms, tables and invoices.

Rather than relying on off-the-shelf annotations, we carefully reviewed and curated the dataset internally to align with how anyformat parses and extracts. Each document was processed end to end and manually validated to ensure that:

• The parsed representation reflects the true document structure.
• Extracted values follow a consistent and well-defined schema.
• Correctness labels are precise and reliable.

This dataset enables us to calibrate and evaluate both parsing confidence and extraction confidence in a controlled and statistically meaningful way.

We plan to disclose more details about this benchmark in the future.

From Documents to Labeled Calibration Units

Once we have benchmark documents and model predictions, we decompose both into business-relevant units:

• Parsing operates at semantic span level.
• Extraction operates at JSON leaf value level.

These units reflect how data is consumed in real workflows. Each predicted unit is matched against ground truth and labeled as correct or incorrect. This creates a labeled dataset of meaningful units.

Only at this point can calibration begin.

Two Separate Problems: Discrimination and Calibration

It is important to distinguish between two steps.

1. Score construction (discrimination)

First, we build a signal that separates correct units from incorrect ones.

We maintain the spirit of our original signal-based approach and use interpretable token-derived features such as:

• Average top logprob
• Minimum logprob within the unit
• Average and minimum top-token gap
• Normalized entropy

These features capture overall model confidence, the weakest point of hesitation, decisiveness vs ambiguity, and distributional uncertainty.

We construct a parametric formula combining these terms. We intentionally keep the expression interpretable and low-dimensional, rather than training a high-capacity black-box classifier, in order to preserve interpretability, reduce overfitting risk, and ensure robustness under distribution shift and domain transfer.

Then we choose the parameters that maximize our chosen separation metric (for example, ROC AUC or PR AUC).

At this stage, we are solving a standard classification problem:

Rank correct units above incorrect ones as effectively as possible.

This produces a score, but not necessarily a calibrated probability yet.

2. Calibration (probability mapping)

Once we have a discriminative score, we apply a calibration function that maps it to a real probability. We use post-hoc calibration (for example, logistic or Platt scaling) learned from labeled data.

After calibration, the interpretation becomes operational:

If we output confidence 0.9, approximately 90% of such units are correct.

To evaluate calibration quality, we use adaptive reliability diagrams and Adaptive Expected Calibration Error (Adaptive ECE). Adaptive binning ensures stable estimates even when confidence distributions are skewed.

Calibration reduces the gap between predicted confidence and empirical accuracy.

This is what turns a ranking signal into a decision-making tool.

Why This Matters in Production

With calibrated confidence:

• You can define thresholds that reflect real risk.
• You can trigger human review only when confidence drops below a certain probability.
• You can safely automate high-confidence values.
• You can compare models without changing your acceptance criteria.

Calibration decouples model uncertainty from operational decision thresholds. The model can evolve; your risk policy remains stable.

This article is also published on LinkedIn, along with the rest of our blog. Follow anyformat there to keep up with new posts.