Suppose an analyst makes a hundred calls and eighty come out right. Impressive. Now suppose every one of those calls was stated with near-certainty. That analyst is accurate and badly calibrated, because they claimed ninety-five and delivered eighty, and anyone sizing on their confidence has been over-betting for a hundred trades.
What calibration means
Take everything said with about seventy per cent confidence. If roughly seventy per cent of those came true, you are calibrated at that level. Do it across the confidence range and you have a calibration curve: stated confidence against realised frequency.
A well-calibrated forecaster can be barely better than chance and still useful, because you know exactly how much weight to put on each statement. A poorly calibrated one is dangerous in proportion to how accurate they are, because the accuracy earns trust that the confidence then misuses.
Why it matters more than accuracy for sizing
Position sizing is a function of confidence. If the stated confidence is systematically too high, every position is systematically too large, and the errors compound in the direction that hurts.
This is why a forecaster who says they do not know is providing information rather than dodging. An honest low-confidence call is more useful than a confident one that is right slightly more often.
It requires the record
Calibration cannot be assessed from memory or from a handful of examples. It needs every call recorded at the time with its confidence attached, and then scored against what happened by a rule fixed in advance.
Any part of that missing and the exercise fails. Confidence assigned afterwards is not confidence. Calls selected for scoring are not a sample.
What to do with a miscalibrated source
Adjust rather than discard. A source that is consistently overconfident by a known margin is still informative once you discount it. Consistency is the useful property, and it is only visible in a long record.