Probability and Prediction
"EPSS failed to predict ..." is usually said with a misunderstanding of what prediction (and probability) really is, and may be followed by some counter-claim made with a "just trust me" vibe.
People tend to think that prediction is an exact statement about the future - "Give me the list of vulnerabilities that will be exploited in the next 30 days" is an impossible task that no person or method can deliver on. Yet, for some reason this is exactly what people expect. Even if someone knew all of vulnerabilities that were exploited in the *last* 30 days (which is also not possible, but that's a different discussion), it's nearly certain that the list of vulnerabilities in the *next* 30 days wouldn't match.
Enter probability. Bertrand Russell once said, "Probability is the most important concept in modern science, especially as nobody has the slightest notion of what it means."
EPSS produces a probability. I know on the face this may seem like a cop-out because even if the absolutely lowest rated vulnerability was exploited we could hide behind probability and say "but I said there was at some chance of that happening" and we could never be wrong. But that's not how probability works and clearly that's not how probabilistic estimates are (in)validated. It's impossible to measure the accuracy of a prediction from any single event. But given enough predictions and observed outcomes we absolutely can measure accuracy.
Luckily EPSS v4 is scoring well over 250,000 vulnerabilities and each one of those is a probabilistic statement. We can measure the accuracy of the EPSS predictions by looking at *all* of those statements *as a whole*. One method of doing that is a calibration plot, which plots the prediction against reality. The horizontal on this plot bins up the predicted probabilities and the vertical looks up the proportion of each bin that were actually exploited in the wild.
So next time you see someone say "EPSS failed to predict..." or "EPSS got this wrong", see if it's a single anecdotal claim you will hopefully spot this silly and sneaky tactic to trick you.
We should stop taking vendors and the loud-speaking folks at their word when they claim they could've done better on this or that. Demand the data, ask for the statistics because single anecdotal use cases are not enough. Ask about false positive and true positive rates (or other evidence). If they can't answer, every other claim out of their mouth should be met with skepticism. If they are able answer, just as with everything in life, do not expect perfection, otherwise you will always be disappointed.
The future of cybersecurity needs empirical evidence.
