Philip E. Tetlock, 2023

“We learn a lot when we get serious about keeping score.”

Philip E. Tetlock used this phrase to summarize decades of forecasting tournaments. Clear probabilistic predictions can be compared with outcomes, making it possible to see which people, teams, methods, and habits improve judgment rather than reward confidence or reputation. Keeping score changes forecasting from commentary into a learning discipline. Questions need defined time frames and resolution criteria, while probabilities preserve uncertainty. Calibration reveals whether events assigned a given likelihood occur at roughly that rate over time. The business value of the line appears when it changes a real decision. It directs attention toward assumptions that routine can hide and invites a team to examine the system behind an outcome instead of treating the visible result as a complete explanation.

Organizations can maintain a forecast register for strategic assumptions, assign probabilities, update them when evidence changes, and score results. Teams should compare accuracy with simple baselines and study the reasoning habits behind both strong and weak forecasts. Application therefore requires an explicit mechanism: what should people do differently, what signal should move, and when will the team examine the result? Writing those elements down turns general agreement into a testable management practice.

Metrics can be gamed when questions are ambiguous or forecasters avoid bold but useful judgments. Not every important issue resolves cleanly, and a good score does not settle values. Governance must protect question quality, transparency, and learning over competition. Context matters because an idea that improves one situation can create harm in another. Leaders need boundaries, dissent, representative evidence, and a way to correct course. Accountability cannot be delegated to the quotation or the tool used to implement it.

AI systems can generate forecasts rapidly, creating a new need for common benchmarks and continuous scoring against human teams and base rates. Claims of superiority should be tied to defined domains, dates, costs, and reproducible evaluation. A strategy meeting can convert three decisive assumptions into dated probability questions, name the evidence that would trigger an update, and review the scores before the next planning cycle. The review should preserve what was known when the choice was made, document what changed, and share the learning beyond one team. Repeated with discipline, that cycle converts a memorable sentence into better judgment rather than another short-lived management theme. It also makes the cost of assumptions visible, allowing future teams to reuse proven knowledge while recognizing the point at which a different customer, risk, or operating environment requires fresh testing.