clawhub

Probabilistic Thinking

Activate when: user says 'what are the odds,' 'base rate,' 'Bayesian update,' 'calibration,' or 'what's the probability'; user is making a forecast or estima...

view source

installs

257

stars

karma

SkillRank score ↗

8.3/ 10

evaluated by implexa, claude-haiku-4-5 · 2026-07-07

probabilistic-thinking covers replacing binary yes/no reasoning with calibrated probability estimates anchored in base rates, updated with evidence, and scored against outcomes. rooted in bayes, knight's risk-uncertainty distinction, and tetlock's empirical work on forecast calibration.

structure

9.0

trigger phrases

9.0

procedure

9.0

edge cases

7.0

documentation

8.0

view original SKILL.md from clawhubclick to expand

---
name: probabilistic-thinking
description: "Activate when: user says 'what are the odds,' 'base rate,' 'Bayesian update,' 'calibration,' or 'what's the probability'; user is making a forecast or estimating a conversion/close/hire/outcome probability; someone claims 'I'm 90% sure' without any evidence grounding; a vivid story is being used in place of a prior frequency; team is treating an uncertain outcome as binary will/won't.
  Do NOT activate when: the problem has a deterministic answer from its inputs (math, well-defined engineering); the question is about identity, ethics, or meaning rather than fact. More: deciqai.com/c/probabilistic-thinking"
---

# Probabilistic Thinking

## Overview

Most reasoning is binary: *will it happen, or won't it?* That framing discards the most useful information — the **degree** of confidence — and produces predictions that cannot be checked, updated, or scored. Probabilistic thinking replaces binary with **calibrated probability estimates**: numbers anchored in base rates, updated with evidence, and scored after the fact. Rooted in Bayes (1763), Knight's risk-vs-uncertainty distinction (1921), and Tetlock's empirical work showing calibration is a trainable skill.

Composable neighbors: first-principles · occams-razor · second-order-thinking · inversion · regret-minimization · expected-value-and-kelly. This skill is the *upstream input* the others depend on — the probability estimate here feeds EV-Kelly, calibrates inversion's failure-path weights, and gives second-order's hops their confidence decay.

## When to Use

Use when reasoning about an uncertain outcome (forecast, diagnosis, pipeline conversion, hire, deal close, geopolitical event); when binary "will/won't" predictions are being made; when a vivid story is replacing a base rate; when "I'm 90% sure" appears with no calibration evidence; when forecasting AI timelines / AGI arrival / agentic reliability, or judging whether AI capex, AI valuations, or AI adoption rates justify a point-estimate bet amid genuine uncertainty.

**When NOT to use:** deterministic problems (math, well-defined engineering); pure Knightian uncertainty with no usable base rate (give a range + humility statement instead); decision is robust across all likely probabilities; question is identity/ethics/meaning (→ regret-minimization).

## Coaching Novices (Adaptive Front Door)

- **Engine mode:** user has a concrete forecasting question → run The Process directly.
- **Coach mode:** user is unfamiliar or has no concrete case → guide step by step.

In Coach mode, respond one step at a time. Each [WAIT] is a hard stop — output only that step's question, then stop.

1. **What-it-is.** Probabilistic thinking replaces "will it happen or not" with a number (0–1) anchored in base rates, updated with evidence, and scored after the fact.
2. **Check fit.** Match against *When to Use / When NOT to use*. Redirect if deterministic or pure Knightian.
3. **Elicit their real question.** "Odds of success" is vague; "probability customer X signs by Q3 given yesterday's call" is a question.
> **[WAIT — do not advance until user responds]**
4. **Walk The Process one step per turn.** Base rate first, then evidence, then update *with* them.
> **[WAIT — do not advance until user responds]**
5. **Close.** State the probability number and the one piece of evidence that would move it most.
> **[WAIT — do not advance until user responds]**

## The Process

Run the **Probability Estimate**. Base rate first, then evidence, then update, then calibration check.

1. **Precise question + deadline.** "Will the deal close?" → "Will customer X sign ≥$50K by 2026-09-30?"
2. **Anchor in a base rate.** Historical fraction of similar situations. No base rate = Knightian territory → report range, not point.
3. **Evidence for and against.** Each signal moves estimate ↑ or ↓. Be uncharitable about both sides.
4. **Bayesian update (plain language).** For each signal: P(evidence | outcome happens) vs P(evidence | doesn't happen). The ratio drives the shift.
5. **Number + confidence interval.** Not "70-ish" — "68%, 80% CI 55–80%."
6. **Most-informative next evidence.** If nothing would move your estimate, you have a belief, not an estimate.
7. **Calibration log.** Record estimate, date, resolution criteria. Score after: did 70%-calls land 70% of the time?

### Output: the Probability Estimate

```
Question (precise): <outcome, deadline>
Base rate: <reference class> → <fraction> (source, n=)
Evidence: <signal> ↑/↓ strong/moderate/weak  [repeat per signal]
Bayesian shift: net <↑/↓ to X%> — rationale in one paragraph
Estimate: <point %>  |  80% CI: <%–%>  |  Knightian caveat if needed
Next evidence: <observable> → <% if X> / <% if Y>
Calibration log: date | question | resolution criteria | Brier score after
```

*→ Method in Action: [Tetlock, IARPA, and the Good Judgment Project (2011–2015)](examples/tetlock-iarpa-good-judgment-project-2011-2015.md)*
*→ 2026 lens: [Forecasting AI Timelines and Agentic Reliability (2023–2026)](examples/forecasting-ai-timelines-agentic-reliability-2023-2026.md)*

## Calibration Packs

| Domain | Base rate source | Classic failure |
|---|---|---|
| Medical | Disease prevalence in population | Base-rate neglect → false positives |
| Sales | Conversion-by-stage history | Anchoring on preferred deal |
| Legal | Crime/suspect-pool frequencies | Prosecutor's fallacy |
| Product/startup | Cohort retention, vintage distributions | Survivorship bias |

## Applying It Well

- **Base rate first.** A story without a base rate is fiction with a number attached.
- **Incremental updates.** Superforecasters update *more often* but *less drastically* than amateurs.
- **Risk ≠ uncertainty (Knight 1921).** For Knightian situations give a range + humility statement, not false precision.
- **Name what would change your mind.** If nothing would move your estimate, you have a position, not an estimate.
- **Score yourself.** Write forecasts down and check them — calibration is trainable only this way.

*→ Primary sources: [references/sources.md](references/sources.md)*

## Common Rationalizations

**[D] = designed upfront | [O] = observed in real use. [O] entries are more valuable.**

| Fake move | Reality |
|---|---|
| [D] **Base-rate neglect** | Reaching for a vivid story while ignoring the prior frequency of the outcome class. *Always anchor in a base rate first; updates come from there.* |
| [D] Confusing **P(evidence \| outcome)** with **P(outcome \| evidence)** (prosecutor's fallacy) | A test triggering on 99% of cases of a rare disease will, in a low-prevalence population, produce mostly false positives. Bayes' theorem connects them; they are not interchangeable. |
| [D] Treating "very likely" as a **binary** | "I'm 90% sure" with no calibration history and no number for "what would make it 50%" is a vibe, not an estimate. |
| [D] Confusing **risk** with **uncertainty** (Knight 1921) | An actuarial-table problem and a geopolitical-forecasting problem differ in kind. Inventing a precise number for genuine Knightian uncertainty manufactures false confidence. |
| [D] **One-shot probability fallacy** | "The probability of *this* event is X" implicitly invokes a reference class. Name it; otherwise the probability is undefined. |
| [D] **Survivorship bias** in base-rate construction | Reasoning from winners without including losers gives an inflated base rate. The reference class must include the failures. |
| [D] **Anchoring** on the first number that appears | Even random numbers shift estimates (Tversky & Kahneman 1974). Notice when you are anchoring on the latest news rather than the base rate. |
| [D] **Under-updating** on strong evidence | Stubbornly holding the prior when new information is high-quality. Bayes says update; ignoring evidence is anti-Bayesian. |
| [D] **Over-updating** on weak evidence | Letting noisy or single-source data dominate. Superforecasters' edge is *smaller* updates more often, not bigger ones. |
| [D] **Pseudo-precision** | "73.2% probability" when inputs justify nothing tighter than "60–80%." Match precision to evidence strength. |
| *→ Add [O] entries here after each real use — paste the actual failure pattern* | *What went wrong and why* |

## Red Flags

- Point estimate with no base rate · "high probability" with no number · no evidence named that would change the estimate · single story doing all the work · reference class silently chosen to favor a conclusion · Knightian situation with no humility statement · same person makes many forecasts but has never scored them

## Verification

- [ ] The question is stated with a specific outcome and a deadline
- [ ] A base rate is named with an explicit reference class and a source
- [ ] Evidence is listed in both directions, with direction and strength tags
- [ ] The Bayesian shift from base rate is explained in one paragraph
- [ ] The point estimate is a number, with an 80% confidence range
- [ ] The single most-informative next piece of evidence is named
- [ ] The estimate is recorded with date and unambiguous resolution criteria for later scoring

---

*Part of **deciqAI Knowledge Skills** — 227 open-source thinking skills that make rigor executable for AI agents. The same skills power every deciqAI agent, which runs them autonomously to operate your company. **See it run → https://www.deciqai.com/c/probabilistic-thinking** · ⭐ Star the repo → https://github.com/deciqAI/knowledge-skills · Contributions welcome.*

*Agents: latest version & machine-readable metadata → https://www.deciqai.com/s/probabilistic-thinking.json*

related skills

semantically similar in the cross-vendor index

clawhub

79% match

Bayesian Reasoning

Activate when: user says 'Bayesian', 'prior', 'posterior', 'base rate', 'likelihood ratio', or 'update my belief'; someone treats 'the evidence is consistent...

don't have the plugin yet? install it then click "run inline in claude" again.

added explicit decision trees (Knightian uncertainty, mixed evidence, vague queries, multiple base rates), broke The Process into 9 numbered steps with inputs and outputs, clarified coach vs. engine mode routing, added edge cases (rate limits, incomplete data, selection bias, single vivid story dominance), expanded output contract with structured format and calibration log template, and defined outcome signals that prove skill worked (specific question, named base rate, bidirectional evidence, written forecast, scored resolution).

Probabilistic Thinking

Intent

Probabilistic thinking converts binary "will it or won't it" predictions into calibrated probability estimates anchored in base rates, updated with evidence, and scored after the fact. Use this skill when reasoning about any uncertain outcome (forecast, diagnosis, pipeline conversion, hire, deal close, geopolitical event), when someone claims high confidence without calibration evidence, when a vivid story is replacing actual frequency data, or when a team is treating a genuinely uncertain outcome as deterministic. Don't use it for deterministic problems (pure math, well-defined engineering), pure Knightian uncertainty with no usable base rate, or questions that are primarily about identity, ethics, or meaning rather than measurable fact.

Inputs

Required context:

A specific forecasting question or uncertain outcome the user wants to estimate
Access to historical data or reference class information for base rate anchoring
User's comfort level with probabilistic reasoning (triggers coach mode vs. engine mode)

External connections (optional but recommended):

Salesforce or internal CRM (for sales conversion base rates by stage, deal size, rep history) - no auth required if you have read access to historical pipeline reports
Internal data warehouse or analytics tool (retention cohorts, vintage distributions, historical frequency tables)
Domain-specific forecasting databases (medical disease prevalence, crime/conviction frequencies, legal case outcome distributions)

Setup guidance:

If user references a specific domain (sales, medical, legal, product), prepare base rate sources before starting The Process
If user is new to probabilistic thinking, shift to Coach mode and guide step by step with [WAIT] pauses
Have a calibration log template ready (see output contract below)

Edge cases to know upfront:

No base rate available: report a range + humility statement instead of a false point estimate (Knightian uncertainty)
Rate-limited or incomplete historical data: adjust confidence interval to reflect data sparsity
Reference class too narrow or too broad: explicit discussion of class selection prevents selection bias
Single vivid story dominating reasoning: explicitly list evidence for and against to surface anchoring
User unwilling to name what would change their mind: halt and redirect to actual estimation vs. posturing

Procedure

Step 1: Activate and Route to Mode

Input: User query or context clue. Action: Listen for "what are the odds," "base rate," "probability," "forecast," "calibration," "I'm X% sure," or a team treating uncertain outcome as binary. Check the When to Use / When NOT to use sections. If deterministic or identity-based, redirect. Otherwise, continue. Output: Confirmed: this is a probabilistic reasoning problem. Proceed to Step 2.

Step 2: Assess User Readiness and Choose Mode

Input: User's familiarity with probabilistic thinking (explicit statement or inferred from first question). Action:

If user is new, unfamiliar, or has no concrete case yet: shift to Coach mode. Output Step 1 of the Coaching Novices section and stop at [WAIT].
If user has a concrete forecasting question ready: shift to Engine mode. Proceed to Step 3. Output: Declared mode (Coach or Engine) and next action.

Step 3 (Engine Mode Only): Elicit Precise Question

Input: User's forecasting problem (often vague: "odds of success," "will we hit targets"). Action: Rewrite the question with specific outcome, deadline, and success threshold.

"Odds of deal close?" becomes "Will customer X sign a ≥$50K contract by 2026-09-30?"
"Will we hit retention targets?" becomes "Will cohort 2026-Q1 retain ≥60% of users through 2026-12-31?" Output: Precise question stated. Proceed to Step 4.

Step 4: Anchor in Base Rate

Input: Precise question. Access to historical data, domain knowledge, or reference class sources. Action:

Name the reference class explicitly: "similar deals in this territory," "first-time hires at this level," "geopolitical forecasts in the same region-type."
Find the historical fraction: n successes / n total. Include the source and sample size (n=X).
If no base rate exists: state this as Knightian uncertainty and skip to Step 6 (output range, not point estimate).
If multiple base rates conflict: list all, note which is most relevant, and explain why. Output: Base rate (%, source, n=) or humility statement if Knightian. Proceed to Step 5.

Step 5: Gather Evidence For and Against

Input: The precise question and base rate. Action: Brainstorm all signals that move the estimate up or down. Be uncharitable to both sides.

For each signal, tag direction (↑ or ↓) and strength (strong, moderate, weak).
Strong: directly predictive, high-quality data, high sample size.
Moderate: some correlation, moderate data quality.
Weak: anecdotal, single source, low information.
Example: "Customer asked for custom integration ↑ strong [previous signal of intent-to-buy]" vs. "Competitor in conversation ↓ moderate [we lost 3 of 5 competitive deals this year]." Output: Bulleted list of evidence with direction and strength tags. Proceed to Step 6.

Step 6: Bayesian Update (Plain Language)

Input: Base rate and evidence list. Action: For each piece of evidence, apply Bayes' theorem without jargon.

P(evidence | outcome happens) vs. P(evidence | doesn't happen). The ratio drives the shift.
Example: "Customer asked for custom integration. In our base rate (50% close), how often do closed deals request custom work vs. lost ones? Closed: 70%. Lost: 20%. Ratio 70/20 = 3.5x → strong shift upward."
Net effect: move base rate ↑ or ↓ and state the new estimate.
If evidence is conflicting, describe the net direction and magnitude in one paragraph. Output: Updated point estimate (%). Proceed to Step 7.

Step 7: Confidence Interval and Calibration Bounds

Input: Point estimate and evidence strength. Action:

Express as: "X% | 80% CI: Y% to Z%"
Match precision to evidence. High-quality base rate + multiple strong signals = tighter CI. Weak data = wider CI.
Example: "68% | 80% CI: 55% to 80%" (wider interval, less certain) vs. "72% | 80% CI: 69% to 75%" (tighter interval, more certain).
Do not report false precision. "73.2%" is pseudo-precision unless inputs justify it. Output: Point estimate with confidence interval. Proceed to Step 8.

Step 8: Name Most-Informative Next Evidence

Input: Current estimate and evidence gathered so far. Action: Identify the single piece of information that would move your estimate most.

Estimate two outcomes: "If X happens, estimate becomes Y%. If X doesn't happen, estimate becomes Z%."
This forces clarity: if nothing would move your estimate, you have a belief, not an estimate. Halt and recalibrate. Output: Named next evidence (observable) and two conditional estimates. Proceed to Step 9.

Step 9: Record for Calibration Scoring

Input: Full estimate with all components. Action: Write to calibration log (see output contract for format). Include date, question, resolution criteria, and space for post-resolution Brier score.

Do not skip this step. Calibration is trainable only if estimates are written down and scored. Output: Calibration log entry. Skill complete.

Decision Points

If user query is vague ("What are the odds?") without a specific outcome:

Redirect to Step 3 (Elicit Precise Question). Do not proceed until outcome, deadline, and threshold are explicit.

If user is new to probabilistic thinking and has no concrete case:

Shift to Coach mode. Walk The Process one step per turn with [WAIT] pauses. Do not accelerate.

If a base rate cannot be found (Knightian uncertainty with no usable reference class):

Do not invent a point estimate. Instead, report a range (e.g., "somewhere between 30% and 70%, leaning 50%, but genuine uncertainty here") and add a humility statement: "We have no relevant historical data, so this is a judgment call, not a calibrated estimate."
Proceed to Step 8 anyway (name what would move your mind), but skip Step 7 (no CI required for non-point estimates).

If evidence is mixed and updates cancel each other out (net zero shift from base rate):

State the conflicting signals explicitly and explain why they approximately cancel.
Estimate stays at base rate. Do not artificially widen the CI to hide indecision; instead, state: "Evidence is balanced; estimate holds at base rate, but observe signal X next."

If user names an outcome they've already forecast before:

Check the calibration log: did previous forecasts on similar questions land where predicted? Use that meta-evidence to adjust confidence in current estimate.
If past estimates were well-calibrated, tighter CI is justified. If overconfident, widen CI.

If multiple reference classes apply (e.g., "deals under $50K" vs. "all deals" vs. "this rep's deals"):

List all base rates and explain which is most relevant to the question.
Use the most specific class (e.g., "this rep's deals, $50K range") if n is sufficient (n ≥ 20 at minimum).
If specific class has low n, blend with broader class and state the blend explicitly.

If user claims high confidence ("I'm 90% sure") with no evidence:

Pause and run Steps 5 and 6 (evidence and Bayesian update). Do not accept the claimed number without justification.
If evidence doesn't support 90%, state the discrepancy and update downward.

Output Contract

Probability Estimate (structured format):

Question (precise): <outcome, deadline, success threshold>
Base rate: <reference class> → <X%> (source, n=<count>)
Evidence:
  <signal> ↑ <strong|moderate|weak> [rationale]
  <signal> ↓ <strong|moderate|weak> [rationale]
  [repeat for each signal]
Bayesian shift: net <direction> to X% ,  <one-paragraph rationale explaining P(evidence|outcome) vs P(evidence|no outcome) and why the net effect is <direction>>
Estimate: <point %>  |  80% CI: <Y%, Z%>  |  <Knightian caveat if needed>
Next evidence: <observable> → estimate becomes Y% if true, Z% if false
Calibration log entry: date | question | resolution criteria | [post-resolution: outcome, Brier score]

Calibration log (maintained across forecasts, scored after resolution):

Date	Question	Resolution Criteria	Estimate	Outcome (Y/N)	Brier Score	Notes
2026-01-15	Will customer X sign ≥$50K by 2026-09-30?	Contract signed and dated by deadline	68% (80% CI: 55, 80%)	[pending]	[pending]	Evidence: custom integration request (↑strong), competitor in conversation (↓moderate)

File location: Store calibration log in version control or a shared analytics tool (e.g., Google Sheets, Airtable, internal wiki). Do not rely on memory.

Metadata output (for agents): If running on Implexa or deciqAI platform, emit JSON:

{
  "skill": "probabilistic-thinking",
  "question": "<precise question>",
  "base_rate_pct": <X>,
  "estimate_pct": <Y>,
  "confidence_interval_80_low": <A>,
  "confidence_interval_80_high": <B>,
  "next_evidence": "<observable>",
  "knightian_flag": <true|false>,
  "timestamp": "<ISO 8601>",
  "reference_class": "<class name>"
}

Outcome Signal

The skill worked if:

User can state the outcome they're forecasting with a specific deadline and threshold, not "odds of success" but "Will customer X sign ≥$50K by 2026-09-30?"
A base rate is named with an explicit reference class and source, not pulled from thin air or a single story.
Evidence is listed in both directions (for and against) with direction and strength tags, forcing honest consideration of contradictions.
The estimate is a number with an 80% confidence interval, not a vibe ("pretty likely") and not false precision ("73.2%").
The user can name one piece of information that would move the estimate meaningfully, proving they have an estimate, not a belief.
The forecast is written down in a calibration log with date, question, resolution criteria, and a reserved spot for post-resolution scoring.
After resolution (when the deadline passes or the outcome is known), the forecast is scored and compared against historical accuracy (Brier score or calibration curves). The user's estimates at the 70% level should resolve true 70% of the time, not 50% or 90%.
Over multiple forecasts, the user's calibration improves. Overconfident early estimates tighten; weak evidence is weighted less; reference classes are refined.

Original source and attribution: deciqAI Knowledge Skills (deciqai.com/c/probabilistic-thinking). Rooted in Bayes (1763), Knight's risk-vs-uncertainty distinction (1921), and Tetlock's empirical work on the Good Judgment Project (2011-2015). Method draws on Tversky & Kahneman (1974) on anchoring, and superforecasting practices from Tetlock & Gardner (2015).

Part of 227 open-source thinking skills for AI agents. See it run at deciqai.com/c/probabilistic-thinking. Contributions welcome at github.com/deciqAI/knowledge-skills.