Scout Mindset & Bias Check

Table of Contents

• Interactive Menu
• Quick Reference
• Resource Files

Core Principle: Map the territory accurately rather than defending a position. Forecasting requires intellectual honesty -- biases systematically distort probabilities, emotional attachment clouds judgment, and motivated reasoning leads to overconfidence.

Interactive Menu

What would you like to do?

Core Workflows

1. Run the Reversal Test - Check if you'd accept opposite evidence

• Detect motivated reasoning
• Validate evidence standards
• Expose special pleading

2. Check Scope Sensitivity - Ensure probabilities scale with inputs

• Linear scaling test
• Reference point calibration
• Magnitude assessment

3. Test Status Quo Bias - Challenge "no change" assumptions

• Entropy principle
• Change vs stability energy
• Default state inversion

4. Audit Confidence Intervals - Validate CI width

• Surprise test
• Historical calibration
• Overconfidence check

5. Run Full Bias Audit - Comprehensive bias scan

• All major cognitive biases
• Systematic checklist
• Prioritized remediation

6. Learn the Framework - Deep dive into methodology

• Read Scout vs Soldier Mindset
• Read Cognitive Bias Catalog
• Read Debiasing Techniques

7. Exit - Return to main forecasting workflow

1. Run the Reversal Test

Check if you'd accept evidence pointing the opposite direction.

Reversal Test Progress:
- [ ] Step 1: State your current conclusion
- [ ] Step 2: Identify supporting evidence
- [ ] Step 3: Reverse the evidence
- [ ] Step 4: Ask "Would I still accept it?"
- [ ] Step 5: Adjust for double standards

Step 1: State your current conclusion

What are you predicting?

• Prediction: [Event]
• Probability: [X]%
• Direction: [High/Low confidence]

Step 2: Identify supporting evidence

List the evidence that supports your conclusion.

Example: Candidate A will win (75%)

1. Polls show A ahead by 5%
2. A has more campaign funding
3. Expert pundits favor A
4. A has better debate ratings

Step 3: Reverse the evidence

Imagine the same evidence pointed the OTHER way.

Reversed: What if polls showed B ahead, B had more funding, experts favored B, and B had better ratings?

Step 4: Ask "Would I still accept it?"

The Critical Question:

If this reversed evidence existed, would I accept it as valid and change my prediction?

Three possible answers:

A) YES - I would accept reversed evidence ✓ No bias detected, continue with current reasoning

B) NO - I would dismiss reversed evidence ⚠ Warning: Motivated reasoning - you're accepting evidence when it supports you, dismissing equivalent evidence when it doesn't (special pleading)

C) UNSURE - I'd need to think about it ⚠ Warning: Asymmetric evidence standards suggest rationalizing, not reasoning

Step 5: Adjust for double standards

If you answered B or C:

Ask: Why do I dismiss this evidence in one direction but accept it in the other? Is there an objective reason, or am I motivated by preference?

Common rationalizations:

• "This source is biased" (only when it disagrees)
• "Sample size too small" (only for unfavorable polls)
• "Outlier data" (only for data you dislike)
• "Context matters" (invoked selectively)

The Fix:

• Option 1: Reject the evidence entirely (if you wouldn't trust it reversed, don't trust it now)
• Option 2: Accept it in both directions (trust evidence regardless of direction)
• Option 3: Weight it appropriately (maybe it's weak evidence both ways)

Probability adjustment: If you detected double standards, move probability 10-15% toward 50%

Next: Return to menu

2. Check Scope Sensitivity

Ensure your probabilities scale appropriately with magnitude.

Scope Sensitivity Progress:
- [ ] Step 1: Identify the variable scale
- [ ] Step 2: Test linear scaling
- [ ] Step 3: Check reference point calibration
- [ ] Step 4: Validate magnitude assessment
- [ ] Step 5: Adjust for scope insensitivity

Step 1: Identify the variable scale

What dimension has magnitude?

• Number of people (100 vs 10,000 vs 1,000,000)
• Dollar amounts ($1K vs $100K vs $10M)
• Time duration (1 month vs 1 year vs 10 years)

Step 2: Test linear scaling

The Linearity Test: Double the input, check if impact doubles.

Example: Startup funding

• If raised $1M: ___%
• If raised $10M: ___%
• If raised $100M: ___%

Scope sensitivity check: Did probabilities scale reasonably? If they barely changed → Scope insensitive

Step 3: Check reference point calibration

The Anchoring Test: Did you start with a number (base rate, someone else's forecast, round number) and insufficiently adjust?

The fix:

• Generate probability from scratch without looking at others
• Then compare and reconcile differences
• Don't just "split the difference" - reason about why estimates differ

Step 4: Validate magnitude assessment

The "1 vs 10 vs 100" Test: For your forecast, vary the scale by 10×.

Example: Project timeline

• 1 month: P(success) = ___%
• 10 months: P(success) = ___%
• 100 months: P(success) = ___%

Expected: Probability should change significantly. If all three estimates are within 10 percentage points → Scope insensitivity

Step 5: Adjust for scope insensitivity

The problem: Your emotional system responds to the category, not the magnitude.

The fix:

Method 1: Logarithmic scaling - Use log scale for intuition

Method 2: Reference class by scale - Don't use "startups" as reference class. Use "Startups that raised $1M" (10% success) vs "Startups that raised $100M" (60% success)

Method 3: Explicit calibration - Use a formula: P(success) = base_rate + k × log(amount)

Next: Return to menu

3. Test Status Quo Bias

Challenge the assumption that "no change" is the default.

Status Quo Bias Progress:
- [ ] Step 1: Identify status quo prediction
- [ ] Step 2: Calculate energy to maintain status quo
- [ ] Step 3: Invert the default
- [ ] Step 4: Apply entropy principle
- [ ] Step 5: Adjust probabilities

Step 1: Identify status quo prediction

Are you predicting "no change"? Examples: "This trend will continue," "Market share will stay the same," "Policy won't change"

Status quo predictions often get inflated probabilities because change feels risky.

Step 2: Calculate energy to maintain status quo

The Entropy Principle: In the absence of active energy input, systems decay toward disorder.

Question: "What effort is required to keep things the same?"

Examples:

• Market share: To maintain requires matching competitor innovation → Energy required: High → Status quo is HARD
• Policy: To maintain requires no proposals for change → Energy required: Low → Status quo is easier

Step 3: Invert the default

Mental Exercise:

• Normal framing: "Will X change?" (Default = no)
• Inverted framing: "Will X stay the same?" (Default = no)

Bias check: If P(change) + P(same) ≠ 100%, you have status quo bias.

Step 4: Apply entropy principle

Second Law of Thermodynamics (applied to forecasting):

Ask:

1. Is this system open or closed?
2. Is energy being input to maintain/improve?
3. Is that energy sufficient?

Step 5: Adjust probabilities

If you detected status quo bias:

For "no change" predictions that require high energy:

• Reduce P(status quo) by 10-20%
• Increase P(change) correspondingly

For predictions where inertia truly helps: No adjustment needed

The heuristic: If maintaining status quo requires active effort, decay is more likely than you think.

Next: Return to menu

4. Audit Confidence Intervals

Validate that your CI width reflects true uncertainty.

Confidence Interval Audit Progress:
- [ ] Step 1: State current CI
- [ ] Step 2: Run surprise test
- [ ] Step 3: Check historical calibration
- [ ] Step 4: Compare to reference class variance
- [ ] Step 5: Adjust CI width

Step 1: State current CI

Current confidence interval:

• Point estimate: ___%
• Lower bound: ___%
• Upper bound: ___%
• Width: ___ percentage points
• Confidence level: ___ (usually 80% or 90%)

Step 2: Run surprise test

The Surprise Test: "Would I be genuinely shocked if the true value fell outside my confidence interval?"

Calibration:

• 80% CI → Should be shocked 20% of the time
• 90% CI → Should be shocked 10% of the time

Test: Imagine the outcome lands just below your lower bound or just above your upper bound.

Three possible answers:

• A) "Yes, I'd be very surprised" - ✓ CI appropriately calibrated
• B) "No, not that surprised" - ⚠ CI too narrow (overconfident) → Widen interval
• C) "I'd be amazed if it landed in the range" - ⚠ CI too wide → Narrow interval

Step 3: Check historical calibration

Look at your past forecasts:

1. Collect last 20-50 forecasts with CIs
2. Count how many actual outcomes fell outside your CIs
3. Compare to theoretical expectation

Diagnosis: Actual > Expected → CIs too narrow (overconfident) - Most common

Step 4: Compare to reference class variance

If you have reference class data:

1. Calculate standard deviation of reference class outcomes
2. Your CI should roughly match that variance

Example: Reference class SD = 12%, your 80% CI ≈ Point estimate ± 15%

If your CI is narrower than reference class variance, you're claiming to know more than average. Justify why, or widen CI.

Step 5: Adjust CI width

Adjustment rules:

• If overconfident: Multiply current width by 1.5× to 2×
• If underconfident: Reduce width by 0.5× to 0.75×

Next: Return to menu

5. Run Full Bias Audit

Comprehensive scan of major cognitive biases.

Full Bias Audit Progress:
- [ ] Step 1: Confirmation bias check
- [ ] Step 2: Availability bias check
- [ ] Step 3: Anchoring bias check
- [ ] Step 4: Affect heuristic check
- [ ] Step 5: Overconfidence check
- [ ] Step 6: Attribution error check
- [ ] Step 7: Prioritize and remediate

See Cognitive Bias Catalog for detailed descriptions.

Quick audit questions:

1. Confirmation Bias

• Did I seek out disconfirming evidence?
• Did I give equal weight to evidence against my position?
• Did I actively try to prove myself wrong?

If NO to any → Confirmation bias detected

2. Availability Bias

• Did I rely on recent/memorable examples?
• Did I use systematic data vs "what comes to mind"?
• Did I check if my examples are representative?

If NO to any → Availability bias detected

3. Anchoring Bias

• Did I generate my estimate independently first?
• Did I avoid being influenced by others' numbers?
• Did I adjust sufficiently from initial anchor?

If NO to any → Anchoring bias detected

4. Affect Heuristic

• Do I have an emotional preference for the outcome?
• Did I separate "what I want" from "what will happen"?
• Would I make the same forecast if incentives were reversed?

If NO to any → Affect heuristic detected

5. Overconfidence

• Did I run a premortem?
• Are my CIs wide enough (surprise test)?
• Did I identify ways I could be wrong?

If NO to any → Overconfidence detected

6. Fundamental Attribution Error

• Did I attribute success to skill vs luck appropriately?
• Did I consider situational factors, not just personal traits?
• Did I avoid "great man" narratives?

If NO to any → Attribution error detected

Step 7: Prioritize and remediate

For each detected bias:

1. Severity: High / Medium / Low
2. Direction: Pushing probability up or down?
3. Magnitude: Estimated percentage point impact

Remediation example:

Net adjustment: -45% → Move probability down by 45 points (e.g., 80% → 35%)

Next: Return to menu

6. Learn the Framework

Deep dive into the methodology.

Resource Files

📄 Scout vs Soldier Mindset

• Julia Galef's framework
• Motivated reasoning
• Intellectual honesty
• Identity and beliefs

📄 Cognitive Bias Catalog

• 20+ major biases
• How they affect forecasting
• Detection methods
• Remediation strategies

📄 Debiasing Techniques

• Systematic debiasing process
• Pre-commitment strategies
• External accountability
• Algorithmic aids

Next: Return to menu

Quick Reference

The Scout Commandments

1. Truth over comfort - Accuracy beats wishful thinking
2. Seek disconfirmation - Try to prove yourself wrong
3. Hold beliefs lightly - Probabilistic, not binary
4. Update incrementally - Change mind with evidence
5. Separate wanting from expecting - Desire ≠ Forecast
6. Check your work - Run bias audits routinely
7. Stay calibrated - Track accuracy over time

Scout mindset is the drive to see things as they are, not as you wish them to be.

Resource Files

📁 resources/

• scout-vs-soldier.md - Mindset framework
• cognitive-bias-catalog.md - Comprehensive bias reference
• debiasing-techniques.md - Remediation strategies

Ready to start? Choose a number from the menu above.