iTranslated by AI

The content below is an AI-generated translation. This is an experimental feature, and may contain errors. View original article
🌪️

Causal Inference with Python: Demystifying the 'Wind and Barrel Maker' Proverb via Spurious Correlations and Confounders

に公開
2025/12/10

Introduction

"If the wind blows, the barrel maker prospers."

Does this famous Japanese proverb raise a question in the minds of engineers and data scientists?

"Is there truly a causal relationship? Or is it just a correlation (or a spurious correlation)?"

To resolve this question, I performed the following experiments using Python:

  1. The Trap of Correlation: Creating a state where "wind" and "sales" show a strong correlation based solely on the data.
  2. Uncovering the Truth: Exposing that the correlation is a mirage using statistical analysis (regression analysis).
  3. Automatic Structure Discovery: Using Bayesian networks to let the data generate a "true causal graph (DAG)" on its own.

In the third step, "Structure Learning," the algorithm successfully deduced that "the wind and the barrel maker are unrelated," which was a fascinating result.

This should demonstrate the folly of dumping data into generative AI and demanding insights without critical thinking. I believe this highlights the true value of humans learning data science, including causal inference.

Architecture (Validation Environment)

https://github.com/shayate811/causal-inference

To prevent the common "it worked on my machine but broke after library updates" scenario, I introduced Dev Containers to ensure perfect reproducibility.

Since causal inference libraries like pgmpy frequently introduce breaking API changes, I have strictly pinned the versions in requirements.txt.

  • Language: Python 3.11
  • Libraries:
    • pgmpy==0.1.25 (Version Pinned)
    • pandas, statsmodels, networkx
  • Environment: Docker (Dev Containers)

1. Data Generation: Planting the "True Culprit"

First, I generate dummy data by assuming the "true mechanism" behind the proverb as follows:

  • True Structure: The "wind" does not make the "barrel maker" rich. "Temperature (winter cold)" is the common cause (confounding factor).
    • Getting colder \to Northern winds get stronger (Wind speed UP)
    • Getting colder \to Air becomes dry and barrels break \to Replacement demand (Sales UP)

In short, there is no direct causal relationship between wind speed and sales.

data_generation.py
import numpy as np
import pandas as pd

np.random.seed(42)
n_samples = 2000

# Common cause (confounding factor): Temperature
temperature = np.random.normal(0, 1, n_samples)

# Lower temperature leads to stronger wind (negative correlation) + noise
wind_speed = -0.7 * temperature + np.random.normal(0, 2.0, n_samples)

# Lower temperature leads to higher barrel sales (negative correlation) + noise
# ★Important: wind_speed is not involved here at all!
barrel_sales = -0.8 * temperature + np.random.normal(0, 2.0, n_samples)

df = pd.DataFrame({
    'Temperature': temperature,
    'Wind_Speed': wind_speed,
    'Barrel_Sales': barrel_sales
})

2. Falling into the Trap: Simple Correlation Check

First, acting as if I know nothing, I look at the relationship between "wind speed" and "sales" using simple linear regression (OLS).

import statsmodels.api as sm

X = sm.add_constant(df['Wind_Speed'])
y = df['Barrel_Sales']
model = sm.OLS(y, X).fit()
print(model.summary())

Simple correlation coefficient between wind speed and sales: 0.686

What is the result? The correlation coefficient is positive, and the p-value is 0.000 (P < 0.05). Statistically, the result indicates a "significant correlation." Looking only at the data, "if the wind blows, the barrel maker prospers" appears to be correct. This is the trap of spurious correlation.

3. The Big Reveal: Controlling for Confounding Factors

Next, I perform multiple regression analysis by adding the true mastermind, "Temperature," to the model.

X = sm.add_constant(df[['Wind_Speed', 'Temperature']])
model = sm.OLS(y, X).fit()

Then, a surprising change occurs.

--- Model 1: Simple Regression (Sales ~ Wind Speed) ---
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const         -0.0285      0.021     -1.350      0.177      -0.070       0.013
Wind_Speed     0.7271      0.024     29.779      0.000       0.679       0.775
==============================================================================

--- Model 2: Multiple Regression (Sales ~ Wind Speed + Temperature) ---
===============================================================================
                  coef    std err          t      P>|t|      [0.025      0.975]
-------------------------------------------------------------------------------
const           0.0031      0.016      0.197      0.844      -0.028       0.034
Wind_Speed     -0.0102      0.031     -0.326      0.745      -0.071       0.051
Temperature    -0.7962      0.028    -28.890      0.000      -0.850      -0.742
===============================================================================

Temperature coefficient: Shows a significant value.

Wind_Speed coefficient: Becomes nearly 0, and the p-value becomes large (not significant).

When we control for "temperature," it is statistically proven that "wind" has no influence whatsoever on "sales."

4. Structure Learning: Reverse-Engineering the "Blueprint" from Data

Now for the main point. Can we make the computer discover the structure of the data (causal graph) itself without humans having to hypothesize that "temperature is suspicious"?

I will use the Python library pgmpy to perform structural learning on a Bayesian network.

from pgmpy.estimators import HillClimbSearch, BicScore
import networkx as nx
import matplotlib.pyplot as plt

# Preprocessing such as discretization (see repository for details)
# ...

# Execute structural learning
hc = HillClimbSearch(df_discrete)
best_model = hc.estimate(scoring_method=BicScore(df_discrete))

# Graph drawing
G = nx.DiGraph()
G.add_edges_from(best_model.edges())
# ... (drawing code)

Results

The output graph is shown below.

An amazing structure of "conditional independence" has emerged! (*Theoretically, it should be a 'Fork' showing a common cause, but due to Markov equivalence, it appears as a 'Chain' in this instance. However, the conclusion that 'there is no direct causation' remains unchanged.)

Temperature \to Wind_Speed

Barrel_Sales \to Temperature

There is no line between Wind_Speed and Barrel_Sales

The algorithm calculated the probabilistic independence hidden within the data and discovered that "once temperature is known, wind and sales are unrelated (conditionally independent)."

Discussion and Conclusion

Through this validation, we confirmed the following:

Correlation \neq Causation:

Even without a direct relationship, strong correlation appears if there is a common parent.

Importance of Structure:

By using Bayesian networks, one can estimate the "generative process" behind the data to some extent. While I used a simple discretization approach for this demo, the following methods are effective for exploratory data analysis in the real world (with continuous data):

LiNGAM:

A method that treats continuous values as they are and uses non-Gaussianity to uniquely identify the direction of causality.

Note that depending on the results, you may occasionally see a physically impossible reversed arrow such as 'Sales \to Temperature.' This occurs because statistically, the 'correlation between A and B' is symmetric, and in some cases, the direction of the arrow (causal direction) cannot be identified from data alone (Markov equivalence). This is why incorporating human domain knowledge (blacklist) into the algorithm is indispensable in practice.

Injecting Domain Knowledge:

Provide physical constraints like "sales do not change the weather" as a blacklist. In data analysis, I felt that being conscious of the structure (causality) behind the numbers (correlation) is the first step toward correct decision-making.

Humans (Data Scientists) are the Judges of Causality

It is foolish to blindly ask AI for answers, but having AI find "seeds of correlation (hypotheses)" that humans would never think of, and then having humans dissect them with the knife of causal inference—perhaps this symbiotic relationship is the future of data science.

GitHubで編集を提案

Discussion