Incorrect Statistical Interpretation in RDD Analysis (T_4_Regression_Discontinuity_on_Progresa_Data.ipynb)

[myeongseok-gwon]

Summary

The notebook contains a significant error in the statistical interpretation of regression discontinuity results, claiming "significant positive effects" when the confidence intervals suggest no statistically significant effects.

Problem Description

1. Incomplete Reporting of Results

In cell 9, the code only extracts coefficients and standard errors:

result.append([rdd_result.coef.iloc[0].values[0], rdd_result.se.iloc[2].values[0]])

Issue: The rdrobust function provides confidence intervals, which are crucial for determining statistical significance. Only reporting standard errors without confidence intervals makes it difficult to assess the statistical significance of the results.

2. Incorrect Statistical Interpretation

The notebook states:

"While the effects in the first year after the intervention are negative, we observe significant positive effects in the second year after an individual or household was accepted in the Progresa program."

Issue: When examining the confidence intervals (robust, bias-corrected, or conventional), all results show confidence intervals that include zero (Below is the result with Robust confidence interval):

Outcome	LATE	CI Lower	CI Upper
Food T_1	-22.16	-48.65	58.98
Non-Food T_1	-9.14	-44.85	41.13
Food T_2	54.96	-39.69	148.94
Non-Food T_2	43.81	-29.69	97.13

Since all confidence intervals include zero, none of the effects are statistically significant, contradicting the claim of "significant positive effects."

Suggested Fixes

1. Include Confidence Intervals in Results

Modify the result extraction to include confidence intervals:

result = []
for outcome in ["conspcfood_t1", "conspcnonfood_t1", "conspcfood_t2", "conspcnonfood_t2"]:
    rdd_result = rdrobust(x=df.pov_index, y=df[outcome], rho=1, masspoints="off")
    ci_bc_lower = rdd_result.ci.loc["Robust", "CI Lower"] # It can be one of following: ["Conventional","Bias-Corrected","Robust"]
    ci_bc_upper = rdd_result.ci.loc["Robust", "CI Upper"] # It can be one of following: ["Conventional","Bias-Corrected","Robust"]
    result.append([rdd_result.coef.iloc[0].values[0], ci_bc_lower, ci_bc_upper])
res_dataframe = pd.DataFrame(result, columns=["LATE", "CI Lower", "CI Upper"],
                             index=["Food T_1", "Non-Food T_1", "Food T_2", "Non-Food T_2"])

2. Correct the Statistical Interpretation

Replace the current interpretation with:

"The RDD analysis shows that none of the estimated effects are statistically significant at conventional levels. While the point estimates for the second year (T_2) are positive, the confidence intervals include zero, indicating that we cannot conclude there are statistically significant effects of the Progresa program on consumption in either the first or second year after intervention."

Impact

This error could mislead readers about the effectiveness of the Progresa program and demonstrates a fundamental misunderstanding of statistical significance testing in causal inference.

Files Affected

• T/T_4_Regression_Discontinuity_on_Progresa_Data.ipynb (Cell 9 and subsequent interpretation)

Priority

High - This is a methodological error that affects the core conclusions of the analysis.