In the realm of modern empirical economics, few tools have gained as much prominence as the "synthetic control method." Often heralded for its ability to create a credible counterfactual by meticulously replicating the pretreatment trends of a specific jurisdiction, the method has become a staple for researchers evaluating the real-world impact of public policy. However, a new working paper released by the National Bureau of Economic Research (NBER)—Working Paper 35931, published in October 2026—casts a cautionary light on the reliability of these models, suggesting that the "degrees of freedom" available to researchers can significantly sway final results.
The study, which subjects synthetic control methods to an intense stress test, specifically examines the controversial economic impact of California’s AB 1228, the legislation that mandated a $20 minimum wage for fast-food workers. By constructing 1,440 "plausibly defensible" specifications, the authors of the paper demonstrate that while the majority of outcomes point toward negative employment effects, the variance between those estimates is vast. This range suggests that the final output is highly sensitive to the subjective choices made by researchers during the modeling process.
The Problem of Researcher Discretion
At the heart of the NBER study is the issue of "researcher degrees of freedom." When an economist builds a synthetic control model, they must make numerous technical choices: which variables to include as predictors, which "donor pool" of other states to use for comparison, and how to weight different time periods before the policy intervention. While each individual decision may be justified as "defensible" in a vacuum, the collective impact of these choices can be transformative.
The authors of Working Paper 35931 sought to quantify this impact by intentionally varying these parameters across 1,440 unique model specifications. The goal was not to determine the absolute truth of California’s employment figures, but rather to reveal how much "noise" is inherent in the methodology itself. The findings are striking: while approximately 96 percent of the generated models produced a negative employment effect, the magnitude of those effects spanned a wide spectrum.
This variability underscores a growing concern in social science research regarding the reproducibility and robustness of policy evaluation. If a single dataset and a single policy intervention can produce such a wide range of conclusions depending on the technical configuration of the model, it challenges the perception of synthetic control methods as an objective, plug-and-play solution for causal inference.
Comparing Synthetic Control to Traditional Methods
One of the most revealing aspects of the NBER study is how it positions synthetic control methods against more traditional, time-honored approaches. Specifically, the authors compared their 1,440 synthetic control simulations against the results produced by "difference-in-differences" (DiD) models.
Difference-in-differences is a classic econometric approach that compares the change in outcomes over time between a treatment group and a control group. While it is often criticized for being simpler and potentially less nuanced than the sophisticated synthetic control approach, the study found that the DiD estimates tended to cluster near the center of the distribution of the synthetic control estimates.
This finding offers a provocative takeaway: the added complexity and computational intensity of synthetic control methods do not necessarily provide a more accurate or "true" result. Instead, they may simply introduce more opportunities for researcher bias—whether conscious or subconscious—to influence the outcome. By moving away from the simplicity of DiD, researchers may be trading transparency for a false sense of precision.
The Context of California’s AB 1228
The choice of California’s $20 fast-food minimum wage as the subject of this methodological critique is particularly apt, given the intense policy debate surrounding the legislation. AB 1228 has been a flashpoint for economists and policymakers alike, serving as a high-profile test case for the effects of aggressive wage floors on labor market outcomes.
Because of the high stakes, the literature on the law has been contentious, with different research groups arriving at starkly different conclusions. This NBER study serves as a meta-analysis of sorts, suggesting that the conflicting reports on the law’s success or failure may be as much a product of methodological variance as they are a reflection of the actual labor market reality.
Indeed, the study notes that the labor market effects of California’s AB 1228 have been the subject of significant scrutiny in other recent NBER papers. The working paper specifically references NBER Working Paper 34033—both its initial and revised versions—and NBER Working Paper 35171. Each of these documents contributes to a growing body of literature that, when taken together, highlights the difficulty of achieving consensus when evaluating complex, real-world economic shifts.
Broader Implications for Economic Research
The release of Working Paper 35931 comes at a time when the economics profession is undergoing a broader reckoning regarding the credibility of empirical research. The "credibility revolution" in economics, which emphasized the importance of rigorous causal identification, has led to the adoption of increasingly complex statistical techniques. However, as this study suggests, the pendulum may have swung toward a level of complexity that obscures rather than illuminates.
For policymakers who rely on these studies to craft legislation, the lesson is one of healthy skepticism. When an economic report claims to provide a definitive answer on the employment effects of a policy, it is essential to ask not just what the result is, but how the model was built and how many other "plausibly defensible" models were discarded to reach that conclusion.
The authors do not suggest that synthetic control methods are useless, but they do argue that they should be treated with more caution. Transparency—specifically the practice of sharing the full range of potential specifications—may be the only way to restore confidence in these types of evaluations. By acknowledging that a single "point estimate" is often an illusion, researchers can provide a more honest and nuanced picture of the likely range of outcomes for any given policy.
The Path Forward
As the academic community continues to grapple with these methodological challenges, the NBER remains a focal point for the discussion. The organization regularly hosts high-level lectures and research presentations aimed at improving the standards of empirical analysis. Recent events, such as the 18th Annual Feldstein Lecture by Mark Duggan on Social Security reform and fiscal deficits, as well as the 2026 Methods Lecture delivered by Melissa Dell of Harvard University and Ashesh Rambachan of MIT, highlight the profession’s ongoing commitment to refining its tools.
In the case of Dell and Rambachan, their focus on estimation and inference in the age of AI-generated data demonstrates that the questions of validity and methodology are evolving rapidly. As the tools available to economists grow more powerful—and more capable of generating massive amounts of data and simulations—the need for rigorous, transparent, and humble interpretation of those results becomes even more critical.
Ultimately, the study on California’s minimum wage serves as a necessary reality check. It reminds the field that data is not objective, and that the "science" of economics is fundamentally shaped by the choices of those who practice it. Whether it is a minimum wage law or any other major policy shift, the "truth" is rarely found in a single, perfectly tuned model, but rather in the honest admission of what we do not know and the range of possibilities that remain on the table. As researchers move forward, the focus will likely shift from finding the "best" model to acknowledging the inherent limitations of the ones we currently use.
