Jim Manzi's book Uncontrolled is a bold attempt to apply the scientific experimental model to public policy. To complete his application Mr. Manzi should first provide a more complete description of the scientific method. He fails to define the concept of hypothesis, but proceeds to intersperse the use of hypothesis and theory. The scientific method requires the developing of a hypothesis or several hypotheses and then testing those hypotheses with data. Most often in testing a hypothesis the experimenter describes two hypotheses, an affirmative hypothesis and a null hypothesis. The data collected is compared for the experimental group relative to a control group. Either random selected or otherwise carefully selected cohorts are selected to minimize the potential for bias. The results may demonstrate a difference, determined by a statistically significance test.
When determining whether the data sets are significantly different requires that the experimenter to determine, preferably beforehand, what level of significance is acceptable. A 99% significance level may be preferable to a 95% or 90% since there is a greater confidence in the statement that the outcomes are differences. In some situations, and often in pharmaceutical clinical trials, a 95% confidence levels is considered acceptable to proceed to make a professional judgement.
In Mr. Manzi's discussion he fails to explain the difference between correlation and causality. Typically a single experiment only demonstrates a correlation. Additional data obtained from separate experiments are needed to develop a causal relationship. Mr. Manzi references John Mill and Sir Austin Hill on how to determine likely causality. In fact Hill outlines nine criteria that could be employed to determine whether statistical associations are indeed causal associations. Failing to discuss these important aspects weakens Manzi's argument. Regression analysis is a process of showing multiple relationships between possible independent variables and a single dependent variable. Again no causal relationship can be concluded based on the mathematical calculations, as implied by Manzi. Perhaps Manzi is too quick to jump to the same conclusions that imprudent readers jump to in reviewing well constructed scientific papers.
An example of the conclusion jumping occurs on p. 88 where Manzi discusses studies conducted in Ghana. In reviewing the hypotheses put forward as a result of the study, he concludes they are all false. The data does not appear to refute the null hypothesis and therefore no conclusions should be drawn. Researchers would have to either collect more data to demonstrate a statistical difference or reformulate the affirmative hypothesis. With the possibility of many confounding variables (what Manzi calls high causal density) the possibility of the outcome resulting in affirmation of the affirmative hypothesis becomes less likely.
Again on p.159 Manzi's statement "In science, theory precedes experiment" is false. In science initial data encourages development of hypotheses which then need to be tested. Science is a continuous iterative process of attempting to prove or improve on a hypothesis. When there is a large body of information that can be culled together to develop an over arching concept this information is pulled together to form a theory.
There is a limited discussion of the importance of blinding to limit or attempt to control bias. In clinical trials single blinding occurs when the patient does not know whether she is being treated or has received a control or placebo. In double blind, neither the patient nor the direct experimenter knows which cohorts are being treated. In triple blind experiments not only do the patient , experimenter, but also the person(s) doing the statistics not know who has been treated and who is a control.
When Manzi gets to Chapter 13 he wanders off into another world. It is not clear to me whether he chose to enter a political arena to sell the book or whether the gobbledegook that he writes is written to try to tie his arguments into the latter chapters. Whichever, the chapter seems totally irrelevant and leads to a weakness of his credibility in his final chapter 15 recommending "Sustainable Innovation."
Chapter 15 provides a reasoned set of recommendations to improve public policy. Manzi claims his argument is based on his previous chapters. In fact some of his suggestions seem to be disconnected from previous chapters. He states there is a need for decentralized innovation, however his examples argue for innovation with no justification for decentralization. In fact, he argues for strong oversight by a federal authority similar to the FDA for studying public policy issues such a medical delivery, education and criminal justice. Innovation requires some form of supervision of the studies to coordinate and evaluate the study results. All human studies require an Institutional Review Board so that we do not repeat unethical or illegal tragedies such as the famous Tuskegee syphillis study.
Some of the proposals he suggested have merit. He advocates increasing the number of students pursuing science and math. He recommends automatically issuing H-1B visa to international students graduating with appropriate math and science degrees.
However, others have problems. For instance, his suggestion to privatize schools are structured to benefit the stockholders. How is that an improvement over union contracts benefiting the teachers? Establishing a financial mutually funded institutions where the parents (or students) make the investment and reap the benefits. We need a common knowledge bases for upper level learning (high school or college) or else colleges will spend even more time than they do now in remediation education. Funding at a limited number of research institution will only exacerbate an already existing problem. Federal funding that funnels money to a limited number of "successful " institutions restricting innovative thinking. Many good ideas have sprung from smaller institutions and projects started on a shoe string. We need to be able to foster those innovative ideas with a spreading of federal funding, not shrinking the institutions who are considered to be "eligible."
Overall this book would have greater value if the political rhetoric (libertarian statements) would have been left out. A clearer statement of the scientific method and how it could be applied to public policy could turn this book into a useful and more widely read reference source.
— William S. Carter, Ph. D. CIH
Emeritus Professor Environmental Safety and HealthThe University of Findlay
Findlay, OH 45840
