Articles on Law, Science, and Engineering

Book Review: Judging Science, by Kenneth R. Foster and Peter W. Huber, Cambridge, MIT Press, 1997.

By Edward P. Richards

Introduction

This is a flawed, but interesting and valuable book.  The previous book edited by these co-authors, "Phantom Risk - Phantom Justice?" (reviewed at 13 EMB #3 p427, 1994), was an excellent collection of essays on scientific-legal controversies by experts in various scientific disciplines.  The volume under review is structured as a discussion of the Dauber case, the 1993 U.S. Supreme Court decisions establishing the standards for admitting scientific evidence into the courtroom.  (The case concerned whether there was adequate scientific evidence to conclude that Bendectin caused birth defects when taken during pregnancy.)

The authors tell us much that is valuable, but without an overall organization scheme that makes the material hang together, and with no clear conclusions at the end.  In their defense, this is an unsettled and contradictory area.  It may be impossible to impose order on it without misrepresenting its complexity.  Despite certain misgivings about the organization and some of the examples there were chosen, I highly recommend this book to anyone interested in broadening their knowledge of the use of scientific evidence in the courtroom.

Can Juries Know Enough?

The role of lawyers in a jury trial is to make the facts fit into a simple story.  If this story deals with matters that are not in the common knowledge of the jury, then the lawyer is not allowed to tell the story him/herself.  The rules of evidence require that the lawyer have an expert in the subject matter tell the story.  In an auto accident case, the story of the patient's medical care and the long term prognosis must be told by a physician.  In the Daubert litigation, the epidemiologic studies on Bendectin had to be explained and presented to the jury by experts.

There are two problems with experts in trial.  The first is structural: it is very difficult, if not impossible, to present a fair picture of a complex scientific problem to a jury generally selected to have little education or knowledge about the area in general.  The best the jury can do is judge the creditability of the experts through their ability to present clear, simple explanations, their previous training and experience, and, most importantly, their grasp of the theatrical demands of the courtroom.  The jury cannot evaluate the underlying truth of the testimony because they do not have knowledge of the subject matter and they are not allowed to consult with either outside experts or reference materials.  This is an intractable problem in a jury system.  It was address in Daubert only as it related to the judge's duty to keep bad testimony from the jury.  The court, and thus the authors, gave little attention to the question of whether juries can make proper decisions about scientific questions, even when the expert testimony has been properly filtered.

Adverse Selection For Experts

The second problem is the selection bias for experts.  The authors do a good job in dealing with the philosophical part of the problem - experts must be prepared to say that things are black and white when they are only gray.  In the Bendectin case, the defendant was faced with proving a negative in epidemiology.  No matter how many studies you do, you never eliminate the possibility that Bendectin causes birth defects.  You can only improve the estimate of how rare those defects are.  For a substance like Bendectin, which is accused of causing a birth defects that are already present in significant numbers, any realistic study can only detect changes in risk ratios on the order of 2.  If Bendectin only doubles the risk of birth defects, the effect will be impossible to detect.  Conversely, the effect will be impossible to disprove.

Plaintiffs must, and do, find experts who are willing to say that Bendectin causes birth defects.  The authors explain, with many examples drawn from the testimony in the case, how the plaintiff's experts exploit the limitations in the epidemiologic method to do this. They also show how the defense experts are unable to say what the defense demands them to say: The evidence shows that Bendectin never causes birth defects.  The defense experts can only say that there is no evidence to show that it does cause birth defects.

The Daubert case, with its impeccably credentialed defense experts who are constrained by the truth, is a poor vehicle for seeing both sides of the expert witness problem.  The authors inadvertently portray this as a problem of disingenuous plaintiff's attorneys.  From this author's experience, both sides play this game. It is less that one side is disadvantaged by being honest as it is that science it mocked by both sides, leaving the jury to choose between two misstatements of the known science applicable to the case. Defense attorneys are as willing to select experts who will tell the story their clients need as are plaintiffs.

The Daubert's court's solution is to have the judge evaluate the credibility of the scientific evidence and exclude testimony and other evidence that may be misleading or wrong.  The authors try to help judges understand how to do this independent appraisal of the evidence. The authors present much useful information on how to do this.  Their presentation would have been strengthened if they had worked out an explicit "by-the-numbers" guide to evaluating specific testimony.  Such a flow chart would highlight the complexity of the task and illustrate the pitfalls in having judges with no scientific expertise evaluate scientific evidence.

Culture Clashes

The authors discuss the culture clashes between scientists and lawyers, but do not really convey the chasm that separates them.  Lawyers are expected to do the best for their client, within the bounds of the law.  While they cannot lie, they have no duty to present a complete or fair picture of the facts or science in a case. Scientists have an obligation to be critical, to constantly reevaluate their hypotheses, and to be open and honest about limitations in their theories.  The authors do a good job describing this clash. 

What they do not convey in this book is how differently lawyers and legal academics view research and publication.  The authors present an excellent discussion of peer review.  They explain why individual published articles, especially ones that are early in the development of a new theory, can be dead wrong while the body of published scientific research, taken together, is very accurate. They then discuss how difficult this is to deal with in litigation, where litigants rely on one or a few studies, taken out of the larger context of research, to support their case.  They approach this as if research and publication were a foreign area to lawyers, one that they do not really understand.

The truth is more bizarre.  Law has its own scholarly publication industry.  Law school professors grind out an unending stream of articles, and do practicing lawyers and judges.  The difference is that the journals are run and edited by students at law schools, with little faculty involvement.  Few have peer review or blind review, and the students feel free to "improve" the articles with their editorial changes.  Who you are and what school you are at has as much to do with whether the top journals publish your article as does its content.  Since the student editors have no expertise in the substantive content of the articles, they are little better than jurors in separating glib charlatanism from brilliant insight.

Lawyers see the world from their own frame of reference.  Since their scholarly publications are essentially political and status/fashion driven, they assume that scientific publication is the same.  To accept the validity of scientific publishing implicitly questions their own model.  This makes the gulf between the professions much more difficult to bridge than even the authors of this book convey.

Caught in Your Own Trap

Knowing what is worth reading is a key skill for scientists and persons evaluating scientific research.  If you work in a popular field, there is much more published than one individual can read.  One of the ways to sort the wheat from the chaff is the howler test: if you come across an egregious error, you stop reading.  Such an error in a book on how to figure out what is good science is especially troublesome.  There is a howler in this book involving how to make policy decisions on screening for human immunodeficiency virus (HIV).  This is perhaps forgivable because it ironically illustrates a major point of the book - that it is science in the aggregate that is accurate, any given paper can be egregiously wrong.

The authors choose to use the test for HIV as an example of how the sensitivity and specificity of a test only gives you useful information if you also consider the prevalence of the condition being detected in the population being screened.  They based their analysis on the data in a 1987 article that argued against general screening for HIV.  Unfortunately, the authors appear to be unaware that, at least until recently, the science behind HIV screening policy has been as political as the science behind the health effects of silicone or whether Bendectin causes birth defects.  They also appear to be unaware that this area has undergone massive change since 1987.

The article they relied on was written to argue against screening for HIV and the data was selected to support this hypothesis. The authors uncritically used this data to illustrate how a test that is both very sensitive and specific, i.e., it is very good at detecting HIV and it does not have many false positives, can still have an unacceptably high incidence of false positives if the condition is very rare in the population being screened.  The key flaw in the data is the estimate of the prevalence of HIV in the general population of the United States.  They used an estimate of one person in 3000, or roughly 83,000 cases in a population of 250,000,000.  Given the estimates of the sensitivity and specificity of the screening test for HIV, the test would have about 5 times as many false positives as true positives.

The problem is two fold.  The estimates used by the authors for sensitivity and specificity for the testing process as a whole are lower than is currently accepted.  More seriously, the estimated prevalence of HIV is dramatically higher.  The latest prevalence data for AIDS, which is a subset of all the persons infected with HIV, is about 225,000 cases.  It is conservatively estimated that there are at least 2.5 persons infected with HIV for every reported case of AIDS, for an estimated total of at least 562,500 cases of HIV, or an incidence of one case per 444 persons, approximately seven times the authors' estimate.  (This has probably changed little since 1987.) The revised numbers dramatically shift the balance from a substantial excess of false positive tests to a clear excess of true positive tests.

The authors also fail to consider the context of their example, another key question in the vetting of scientific conclusions.  When you are dealing with a fatal communicable disease, one that is now very responsive to treatment, you will tolerate a lot more false positive tests than you will for a less serious condition.  Adding in the risk of transmission further increases the tolerance for false positive tests.

By failing to follow their own recommendations - look at the publication record as a whole, especially the later work, make sure to get the proper context for the data, and be aware of controversial theories - the authors underscore the complexity of making good scientific policy decisions.  Unluckily, their error involved a critical public health policy issue, one that most of their readers will not have the background to detect.

Organizational Issues

"Judging Science" is a combination of the author's own work and selections from previously published articles by a diverse group of authors.  Perhaps 50% of the text is made of these excerpts.  Each is a page and a half or less in length and is placed in a text box. While the excerpts are individually interesting, the effect cries out for hypertext: if this were a WWW page and each excerpt a hotlink, the flow would be much more effective.

Conclusions

"Judging Science" is a useful addition to the literature of science in the courtroom.  While the book has some organization problems, and some unreliable examples, these do not detract significantly from the book's value.  This is an evolving area and it is unreasonable to expect books on the subject to be definitive or to have solutions for the problems they identify.  This author hopes that we will see further works on science in the courtroom from the team of Foster and Huber.