The JFK assassination plus FBI compositional analysis of ...

The JFK assassination plus FBI compositional analysis of ...

Data, integrity, and the scientific method: The case of bullet lead as forensic evidence and the JFK assassination Cliff Spiegelman TAMU Karen Kafadar, Bill Tobin, Dennis James, Simon Sheather, Stu Wexler, and Max Roundhill 1 From FBI Affidavit in Kulbicki case--2006 27. The FBI Laboratory established the comparative bullet lead program in response to requests for examination of the bullet fragments

recovered from the body of President John F. Kennedy and the ammunition recovered from the firearm believed to have been used in his assassination. Establishment of the examination for routine casework followed from those initial efforts to determine if an association could be made between fragmented bullets and functional ammunition. 2 November 22, 1963

A truly traumatic event event occurred on November 22, 1963. The nation and beyond has been wondering for decades was Lee Oswald the person solely responsible for the assassination. We will look closely at the testimony and science underlying the testimony of Dr. Guinn to the HSCA. 3

Testimony of Dr. Vincent Guinn Excerpts--each bullet is unique Page 494 So we have usually found [ in the general bullet population] much higher antimony levels than in the WCC Mannlicher-Carcano bullets. The other unusual feature of the WCC Mannlicher-Carcano is that there seems to be no uniformity within a production lot. That is, even when we would take a box of cartridges all from a given production lot, take 1 cartridge out and then another and then another and then another, all out of the same box-boxes of 20, these were-and analyze them, they all in general look different and widely different, particularly

in their antimony content. This is not true of most bullet leads that we have ever looked at before, which are very uniform. In general, if you take most boxes 4 What is the null hypothesis? If you want to convince people that there was a conspiracy then H0: There was no conspiracy If you want to convince people there was no conspiracy then H0: There was a conspiracy Thus folks who say on TV you have to believe there was/was not a conspiracy unless you can prove otherwise usally have their null hypothesis wrong. (Aside: This is a major issue in the FBIs application of

CBLA.) 5 6 So are bullets essentially unique? Does not look like that does it? 7 FBI measurements

The next few slides show antimony measurements from 4 boxes of of the 4 largest domestic bullet manufacturers Does they look qualitatively different from the TAMU measurements of the WCC bullets? 8 9 10

11 12 Testimony of Dr. Vincent Guinn-- excepts Continued Two and only two bullets

From Page 503 Mr. WOLF. Is it your opinion then that these all are fragments from WCC Mannlicher-Carcano bullets? Dr. GUINN. I think that is their most likely origin, yes. Mr. WOLF. Looking at these results, can you determine how many bullets these evidence specimens came from? From Page 504 Dr. GUINN. Yes, sir, I can. Mr. WOLF. What is the number of bullets, in your opinion? Dr. GUINN. These numbers correspond to two bullets . Two of the samples have indistinguishable compositions, indicating that they came from the same bullet, and the other three particles are evidently samples from another bullet. Mr. WOLF. So it is your opinion that the evidence specimens represent only evidence of two bullets, is that correct?

Dr. GUINN. Yes, sir, there is no evidence for three bullets, four bullets, or anything more than two, but there is clear evidence that there are two. 13 Description of our experiment We measured 30 WCC bullets from 3 separate boxes (2 lots) Lot 6000 and 6003P bullets we measure 3 times each--the FBI stated number of fragments

Lot 6003 bullets were measured more than 10 times each with some bullets measured in many over 15 times including 3 separate radial locations 3 times each. The purpose of the multiple measurements is to get a sense of compositional bullet homogeneity 4 elements measured antimony, silver, arsenic, and copper (all the relevant elements that NAA can reasonably handle). 14 Guinns revised measurements 15 16

Pr(T|E )= P r (E |T )P r (T ) What is the likelihood of a 3rd bullet within the assassination fragments? The ratio on the left hand side of the equation is a ratio of

the probabilities for two bullets as the source for the five JFK fragments in question versus more than two bullets being the source for the five JFK fragments. The first ratio on the right hand side is the ratio of probabilities for the evidence given two bullets versus having more than two bullets. This ratio allows to decide whether, given the evidence, two bullets or more than two bullets is more likely. 17 Pr(E|T).8503 The calculated ratio is: =

18 Our finding based upon our experience and NRC report JFK fragments need to be remeasured following NRC bullet committee guidelines The necessary science needs to be established for counting bullets contributing to the assassination fragments This includes: characterizing within bullet homogeneity/heterogeneity between bullet homogeneity/heterogeneity other types of bullet science not considered by the NRC

committee -- metallurgical, 19 Some Aspects of Work on the NRC Bullet Committee (This part is Joint with Karen Kafadar) 20 Committee members A special thanks to Mike Cohen of the NRC for help editing the statistics sections, Changfu Cxaio for computer support, and Bill Tobin for background and sharing some of his case information.

21 Background for CABL CABL uses seven selected elementsarsenic (As), antimony (Sb), tin (Sn), copper (Cu), bismuth (Bi), silver (Ag), and cadmium (Cd)in the bullet lead alloy of both the crime-scene and suspects bullets. Measured by ICP-OES Various methods akin to Z or t tests done element by element

comparing crime scene and suspects bullets. Sometimes clustering (called chaining by the FBI) is used. If two bullets are alike for all 7 elements a match is declared. The chance of a false match depends upon how close a truly non-matching bullet is Error ranges from nearly zero to over 40% if true non match is within 2 SD. 2 = 2 (measurement error)+ 2 (between bullet)+ 2 (within bullet fragments) 22 Some Committee Recommendations

Recommendation: The FBI should continue to measure the seven elements As, Sb, Sn, Cu, Bi, Ag, and Cd through ICP-OES as stated in the current analytical protocol. Also, the FBI should evaluate the potential gain from the use of high-performance ICP-OES because improvement in analytical precision may provide better discrimination. Recommendation: The FBIs documented analytical protocol should be applied to all samples and should be followed by all examiners for every

M case. Recommendation: The FBIs statistical protocol should be properly documented and followed by all examiners in every case. Finding: Variations among and within lead bullet manufacturers makes any modeling of the general manufacturing process unreliable and potentially misleading in CABL comparisons. Recommendation: Interpretation and testimony of examiners should be M assessed regularly. Recommendation: Laboratory reports and expert witnesses on direct examination should acknowledge that different CIVLs can sometimes coincidentally be analytically indistinguishable. 23

Requested 71k case data was never made available to the committee Committee requested full data log of ~71,000 bullet lead measurements. Two files purportedly fulfilling this request were delivered on CD-ROM at the final committee meeting. The two files together contain 56,260 measurements. These files were incomplete without a record of bullets in a case (if any) that were found by the FBI crime lab to match. An partial analysis of these data later this month. 24 Why case data would be useful

Compare findings to testimony Compare frequency of matches among examiners Compare relative standard errors among tech (may not exist) Compare matching algorithm among examiners

Find trends in measurements and bullet compositions over time See if practice matches the descriptions given to panel 25 Additional reasons that the data is useful

Does the data support inter-examiner reliability and consistency? Does the data support identification of patterns or inconsistencies in testimonies? Did the FBI use different protocols: number of ``slivers'' per bullet? How many matches are attributed to the different match criteria that were used? What were the different match criteria used? Number of measurements per bullet? ("choose 3 of the most consistent matches"?) 26

Scientific Method The scientific method is supported by common sense Is the nondisclosure of case data to the NRC panel or failing to make all case data generally available to scientific bodies sensible? In at least one field of study the NRC has made recommendations about data disclosure. The results can be found in: Sharing Publication-Related Data and Materials: Responsibilities of Authorship in the Life Sciences http://books.nap.edu/catalog/10613.html

27 Why the scientific method matters In Daubert v. Merrell Dow Pharmaceuticals, Inc., the Supreme Court interpreted an earlier version of Rule 702 to require that scientific evidence meet a reliability test. The Court wrote that in order to qualify as scientific knowledge, an inference or assertion must be derived by the scientific method. Proposed testimony must be supported by appropriate validationi.e., good grounds, based on what is known. In short, the requirement that an experts testimony pertain to scientific knowledge establishes a standard of evidentiary reliability.. 28 The scientific method depends upon common sense

(From Scientific Method In Practice, by Gauch, Cambridge University Press, 2003) 29 Sharing Publication-Related Data and Materials: Responsibilities of Authorship in the Life Sciences Community standards for sharing publication-related data and materials should flow from the general principle that the publication of scientific information is intended to move science forward. More specifically, the act of publishing is a quid pro quo in which authors receive credit and acknowledgment in exchange for disclosure of their scientific findings. An authors obligation is not only to release data and materials to

enable others to verify or replicate published findings (as journals already implicitly or explicitly require) but also to provide them in a form on which other scientists can build with further research. All members of the scientific communitywhether working in academia, government, or a commercial enterprisehave equal responsibility for upholding community standards as participants in the publication system, and all should be equally able to derive benefits from it. 30 Some of what we have found with limited data sets provided at the last NRC committee meeting

No case data were provided indicating whether a match was found despite numerous requests from all levels of the NRC committee The data that were provided at the end of the last NRC committee meeting contained chemical measurements for 1079 of the roughly 2500 cases that the FBI has handled since 1980. (It has been doing CABL for a much longer period.) It was incomplete with over 85% of cases missing at least all K bullets or all Q bullets. Some cases, as well, appear to be missing all K bullets and some Q bullets. Out of these 1079 cases only 138 cases had both K and Q bullets There are many extreme outliers with standard errors > 10 12 , (> 1.2% of the records had such outliers). No data given to the NRC committee before final deliberations had such outliers. So we never asked how they were handled.

No indication from the FBI, to the committee on a uniform policy for handling outliers. Hopefully they did not use them. Other findings follow 31 Observations

No cases identified whether there was a match (or a method(s) for how a match was obtained) There are massive inconsistencies among FBI data sets and FBI descriptions of the data sets There are differences in the chemical compositions patterns of bullets according to the caliber of the bullets- relevant to how the FBI calculated the probability of false matches. Only a few cases, 138, in the data provided have both K and Q bullets K bullets are those found at the crime scene Q bullets are suspect bullets The more K and Q bullets the more matches ( This is a point that is not brought out by the FBI). 32

How the number of matches varies with the number of K & Q bullets (from lead2.xls) As expected the number of 2 SD overlap matches increases as the number of K*Q (the Bonferroni count) increases. A similar increase is seen in the percentage of cases that have matches when plotted against K*Q. 33 U.S. Department of Justice Federal Bureau of Investigation

October 1999 Volume 1 Number 3 Statistics and Trace Evidence: The Tyranny of Numbers Max M. Houck Supervisory Physical Scientist Trace Evidence Unit Federal Bureau of Investigation Washington, DC 34

Excerpts from The Tyranny of Numbers How Do We Know All Ravens Are Black? The tyranny of numbers is a consequence of an over reliance on deduction and mathematics, and these ultimately limit a discipline by requiring it to fit into a preordained model. Equating quantification with science to justify and validate its "science-ness" indicates that a "faulty notion of science, or no notion at all," is at the heart of the tyranny (Mayr 1982) It isn't that they can't see the solution. It is that they can't see the

problem." G. K. Chesterton 35 Excerpts Cont How does one determine the frequency of a fiber type? You could look for a specific fiber on unrelated garments (Houck and Siegel 1999) to gain an idea of its commonness or rarity. It could also be possible to determine the frequency of fiber types within a textile population, either by cataloging the fibers (Home and Dudley 1980) or the textile population (Biermann and Grieve 1996a, 1996b, 1998) or by sampling selected environments to determine chance matches (Palmer and Chinherende 1996). Small ad hoc studies could also be performed to answer specific questions of frequency (Deedrick 1998). The difficulty with all

of these approaches is that they are not universal in their application, if they can be applied to casework at all. Questions of sampling, randomness, and relationships impinge on this process and complicate matters. Before trace analysts can comfortably approach using statistics to present their results to a judge or jury, it is critical that these fundamental questions be answered and those answers be accepted by the trace evidence community. 36 How the FBI crime lab takes the committees recommendations Forensic Significance of Bullet Lead by Koons and Buscaglia, JFS 2005. 37

So is the 1837 dataset is a reasonable representation of the FBI case load? 38 Conclusions Forensic science laboratories need to make their data, materials, and methods available to responsible organizations and individuals upon request The FBI crime lab needs to embrace all aspects of the scientific method

including: Recommendations from major scientific bodies such as the NRC Mathematical aspects of the scientific method such as sampling, data analysis, inference, Creating an atmosphere of openness 39 Prologue

Dwight, I hear what you're saying and understand your concern. I would add, however, that this is a very tight examination now. There will be no such thing as an inconclusive, provided we could do the examination. Either they'll match or they won't using our revised statistical protocol. The exception may be with fragments where we can at least rule out a common source of the lead, but the new technique will not allow us to use a single measurement to claim that we have a match. The only reason that we had issues in the past was because we allowed for a subjective variable ... namely "experience "....to somehow figure into whether a call was made or not. Now it is all done behind the scenes with our statistical program. Looking at all the cases that we reported since 1996, only 1.4% of the cases resulted in a different report being issued. To the best of our

ability, we have identified why those 7 cases have a different result today and it is mainly due to the use of 'chaining 'I... some of it used quite liberally. I think some of your concern may be due to discussions that were held last year when we were looking at the use of the "equivalence test". If we had continued with that method of assessing the data, we would have been making a lot more false exclusions. As you will recall, we switched horses at that point and did the false positive probability study that will allow us to use a very traditional statistical technique ... namely the student t test. I can't imagine anyone really questioning the use of this statistical technique in the scientific community. 40

Recently Viewed Presentations

  • Patterns of Evolution - H.S.H.P. Biology

    Patterns of Evolution - H.S.H.P. Biology

    Patterns of evolution. A. Mass Extinctions. B. Adaptive Radiation. C. Convergent Evolution. D. Coevolution. E. Gradualism. F. Punctuated Equilibrium. These are
  • Cellular Transport - Metcalfe County Schools

    Cellular Transport - Metcalfe County Schools

    Primary Active Transport. ATP is hydrolyzed and drives the movement of ions against the concentration gradient. Sodium potassium pump is an example of 1 AT. Because the ions move against the concentration gradient. (Na leaves cell, although more Na outside...
  • Lecture 1 - Digilentinc

    Lecture 1 - Digilentinc

    Additional nodal analysis examples ... review Identify mesh loops The currents around these loops are the mesh currents Use Ohm's Law to write KVL around each loop in terms of the mesh currents Solve these equations to determine the mesh...
  • Ceng 334 - Operating Systems

    Ceng 334 - Operating Systems

    A fixed portion of memory is used for mapping regardless of the number of processes or virtual pages This approach is used by IBM's AS/400 and RISC System/6000 computers Translation Lookaside Buffer Translation Lookaside Buffer (Cont.) Translation lookaside buffer (TLB)...
  • Sámánság neurofenomenologiája

    Sámánság neurofenomenologiája

    Módosult tudatállapotok - Általában A hős útjának templátja Joseph Campbell alkotta meg a "Hős ezer arcának" kompozit portréját meghatározván, hogy a különböző mitológiákban mi a közös motívum a harcos, az uralkodó, a gyógyító, a szent és a félisten "életpályájában" öt...
  • Aucun titre de diapositive - LeWebPédagogique

    Aucun titre de diapositive - LeWebPédagogique

    Title: Aucun titre de diapositive Author: Gilles Gallois Last modified by: gallois gilles Created Date: 6/8/2001 12:26:56 PM Document presentation format
  • Technician License Class Technician Class Chapter 3 Electricity,

    Technician License Class Technician Class Chapter 3 Electricity,

    An analogy is a check valve in a water pipe. Components and Units Diodes, Transistors, and Integrated Circuits. Diodes. Components and Units Diodes, Transistors, and Integrated Circuits. Diodes. High voltage and/or high current diodes designed for use in power supplies...
  • Thinking, reasoning and working mathematically

    Thinking, reasoning and working mathematically

    Thinking, reasoning and working mathematically Merrilyn Goos The University of Queensland Why is mathematics important? Mathematics is used in daily living, in civic life, and at work (National Statement) Mathematics helps students develop attributes of a lifelong learner (Qld Years...