Search: GO!

The Scientific Method of Investigation

The goal of this section is to help health consumers (at one time or another that is all of us) better understand biomedical research, and to sort out scientific fact from fiction. It will help in making sense of new research as distilled in the popular press or in advertising claims for new products, as well as presented in scientific articles.

It can be confusing to read reports that tend to be mutually contradictory. Yesterday’s dictum seems to be today’s derision. As you read this section, you may find yourself formulating questions to ask when reviewing research or advertising claims.

The scientific method of investigation is the process of arriving at theories to help us explain phenomena. Value of the theorem produced in the process is dependent upon the quality of the steps taken in producing that theorem.

The first step is making careful observations.
The second is asking a question based upon those observations.
The third is formulating a hypothesis to attempt to answer the question.
Fourth is experimentation to test that hypothesis.
Fifth is to propound a theorem that interprets the tested hypothesis in light of the results of the experiments.

The theorem is then used as a test that is continually validated. If over time, the theory is not found to be useful, it may need to be revised or discarded.

Step 1: Observations

Two illustrations may elucidate the importance of these steps. A few years ago, a significant number of elderly people in the Georgian Republic were reported to have lived over 110 years; longevity was attributed to daily consumption of yogurt. This resulted in quite a bit of interest by American consumers. It turned out that claims of longevity were exaggerated or could not be verified. This is an example of incautious observations. Another example illustrates misinformation that might have repercussions that are more serious.

Steps 2 and 3: Asking questions and formulating hypotheses

Studies showed consumption of polyunsaturated vegetable oils results in lowering serum cholesterol levels (1). A plethora of advertising appeared touting the benefits of vegetable oils for preventing heart disease. Consumption of vegetable oils, margarine, and shortenings dramatically increased (2). Lost among the ballyhoo were studies addressing where the cholesterol went, or what side effects might occur in increased consumption of debulked, defibered vegetable oils. Research on polyunsaturated fatty acids, a frequent component of vegetable oils, has investigated free radical generation of oxidized low density lipoproteins and atherosclerosis (3), and promotion of various cancers (4). The FDA stopped manufacturers from making some of the more outlandish claims, but the myths persist.

Step 4: Testing the hypothesis

Even if careful observations are made, good questions asked, and sound hypotheses proposed it might still be possible to fall into the experimental pit. Laboratory or basic research is performed in animals or in vitro (literally, within glass), but not in humans. These experiments can often be tightly controlled; however, it is not always possible to extrapolate to human beings.

In any study, investigative bias can easily creep in and invalidate the best-intentioned research. Neutrality of investigators is paramount. For example, it might be difficult to evaluate the effects of tobacco if a tobacco company is funding the investigation. A more subtle influence to assess is the impact on research if the continued funding of research is dependent upon generation of statistically relevant results. Facing loss of funding for a project that the principal investigator may have dedicated years to, may be a difficult milieu in which to function without bias.

Step 5: Data interpretation

So far, we have touched upon observations, questions, hypotheses, and experimentation. Next is analysis of experimental data. It is this statistical analysis that may help to determine the validity of our hypothesis, and may lead us to a theorem.

After data is collected, statistical analysis can begin. Statistical analysis may be able to tell us if results are significant or merely the product of chance. Experimental design determines the type of analysis employed. Degree of certainty may be expressed in several ways; conventionally, a relationship is statistically significant if the probability of it occurring by chance is less than 5 out of 100. For more details on statistical analysis, please see the section on biostatistics.

Not all experiments will lead us to conclusions. It may simply be that we discover inadequacy of our current methods of evaluating the problem. Ideally, the whole population should be studied, but this is usually not possible. Instead, representative samples must be taken. Sampling can be the downfall of experiments.

It is difficult to avoid bias and maintain randomness while keeping study populations small enough to handle. A prominent example of this problem may be seen in a recent study (1995) reported in a well-respected journal () and widely quoted in the popular press. The study attempted to evaluate effects of fruits, vegetables and olive oil on breast cancer. It was conducted on 820 Greek women with breast cancer and 1,548 Greek women without breast cancer. The group who consumed the most vegetables had about 1/2 the risk, the group who ate the most fruit had about 1/3 less risk, and the group who consumed olive oil more than once per day had about 1/4 the risk of breast cancer. Unfortunately, scientists found several flaws with this study.

The study population who consumed olive oil was only 99 women, mainly postmenopausal who likely had consumed olive oil (in unknown quantities) for much longer than the year covered in the study. Information about diet came from one retrospective questionnaire in which the women were asked to estimate their nutritional intake for the past year and no questions were asked about the amount of olive oil consumed. Estimates of dietary intake are often erroneous and very difficult to perform for a year in the past. Further, psychological investment for particular eating regimens was not controlled. Breast cancer rates in the Mediterranean countries are quite a bit lower than in the USA; and, less animal fats, more fruits, vegetables, and monounsaturated olive oil are consumed. So, it is tempting when looking for a dietary connection to cancer, to look at these dietary parameters. Alas, linkage to any of these factors is not yet clear.

Journals

It is also important to consider the journal in which a study has been published. The best journals are peer reviewed. This is a process where experts in the field critically review the paper before it is accepted for publication. It is common for a paper to be revised in a way that takes into account the critique presented by the reviewers. In addition, a fair number of papers are rejected. One type of publication that may have excellent studies but are not peer reviewed, are symposium volumes. These publications publish work that has been presented at a conference, usually just as they were submitted. Also, there is grey literature, literature that does not show up on databases, or are not published in regular accessible journals. These would include conference proceedings that contain abstracts of presentations, and unpublished theses. It is important to be careful when evaluating these articles.

Types of studies

The types of studies can be ranked in terms of quality of evidence based upon research design and the validity of results (5). Generally, at the top is research involving at least one properly randomized, controlled clinical trial. Next are well-designed controlled studies that were not randomized. Then comes observational studies (both cohort and case-control epidemiological studies), preferably from more than one research group. The penultimate type is uncontrolled multiple time series with or without experimental intervention. At the bottom are descriptive studies (case reports or studies, and case series) and expert opinion.

Conducting experiments where people are subjected to some intervention and carefully observed in clinical trials are not always possible. If it were possible always to perform clinical trials where all variables are carefully controlled, experimental results might be much easier to interpret. Due to ethical concerns and the long time to get results, it is often not possible.

Humans are heir to the placebo effect, reporting errors, memory lapses, and the desire to please among other things. The placebo effect has been claimed to account for up to 30% false positives in some studies. To help eliminate some of these effects, the best studies are performed double blind where neither the subject nor the experimenter knows who is receiving a real treatment or who receives a placebo. More often, the test is performed single blind, where only the subjects do not know who received the placebo.

Researchers often rely upon epidemiological studies. These may be prospective or retrospective (looking forward or backward respectively). In prospective (also called longitudinal) studies, cohorts are used. A cohort is a defined population group followed prospectively. Participants are carefully selected and extensively questioned about such things as life styles. Over the course of the study, individuals are monitored for disease, and attempts are made to identify factors associated with the malady. In retrospective studies, or case-control studies, people afflicted with a particular disease (cases) are carefully matched (by gender, age, etc.) to healthy controls. Researchers then attempt to identify factors that determined who got sick. These studies take less time and money to perform, but can often be difficult to interpret.

Much of the scientific literature until 1980, were case studies, comparatively few cohort studies and nonrandomized trials, and no randomized trials. In the 1980s, randomized controlled trials (RCTs) became the preferred design. Because the RCT and especially the systematic reviews or overviews of multiple RCTs are more likely to accurately tell us if a treatment does more harm than good, they have become the gold standards. Non-experimental approaches to questions on therapy can too easily lead to false positives.

Reviews often but not always use meta-analysis, a statistical technique to summarize the results, or pool the data of multiple RCTs. Meta-analysis and confidence intervals is discussed in more detail in the section on biostatistics in Psychoneuroimmunology for the Health Sciences.

However, RCTs and meta-analyses are not always the best approach to get the best answers to clinical questions. Basic research in immunology, genetics, or endocrinology may be the best place to look for answers to questions that have not made it to clinical research.

For assessing the utility of a diagnostic test, a cross-sectional study selecting those people with a malady for whom the diagnostic test might have some relevance can be the most appropriate: a randomized trial would seem to be antithetical. For information on prognosis, follow-up studies conducted at an early stage in a disease might be the most appropriate.

Then, too, sometimes clinicians cannot wait for results of RCTs if none have been conducted that relate to the patient’s problem. In that case, the clinician needs to make do with the next best evidence for efficacy.

I do not want to leave the impression that RCTs are the only way to produce trustworthy results. A comparison between meta-analyses of RCTs and meta-analyses of observational studies were conducted on 5 topics (). They found that the results of well-designed observational studies did not systematically overestimate the magnitude of the effects of treatment when compared to RCTs. They found that the results were remarkably similar for each of the 5 clinical topics. Indeed, it is possible to use sophisticated methods such as the restricted cohort design to strengthen observational studies ().

And as we shall see later, RCTs can also produce inconsistent or contradictory results. Pooled analyses of conflicting results of RCTs (meta-analysis) can also be misleading as they can blur critical distinctions among individual RCTs (). Six methodological sources of variations occurring because of design complexity and 3 because of interpretations were identified by one expert reviewer of over 200 RCTs (). Design issues included eligibility criteria, selection of study groups, baseline differences in the available study population, variability in indication for the therapies, protocol requirements, and management of intermediate outcomes. Interpretation problems included regulatory effects of treatments, vulnerability of double blinding, and the occurrence of unexpected outcomes.

Abduction, Deduction, and Induction

There are three types of reasoning that are used in scientific studies. These are abductive, deductive, and inductive reasoning. Abduction and induction are used in creating hypotheses.

Abduction makes inferences from a single case to reach a hypothesis about a specific case.
Induction brings together a number of cases, which taken together suggest the plausibility of a general rule (hypothesis): what is likely to be true. The strength of the hypothesis lies in the quality and quantity of observations that laid the foundation for the hypothesis.
Deduction begins with a general rule (a hypothesis produced by inductive reasoning), looks at a specific case that is believed to be true, and then forms a conclusion (not a hypothesis) based upon the rules of logic: what must be true if the hypothesis was true.

The general rule is sometimes called the major premise and the specific case is sometimes called the minor premise. If both the general rule and the specific case are true, the result will be true. Of course, if either of these is not true, the result will be suspect. Indeed, the utility of the conclusion rests upon the strength of the major premise. In fact, this is where abduction is commonly used. The result of abduction simply states that the putative “conclusion” is a hypothesis, one of various plausible explanations of the original observations. Actually, in abductive reasoning the fact that begins our search for the action that caused it is in itself a hypothesis (a hypothesis leading to another hypothesis). Here is a comparison of the three types of reasoning:

Abductive Reasoning:
Example 1:
Initial observation (A given rule): No traffic passes by here when the streets are flooded.:
Second relevant observation: There is no traffic passing by here.:
Hypothesis: The streets are flooded. :
Strength of hypothesis: This is one logical possibility: this may be the case. A careful scientist would say, “There may be other explanations, but based upon my experience, flooded streets would explain the lack of traffic”.

Example 2:
Initial observation (A given rule): Almost no traffic passes by here when the streets flood.
Second relevant observation: There is no traffic passing by here.
Hypothesis: Perhaps the streets are flooded.
Strength of hypothesis: This is a more careful appraisal, stating that this may be the case, based upon an initial observation with uncertainties. Much of scientific investigation begins with such a measure of uncertainty.

Inductive Reasoning:
Initial observation (A given case): The streets are flooded.
Second relevant observation: There is no traffic passing by here.
Hypothesis: When the streets flood, no traffic passes by here.
Strength of hypothesis: The strength of this hypothesis rests largely upon the number of times that the given case is associated with the second relevant observation. This is the process for qualitative research. And the hypothesized conclusion is the point of departure for beginning exploration of the hypothesis through deductive reasoning.

Deductive Reasoning:
Hypothesis (A given rule). When the streets flood, no traffic passes by here.
First observation: The streets are flooded.
Conclusion (Result): No traffic is passing by here.
Strength of conclusion: If the hypothesis is correct, the result will always occur. If we see a different result than what has been predicted, we need to explore the problem. The difficulty may be with the initial hypothesis, or our means of testing it or analyzing our results. If the hypothesis always, in at least a sufficient number of times for comfort, leads to the same conclusion, then it takes on the force of a theory.

One Final Example

One last example is illustrative of paradoxical results from apparently well-designed studies. You may wish to get copies of these three studies and try out your scientific sleuthing abilities. Not long ago, three very-well-designed studies on effects of hormone replacement therapy and cancer risk were published (, ). Unfortunately, the results seem to be contradictory. The first study published in 1995, looked at almost 70,000 postmenopausal nurses, approximately 60% of whom took estrogen alone (estrogen replacement therapy, ERT) or estrogen plus progestin (hormone replacement therapy, HRT). Women on HRT for more than 5 years had a 40% increase in breast cancer risk over the control group of women who did not take either ERT or HRT. Women on ERT had slightly less increase over control.

The second study, also published in 1995, included 500 women with breast cancer and 500 without. They were followed for eight years and no increased risk of breast cancer was found in postmenopausal women taking ERT or HRT. Indeed, a slight decrease in risk was found in women taking HRT for greater than eight years. The answers to the apparent contradictory results may lie in the characteristics of the sample groups.

The third study published in JAMA in 2002 was designed to check the impact of HRT on coronary heart disease (CHD) (non-fatal myocardial infarction and CHD death) and invasive breast cancer as the primary adverse outcome. It was a multi-centered, randomized controlled primary prevention trial (begun between 1993 and 1998), designed to run 8.5 years and test 16,608 postmenopausal women with an intact uterus (50 to 79 years of age) (). On May 31, 2002 after a mean 5.2 years of follow up, the committee monitoring the data and safety stopped the trial because the risk for invasive breast cancer exceeded the stopping boundary. They also found significantly increased risks of CHD, stroke, and pulmonary embolisms ().

Copies of these articles are readily available. It can be stimulating to critically read studies such as these. It can be as entertaining as unraveling a mystery story, but more rewarding. The answers to these real life mysteries can add to your health and well-being.

Summary

So, to summarize what we have mentioned to this point, when evaluating claims, ask at least the following questions.

Are claims based upon the scientific method of investigation?
Is this a single study or are there related corroborating studies?
Was the study conducted in vitro, in animals, or in humans?
Was this a randomized clinical trial, a retrospective or a prospective study, or some less reliable type?
Was evidence published in a refereed journal (i.e., one where specialists in the field critically reviewed the study prior to publication)?
How large was the study population?
Was the study population representative of the population as a whole or are the results likely to be valid only for people with specific characteristics?
Was the study conducted for enough time for relevant effects to be observed?
Can you identify any sources that might have biased the results?
Is there some other reasonable explanation that might explain the results, or were other hypotheses possible but not tested?
If this is an advertising claim, what research is referenced to support these claims?

References (Not PubMed indexed)

Wissler RW, Vesselinovitch D. Am J Cardiol Jul 20;52(2):2A-7A, 1983.
Rizek RL, et al. J Amer Oil Chem Soc 51:244, 1974.
Jialal I, Devaraj S. Clinical Chemistry 42(4):498-506, 1996.
National Research Council: Diet Nutrition and Cancer, Washington, National Academy Press, 1982.
Preventive Services Task Force. Guide to clinical preventive services: report of the U.S. Preventive Services Task Force. 2nd edition. Baltimore: Williams and Wilkins, 1996.


Our Workshops Workshop Overview Our Staff Our Location About Heather Hills Our Endorsements Free Stress Test Endorsements: "I will always appreciate the support Elizabeth provided during my time of distress." Read More		Heather Hills Evidence Base: Health Consumer PNI Evidence Articles Health Care Professional PNI Evidence Articles Psychoneuroimmunology for the Health Sciences