How a researcher performs the study determines if the data or statistics are reliable and reproducible by other researchers and doctors in practice. The design of the study may have biases, which are ways that researchers unintentionally introduce their influence or opinions on the results of the study. For example, if you are studying how Vitamin B affects depression in a group of patients, your selection process of that group of patients based on their mental health introduces your biases. If you stand to make a lot of money on your new Vitamin B patented pill – or you just want Vitamin B to work to help everyone – you may choose less depressed patients to take the pill so the results look better. How researchers eliminate these unintentional biases are in the methods they choose for conducting a study.

Evidence-based medicine uses a universally accepted measurement similar to a grading system used in school to determine how much a patient or physician should rely on the results for the study. In particular, to determine if one treatment is better than another, Level I is the highest receiving an “A” and Level V is the lowest receiving an “F.” Now the Level V – which is merely an opinion of a doctor about something – can be valuable for other reasons such as stimulating ideas, just not for deciding about one therapy over another. The Levels do not determine if the treatment being studied – for example chemotherapy – is safe, citing toxic side effects. Instead, it determines the level of accuracy and reliability of results, regardless of whether they show the treatment is good or bad.

So let’s say a Level I study says that the medical treatment – for example “Chemotherapy B2” – compared to another treatment is not as safe, citing toxic side effects; and the researchers of a Level III study say “Chemotherapy B2” is safe. The researchers of the Level I study received an A for this type of therapy decision whereas the researchers of the Level III study received a C. The decision for which therapy a patient receives should be based on the Level I study. Below is an example of design and methods of Level I through V studies. One will be the most objective, valid study for determining one treatment over another and the last described here is just an opinion. Although opinions and patient results of one treatment without comparison to anything else may have value for other reasons, it is not for a “therapeutic decision” of one treatment over another. Here is the question for these studies:

"Does drinking pomegranate juice make your back pain get better?"

Level I

Randomized, controlled clinical trials. Researchers would use a computer program to randomly assign patients with back pain into two groups of 20. The first group (placebo group known as the control) would drink water (that only tasted like pomegranate juice but was not) for 10 days. The second group would drink real pomegranate juice for 10 days. None of the patients would know if they were drinking the real pomegranate juice or not. (This is called a blinded study). Then a researcher who does not know which person drank what (which now makes this a “double blinded” study) would interview the patients to determine if their back pain was reduced, worsened or stayed the same. After this was all done, the information about which patients drank what would then be revealed. One could then see if those who drank the real pomegranate juice were better or not than the water-drinking group.

Level II

Non-randomized, prospective comparative study. A researcher looks at the medical records of 40 patients to select 20 patients for the pomegranate drinking group and 20 patients for the control group who will drink water. Here the researcher may introduce his own bias whether he intends to or not. If he believes pomegranate is a safe, effective treatment for back pain, he may put the healthier patients with less back pain in the pomegranate group, whether he intends to or not. (This particular bias is called “selection bias.” See section on types of research bias.)

Level III

Retrospective (already occurred) comparative study or case controlled study (each “experimental” patient is matched to a patient that never had the treatment). This is not a reliable standard for determining one treatment over another. This method can be helpful to report the number of complications in certain treatments. Researchers do a retrospective study, for example, reviewing the records of 20 patients who reported they have been drinking pomegranate juice in the past, and then the records of 20 patients who have not reported drinking pomegranate juice. Then the researchers review the patients’ medical records to determine if the back pain was reported better, worse or stayed the same. Here again, the selection process may introduce bias, intentionally or not. In this case it may not only be selection bias, but could involve “recall bias,” or “expectation bias” or “attention bias.” (See Bias in Research section)

Level IV

Level IV consists of case studies. Case studies do not determine success or failure of a treatment compared to other treatments or no treatment at all. Researchers and a physician conduct a case study on 20 patients who drink pomegranate juice for 10 days and then report the results. In this case there is no control group or comparison to patients who are not drinking pomegranate juice. It does not take into consideration that back pain could get better in 10 days if the patient takes nothing at all for the pain. These studies are easier and cheaper. They can be of value to determine better methods of doing a particular treatment, or what the complications of a certain treatment are, but NOT for determining if one treatment is better than another.

Level V

Level V relies upon expert opinion—one physician's opinion regarding whether pomegranate juice helps reduce back pain. No original research is conducted. Instead, just a written opinion or editorial that may talk about other research and give opinions, but no clinical study is conducted. AME considers this to be one step above hearsay for determining one treatment over another, though it may be valuable for stimulating discussion and ideas on a particular topic.

Level of Evidence