History of Evidence Based Medicine - Association For Medical Ethics

AME Star Rating System for Validity of Data on New Devices and Procedures

Evidence Based Medicine (EBM) is a universally accepted system to objectively rate the validity of any medical study in terms of quality. For example, two different studies may have opposite conclusions about whether a new device is good for patients or not. Yet, one study may just be a doctor documenting he did 10 of the procedures (or “cases”) and said the results are good and patients happy. The other could be where patients were compared to a group that had a standard procedure, or maybe nothing at all, and then evaluated by a third independent party for comparing the groups. The second study’s conclusion may be infinitely more valid and therefore predict what one’s real chances of doing well with a new procedure are. Patients, and even many doctors, may think both are equal, but they are not. One is an objective valid study for predicting results, and the other may be one step above hearsay.

To make matters worse, the studies may have authors that receive hundreds of thousands, or even millions, of dollars from the company that makes the device, yet people only know of a vague “disclosure,” as its called, that just says the name of a company the author “works with.” This may create bias whether intended or not.

To address the first problem, namely the quality and predictive value of a study, we use Evidence Based Medicine’s five levels of rating for categorizing a study. They are as noted below. We have reformatted the “levels” into a patient friendlier format where 5 stars is the best type of paper, a level 1, and the worst is a 1 star rating, which is a level V. We will only talk in star ratings however to make it simple. Reversing only the numbers but not the meaning seems to be more easily understandable for each paper as well as the overall final rating since people are familiar with hotel and restaurant rating star rating. We do this for EVERY study published in the English literature in peer reviewed publications as found on Pub Med, the NIH site (National Institute for Health , of the Dept. of Health and Human Services) for citing medical publications. We then rate the studies AS A GROUP under our AME system of 5 stars where 5 is the best and most predictable data, and 1 is the worst. For example, if a dozen studies of a new procedure are half rated as the highest level of Evidence Based Medicine, our 5 star rating, and the rest are the next best levels, our 4 star rating, then this would be an OVERALL AME 5 star rating which has a excellent level of predictive value for success if one chooses to get the procedure. If, on the other hand, the studies were all the lowest levels of star rating such as 1 and 2 stars, then this would get an overall rating of 1 star which has a very poor level of prediction for success. In that case, one should give very serious pause as to having such a procedure done.

OVERALL AME STAR RATING for Prediction of Success for a Surgical Procedure or Device

Predictive value for success for the device or surgical procedure	Star Rating
Excellent	5 stars ***** > 25% of all studies are rated at 5 stars > 25% of all studies are rated at 2 stars
Good	4 stars **** > 10% of all studies are rated at 5 stars > 20% are rated at 2 stars
Fair	3 stars *** > 10% of all studies are rated at 5 or 4 stars > 20% are rated at 3 stars or better
Poor	2 stars *** > 50% of all studies are rated at 3 stars or better
Very Poor	1 star * < 50% of all studies are rated at 3 stars or better

Note: 5, 4, and 3 star only applicable if minimum of 10 studies published

INDIVIDUAL AME STAR RATING for Individual Studies

***** Five Stars Rating
Randomized, controlled clinical trials
This is the most objective and valid study results. This study has posted the highest level of quality for evidence based medicine.

**** Four Stars Rating
Prospective non-randomized, comparative studies
This is the second most objective and valid study results.

*** Three Stars Rating
Retrospective comparative or case controlled studies
This third level is not a reliable standard for determining the validity of a study

** Two Stars Rating
Case Series
This two level rating means that the study was (need to describe what this is)

*One Star Rating
Expert Opinion
This means that the author’s the study relied on their own patient experiences as to what happened. AME considers this hearsay and the least reliable indicator of evidence based medicine.

Detailed Summary of INDIVIDUAL AME Star Rating for Individual Studies:

Types of Studies
	Therapeutic Studies - Investigating the Results of Treatment	Prognostic Studies - Investigating the Outcome of Disease	Diagnostic Studies - Investigating a Diagnostic Test	Economic and Decision Analyses - Developing an Economic or Decision Model

5 Star	1. Randomized controlled trial a. Significant difference b. No significant difference but narrow confidence intervals 2. Systematic review2 of 5 star randomized controlled trials (studies were homogeneous)	1. Prospective study1 2. Systematic review2 of 5 star studies	1. Testing of previously developed diagnostic criteria in series of consecutive patients (with universally applied reference "gold" standard) 2. Systematic review2 of 5 star studies	1. Clinically sensible costs and alternatives; values obtained from many studies; multiway sensitivity analyses 2. Systematic review2 of 5 star studies
4 Star	1. Prospective cohort study 3 2. Poor-quality randomized controlled trial (e.g., <80% follow-up) 3. Systematic review2 a. 4 star studies b. nonhomogeneous 5 star studies	1. Retrospective study4 2. Study of untreated controls from a previous randomized controlled trial 3. Systematic review2 of 4 star studies	1. Development of diagnostic criteria on basis of consecutive patients (with universally applied reference "gold" standard) 2. Systematic review2 of 4 star studies	1. Clinically sensible costs and alternatives; values obtained from limited studies; multiway sensitivity analyses 2. Systematic review2 of 4 star studies
3 Star	1. Case-control study5 2. Retrospective cohort study4 3. Systematic review2 of 3 star studies		1. Study of nonconsecutive patients (no consistently applied reference "gold" standard) 2. Systematic review2 of 3 star studies	1. Limited alternatives and costs; poor estimates 2. Systematic review2 of 3 star studies
2 Star	Case series (no, or historical, control group)	Case series	1. Case-control study 2. Poor reference standard	No sensitivity analyses
1 Star	Expert opinion	Expert opinion	Expert opinion	Expert opinion

1.- All patients were enrolled at the same point in their disease course (inception cohort) with 80% follow-up of enrolled patients.
2.- A study of results from two or more previous studies.
3.- Patients were compared with a control group of patients treated at the same time and institution.
4.- The study was initiated after treatment was performed.
5.- Patients with a particular outcome ("cases" with, for example, a failed total arthroplasty) were compared with those who did not have the outcome ("controls" with, for example, a total hip arthroplasty that did not fail).

Introducing Levels of Evidence to The Journal

Note: Levels converted to “Star” rating as indicated in text.

James G. Wright, MD, MPH, FRCSC, Marc F. Swiontkowski, MD and James D. Heckman, MD

Click here for the Journal Review on Lumbar Disc Replacement.

Click here for the Journal Review on X-Stop.