Five Must-Follow Recommendations for Systematic Reviewers

Systematic reviews and meta-analyses are optimal ways to amalgamate data from many studies and translate them into usable knowledge. And the number of published systematic reviews has increased by an order of magnitude over the past decade. However, systematic reviews are vitally dependent on the quality of work done by the reviewers. So, although being asked to serve as a reviewer in a systematic review can be an exciting time for a researcher, the decision to participate should not be a casual one.

I spoke with two of my colleagues from the University of Alberta Emergency Medicine Research Group (EMERG). Dr Uirá (Mia) Wisnesky is a Postdoctoral Fellow and Scott Kirkland is the Project Research Coordinator. Both are experts in systematic review and meta-analysis and have extensive experience as reviewers. They shared their best recommendations for becoming a skilled reviewer.

1. Know Your Role

Both experts admitted that starting a systematic review can be a "daunting task." They suggest that you first ask what your role will be. Are you a reviewer or part of the analysis team? Will you be doing the data extraction? The bias assessment? This is especially important for new reviewers to ensure that you focus your efforts.

Both of my colleagues recommend that reviewers are honest with themselves about time commitment, as reviewing papers often takes much longer than you anticipate. "Some papers you can spend three to four hours," Dr Wisnesky admits. While more recent papers that follow a standardized reporting structure may take only a few minutes to review, older papers may be much more tedious. Both experts admit that there is a high variability in the time it will take to review papers, and that it is difficult to estimate at the beginning of the study how long it will take. Asking about deadlines is also important. "It is better to say no then to delay the team," according to Dr Wisnesky.

"Quality assessment is the worst in terms of time," Scott explains, "some tools are very complex and require good knowledge of the tools." For instance, while they both like the Cochrane Risk of Bias Tool, they feel that many aspects of the tool can be quite subjective. "It is rare that we agree on everything," Dr Wisnesky explains, and this leads to many meetings to sort out a consensus. These meetings can be a very time-consuming part of the review process.

2. Know the Protocol

The experts recommend that you read the protocol in detail before starting the review process. Both feel that understanding the research question is the most important first step. "I print off the research question and post it in front of me, so I don't get lost," says Dr Wisnesky. She explains that it is easy for reviewers to become lost in the details of the paper, and focussing on the research question can ensure that vital information is kept while excess details are not.

It is also important that all team members are using the same study protocol. This includes the librarian, the analysis team, and the reviewers. Scott feels strongly that knowing the primary outcome is critical in the study screening process: "you will inevitably run into studies that are in the grey area," and referencing back to the study objectives will make the process much easier. Scott likes to print off the exclusion and inclusion criteria for reference during the initial screening process. Many studies have extensive lists of inclusion and exclusion criteria and having them in sight during the screening process can allow you to quickly exclude papers when they do not meet the protocol criteria.

3. Keep Organized

Systematic reviews and meta-analyses can explode in size rapidly. Often hundreds or thousands of paper will appear in the initial literature review. Keeping organized initially is mandatory. Scott recommends asking both the team and yourself: "what are you going to do to keep organized?"

The experts recommend using an online tool for study screening and review. And, because the most current version of the original manuscript can change, all manuscripts should be stored in a single common area. "Every team has its own way to organize," Scott explains, "so try to adapt to the team, so that everyone is on the same page."

4. Standardize the Tools

Making sure that there are standardized tools for data collection is critical according to my colleagues. They also feel that it is optimal for the reviewers to be involved in the design of the data collection tool when possible. This ensures that the tool captures all required information but also remains implementable.

Dr Wisnesky recounts that data collection tools should be explicit about how numbers are recorded, particularly for meta-analyses. For example, numerator and denominator should be stored in different cells. Also, the distinction between standard deviation and standard error should be detailed. "Familiarize yourself with the tool," Scott suggests.

Piloting the data extraction tools is also mandatory. This allows the team to assess the wording of the tool, to ensure that the correct data is extracted, and to confirm that the data is recorded in a standardized fashion. Dr Wisnesky recommends trialing the tool on three to five studies. If reviewers are new to the tool, or if the tool itself is new, the reviewers may need to meet to discuss the tool further to ensure that all reviewers are using the tool in the same manner.

When I spoke to STAT59's own Dr Manuela Verde, she echoed the difficulty of creating an effective data extraction tool. She discussed the importance of ensuring that the statistician who will be analyzing the data also be involved in the tool design process to ensure that the data collected by the reviewers is sufficient and not redundant. She also prefers that the data extraction tool includes a legend with a line-by-line item description of each field. "I like to print out this legend to have beside me during the review process," she says.

5. Work With The Content Expert

"Ask questions early," Scott advises, "it saves you a lot of headaches." He explains that working closely with the study content expert is a critical step to success. Often the first question is "what did you find so far?" Speaking at the very beginning with the content expert is a fast and easy way to put the importance of the study and the expected outcome quickly into focus.

Scott also mentioned that he likes to do his own fast search of the literature to see what type of papers have been written and to better understand the topic.

The protocol should also specify explicitly what to do in the case of a disagreement. Often this will involve clarification with the content expert. It may also involve a third-party mediator - either early or late in the process. In addition, if the protocol is based on "independent" reviews, the reviewers should avoid speaking to one another for clarification of values: speaking to one another to clarify concepts is reasonable.

What Does it Take?

What does it take to be a great reviewer? It is clear that both experts have exceptional dedication to the systematic review process, and great commitment to ensuring that the review process is precise, accurate, and consistent. I was also struck by the level of patience they exhibited in their devotion to ensure that review was done right regardless of time costs. Becoming an expert reviewer does not seem to be a process that can be rushed, but rather a skill developed with the combination of intelligence, wisdom, and experience. Whether you are an experienced reviewer, or just starting your first systematic review, following the five recommendations from these experts is a sure-fire way to improve.

If you would like a quick summary of these five tips to become an awesome reviewer, you can download the infographic.