Systematic Review

Citation management is essential in a systematic review. There are a number of citation management tools available that are compatible with many of the databases and resources you'll use.  Visit the citation management guide to get started. It only takes a few minutes to set up an account with such tools as Zotero, Mendeley and Endnote Online. For more advanced features, you may consider purchasing the desktop version of Endnote. 

Endnote Resources

Below you will find selected resources to assist you with various functions on Endnote.

Zotero Resources

Downloading large results from EBSCO In EBSCO databases, find a link called "Email a link to download exported results." This will allow you to download the results in RIS file format and import that RIS file into Zotero.

Authorship Order

• CRediT taxonomy for authorship - "a high-level taxonomy, including 14 roles, that can be used to represent the roles typically played by contributors to scientific scholarly output. The roles describe each contributor’s specific contribution to the scholarly output."
• International Committee of Medical Journal Editors (ICMJE) - defining the role of authors and contributors

Deduplication

• Systematic Review Accelerator (SRA) offers users a reliable program to remove duplicate records with greater sensitivity and specificity than EndNote. SRA aimed to reduce the amount of time taken to produce systematic reviews by maximizing the efficiency of the various review stages such as optimizing search strategies and screening, finding full-text articles, and removing duplicate citations. This application can save researchers and information specialists time and avoid research waste. Visit the help guide to get you started or read the review article.

• NOTE: Read further about the types of duplicate records in a systematic review context

Screening

• Systematic Review Accelerator (SRA) - Screenatron - allows to blind the screening for reviewers to screen the studies independently. Access the tutorial to learn more on how to use it.
• Excel is the most basic tool for the management of the screening and data extraction stages of the systematic review process. Customized workbooks and spreadsheets can be designed for the review process, and lists of references can be exported from citation managers into Excel format for screening.
  • Basic examples include Database Search Log and Review Matrix.
  • A more advanced way to using Excel for this purpose is the PIECES approach, designed by a librarian at Texas A&M.
• Evidence synthesis tools for Campbell authors - a list of tools for review, screening, data extraction and analysis. 
• Catchii - a systematic literature review screening application. Read more about its features in the article by Halman & Oshlack (2023). 
• Distiller is designed for screening and data extraction phases of systematic reviews. Free for student as a single user for one project for 1 semester with no collaboration with other users.
• JabRef is a free reference manager that helps you to discover, collect, organize and cite your scholarly literature and research in an efficient way. Check this Libguide and a video on how to use JabRef.
• Rayyan is a free online tool that can be used for independent screening and coding of studies in a systematic review. Rayyan uses tagging and filtering to code and organize references. Citations can be uploaded from RefMan, EndNote, BibTex, CSV or zip file formats.
• RevMan is used by Cochrane reviewers to manage the data extraction and analysis process. It is a free tool that is geared toward Cochrane-like reviews.
• RobotAnalyst is a free online tool "for searching and screening reference collections obtained from literature database queries. It combines search engine functionality with machine learning and text mining technology, including topic modelling and relevancy feedback-based text classification models, to minimise the human workload involved in identifying relevant references."
• GradePro is used to create summary of findings tables. Free for single-users and non-profits. 
• Search result formatting - a method for formatting a list of search results in a Word document in which each numbered record contains an openURL link resolver link.
• Sysrev - a collaborative web-based application that uses machine learning to optimize systematic literature reviews. It helps facilitate screening, collaborative extraction of data from academic articles and abstracts, PDF documents.The free platform supports open access/public projects only. There is a fee to apply privacy settings.

• CADIMA - Try this open access tool for all stages of your systematic review project! 

Kohl, C., McIntosh, E. J., Unger, S., Haddaway, N. R., Kecke, S., Schiemann, J., & Wilhelm, R. (2018). Online tools supporting the conduct and reporting of systematic reviews and systematic maps: A case study on CADIMA and review of existing tools. Environmental Evidence, 7, 8. https://doi.org/10.1186/s13750-018-0115-5

Citation searching, chasing, or snowballing

It is recommended to search the reference lists of included papers for potentially relevant ones. Below are guidelines on how to use the proper terminology to describe the process of citation chasing:

• Hirt, J., Nordhausen, T., Fuerst, T., Ewald, H., Group, Tarc. study, & Appenzeller-Herzog, C. (2023). The TARCiS statement: Guidance on terminology, application, and reporting of citation searching. medRxiv. https://doi.org/10.1101/2023.10.25.23297543

Data Extraction

• SRDR+ (Systematic Review Data Repository) - a free tool for data extraction, management, and archival during systematic reviews.

Retraction Watch

• http://retractiondatabase.org/
• User Guide

Writing

• Academic Phrasebank - Get some inspiration and find some terms and phrases for writing your research paper.
• Wordtune - Say exactly what you mean through clear, compelling, and authentic writing.
• WordHippo thesaurus - find synonyms or antonyms when you just can’t seem to think of the exact word you want to use
• Methods Wizard, via the Systematic Review Accelerator, to write your Systematic Review Methods section. Check the Text Recycling Research Project offering best practices for researchers' reuse of materials from their own prior work in new documents. 
• Outwrite can be used for writing your manuscript. Automated high-quality spelling, grammar, and rephrasing corrections using artificial intelligence (AI) to improve the flow of your writing. Free and subscription plans are available starting from $10/month.
• RevManHAL is an extension to RevMan to automate the writing of systematic reviews (created by the Cochrane Collaboration Schizophrenia Group). 
• Oxford English Dictionary - Use to locate word variants.
• Binoculars - a tool that detects AI-generated content. For more information about the tool, read the preprint below (PDF):
  • Hans, A., Schwarzschild, A., Cherepanova, V., Kazemi, H., Saha, A., Goldblum, M., Geiping, J., & Goldstein, T. (2024). Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text (arXiv:2401.12070). arXiv. https://doi.org/10.48550/arXiv.2401.12070 
• Equitable Citation Practice:
  • Power Point slides
  • Include a diversity statement for citations in your manuscript
  • Resources to Help with Critical Citation Practice
    • Black Authors
      • How to Cite Like a Badass Feminist Tech Scholar of Color - An article and printable zine that guides you in excellent critical citation practice.
      • Cite Black Authors - A database of self-identified Black scholars to cite in a wide variety of fields and subjects.
      • Cite Black Women Collective - A group of women who gather resources and push researchers to change their citation practice.
    • Neuroscientists
      • Women in Neuroscience
      • Anne's List
      • BiasWatchNeuro

Text mining

• PubMed Pub ReMiner - Text mining for PubMed
• Yale MeSH Analyzer - Watch this tutorial (7 min.)
• NCBI MeSH on Demand - Identifies MeSH terms in submitted text
• SWIFT-Review - Desktop text mining tool specific to systematic reviews.
• Carrot2 clustering engine - Organizes your web or PubMed search results into topics
• Voyant - General text mining
• TerMine - General text mining
• JSTOR Text Analyzer - Recommends journal articles in JSTOR relevant to text.
• CREBP-SRA Word Frequency Analyser (WFA) - This tool helps determine which words you should use to construct and refine a search strategy

Visualization

• RAWGraphs is an open-source data visualization framework built with the goal of making the visual representation of complex data easy for everyone.
• CitNetExplorer is a software tool for visualizing and analyzing citation networks of scientific publications. The tool allows citation networks to be imported directly from the Web of Science database. Citation networks can be explored interactively, for instance by drilling down into a network and by identifying clusters of closely related publications.

References:

• Van Eck, N. J., & Waltman, L. (2014). CitNetExplorer: A new software tool for analyzing and visualizing citation networks. Journal of Informetrics, 8(4), 802-823.
• Van Eck, N. J., & Waltman, L. (2014). Systematic retrieval of scientific literature based on citation relations: Introducing the CitNetExplorer tool. In BIR@ ECIR (pp. 13-20).

Converting scanned PDFs to text

• Convert PDF to DOC - This free online PDF to DOC converter allows you to save a PDF file as an editable document in Microsoft Word DOC format, ensuring better quality than many other converters.
• Instructions on how to convert scanned PDFs into text using Adobe Acrobat and Microsoft Word

Translation tools

• In Google Docs, click on the Tools > Translate document menu at the top of the page. 
• Google Translate can translate PDFs if they were saved using Optical Character Recognition (OCR).
• Yandex.translate can translate document types DOC, DOCX, PDF, XLS, XLSX, PPT, PPTX, and other files up to 5MB in size.
• DeepL translator - a free online tool for translating text without limits.
• Cochrane TaskExchange - articles are not translated in their entirety but rather checked to see if they meet the eligibility criteria and if so, then the data is extracted. 

Supporting several stages of the review process

• Covidence - Create and maintain your systematic review throughout screening, selection and extraction phases (starting from $240 per year). 
• Distiller is designed for screening and data extraction phases of systematic reviews (starting from $15/month for students and $75/month for faculty). 
• EPPI-Reviewer is a web-based program used to manage your systematic review through all stages of the process from bibliographic management, screening, coding and synthesis (starting from $14/month).
• CMA (Comprehensive Meta-Analysis) is a software that will compute effect size, treatment effects and create forest plots (starting from $125/year).  
• PICO Portal - A systematic review platform that leverages artificial intelligence to accelerate research and innovation. Useful for deduplication, screening and collaboration with researchers from different organizations. Watch video tutorials before you get started. 
• SUMARI is (System for the Unified Management, Assessment and Review of Information) is the Joanna Briggs Institute's premier software for the systematic review of evidence. It is designed to facilitate the entire review process, from protocol development, team management, study selection, critical appraisal, data extraction, data synthesis and writing a systematic review report (annual subscription is $130). 

Summarizing articles

• Scholarcy - an AI-powered article summarizer and a Google Chrome extension. It summarizes research articles, creates interactive flashcards, highlights key points, links to open-access versions of each citation. This extension gives you the key points of any article or research paper you have open in your browser.  It creates a referenced summary with important points highlighted and generates a background reading list for those new to a field. Free with basic features, $7.99/month unlimited access.   

Writing services

• Eloquenti - guidance and resources to help individuals enhance their writing skills across various formats, including scientific communications and medical writing.
• Paper Preflight - although this application checks for errors and issues with the manuscript for the journal Sexually Transmitted Diseases, it will evaluate the paper for free and show the results for a one-time fee of $29.

Author services from Springer Nature

• English Language Editing
• Scientific Editing
• Manuscript Formatting
• Illustration and Figure Services
• Research Data Support Services

Translation services AND MORE (grant application, protocol development, etc.)

• Systematic Review Consultants - a team of freelance information specialists, systematic reviewers, health economists, and research analysts providing a wide range of services for all stages of the systematic review. They offer flexible rates and service levels based on your available time, skills, and resources. Contact Dr. Farhad Shokraneh (farhad@SystematicReview.info) for further information. 
• Springer Nature Academic Translation - price varies depending on the word count
• American Translators Association (ATA) - find certified translation services ( an individual or a company)

Quality Assessment Tools

Quality Assessment and Risk of Bias Tool Repository - a freely available repository of commonly used tools to assess the risk of bias. To best access the tool, follow the steps:

1. Go to the Files section on the left.
2. Download the excel spreadsheet. This will allow you to utilize all the functions such as the drop-down filter to limit to a certain study design type (column F).

Risk of Bias Tools Repository (National Toxicology Program, U.S. Department of Health and Human Services) - a repository of risk of bias tool providing the tools through Tableau, a data visualization platform. The website offers options to filter the tools alphabetically, by domain name, study design, and research area.

Which one to choose? Read the article below for further guidance.

• Munthe-Kaas, H. M., Glenton, C., Booth, A., Noyes, J., & Lewin, S. (2019). Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: First stage in the development of the CAMELOT tool. BMC Medical Research Methodology, 19(1), 113. https://doi.org/10.1186/s12874-019-0728-6. See Additional file 3 for a list of included critical appraisal tools.

• Zeng, X., Zhang, Y., Kwong, J. S. W., Zhang, C., Li, S., Sun, F., Niu, Y., & Du, L. (2015). The methodological quality assessment tools for preclinical and clinical studies, systematic review and meta-analysis, and clinical practice guideline: A systematic review. Journal of Evidence-Based Medicine, 8(1), 2–10. https://doi.org/10.1111/jebm.12141

• Seehra, J., Pandis, N., Koletsi, D., & Fleming, P. S. (2016). Use of quality assessment tools in systematic reviews was varied and inconsistent. Journal of Clinical Epidemiology, 69, 179-184.e5. https://doi.org/10.1016/j.jclinepi.2015.06.023

Critical Appraisal Tools for Blogs, Podcasts, and Grey Literature

• JBI Checklist for Text and Opinion
• Colmers IN, Paterson QS, Lin M, Thoma B, Chan T. The Quality Checklists for Health Professions Blogs and Podcasts. The Winnower 2:e144720.08769 (2015). DOI: 10.15200/winn.144720.08769
  • For Blogs
  • For Podcasts
• The CRAAP Test
• AACODS checklist for grey literature
• The Public Health Ontario Critical Appraisal Tools
  • MataQAT
  • Guide to appraising grey literature