What is critical appraisal?
Evaluation of literature is determining the value or worth of that piece of information, critical appraisal is a more advanced form of evaluation which asks the reader to consider the specifics of the literature in a structured way. Critically appraising a piece of research combines analysis of the design of the study, the validity of the findings in relation to the design of the study, the likelihood of bias, and the relevance of the overall results to other current research.
Specialist tools
Critical appraisal should be done systematically and objectively. To aid with this, there are a number of systems used for critical appraisal, many of which focus on using specific check lists. As there are a variety of available tools you should be able to choose the most appropriate tool for your research.
SIGN:
Scottish Intercollegiate Guidelines Network (SIGN) have detailed guidance on their website for the tools they use when undertaking systematic reviews for guideline development. This includes a flow chart to help you define the study type you are appraising and all relevant checklists and supporting guidance.
CASP:
The Critical Appraisal Skills Programme provides a set of eight checklists which cover some study types which the SIGN checklists do not, such as Cohort Studies.
Center for Evidence-Based Management (CEBMa):
This group provides a small selection of online checklists, but also has a mobile phone app (Android and IOS) which can help with critical appraisal on various published articles.
CEBM:
The Centre for Evidence-Based Medicine at Oxford University has created critical appraisal guidance sheets for 4 main study types, which are available in 4 languages: English, German, Spanish and Lithuanian.
The checklists from STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) aim to provide support in critical appraisal for observational studies rather than clinical trials, and they have 5 checklists available, including one to appraise conference papers (STROBE, 2017).
Aimed at non-randomised study types, with a focus on case control studies, and designed to be easy to use while also providing a clear score for each paper.
PRISMA focus on the critical appraisal of systematic reviews and meta-analyses and have a standard checklist available covering both study types.
The University of Bristol have developed the QUADAS system as a tool to appraise the quality of diagnostic studies.
The GRADE system has been developed to combat the failings often seen in the check lists systems which are often limited to a single study type and encourages a thorough analysis of the paper as a whole via the creation of evidence tables and cross referencing (Grading of Recommendations Assessment Development and Evaluation (GRADE), 2017).
General tools
All the tools above are specific to designated study types, and at times there may be a requirement to critically appraise other forms of information. The following resources do not provide a numerical grade for the quality of the evidence. They do, however, give guidance and advice on critical appraisal and detailed evaluation which are applicable to a variety of resources.
Books and Articles:
Trisha Greenhalgh has published an excellent book on critical appraisal called 'How to read a paper' which is available in the Library.
Additionally, the article from Nature Clinical Practice Gastroenterology & Hepatology by Jane Young and Michael Solomon covers the theory of critical appraisal as well as discussing ten valuable questions to ask when evaluating literature.
This group have produced a general critical appraisal checklist which covers 20 questions to consider as you read through the different sections of a published journal article.
Developed by academic publishers McGraw-Hill in 2001 the CARS checklist is designed to be applied to any type of information, not just scholarly articles. It covers the 4 main elements of evaluation while providing examples of questions to consider when appraising.
Understanding Health Research:
Created by the Social and Public Health Sciences Unit at the University of Glasgow, as well as having a step-by-step tool to walk you through critical appraisal of an article, this site also contains a wealth of information on methods used in health research and understanding bias.
Some forms of critical appraisal of the literature result in each individual piece of literature receiving a score to rate its quality - this is often referred to as grading: the quality of literature is assessed and given a score. Evidence tables are used to list study characteristics and to help compare the literature.
For example, SIGN checklists and the GRADE (Grading of Recommendations Assessment, Performance and Evaluation) system formalise the appraisal of literature by assigning an overall quality rating.