How Research contributes to Evidence based Practice

how to use research evidence to inform practice and what is research evidence on writing and how do writers of research gather evidence how to synthesize research evidence
DrLaurenHepburn Profile Pic
DrLaurenHepburn,United Kingdom,Researcher
Published Date:07-07-2017
Your Website URL(Optional)
Comment
EPPI-Centre The Science of Using Science Researching the Use of Research Evidence in Decision-Making Laurenz Langer, Janice Tripney, David Gough EPPI-Centre Social Science Research Unit UCL Institute of Education University College London Final report • April 2016 PPI CENTRE e This project was led by the Alliance for Useful Evidence, with generous funding and support from Wellcome Trust and the What Works Centre for Wellbeing. The project products include: 1. The Final Report by Langer, Tripney, and Gough (the current document); 2. The Technical Report by Langer, Tripney, and Gough (available at http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3504); 3. A discussion document based on the Final Report and a case study analysis of decision- makers’ use of evidence by Breckon and Dodson (available at http://www.alliance4usefulevidence.org/publication/); 4. A conference to disseminate the findings, held on 12 April 2016 at the Wellcome Trust. The Technical Report is an extended version of the Final Report and provides additional detail on research methods and findings. Information on the extent of such further detail is provided at the start of each chapter. The overall project was supported by a project board with members: David Carr (Wellcome Trust) Nancy Hey (What Works Centre for Wellbeing) Jonathan Breckon and Jane Dodson (Alliance for Useful Evidence) David Gough, Laurenz Langer, and Janice Tripney (EPPI-Centre) The research was undertaken by Laurenz Langer, Janice Tripney, and David Gough of the EPPI-Centre, Social Science Research Unit, UCL Institute of Education, University College London. This report should be referenced as: Langer L, Tripney J, Gough D (2016). The Science of Using Science: Researching the Use of Research Evidence in Decision-Making. London: EPPI-Centre, Social Science Research Unit, UCL Institute of Education, University College London. ISBN: 978-1-907345-88-3 © Copyright 2016 Authors of the reviews on the EPPI-Centre website (http://eppi.ioe.ac.uk/) hold the copyright for the text of their reviews. The EPPI-Centre owns the copyright for all material on the website it has developed, including the contents of the databases, manuals, and keywording and data-extraction systems. The centre and authors give permission for users of the site to display and print the contents of the site for their own non-commercial use, providing that the materials are not modified, copyright and other proprietary notices contained in the materials are retained, and the source of the material is cited clearly following the citation details provided. Otherwise users are not permitted to duplicate, reproduce, re-publish, distribute, or store material from this website without express written permission. i Table of contents Abbreviations........................................................................................... ii Executive summary ................................................................................... 1 Chapter 1. Introduction ............................................................................. 6 1.1 Aim ................................................................................................... 6 1.2 Background ......................................................................................... 6 1.3 Approach taken by this project .................................................................. 7 1.4 Research questions ............................................................................... 11 Chapter 2. Research methods ......................................................................13 2.1 Overview of research structure & process .................................................... 13 2.2 Review 1 methods and process ................................................................. 13 2.3 Review 2 methods and process ................................................................. 14 Chapter 3. Summary of findings ..................................................................16 3.1 Review 1 results (EIDM literature) .............................................................. 16 3.2 Review 2 results (broader social science literature) ........................................ 18 3.3 Summary of results across Review 1 and Review 2 .......................................... 19 M1 interventions (building awareness for, and positive attitudes towards, EIDM) ........... 20 M2 interventions (building agreement on policy-relevant questions and fit-for-purpose evidence) ............................................................................................... 24 M3 interventions (providing communication of, and access to, evidence) .................... 27 M4 interventions (facilitating interactions between decision-makers and researchers) .... 32 M5 interventions (developing skills to access and make sense of evidence).................. 36 M6 interventions (influencing decision-making structures and processes) .................... 41 Chapter 4. Conclusion ...............................................................................47 4.1 Scope of project .................................................................................. 47 4.2 Strengths and limitations ........................................................................ 47 4.3 Suggestions for future EIDM interventions .................................................... 48 4.4 Guidance to facilitate development of a Theory of Change ................................ 49 References .............................................................................................54 Appendix A: Search results Review 1 ............................................................... 56 ii Abbreviations CMO Capability, motivation, and opportunity CoP Communities of practice DPME South African Department of Planning, Monitoring, and Evaluation EIDM Evidence-informed decision-making NICE National Institute for Health and Care Excellence RCT Randomised controlled trial ToC Theory of Change ii Executive summary Introduction Research evidence is just one factor that can influence decision-making at a policy and practice level. While various interventions have been developed to enhance and support the use of research evidence by decision-makers, it is unclear which interventions are effective. This research project set out to review the efficacy of interventions applied to increase decision-makers’ use of research in various decision arenas. The project also examined whether there is additional knowledge in the broader social science literature that is relevant to evidence-informed decision-making (EIDM) and could be applied to help support future interventions in this area. Review methods Two reviews of reviews were conducted: first, a systematic review of reviews of the EIDM literature (Review 1); and, second, a scoping review of the research reported in reviews in the broader social science literature (Review 2). Both reviews applied an explicit review methodology following a structured and transparent process to synthesise the findings reported in both bodies of literature. An overall conceptual research framework was developed to structure the two reviews in a comparable manner and to allow for the integration of the results from both reviews. This framework was used to group interventions according to six mechanisms of change (i.e. the processes by which EIDM might be achieved). Each of the six mechanisms (M1-M6) were also examined in terms of intermediary behavioural outcomes consisting of the capability, motivation, and opportunity (CMO) to act in a way that may increase EIDM. Review 1 results: what works to increase research use by decision-makers? The systematic review of reviews (Review 1) identified 36 existing reviews assessing what interventions work to increase research use. Synthesising the findings of 23 reviews rated moderate to high trustworthiness and relevance, we found: Evidence of effects (evidence use outcome) Interventions facilitating access to research evidence, for example through communication strategies and evidence repositories, conditional on the intervention design simultaneously trying to enhance decision-makers’ opportunity and motivation to use evidence (reliable 1 evidence). Interventions building decision-makers’ skills to access and make sense of evidence (such as critical appraisal training programmes), conditional on the intervention design simultaneously trying to enhance both capability and motivation to use research evidence (reliable evidence). Interventions that foster changes to decision-making structures and processes by formalising and embedding one or more of the other mechanisms of change within existing structures and processes (such as evidence-on-demand services integrating push, user-pull 2 and exchange approaches) (cautious evidence). 1 ‘Reliable’ refers to evidence based on reviews rated high trustworthiness and relevance in the weight of evidence assessment. For details of the weight of evidence assessment, see Section 2.1 below and Chapters 2, 3 and Appendix I in the Technical Report. 2 ‘Cautious’ refers to evidence based on reviews rated moderate trustworthiness and relevance. As above. 1 There is reliable evidence that some individual interventions characterised by a highly intense and complex programme design lead to an increase in evidence use. Overall, however, and based solely on observation, simpler and more defined interventions appear to have a better likelihood of success. Evidence of no effects (evidence use outcome)  Interventions that take a passive approach to communicating evidence that only provide opportunities to use evidence (such as simple dissemination tools) (reliable evidence).  Multi-component interventions that take a passive approach to building EIDM skills (such as seminars and ‘communities of practice’ without active educational components) (cautious evidence).  Skill-building interventions applied at a low intensity (such as a once-off, half a day capacity-building programme) (cautious evidence).  Overall, unstructured interaction and collaboration between decision-makers and researchers tended to have a lower likelihood of success. However, clearly defined, light-touch approaches to facilitating interaction between researchers and decision- makers, engagement in particular, were effective to increase intermediate CMO outcomes (cautious evidence). Absence of evidence  Interventions building awareness of, and positive attitudes towards, EIDM.  Interventions building agreement on policy-relevant questions and what constitutes fit- for-purpose evidence. Review 2 results: insights from social science knowledge to support research use The scoping review of the broader social science literature (Review 2) identified 67 interventions of potential relevance to EIDM. Configuring the insights and, in some cases, the reported effects of these interventions generate a number of contributions that the reviewed social science literature suggests. These contributions illustrate examples of potential applications of social science knowledge to support EIDM interventions and mechanisms. Promote and market behavioural norms  Social science knowledge on the creation of behavioural norms could be used in EIDM to support the formation of social or professional evidence use norms. Effective social science interventions to build such norms included social marketing, social incentives, and identity cues, for example. Engage in advocacy and awareness raising for the concept of EIDM  Social science research suggests that advocacy and awareness-raising campaigns are effective in supporting behavioural change. These strategies could be applied to communicate and popularise the concept of EIDM to increase awareness for the benefits of using evidence during decision-making as well as the risks of not doing so. Effectively frame and formulate communicated messages  Social science literature on effective communication suggested many techniques and strategies that can be used to enhance the communication of research evidence. Framing of messages, tailoring communication including audience segmentation, and regular use of reminders are examples of communication techniques reported as effective in the social sciences that could contribute insights to EIDM interventions as well. 2 Design appealing and user-friendly access platforms and resources  The social science literature features a rapidly growing body of knowledge on information design. Interventions aiming to improve decision-makers’ access to evidence could directly draw from this knowledge to enhance the design of evidence repositories and other resources, as well as to investigate the programming of EIDM apps. Build a professional identity with common practices and standards of conduct  Social science insights on social influence, collaboration, relationship building, and group interaction could be used to improve the design and outcomes of interaction interventions. The literature suggests that interaction among professionals can build a professional identity with common practices and standards of conduct (through, for example, communities of practice, mentoring, and inter-professional education). Making the building of a professional identity relating to evidence use a key objective of future interaction interventions would, in turn, entail a greater emphasis on facilitating interactions between different decision-makers to fully harness the power of social influence and peer-to-peer interaction. Foster adult learning  Social science knowledge on adult learning theories and principles is of direct use and relevance to EIDM capacity-building. Integrating this body of knowledge more closely with EIDM is likely to enhance the long-term performance of interventions supporting decision-makers’ EIDM skills. Build organisational capacities and support organisational change  A large body of knowledge on organisational structures could be transferred to support the design of EIDM interventions. Social science research on organisational learning and cultures, management and leadership techniques, and other changes to organisational processes and structures (for example, facilitation), is of direct benefit to interventions aiming to increase the receptivity of decision-making processes and structure to evidence use. A closer integration of this body of knowledge could enhance the appetite and readiness of organisations to use evidence. Use behavioural techniques, including nudges  A developing body of social science knowledge, one which is currently not integrated within the EIDM literature, investigates the influence of behavioural factors (such as cognitive loads) on individual decision-making processes. It has also developed effective techniques to reduce cognitive biases and enhance decision-makers’ choice architectures. Supporting the use of evidence during decision-making similarly could be subject to these techniques and the design of evidence use nudges could provide a valuable tool in the repertoire of EIDM interventions. Behavioural sciences stress the importance of salience in the design of interventions, which could directly be applied to support the practice of EIDM. Exploit the potential of online and mobile technologies  The application of online and mobile technologies is suggested in the social science literature to increase the reach, convenience, and appeal of interventions. A range of EIDM interventions (e.g. communication, capacity-building, decision aids) could benefit from the integration and regular use of online and mobile technologies. 3 Institutional frameworks and mechanisms  Institutional frameworks and mechanisms can advocate and nurture structural changes at all levels of decision-making. In the context of EIDM, effective examples include accreditation processes, clearinghouses such as the National Institute for Health and Care Excellence (NICE), and government ministries. Overall, however, not enough rigorous evaluation in this area is taking place. Implications from Review 1 and Review 2: The findings from Review 1 and Review 2 suggest a number of implications for EIDM practice and research. We discuss these for each review in turn below, before concluding with some final suggestions based on combined insights from both reviews. Interventions that support the communication of and access to research evidence were only effective to increase evidence use if the intervention design simultaneously tried to enhance decision-makers’ opportunity and motivation to use evidence. It is therefore advisable that future research and practice focus on how to design and tailor interventions that better feature these CMO configurations. In this, social science offers a great deal of knowledge that can be drawn upon. Similarly, interventions building decision-makers’ skills were only effective to increase evidence use if the intervention design simultaneously tried to enhance both capability and motivation to use research evidence. Again, attention should be paid to CMO configurations when designing or tailoring such interventions. Changes to decision-making structures and processes may be an effective mechanism to increase evidence use, but this currently lacks an extensive evidence-base. The results of this review suggest increasing the use of this mechanism in practice, as well as urging future research studies to explore the mechanism’s impact and theory of change more carefully. The majority of the reviewed interventions that focus on unstructured interactions between decision-makers and researchers appear ineffective at improving decision- makers’ evidence use, a finding that may be explained by a lack of conceptual clarity (i.e. what constitutes interaction, relationships, trust) and casual clarity (i.e. purpose of the interaction, theory of change of how interaction supports evidence use). Future research therefore requires an in-depth engagement with the theory of change underlying interaction interventions, and current practice is advised to focus on light-touch and well- defined intervention designs, such as decision-maker engagement, which command a more positive evidence-base. Given the current evidence gap, increased research and practice efforts are required to gain an understanding of interventions promoting the concept of EIDM, as well as those working towards mutual understanding of policy-relevant questions and agreement on what constitutes fit-for-purpose evidence needed to answer them. Unfortunately, the evidence on the relative effectiveness of single and multi-mechanism interventions is limited to observational patterns at this stage. Based on this, however, there is some suggestion that simpler and more defined interventions have an increased likelihood of success. Therefore, it seems sensible to both increase and substantiate research knowledge on simpler interventions, and develop the necessary theory before conducting large studies of multi-mechanism interventions whose casual chain is difficult to disentangle at this early stage of research knowledge. The scoping review identified many areas of social science knowledge that are currently not well-integrated and drawn from in EIDM. This leaves two main implications from Review 2 for future research and practice: first, a closer investigation of the integration of 4 the social science interventions and knowledge suggested as of relevance to EIDM in this scoping review; and second, the creation of a closer link between EIDM and the social science literature. Future research should explore mechanisms to better connect both bodies of knowledge. Thereby, EIDM would be better positioned to benefit from the most up-to-date knowledge base and run less risk of being out of sync with other areas of the social sciences. Finally, in this project we have used levels of intervention, mechanisms of change, and capability, motivation and opportunity to change behaviour as a framework to help understand (a) what interventions are trying to achieve, and (b) the processes they use to try to achieve this (in other words, the ‘theory of change’ of how the intervention is meant to have its effect). We hope that this framework can help others to plan a theory of change when they develop or evaluate interventions to enable EIDM, and we offer guidance on how to develop such a theory of change. 5 Chapter 1. Introduction This chapter is identical in content with Chapter 1 in the Technical Report. 1.1 Aim The results of research studies can be one important component in decision-making by policymakers, professionals, and members of the public. However, such research evidence is not always considered in decision-making, even when relevant research is available. The aim of this research project is to review the evidence-base relevant to increasing the use of research evidence by decision-makers; in other words, to review one aspect of the science of using scientific knowledge. 1.2 Background Over the last twenty years there has been an increasing concern, both in the UK and internationally, to make better use of the evidence produced by research in policy and practice decision-making. This has led to the rapid growth of systematic reviews to bring together, in a rigorous and transparent way, the available research evidence. There have also been a number of initiatives developed to improve the communication, interpretation, and uptake of research with the aim of helping decision-makers of different types make better use of research. In addition, a new area of research activity has developed to study how research interacts with policy and practice, with the intention of enabling such interactions to become more frequent and useful (Nutley et al. 2007). While much of this research has focused on processes of research use and/or the barriers and facilitators to the use of research (for example, Oliver et al. 2014), there is also now a considerable body of research evaluating the effectiveness of strategies promoting evidence-informed decision-making (EIDM). To address the aim of this project, we conducted two separate reviews of the literature. First, we first systematically reviewed existing reviews of the specialist EIDM literature which has evaluated evidence use interventions. Second, as there are also many other aspects of social science research that may be relevant to the study of research use, we undertook a scoping review of the broader social science literature to identify evidence of the effectiveness of additional interventions and any further insights that could be 3 relevant in an EIDM context . Our research therefore brings together the findings reported in two related bodies of literature: Review 1 (review of EIDM literature) and Review 2 (review of the broader social science literature). Definitions For the purpose of this project, EIDM is defined as a process whereby multiple sources of information, including the best available research evidence, are consulted before making a decision to plan, implement, and (where relevant) alter policies, programmes and other services. Our concern is limited to the use of a particular type of evidence in decision-making: that is, research-based evidence. Research may be defined as a systematic investigative process employed to increase or revise current knowledge. For the purposes of this review, we employed a broad conceptualisation of research that included not only 3 In this context, ‘broader’ indicates the research use literature too, as it is also part of the social science literature. 6 scientifically-based research, but also administrative data and statistics collected in the course of service and benefit provision (such as school-level datasets). Research use is understood as a multidimensional construct (Weiss 1979). Two kinds of research use are relevant to this study: instrumental and conceptual. • Instrumental research use is a direct use of research knowledge. It refers to the concrete application of research, such as in the taking of specific policy decisions or implementation of practice interventions. • Conceptual research use highlights its enlightenment function. This is when research influences how policymakers and practitioners think about issues, problems, or potential solutions. Research findings may change their opinion but not necessarily a particular action. The phrase ‘research use’ therefore implies that the research user has engaged with the research and acted upon it in some way. Acting upon it may not necessarily mean that the research has been used to inform policy or practice developments. It could simply mean that the findings have been considered during policy discussions. Throughout the report we use the terms EIDM, evidence use, and research use interchangeably to denote the use of research evidence by decision-makers. 1.3 Approach taken by this project The research project was concerned with interventions able to enhance and support the use of evidence in decision-making. In the absence of an agreed over-arching theory of how EIDM occurs, we developed a conceptual framework to structure both reviews in a comparable manner and to allow for the integration of the results from both reviews. This framework consisted of two different types of intervention, which were grouped according to six identified mechanisms of change (i.e. the processes by which EIDM might be achieved). In addition to the primary outcome behaviour of EIDM, each of the six mechanisms were also examined in terms of intermediary behavioural components consisting of capability, motivation, and opportunity (CMO) to act in a way that may increase EIDM. We are aware that these interventions could occur at different levels, such as targeting behaviour change by individuals or in organisations. Together these four elements of intervention types, the mechanisms, behavioural CMOs, and levels of intervention, provided the overall conceptual framework for examining both the EIDM and broader social science literature, as illustrated in Figure 1.1 (and described in greater detail below). 7 Figure 1.1: Overall conceptual framework for the project 1. Interventions to increase Assessment of the research use research on the efficacy of interventions to 2. Mechanisms by which the increase the use of interventions (to increase research evidence by research use) have their impact decision makers: (i) Review 1: From 3. Behaviour change components the EIDM (Capability, Motivation, and Opportunity) literature; (ii) Review 2: From the broader 4. Levels on which interventions social science are applied and on which they literature. have an impact (1) Specific interventions from the EIDM and the broader social science literatures The project focused on two main types of interventions. First were those interventions designed to directly impact on the consideration of research evidence in decision-making (for example, continuing professional development activities to increase policymakers’ awareness of and capacity to use research in developing policy). The second type of intervention were those from the broader social science literature (for example, psychology, management, and behavioural sciences) that could potentially be relevant to increasing EIDM (even if such research has not yet been applied directly to EIDM). So, for example, there may be research on interventions to increase the effectiveness of communication strategies, but not specifically about communicating research evidence or the need to use such evidence. Other examples may include approaches to changing organisational behaviour and the use of marketing in individual behavioural change. As our focus was on interventions to improve consideration of research evidence in the decision-making process, supply-side interventions to improve the research enterprise itself (such as through funding channels) or researchers’ behaviour were not considered. In addition, interventions to support implementation and/or adherence of agreed evidence- based policies, practices or programmes (for example, clinical practice guidelines) were also outside the scope of the project. (2) Mechanisms of evidence use We used the underlying mechanisms driving interventions that have been proposed in the EIDM literature to categorise evidence use interventions. We identified six such intervention mechanisms based on previous studies of mechanisms (for example, Gough et al. 2011; Nutley et al. 2007), research on barriers and facilitators to decision-makers’ use of evidence (for example, Oliver et al. 2014), and existing empirical frameworks for intervention effectiveness (for example Moore et al. 2011). Interventions aiming to 8 increase EIDM were assumed to work through either individual mechanisms or through a combination of mechanisms. Table 1.1 outlines these six evidence use mechanisms. Table 1.1: Identified evidence use mechanisms Evidence use mechanisms Building awareness for, and positive attitudes toward, evidence-informed decision-making (EIDM). AWARENESS (M1) This mechanism emphasises the importance of decision- makers’ valuing the concept of EIDM. Building mutual understanding and agreement on policy- relevant questions and the kind of evidence needed to answer AGREE them. (M2) This mechanism emphasises the importance of building mutual understanding and agreement on policy questions and what constitutes fit-for-purpose evidence. Providing communication of, and access to, evidence. COMMUNICATION This mechanism emphasises the importance of decision- & ACCESS makers receiving effective communication of evidence and convenient access to evidence. (M3) 4 Interaction between decision-makers and researchers. INTERACT This mechanism emphasises the importance of decision- (M4) makers interacting with researchers in order to build trusted relationships, collaborate, and gain exposure to a different type of social influence. Supporting decision-makers to develop skills in accessing and making sense of evidence. SKILLS (M5) This mechanism emphasises the importance of decision- makers having the necessary skills to locate, appraise, synthesise evidence, and integrate it with other information and political needs etc. Influencing decision-making structures and processes. STRUCTURE & This mechanism emphasises the importance of decision- PROCESS makers’ psychological, social, and environmental structures (M6) and processes (for example, mental models, professional norms, habits, organisational and institutional rules) in providing means and barriers to action. To enhance accessibility we have structured the mechanisms using a numerical list and abbreviation (M1–M6). However, this does not reflect a hierarchical order of the mechanisms and we assume each mechanism to be of equal importance in supporting decision-makers’ use of evidence. 4 Use of the term researcher denotes anyone conducting research and is not confined to appointed individuals in official research positions. 9 (3) Components of behaviour change Increasing the use of research evidence by decision-makers depends on behaviour change: in this instance, the use of such evidence to influence policy debates, the resulting policy choices, and the practical implementation of those choices. The components of such behaviour change provide us with intermediary outcomes, in addition to the primary outcome behaviour of EIDM. Based on a review of existing frameworks for understanding behaviour change, Michie and colleagues (2011) developed a method for characterising interventions and linking them to an analysis of the targeted behaviour. In this ‘behaviour system’, three essential conditions—capability, motivation, and opportunity (CMO)—interact to generate behaviour that in turn influences these components. Any given intervention might change one or more components in this ‘behaviour system’ (see Figure 1.2). Our review has retained 5 Michie’s definition of capability, motivation, and opportunity. Figure 1.2: Components of behaviour change (source: Michie et al. 2011) 0 (4) Level of intervention The change in behaviour may be in organisations or by individuals, and organisations can vary in terms of their scope and responsibilities. For the purposes of this review, behaviour has been organised into four levels consisting of:  individual behaviour;  immediate organisational context (such as where people live or work);  broader organisational context (such as local government);  national and international organisations. 5 Capability is defined as the individual’s psychological and physical capacity to engage in the activity concerned. It includes having the necessary knowledge and skills. Motivation is defined as all those brain processes that energise and direct behaviour, not just goals and conscious decision- making. It includes habitual processes, emotional responding, as well as analytical decision-making. Opportunity is defined as all the factors that lie outside the individual that make the behaviour possible or prompt it (Michie et al. 2011). 10 Logic model As noted above, there is no agreed theory of how interventions can effectively influence decision-makers’ use of evidence. We therefore brought together the individual components of our conceptual framework to create a basic logic model that sets out how evidence use interventions are assumed to influence decision-makers’ consideration of research evidence (Figure 1.3). Figure 1.3: Intervention logic model – for each level of intervention M1 Capability M2 M3 Motivation Behaviour change: Evidence use M4 M5 Opportunity M6 The model illustrates how interventions may influence evidence use, either through a single mechanism or through multi-mechanism combinations. Applying these mechanisms allows interventions to influence one or more components of behaviour change, i.e. capability, motivation, and/or opportunity to use evidence. These CMOs then facilitate the final outcome of evidence use. A CMO component can therefore be understood as an intermediate outcome on the causal pathway to the final outcome. CMOs can work either in isolation or in combination. The logic model allowed us to structure the interventions according to the applied intervention mechanisms (outlined in Table 1.1). We could then unpack the impact of these interventions on evidence use through a CMO configuration as an intermediate outcome. Structuring interventions according to mechanisms, and outcomes according to behaviour change components, allowed us to create a structure that equally applied to the EIDM and broader social science literature. 1.4 Research questions To review the evidence-base relevant to increasing the use of research evidence by decision-makers in a systematic and transparent manner, we constructed the following research questions for this project. 11 Interventions applied to increase decision-makers’ use of evidence All levels of intervention Review 1: (RQ1) What is the quantity and type of studies that have been undertaken on the efficacy of interventions used to increase the use of research evidence by decision makers? (RQ2) What evidence is there for the efficacy of interventions used to increase the use of research evidence by decision makers? Review 2: (RQ3) What interventions are suggested in the social science literature that might be relevant to the evidence use mechanisms mapped in Review 1? (RQ4) What evidence is there for the efficacy of these broader social science interventions and how might they be relevant to EIDM? 12 Chapter 2. Research methods This chapter provides a summary of the methods used in this research project. For an in- depth discussion and additional content, please see Chapter 2 in the Technical Report, available at http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3504. 2.1 Overview of research structure & process This research project reviewed two related bodies of literature that have investigated which interventions are effective to increase decision-makers’ use of evidence. We used the logic and methods of systematic review (Gough et al. 2012) to guide our research and adopted a distinct methodological review approach for each body of literature. 6 (i) Review 1: systematic review of reviews of the EIDM literature. 7 (ii) Review 2: scoping review of reviews of the broader social science literature. Both Review 1 and Review 2 were conducted using a similar conceptual structure and research process as described in Chapter 1 and illustrated in Figure 1.3. Applying the same framework across both reviews allowed us to integrate their findings in a transparent and structured manner. For both reviews the process of bringing together the relevant literature included the following steps:  definition of criteria to include relevant research;  search of academic and grey literature for relevant research;  screening and inclusion of relevant research (research selection);  data extraction and trustworthiness/relevance appraisal of included research;  synthesis of research findings. We used bibliographic management software (EPPI-Reviewer 4) to manage the review process. 2.2 Review 1 methods and process Review 1 is a systematic review of reviews on the efficacy of interventions applied to increase the use of research evidence by decision-makers. It includes a systematic map and synthesis of the findings of existing systematic reviews of relevant EIDM literature. The search for relevant systematic reviews followed a detailed search strategy based on an a priori master search string of keywords related to research use, which was applied in 8 a range of academic databases and grey literature sources. To be included, reviews had to present a systematic review of the literature; non- systematic literature reviews and primary research were excluded. Systematic reviews also had to measure the effectiveness of interventions, which excluded conceptual and theoretical reviews as well as reviews that investigated barriers and facilitators to research use, such as decision-makers’ perception of evidence. The main inclusion criteria 6 A systematic review is defined as a review of research literature that uses systematic, explicit and accountable methods. It involves three key activities: identifying and describing the relevant literature; critically appraising relevant reports; and bringing together the findings in a systematic way, in a process known as synthesis (Gough et al. 2012). 7 A scoping review is an exploratory review that is predominantly rigorous and explicit yet is not fully systematic in its methods. 8 The full search strategy can be found in Appendix B in the Technical Report. 13 here refer to each reviews’ ability to investigate the attribution of the evidence use outcomes to the reviewed interventions. For this reason, reviews were required to include primary evidence that could lay a reasonable claim to evaluate the effects of interventions on evidence use outcomes minimising possible biases in the attribution of the effects to interventions. Eligible designs included, for example, randomised controlled trials (RCTs), quasi-experiments and single-group pre/post-test evaluations. The ability of research designs to minimise biases in attribution further influenced the weight of evidence rating of each review. Relevant outcomes included both the primary outcome of decision-makers’ use of research evidence (for example, increased references to research in policy documents), and intermediate outcomes in the form of changes in decision- makers’ CMOs—that is, changes in their capability, motivation and/or opportunity to use research evidence (for example, increased attitudes toward research evidence). Practitioner outcomes, such as the uptake and implementation of an evidence-based practice, were outside the scope of this review. Outcomes could be measured at any level of analysis presented in Section 1.3. A full list of inclusion criteria is available in the Technical Report. Eligible systematic reviews were subject to a detailed process of data extraction and quality appraisal. Data extraction included coding reviews for intervention, outcome measures, and results. Interventions reported in the included reviews were categorised and coded according to the six underlying mechanisms of change. This coding depended on the description of the interventions in the reviews, and interventions often applied multiple mechanisms. Coding of interventions and mechanisms followed a systematic and transparent method and the individual codes for each intervention are presented in Appendix A in the Technical Report. Notwithstanding, the coding process entails a degree of interpretation. The same disclaimer applies for the coding of the CMOs. To guide a transparent and comparable quality appraisal of the included reviews, we developed a weight-of-evidence assessment tool (Gough 2007), which examined the trustworthiness and relevance of the reviews’ findings. This tool took into consideration issues of the fitness-for-purpose of the review design, for example ability of included primary studies to minimise confounding bias, as well as the relevance of the review’s findings and approach to the project’s research questions. Findings drawn from reviews of high trustworthiness and relevance were classified as reliable evidence; while findings drawn from reviews of moderate trustworthiness and relevance were classified as cautious evidence. Relevant extracted information was captured in summary tables for each review, which were compared and fed into an overall summary of findings table (see Technical Report: Table 4.1 and Appendix A). In this process, review findings were aggregated and structured according to the applied intervention mechanisms. The review findings did not allow for a statistical meta-analysis, therefore we conducted a structured framework synthesis of review findings to investigate the effects of intervention mechanisms on CMOs and decision-makers’ use of evidence. The directions of the effects are included in summary tables for each mechanism (see, for example, Table 3.1 in Chapter 3). For the purposes of this review, the term ‘reliable’ evidence is used to denote evidence from reviews rated as high weight-of-evidence and ‘cautious’ for moderate weight-of-evidence rated reviews. 2.3 Review 2 methods and process The social science research literature is extremely large and diverse, and so it was understandably not possible within the resources of the review to systematically search and review the entire literature. Instead, Review 2 is a broad scoping review of social science research, with a specific focus on interventions that may be relevant to EIDM. 14 Interventions in Review 2 could refer to:  individual programme components (for example, sending reminders as a component of communication interventions);  interventions (for example, social marketing as a communication intervention); and  concepts from which future interventions might be derived (for example, information design as a scientific concept). We conducted a two-stage search of the social science literature. The first stage followed an iterative process, in which the six evidence use mechanisms guided a first scoping of areas of social literature relevant to each mechanism. Guided by the evidence use mechanisms, we engaged in an iterative search of these areas combining keyword searches, snowballing, and hand-searches of academic journals to identify concepts and interventions relevant to EIDM in the social science literature. The second stage (having identified relevant social science interventions) was to search for existing reviews on the impact of these interventions. The search for reviews of the effects of these relevant social science interventions was conducted using keyword searches in academic databases and Google Scholar. This iterative search process is explained in Sections 2.2 and 5.1 of the Technical Report. Any type of research (i.e., both empirical and conceptual studies) was eligible for inclusion in stage 1 of the search, providing they were relevant to supporting the application of evidence use mechanisms. For stage 2, reviews had to provide syntheses of impact evaluations of the interventions of interest. Reviews identified in stage 2 of the search were appraised for their trustworthiness using the same weight-of-evidence tool applied in Review 1. Unlike Review 1 however, we did not exclude low-trustworthiness studies from the synthesis, on the grounds that different research traditions within the social sciences subscribe to different methodological approaches. Review findings rated as of low trustworthiness were considered and labelled as ‘literature review’ findings. We extracted data on the direction of the effects reported in the social science reviews and these are included in a summary table of relevant social science interventions for each mechanism (see, for example, Table 3.1 in Chapter 3). However, unlike for Review 1, detailed information on each review was not collected and therefore is not provided in this report. Due to both the lack of sufficient and appropriate data available for statistical synthesis, and the iterative nature of the review process, we conducted a narrative synthesis based on the summary tables in Chapter 3 to assess the contribution and likely effects of social science interventions on CMOs and behaviour change outcomes if applied in the context of EIDM. This scoping exercise was not exhaustive and some of the identified concepts and interventions may have been suggested to be of relevance to support EIDM in theoretical papers, primary studies, and/or practice reports, all of which were outside the scope of 9 this project. 9 At the end of the discussion of each mechanism below, we provide a list of suggestions based on our project’s findings and point the reader to some examples of primary EIDM literature that raise similar points. 15 Chapter 3. Summary of findings This chapter is a summarised version of the findings reported in Chapters 3, 4, 5 and 6 in the Technical Report, available at http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3504. The content reported in section 3.3 is identical to Chapter 6 in the Technical Report. This chapter reports a summary of the individual results of Review 1 (Section 3.1) and 10 Review 2 (Section 3.2) and their combined findings (Section 3.3). 3.1 Review 1 results (EIDM literature) Review 1 identified a large body of evidence assessing what works to increase decision- makers’ use of research evidence. The systematic search yielded 6786 unique citations, which were screened on title and abstract against the inclusion criteria and led to the inclusion of 36 reviews (see Appendix A for more detail). The interventions included in these reviews are heterogeneous and few applied a common definition of EIDM outcomes and indicators to capture changes in EIDM. The identified interventions were underpinned by a range of mechanisms. The most common were M3 (communication and access) and M5 (skills to access and make sense of evidence), followed by M4 (interaction between researchers and decision-makers) and M6 (changes to decision-making processes and structures). M3 and M5 were the mechanisms most frequently applied in isolation. The majority of interventions applied multiple mechanisms, in particular, the mechanisms M3, M4, and M5 in combination. Finally, in terms of CMOs, opportunity to use evidence was the most commonly targeted component of behaviour change, followed by capability and then motivation. The majority of the interventions were aimed at the health professions. The 36 included reviews examined primary studies published from the mid-2000s onwards. The relevance and methodological quality of 23 reviews was judged appropriate, and the findings from these reviews were included in the synthesis. The remaining 13 reviews were 11 excluded on the grounds of low relevance and/or low trustworthiness . The narrative synthesis was structured according to the mechanism(s) to which the reviewed interventions applied, and their effect on CMOs and decision-makers’ use of 12 research evidence. As the included reviews pooled the results of different interventions in their synthesis, it has not always been possible to discuss the results of individual interventions. In addition, categories of pooled interventions in the included reviews varied, which further challenged the organisation of our synthesis according to individual interventions. We provide additional detail on individual interventions in the Technical Report in Table 4.1 and Appendix A. The main findings of Review 1 are as follows: Interventions related to M1 (awareness) and M2 (agree): We currently cannot comment on the efficacy of interventions applying M1 (awareness) and M2 (agree) as there is an absence of review evidence. Interventions related to M3 (access to and communication of evidence): In relation to CMOs, there is reliable evidence (i.e. from high weight-of-evidence rated reviews) indicating that interventions applying this mechanism can improve both decision-makers’ 10 For an in-depth discussion of the review findings, please see the Technical Report. 11 For a detailed discussion of the weight of evidence outcomes, please see section 3.3 in the Technical Report. 16

Advise: Why You Wasting Money in Costly SEO Tools, Use World's Best Free SEO Tool Ubersuggest.