Poster No. 1 Determinants of policy influence potential in impact evaluation studies
Sofia Estevez, CIPPEC, Argentina
CIPPEC has a vast experience in the study of the use of evidence from impact evaluations in policy making. This experience, which includes CIPPEC’s participation in the Policy Influence Monitoring project led by 3ie led us to develop a framework to understand the conditions under which impact evaluations are more likely to influence the policy process. This poster presents the main conclusions and recommendations to promote an effective design of the policy influence strategy of these type of studies. We identify three main drivers: institutional-political context, policy influence plan and the composition of the research team. We believe that this information could be useful for researchers, donors and civil society organisations as it can improve their capacity to achieve their policy influence goals.
replacing CANCELLED “What factors affect evidence-informed policymaking in public health? A systematic review of qualitative evidence using thematic synthesis”
Poster No. 3 Sharing knowledge at last mile
Rob Cartridge, Practical Action, UK
The gift of knowledge is far more effective than the gift of material goods. So said Fritz Schumacher in the 1970s. But how do we share knowledge meaningfully with marginalised communities? With big data and ICTs there is more knowledge available than ever before, but how do we really make it accessible. Across the world we have used knowledge systems to enhance well-being and promote resilience. From podcasting in Zimbabwe to focus group discussions in Nepal, and from websites in Peru to call centres in Bangladesh. Not everything works, and in this poster we present the common factors in great knowledge sharing at the last mile.
Poster No. 4 Using methods aligned with quality improvement to develop a ‘researcher in residence’ role within a quality improvement programme team in the Scottish Government
Anna Milsom, Healthcare Improvement Scotland, UK
Combining academic research and quality improvement (QI) work may have the potential to produce a broader perspective of evidence for practice: this has been described by Glasziou and colleagues as combining ‘doing the right things’ and ‘doing things right’ so that we learn how to ‘do the right things right’. However, partnership between research and QI disciplines has been characterised as a challenge. This poster describes a commitment to collaboration via the appointment of a health services researcher embedded within a national health quality improvement support team. For 2016/17 a researcher who had previously worked on a remote basis with the team has been appointed a ‘researcher in residence’ within the Delivering Outpatient Integration Together (DOIT) and Technology Enabled Care (TEC) programmes within the Scottish Government. The researcher chose to use methods aligned with QI as a shared negotiation tool to plan upcoming work for the year ahead. The selected methods included an ‘after action review’ of previous involvement including retrospect analysis, a ‘plan, do, study, act’ cycle, and an ‘action effect model’ driver diagram. These methods were initially selected to deliberately orientate the new role towards association of research and QI, but an unexpected benefit of alternative perspective taking also became rapidly apparent to the researcher. Evaluation of the impact of the approach will take place throughout the year, and to date improvements to research work formats have already been collectively navigated.
Poster No. 5 Building relationships to support the use of evidence in Africa: the Africa Evidence Network
Laurenz Langer, University of Johannesburg, South Africa
In 2012, 23 delegates from Africa attended an Asian-specific conference on the production of useful evidence organised by 3ie and the Campbell Collaboration. At this event, the need for a unique African community of practice for the production of policy-relevant evidence and its use during decision-making was realised; and led to the creation of the Africa Evidence Network (AEN). Since 2012, the AEN has expanded to a Network of over 700 people from across the region, including government colleagues, academics, and knowledge brokers. The Network runs a website, regular newsletters, blogs and shares a range of different evidence-informed decision-making (EDIM) resources. The AEN has hosted roadshows to three countries and we will be hosting our second regional Evidence conference in September. This presentation will introduce the AEN and report on our membership profile, activities, and plans. It will draw on a full membership survey, a set of over 40 interviews, and an analysis of related networks. The poster presentation will highlight a number of case studies to illustrate how the Network has made a difference to its members, telling the stories behind the numbers. It also will report an analysis of the purpose and value of such a network to its members and to the EIDM community across the continent. In particular, we will consider the importance of communities of practice such as this in resource poor environments where the use of evidence is arguably most important, and yet where researchers, practitioners, and policy-makers are relatively unsupported in their work.
Poster No. 6 Beginning with the end in mind: A partnership strategy for improving research uptake
Christine Kim, FHI 360, USA
Even as scientific research and program experience offer valuable information about promising and proven strategies, too often critical gaps remain between evidence and real-world practice. Knowledge brokering is one research utilization (RU) strategy that aims to facilitate and improve the application of evidence to policy and practice. FHI 360 implemented an RU partnership model as an innovative knowledge brokering strategy that paired specialists in research uptake with researchers throughout the course of a study, from conceptualization to the post-study period. We present a retrospective case study of the RU partnership model that describes the role RU partners played, the challenges experienced during the partnership lifecycle, and examples of its outcomes. This partnership model offers a promising strategy for decreasing the gap between research generation and uptake, and can help funders, researchers, and policymakers use evidence to inform and improve decision-making.
Poster No. 7 Research utilization strategies for improving evidence uptake in policies and programs: a scoping study
Christine Kim, FHI 360, USA
Evidence of effective research utilization (RU) strategies, especially in low- and middle-income countries (LMIC), remains limited. We conducted a scoping study on RU strategies that aim to improve the uptake of evidence by relevant decision makers across various sectors globally. We identified primary studies in peer-reviewed literature published between January 2011 and June 2015 using electronic databases including Pubmed, Web of Science, Scopus, and Embase. Articles were screened according to selection criteria and relevant data charted; strategies were examined along an RU framework. We identified 5,471 articles after excluding duplicates, 357 articles were considered after screening, and 63 articles were included. Only 20 articles (< 1/3) came from LMICs and most studies had a qualitative or case study design (n=46). A total of 32 distinct strategies were identified, of which the most common RU themes addressed were knowledge transfer, capacity building, and tools to facilitate or monitor RU. Policymakers and researchers were the main actors engaged in RU strategies and the most common RU phases reported were the research and translation phases, with limited strategies implemented to address the institutionalization phase. RU outputs ranged widely but focused on types of evidence summaries or reports and knowledge translation tools. While documentation of RU strategies has increased over the years, strategies vary widely and evidence for their effectiveness remains weak globally. There is limited engagement of diverse research end users beyond policymakers and researchers. Improved evaluation methods of RU strategies may need to be developed to show their effectiveness.
Poster No. 8 How do decision makers engage with evidence when redesigning care models?
Alison Turner, NHS, UK
This poster will report on a qualitative case study, exploring how evidence is valued and applied in the design of new models to deliver health and social care in England. There is a growing recognition of the complexity of large scale change with observers advocating a systems approach favouring iterative and co-produced change. This study explores the role of evidence in designing complex change, capturing perspectives of a range of senior decision makers and exploring processes used. Analysis is based around four key themes identified from a literature review: the context in which individuals and programmes are working; personal perspectives on evidence;
processes for using evidence of what works; and experiences of using evidence. Evidence is perceived differently, influenced by culture, experience, professional background. This can have positive implications, as a diverse group working across health and social care, can together build a diverse and robust evidence base. However, there are risks that different values may be applied to different types of evidence. The study demonstrates the importance of evidence to inform the design of change interventions, helping to challenge assumptions and build consensus. Whilst evidence seems to be used to inform design, it is unclear if this is sustained throughout the lifecycle of the change programme, suggesting opportunities to facilitate the use of evidence to inform implementation and evaluation. The systems approach seems to have changed the “demand” for evidence, setting a challenge for the “supply” side to respond with new ways of creating and presenting evidence.
Poster No. 9 Can evidence based medicine be implemented into treatment of rare diseases? Lessons from a Cochrane review
Elad Shemesh, Israel
Background: Gaucher disease (GD) is an ultra-orphan rare metabolic disorder, caused by a deficient/malfunctioning enzyme. Untreated, GD may lead to significant disability and death. A breakthrough discovery 30 years ago revolutionized the lives of patients by providing the replaced enzyme. Despite the emergence of various treatment options in the last decade, there are no evidence based recommendations regarding treatment regimens/drugs, and treatment costs remain very high. Other inborn metabolic disorders suffer from similar unanswered questions regarding treatment- currently, there are 17 published Cochrane reviews on other rare inborn errors of metabolism, and 16 are listed as high priority titles for analyses (defined by the genetic disorders group together with the NHS). Methods: Applying Cochrane criteria, 8 randomized clinical trials (RCT) (300 patients) were filtered after an extensive medical database searching. Numerical data regarding organ volumes, disease activity markers and blood counts were collected, as well as data about possible biases. Different drugs and doses were compared. Conclusions: We contend that limiting analyses to RCTs in fields where these studies represent only small proportion of the total body of literature (such as in the case of rare diseases) may distort the conclusions and significantly constrain the recommendations that can be concluded. Therefore, despite being labeled as inferior to RCTs, inclusion of non-randomized trials should be positively considered when attempting to answer delicate questions (such as the optimization of treatment doses), when dealing with a high non-RCT/RCT ratio, or when discussing a disease affecting few patients(as in the case of rare diseases).
Poster No. 10 Application of the “Routine Piloting in Systematic Reviews” Method Across Academic Institutions to Improve Usability of Evidence in Public Health Research: A Case Study
Linda Long, University of Exeter, UK
Background: In 2014, Linda Long proposed routine piloting of systematic reviews through to evidence synthesis using data from a sample of included papers to improve review efficiency and validity. Objectives: This paper describes routine piloting in a systematic review of low intensity interventions to prevent sexually transmitted infections in young people and men who have sex with men (MSM). Methods: Seven databases were searched to October 2014. 23 young people RCTs and 10 MSM RCTs were identified for inclusion. A sample of five young people studies was piloted through to evidence synthesis, undergoing data extraction and quality appraisal using the Cochrane ‘risk of bias’ tool. Summary of Findings tables were created and circulated to researchers in participating research institutions and feedback sought on usefulness to inform the next stage of the programme. MSM papers were not subject to the piloting method and were processed as usual. Results: Following piloting, a number of criteria were identified as needing modification. After amendments, all relevant data from the remaining young people RCTs were efficiently extracted in one phase. All ten MSM RCTs were data extracted and quality appraised. However, following criteria modification, a second phase of data extraction was required. Conclusions: Routine piloting facilitated a “bespoke” review, with time saved through efficient data extraction. The mini synthesis provided a version of the full review that could be agreed across institutions at an early stage. This supported review project management, improved efficiency, and ensured optimal usability by researchers involved in the wider research program.
Poster No. 11 Systematic review: policy related drivers of diet in low and middle-income countries
Darlena David, LSHTM, UK
A systematic review was carried out to identify upstream policy determinants and the pathways through which they affect changes in diet and nutrition in low- and middle- income countries. Search was carried for published and unpublished papers in English that examined association of policies such as agriculture, or trade, or other policies with changes in diets in low- and middle-income countries. Nineteen databases were searched for published and unpublished papers in English (inception – 2011) and updated in 2014 (2011-2014 on 6 databases that contained included papers). The key policies that affected dietary change were economic, trade, social protection and food policies, and agriculture policies. The main pathways were through food prices, total expenditure, and availability. Economic policies (28 studies, 1947-2011) were linked change in diet (loss of nutrition gains, stunting, reduction in micro-nutrients— Vitamin C, Vitamin A, calcium— higher anaemia and obesity. Trade policies (19 of 21 studies, 1942-2011) increased rural and urban differences in food intake and promoted unhealthy food consumption patterns. While a majority of studies linked social protection and food policies with unhealthy dietary changes, a few like India’s employment guarantee scheme was associated with reducing food consumption inequities and enhanced dietary diversification. Agriculture policies were sometimes associated with encouraging more healthy diets. Agriculture policies prioritizing domestic nutrition security by improving availability, affordability and access to year-round supply of fruits and vegetables in local markets can ensure dietary diversity and nutrition security in low- and middle- income countries.
Poster No. 12 Immunization uptake in Nomadic and pastoralist communities in Kenya
Careena Otieno, Great Lakes University, Kenya
This is an ongoing research that seeks to assess the impact of a government strategy to improve primary health care, especially immunization uptake in nomadic and pastoral communities in Kenya, with an additional component to improve access to immunization health services for these communities. The communities in which this intervention is being carried out are classified as hard to reach areas, have been marginalized since independence and continue to produce poor health indicators, among them being immunization uptake for children below 23 months. The project seeks to identify households with children below 23 months using community health assistants who will then follow up on them providing health education to the households, defaulter tracing, and referral of children to ensure each child within their sites is fully immunized by the age of 23 months. The research project has a quasi-experimental design with intervention and control sites in which before and after data will be collected to measure the impact of the intervention
Poster No. 13 What is the evidence base behind the work Evidence Aid is undertaking? Substantiating the proposed theory of change with a review of the knowledge translation literature
Dominic Mellon, Evidence Aid, UK
Evidence Aid is a co-ordinated, international initiative to create and satisfy an increasing demand for evidence to improve the impact of humanitarian aid by stimulating the use of an evidence-based approach. In 2015, Evidence Aid reviewed its approach to translating knowledge into action in humanitarian response, developing a new theory of change with specific targeted interventions. Evidence Aid believes that to effect change, it needs to prioritise and focus its target audience to those who influence and steer practitioner behaviour. This is a significant departure from a programme based on improving access to research evidence through broadcast dissemination of systematic review summaries to all parts of the humanitarian sector. The four strategic objectives for Evidence Aid in 2015-2020 will be to: 1. Establish Evidence Aid as an influential organisation, targeting, engaging and inspiring influential organisations to use evidence in the humanitarian sector. 2. Identify priority evidence relating to broad health outcomes across sectors, generate evidence and facilitate its delivery. 3. Influence quality standards for evidence in health outcomes across sectors and for broader humanitarian outcomes. Partner with the sector to support and develop the supply of and capacity to use evidence in the longer term. 4. Target influential organisations and build their capacity to specify the use of evidence in decision-making and humanitarian response. We present a review of the evidence for a revised theory of change to promote transparency in our approach and invite constructive challenge from our partners, stakeholders and colleagues in this field.
Poster No. 14 Evidence Aid special collection for refugee health
Claire Allen, Evidence Aid, UK
Background: In 2015 over one million people arrived in Europe by sea, mostly originating from Syria. In the same year 3,771 people went missing or died attempting to reach safety in Europe. In 2016 people continue to make the hazardous journey across the sea and at the beginning of February 67,072 people made it across, while 357 were reported dead or missing. Objectives: To build collections of healthcare evidence to provide those addressing the health of refugees with some guidance, collections divided between Evidence Aid and Cochrane. Methods: Both collections focus on some of the most relevant medical conditions as perceived by experts involved either in guideline development or on the frontline, directly addressing the healthcare needs of refugees and asylum seekers. In the first instance, the work-group addressed five priority conditions. Results: The collection, ‘Health of refugees and asylum seekers in Europe’ was published 12/02/16, hosting curated resources from the Cochrane Library and other research outputs, categorized into guidelines; systematic reviews; articles; and other information. Conclusions: Since publication, the refugee health collection on evidenceaid.org has received almost 600 pageviews, ranking it third amongst most viewed pages, after the homepage and the resources tab, for that period. On average, users have been spending 2:30 minutes on the page, suggesting the content is commanding attention. We will continue to encourage an evidence-based response to this crisis, and will report on usage of both collections at the conference.
Poster No. 15 New approaches to Evidence Synthesis and Uptake
Kerry Albright, and Nikola Balvin UNICEF, Italy
UNICEF’s current Strategic Plan (2014-2017) includes an explicit commitment to ‘use knowledge and research in support of results for the most disadvantaged children’ Since 2012, UNICEF’s Office of Research-Innocenti based in Florence, Italy has had a mandate to provide strategic leadership to the rest of the organization in both identifying global research priorities of relevance to children but also in strengthening institutional capacity to improve the quality and use of research evidence. A dedicated ‘Research Facilitation and Knowledge Management Unit’ has been established to help ‘make evidence part of UNICEF’s DNA’, working alongside evaluation and data colleagues. This poster session will highlight a range of evidence-related products and activities being managed and/or undertaken in partnership with this Unit including (i) evidence synthesis products including a systematic review on gender socialization and an evidence gap map on adolescent wellbeing (ii) methodological briefs in impact evaluation and conducting research with adolescents (iii) research digests to highlight emerging research findings (iv) establishment of the annual ‘Best of UNICEF Research’ awards to highlight good practice, exchange knowledge and reward excellence (v) development of institutional standards and procedures on quality assurance and ethics in evidence generation (vi) training on research management and methods throughout the entire research process from developing a good research question, appraising evidence quality and validity, synthesizing and communicating findings, understanding evidence-informed policymaking, through to assessing uptake and impact.(vii) Identifying innovative ways to communicate findings including through data visualization and tools to better capture research impact both online and offline.
Poster No. 16 Minding the ‘know-do’ gap: Experiences of policy and practice uptake from a systematic review of parenting programmes for child disruptive behaviour
Muiread Furlong, Maynooth University, Ireland
Objective: Producing a high quality systematic review is only the first step in translating evidence into policy and practice. This rapid review assessed the academic, policy and practice uptake of a Campbell and Cochrane systematic review of parenting programmes for child disruptive behaviour (published in 2012). Methods: Review impact was searched using Altmetric, Google scholar, and the first twenty pages of Google. Experiences of the authors in evidence distribution and engagement are also reported. Results: Review findings were highly cited in academic articles, books, policy documents and guidelines, and had high visibility on media and social media, e.g. ABC television, newspapers, Twitter, blogs, and Wikipedia. Evidence distribution and transformation was also enhanced by efforts of the authors in engaging with media, child advocacy groups, service providers, government in the UK and Ireland, and reporting findings at conferences. Challenges in evidence engagement and implementation included: lack of nuance from policymakers in using review findings; the review not answering questions of costing the broad-scale implementation of the intervention; and government unwillingness to allocate extra funds to parent training within an era of ‘austerity’ politics. Conclusions: Electronic dissemination of the review led to significant evidence distribution and transformation with only a little effort required from authors. Review impact was successful in terms of enhancing the priority of parent training on the policy agenda. Implementation of evidence in practice may be enhanced by dedicating a knowledge broker to evidence engagement and discussion and through developing a costed implementation strategy for parent training.
Poster No. 17 An Evaluation of the Effectiveness of Early Intervention Support Services in Northern Ireland
Maurice Meehan, HSCNI, Ireland
The Early Intervention Transformation Programme (EITP) is a Northern Ireland Executive/Atlantic Philanthropies Delivering Social Change Signature Programme, funded jointly by the Delivering Social Change fund, DHSSPS, DE, DoJ, DSD, DEL and The Atlantic Philanthropies. EITP aims to improve outcomes for children and young people across Northern Ireland through embedding early intervention approaches. It’s widely recognised that early intervention in the lives of children and their families can pay dividends in preventing longer term difficulties which, left untreated, can escalate in severity. By supporting families and generating positive intergenerational interactions later onset difficulties can be avoided and substantial public spending savings can be made (Allen, 2015). Early Intervention Support Services (EISS) in Northern Ireland aims to support and empower Tier 2 Families by intervening early with evidence informed services before difficulties become intractable. The effectiveness of the EISS will be evaluated by a research team by analysing outcomes measured across the five pilot sites over a thirty month period. It’s expected the EISS will support approximately 1925 families over this time. Outcome data will be collected using a range of standardised measures. Alongside the quantitative work, a qualitative strand will be used to explore the experiences of families and key workers taking part in the EISS. The presentation team will include the research partner, the commissioner and the policy developer, the vision is to refine and replicate the EISS model based on the basis of what has worked and to ensure availability of an EISS across Northern Ireland, if effective.
Poster No. 18 The Establishment and Application of Chinese Clearinghouse for Evidence Translation in Child & Aging Health
Shaokun Liu, Evidence-Based Medicine Center of Lanzhou University, China
Objective:To develop Chinese Clearinghouse for Evidence Translation in Child & Aging Health (CCET) which combines evidence-based research with evidence-based decision making and evidence-based practice. Methods:Child and Aging Health Advisory Committee were created to appraise and select health programs that will be appropriate for implementation in China, mainly from National Registry of Evidence-based Programs and Practices, California Evidence-Based Clearinghouse for Child Welfare and Campbell Library. The scientificity of eligible programs was rated and the applicability will be given in specific implementation context with strength of recommendation. Devote to disseminate qualitative evidence classification method and conduct methodology training of evidence-based social work. Results:CCET website consists of four core module, child health, aging health, dissemination & implementation and evidence-based research. 16 child health programs and 30 aging health programs with strong scientificity were translated. Implementation mode and strategies were analyzed from five aspects which include research institutions, service providers, decision-making bodies, users and health framework. An approach to assessing confidence in findings from qualitative evidence syntheses——CERQual (Confidence in the Evidence from Reviews of Qualitative research) was introduced. CCET has been a co-organizer for the 1st Senior Seminar of Research Methods for Evidence-Based Social Work in China. Conclusion: CCET is the first clearinghouse in China which devotes to the translation and appraisal of child and aging health, research methods training for evidence-based social work and intervention training and development. It will play an important role in knowledge translation of child and aging health in China.
Poster No. 19 The Status Analysis of the Development of Evidence-based Concept and Methods in the Field of Domestic Correction
Xiaoling Zhong, Evidence-Based Medicine Center of Lanzhou University, China
Objective Based on the analysis of the current development of evidence-based concept and methods in the field of correction, the study revealed the situation and development trend in China to provide the direction and guidance for more researchers. Methods The database CNKI was searched with terms”evidence-based””evidence-informed” from inception to October 2015.We sifted references by EndnoteX7.Inclusion criteria: the literature related to evidence-based correction. Exclusion criteria: conference abstracts reports and communications etc. Information of included articles were extracted and analysed (using Tableau 9.3 software and Ucinet software) according to published time and region, etc. Results Totally 653 references were retrieved and 52 were included finally. According to published time:evidence-based correction study was started in 2012,and then it increased. According to region, Jiangsu province and Beijing were the most, others were less. According to the study quality: high quality researches were less than low quality. Also, just a few projects were supported by government. Through the co-word analysis of key words: we find “evidence based correction” “correction objects” “correction methods” have been the core, which indicated researchers mainly concerned on how to implement it. Conclusion There are many challenges and limitations in developing evidence-based correction researches in China. Evidence-based correction in domestic has just started. But it’s still the hot of evidence-based researchers and social workers in the future. There will be more and more researchers to concerned on it.
Poster No. 20 Autonomy and agency among NEET young people
Rodrigo Ceni, Universidad de la República, Uruguay
The aim of this paper is to evaluate the impact of Young Network Program (JeR) on the aspirations, autonomy and agency among young people in Uruguay. JeR is a program in which multidisciplinary teams lead and support for a 24-month period groups of young people Not in Education, Employment or Training, in order to enable them for their re integration in the formal education system or in the formal labor market. We focus on the aspirations, autonomy and agency because all of them entail goals and commitment, which are long lasting capabilities for their future life. We estimate the LATE and the Marginal Treatment Effect (MTE) through a two-stage instrumental variable estimator using geo-referencing data. Our main results on LATE show that JeR affect different features regarding aspirations and autonomy. Treated young people indicate work as an important part in their autonomy, considering in a second place the family income. These population adjust their perceptions of the objective life conditions and their subjective well being, indicating a lower position in the income distribution, and their life satisfaction. The estimation of the MTE shows that there is a heterogeneous impact depending on the unobserved cost to participate in the program, those with higher cost consider that young people have lower influence in the society, especially among women and those above 18 years old.
Poster No. 21 Youth Empowerment through Microfinance in Uganda
Sophie Feller, UCLA, USA
Our project evaluated the Youth Livelihood Programme (YLP), sponsored by the Government of Uganda. This microfinance program aims to empower youth to create economic opportunities with the assistance of a government loan. YLP focuses on the provision of skills, sustainable economic opportunities and improving well-being, while decreasing gang activity and substance use, side effects of lack of opportunity. YLP targets youth aged 18-30, who determine an enterprise to pursue, submit a proposal and then receive government funds and oversight to help support the growth of the enterprise. The programme evaluation analyzed the context, objectives and outcomes of YLP within the programme’s paradigm of theory of change. Additionally, we evaluated the factors supporting program implementation and those that were barriers to implementation. We looked at 11 districts and 4 regions for feasibility purposes. We conducted focus group discussions and in-depth interviews with youth and government officials. Of 35 youth groups interviewed who were not newly funded, 18 had successful enterprises, 16 were struggling and 1 had disbanded. Eleven groups were saving money and 12 groups had begun loan repayment. Sixteen groups believed they had achieved an improved quality of life and 15 felt they had sustainable careers and new skills. Thirty-seven groups were empowered with future plans. Successful income-generating activities boost morale, inspire other youth and energize the community. Government officials’ efforts to provide logistical support were limited by the availability of resources. Program areas of improvement include better preparing youth to start and manage an enterprise.
Poster No. 22 Does financial education impact financial behavior, and if so, when? A meta-analysis
Tim Kaiser, DIW, Germany
In a meta-regression analysis of 115 microeconometric impact evaluation studies we find that financial education significantly impacts financial behavior, and to an even larger extent financial literacy. These results also hold for the subsample of RCTs. However, intervention impacts are highly heterogeneous: Financial education is less effective in low- and medium income countries; some target groups, such as low-income clients, or specific behaviors, such as borrowing, are difficult to influence; also mandatory financial education appears to be less effective. Thus, it is even more crucial for success to increase training intensity and offer financial education at a “teachable moment”.
Poster No. 23 Gay-Straight Alliances and Reduced Student Victimization: A Meta-Analysis of the Evidence
Robert Marx, Vanderbilt University, USA
Gay-straight alliances (GSAs) are school-based organizations for lesbian, gay, bisexual, transgender, and queer youth and their allies. Although they may sometimes be politically contentious, GSAs represent one attempt to improve students’ school climate and are one of the Gay, Lesbian, and Straight Education Network’s recommended interventions to create safe and inclusive schools in the United States. The current body of literature is mixed with respect to GSAs’ associations with school climate and lacks a quantitative synthesis that provides compelling evidence for the adoption of GSAs. To that aim, this meta-analysis evaluates the impact of GSAs on students’ self-reports of victimization in school by quantitatively synthesizing 18 primary studies with 54,152 participants. GSAs were associated with statistically significant reductions in students’ self-report of homophobic victimization (g = -0.22, 95% CI [-0.32, -0.11]), fear for safety (g = -0.25, 95% CI [-0.31, -0.180), and hearing homophobic remarks (g = -0.28, 95% CI [-0.42, -0.13]). The findings of this meta-analysis support the creation and funding of GSAs as a means of decreasing student victimization, and the dissemination of this information to school officials, teachers, and policymakers as a means of creating a more informed body of decision-makers.
Poster No. 24 Putting Evidence to Work for Recommended Practices in Early Intervention and Early Childhood Special Education
Brian Riechow, University of Florida, USA
In early intervention and early childhood special education, reviews of the research literature have been conducted to identify the status of best-available empirical evidence in support of recommended intervention practices for use by practitioners or families of young children with or at risk for disabilities. Though work using research literature for informing recommended practices began in the early 1980s, in the last 30 years, the field has further embraced this work and moved forward in defining and refining how research syntheses can be helpful to inform recommended practices. This has culminated in multiple versions of the Council for Exceptional Children Division for Early Childhood Recommended Practices. The intended audience for these practices are practitioners who work with young children with or at risk disabilities or their families. The most recent version of the DEC RP (2014) provided the field with updated recommended practices, but systematic research syntheses for each practice were not conducted. Rather, DEC undertook an initiative to develop a uniform system and accompanying processes for identifying, reviewing, and synthesizing the best-available evidence in support of each practice. Presently, the system has been developed and is being piloted with 10 of the 66 practices. The purpose of this poster is to present the current process and outputs and to solicit suggestions from conference attendees about both the systematic evidence review system as well as knowledge translation and dissemination activities.
Poster No. 25 Changing Minds on Girls Education Through Media: New Evidence from Africa
Woubedle Alemayehu, Social Impact, USA
Evidence from Kenya, Ghana and Nigeria on effects of National Chat Shows aired on TV to influence public dialogue and popular culture indicate that messages from credible sources help develop positive attitudes and practices toward girls’ education. Qualitative data were gathered over a year among young rural and urban women, and mothers and fathers with daughters before and after the shows. Contribution analysis of narratives was conducted for capturing change in knowledge, attitude and practices. Results demonstrate the power of celebrity messages, breadth of viewers’ internalization of messages on value of education, and sustainability of attitude change at various levels among the three countries. Notable positive attitude shifts over time was found in equal consideration of educating boys and girls, support needed for girls’ education and women’s empowerment, especially among fathers. All types of respondents showed thoughtful reflection and elaboration on the content of the programs to change their practices but notably among fathers and rural young women. The rationale for positive behavior change was tied to the show – many male respondents improved their patience, communication and support for daughters and wives, and many women respondents decided to return to school and develop resilience when facing adversity. Unintended consequences noted in the show’s primary focus on guests’ talents and passions, and secondary focus on guests’ education that could potentially undermine the interest in formal education. Many lessons for designing shows that inspire, entertain and educate were documented.
Poster No. 26 When students evaluate teaching: What variables affect teachers’ perception of the usefulness of the evaluations?
Eyvind Elstad, University of Oslo, Norway
Norwegian education authorities pursue the objective that students in upper secondary schools should evaluate teaching. This kind of evaluation scheme has, however, elicited resistance from teachers. The aims of this article are to explain the advent of teaching evaluation policies in Norway and to explore what factors explain teacher resistance to teaching evaluation schemes. Structural equation modelling of teacher survey data shows which factors are statistically associated with the concept of teacher resistance to teaching evaluation. Teacher stress and resistance is positively associated with the perceived controlling purpose of teaching evaluation rather than the evaluation itself. Teacher resistance is clearly negatively associated with their acknowledgement of students’ feedback and their perception of the communication with leadership. Furthermore, the model shows moderately strong negative pathways between perceived control purposes of teaching evaluation and acknowledgement of students’ feedback on the one hand and perceived control purposes of teaching evaluation and perception of clear communication of goals on the other hand. Teachers’ acknowledgement of the students’ feedback is weakly negatively associated with teacher stress. The implications for practice and further research are discussed. Although studies have examined teaching evaluation as a potential source of professional development, not much has been written about teachers’ perception of the usefulness of evaluations done anonymously by upper secondary students. The study explores also antecedents of Norwegian teachers’ perception of the usefulness of evaluations. Findings suggest that student’s evaluations have the potential to provide useful feedback to teachers.
Poster No. 27 “No Excuses” Charter Schools for Increasing Math and Literacy Achievement in Primary and Secondary Education: A Systematic Review and Meta-Analysis
In efforts to minimize the persistent racial and income-based educational achievement gaps, Charter schools in urban communities across the United States have begun employing a No Excuses philosophy, which focuses intensely on improving the math and literacy achievement of students. Enrolling students who come primarily from low-income and minority backgrounds, these schools feature high academic expectations, rigid and consistent disciplinary standards, extended instructional time, intensive teacher training, and parental involvement. Implementation of the No Excuses model is controversial among educators and scholars. We conduct a systematic review and meta-analysis of the impacts of No Excuses charter schools on math and literacy achievement. Preliminary results (N = 17) indicated that the overall mean effect size for both math (g = 0.26, p < 0.000) and literacy (g = 0.14, p < 0.002) outcomes after one year were positive and significant. Future directions will explore if these intervention effects hold over longer follow-up intervals.
Poster No. 28 Academic Interventions for Elementary and Middle School Students with Low Socioeconomic Status: A Systematic Review and Meta-Analysis
Martin Bøg, SFI, Denmark
Socioeconomic status is a major predictor of educational achievement. This systematic review and meta-analysis seeks to identify effective academic interventions for elementary and middle school students with low socioeconomic status. Included studies have used a treatment-control group design, were performed in OECD and EU countries, and measured achievement by standardized tests in mathematics or reading. The analysis included 101 studies performed during 2000-2014, 76 percent of which were randomized controlled trials. The effect sizes (ES) of many interventions indicate that it is possible to substantially improve educational achievement for the target group. Intervention components such as tutoring (ES = 0.36), feedback and progress monitoring (ES = 0.32), and cooperative learning (ES = 0.22) have average ES that are educationally important, statistically significant, and robust. There is also substantial variation in effect sizes, within and between components, which cannot be fully explained by observable study characteristics.
Poster No. 29 Policing and mental health: using research evidence to improve outcomes for all
Alison Booth, University of York, UK
Police officers with North Yorkshire Police and academics at the University of York have been working on the co-production of evidence to help improve decision making at both policy and practice levels. The key aim of the collaboration is to improve outcomes for victims, suspects, witnesses or others who come into contact with the Police and who have a mental vulnerability. Identification, recording, responding, reviewing and referring were the focal points for planning a series of systematic reviews. The process of discussing, scoping and agreeing questions provided a learning experience for all involved. Academics found themselves explaining research methods to police officers and in return finding out the day to day problems in practice for which solutions were needed. The result is a series of systematic reviews and evidence briefings that examine the evidence, and gaps, on what works on a range of topics presented in ways to ensure the messages reach those who need to know. This presentation will outline the learning from both perspectives of the collaboration; how the evolving reviews have been produced and the knowledge translated; and where it has been/will be used to improve outcomes for people who have a mental vulnerability.
Poster No. 30 Evidence Gap Map for interventions to prevent intimate partner violence
Jen Ludwig, 3ie, USA
The intimate partner violence prevention evidence gap map (IPV EGM) is a core element of a global knowledge platform currently under development by Assemblyfor, a strategy and design firm, as per request of a group of donors and implementing agencies working in this area. The IPV EGM provides easy access to rigorous evidence on interventions to prevent IPV, and identifies clusters of evidence that could indicate the opportunity for a systematic review. The evidence gap map groups types of IPV prevention interventions and the different outcomes expected of such programs along the causal chain and displays them in a matrix framework where each cell contains a list of impact evaluation studies with effect sizes of a given type of intervention over a type of outcome. Each record links to a summary of the study on 3ie’s impact evaluation repository. This gap map, complemented with a portfolio review of recent and ongoing initiatives in order to understand the need for evidence of implementing agencies, promises to be an useful tool to prioritize research investment in impact evaluation and systematic reviews; while being part of a global knowledge platform positions it well to respond to the need of implementing agencies to access evidence around current and future programmes.
Poster No. 31 Rapid Evidence Mapping (REM): Piloting a new tool to access evidence for informed policymaking
Jorge Miranda, 3ie, USA
The Rapid Evidence Mapping (REM) is a tool developed by 3ie to respond to the need of different stakeholders, particularly government agencies, to identify and access in a more systematic and timely manner the impact evaluations of interventions around a given topic. To identify relevant evidence for intervention design, policymakers typically face a common constraint: time. The need for expedited access to evidence creates a challenge on the supply side, since swiftness can easily come at the expense of rigor in both finding and interpreting the evidence. Methods for reviewing evidence that take less time, such as literature reviews, are typically not systematic, and involve a relatively high risk of bias. The rapid evidence mapping follows a similar search strategy as in evidence gap maps, but with a focus on finding the existing evidence rather than identifying the gaps in knowledge. The REM takes less time than a standard evidence gap map because it focuses in a narrower sets of questions for a small number of interventions and/or outcomes. A pilot opportunity for REM has been identified with CONEVAL in Mexico, in order to identify evidence around interventions to reduce food insecurity. The evidence map and the experience of this pilot are described in the poster limits, exploring the potential of the tool for policymakers’ access to evidence.
Poster No. 32 Risky Business: Review Quality and Related Factors within an Overview of Reviews
Emily Hennessy, Vanderbilt, USA
Systematic reviews and meta-analyses are highly influential for policy and practice; however, syntheses are prone to flawed conclusions if authors use inappropriate review methods. This study assessed the quality of the synthesis literature included in a review of reviews that examined the relationship between nutrition (e.g., dietary patterns) and mental health (e.g., depression). Additionally, we explored whether journal type, funding, and other publication factors were related to overall synthesis quality. Two authors extracted data from 50 reviews (27 meta-analyses, 23 systematic reviews) included in a parent overview using the ROBIS tool to assess review quality (Whiting et al., 2015). 49% of the reviews demonstrated high risk of bias (RoB). Approximately half of the reviews included publication type (published-only, n=17) or language (English-only, n=20) limitations. Only 12% published a protocol and 40% used a RoB scale appropriate for systematic reviews. These 50 syntheses suggest considerable interest in understanding relationships between nutrition and mental health; however, their impact is limited because authors used inappropriate methods in half of the reviews. Specific publication factors were related to high review quality. For example, being a Cochrane review (r=0.34), publication in a higher-impact journal (r=0.41), and conducting a meta-analysis (r=0.36) were significantly associated with overall low RoB (p < .05). There was no significant relationship between journal type (nutrition, n=20; mental health, n=13; other, n=16) or review funding and overall RoB. Our hope is that these findings contribute to improvements in review quality in this field and provide guidance for practitioners in selecting evidence.
Poster No. 33 Can abstract screening workload be reduced using text mining? User experiences of the tool Rayyan
Agneta Brolund, SBU, Sweden
During 2015, SBU (Swedish Agency for Health Technology Assessment and Assessment of Social Services) evaluated the efficiency of the text mining function of Rayyan, a screening tool developed by Qatar Computing Research Institute(1) . Rayyan was used screening abstracts for six reviews including two systematic reviews (SR). Results: Our evaluation shows that Rayyan helps the screener to identify the relevant records earlier in the screening process. 60 percent of the relevant records were found after screening only the first quarter of the search result. After screening the first half of the search result, 95 percent of the relevant records were identified. After screening three quarters of the total search records, 98 percent of the relevant records were found. This result is well in line with the developers own evaluations. Background: A reproducible and thorough literature search from different sources, designed to identify as many potentially relevant studies as possible, is one important step to minimize bias in the SR process. Taking into account the rapidly growing body of scientific literature, searches to systematic reviews often generates large search results which in turn means an increasing number of abstracts to be screened. SR have thus become very time and resource consuming to produce. The development of text mining tools to support abstract screening is today one of the most promising ways to reduce abstract reviewer’s workload. (1) Elmagarmid A1, Fedorowicz Z2, Hammady H1, Ilyas I3, Khabsa M4, Ouzzani In collaboration with Bahrain Cochrane Branch, University of Waterloo and Pennsylvania State University.
Poster No. 34 Publication Bias in Meta-Analysis: Its Meaning and Analysis
Sung-Dong Hwang, KNU, South Korea
As a new mode of research synthesis, meta-analysis has recently drawn much attention from researchers, having proved to produce less biased results than the traditional research synthesis method. However, scholars may be suspicious about the results of meta-analysis due to publication bias as it occurs when published studies represent a biased selection of the evidence. This study was designed to discuss the meaning of publication bias and conduct publication bias analysis using an example data set so as to provide researchers with a guideline for performing valid meta-analysis. The process of publication bias analysis begins with a careful examination of the effect sizes of individual studies and continues with testing funnel plot asymmetry. With both a graphical test and a statistical one, potential publication bias could not be ruled out. To deal with this bias, adjustments were made using the trim-and-fill method followed by the Copas selection model, and the original result was found to be valid and statistically robust even with the small bias detected. In the academic circle of meta-analysis, publication bias analysis is specified as one of the required items in some guidelines of reporting and evaluating the results of meta-analysis. Therefore, it is highly recommended to conduct publication bias analysis for the validity of meta-analysis as well as the integrity of individual meta-analytic studies.
Poster No. 35 From Smallholders to satellites – collecting better agricultural data
Suraj Nair, IFMR, India
The poster will summarize the usage of medium and high resolution satellite imagery, to obtain data for an ongoing evaluation of a financial services provider in India. The aim of this exercise is to answer the following questions: 1) Is it possible to capture treatment effect of a financial services intervention on agricultural outcomes through the use of high resolution and multi-temporal satellite imagery? 2) Can we use statistics (predictive and actual) extracted from high resolution imagery to validate self-reported data? For example – can we use satellite imagery to capture accurate data on crop yields? If yes, this addresses two major issues – 1) Validity of self reported data 2) Cost effectiveness in data collection – if reliable in capturing data, satellite imagery can allow particular kinds of data collection from really huge sample sizes in a cost effective manner.
Poster No. 36 Monitoring attribution Vs Contribution
Ehtisham Ul Hassan, Save the Children, UK
The world of international development has experienced a major cultural shift in evidence over the past decade. Donor requirements for evidence-based programming and rigorous evaluations, the celebration of 2015 as the “Year of Evaluation” as well as a general shift away from large scale programmes and towards system strengthening and testing innovative approaches to solve problems, are some of the trends that have come together to build momentum for generating and using robust evidence more systematically. Save the Children is a global movement of 30,000 staff in 130 countries, with a decentralised structure and multiple lines of accountability. As an organisation, we have implemented, or are in the process of implementing various institutional approaches to generate and use evidence more systematically across the movement. We would like to present some of those approaches from the perspective of our work on global health. These include, but are not restricted to: • Using unrestricted funding to test innovative approaches and fill evidence gaps; • Improving the quality of the hundreds of evaluations we commission every year; Designing and rolling out knowledge management systems, including (1) a global Health E-Library designed to share knowledge and evidence emanating from our programmes around the world, and (2) a global Health MIS designed to collect consistent data on global indicators from the health areas we support around the world; Integrating evaluation and research plans in the design of new programmes, with a clear learning agenda; Strengthening our monitoring, evaluation and research capacity at all levels of the organisation.
Poster No. 37 3ie systematic review database: bringing together systematic reviews in international development
Jennifer Stevenson, Ami Bhavsar, Birte Snilstveit
This poster presentation will draw on specific examples of 3ie’s Evidence Gap Maps (EGMs) to illustrate how this dynamic and interactive platform can be used to easily access and explore 3ie’s database of systematic reviews, the only database in the world that brings together systematic reviews in international development.
3ie EGMs, as the name suggests, map out thematic collections of evidence covering a particular issue. They identify evidence from systematic reviews and impact evaluations and provide a graphical display of areas with strong, weak or non-existent evidence on the effects of development programmes and interventions. EGMs are useful to policy makers and development practitioners looking for evidence to inform policies and programmes. For donors and researchers, these map scan help identify areas where there is an urgent need for more rigorous research evidence. 3ie has recently developed a new interactive and dynamic online platform which allows users to explore the evidence included in a particular EGM.
These EGMs are a new and innovative way of exploring 3ie’s database of systematic reviews in international development. The EGMs display quality appraisals from the database and link to user-friendly summaries of the systematic reviews in the database. The database itself is an open access, searchable site that is a one-stop shop for policymakers, practitioners and academics looking for systematic reviews of evidence on social and economic development interventions in low- and middle-income countries. In addition to providing an overview of the reviews in the field, the database provides quality appraisal and plain language summaries.