Usage and Impact Monitoring
Dear Usage and Impact Monitoring Group,
In October - November 2011 the Institute of Development Studies' Impact and Learning Team held a fruitful e-discussion: How to evaluate the impact of knowledge brokering work. The aim of this discussion was to harvest what we know about evaluation - what's been done, What works and what's missing. The details below were contained in an email sent early February 2012 providing a summary and resources from the KBF e-discussion on evaluating the impact of knowledge brokering work:
The summary and analysis of the e-discussion about evaluating the impact of knowledge brokering work is now available for download here.
Summary of e-discussion The paper summarises a rich discussion which took place on KBF during October and November 2011. The discussion focused on how to evaluate the impact of Knowledge Brokering work. Members shared experience and insights about evaluating impact in order to be better able to evaluate IDS's own work and build greater understanding of the potential of the sector.
The discussion reached some areas of consensus and began to explore other areas outlined below. Each section contains links to the wide range of resources shared:
- Understanding the purpose of your work is the basis for evaluation: the debate stressed the value of both generic planning frameworks (e.g. Theory of Change, Social Framework) and frameworks specific to knowledge brokering
- Be clear why you are evaluating your work: there are various reasons for undertaking an evaluation that will shape what questions you ask and what methods you use
- Understand what you mean by impact: the changes that KBs are seeking to generate include changes in effectiveness, policies, behaviour, indicators of wellbeing and different kinds of impact
- The challenge of indicators and metrics: there are currently few good indicators and metrics for KB work - indicators will depend on what you are trying to achieve, but there is potential to collectively develop thinking in this area
- Methodologies and approaches: there are a range of different approaches that KBs can use, from case studies and surveys, to propensity score matching.
This debate is intended to feed into and catalyse future action to help strengthen our individual and collective understanding of how to evaluate the impact of KB work. Ideas for future activities that are worth considering include:
- Sharing what we are actually doing, methods indicators metrics etc
- Build a sector wide bank of impact case studies, possibly linked to a theory of change
- Create a guide that suggests different evaluation approaches/questions/metrics and methods that are appropriate to understanding different kinds of impact.
Resources on evaluating the impact of knowledge brokering work The resources shared in this discussion have also been collated on the KBF wiki (http://www.knowledgebrokersforum.org/wiki) and are available to access by topic, as indicated below:
Guides to and definitions of evaluation http://www.knowledgebrokersforum.org/wiki/item/guides-to-and-definitions-of-evaluation
- M & E specific to knowledge brokers and intermediaries
- Evaluating research impact/communication
- Evaluating policy influence
- Evaluating agricultural extension work
- The need/role for knowledge intermediaries
- Measuring the impact of knowledge management
- Ideas for self-tagging
- Theory of change
- General M&E resources
- Realistic evaluation
- M&E/evaluation frameworks
- Reflexive Monitoring in Action
You can also download a pdf version of M&E resources (http://bit.ly/wAzLZc) collated from the e-discussion.
We hope that this summary and the associated resources will prove a useful contribution to people working in this area. We would welcome reflections and feedback on the ideas in this paper and suggestions for how they could be taken forward, particularly from others working in related areas who would like to collaborate. We also hope you will draw on the resources shared and continue to share information and resources on evaluation of KB work and other areas relevant to KBs via the platform, wiki, and email list.
Yaso Kunaratnam Network and Partnerships Convenor IDS Knowledge Services Institute of Development Studies, U.K
Catherine Fisher Capacity Support Coordinator Impact and Learning Team Institute of Development Studies, UK