Does the pressure for impact compromise research?

CIFOR scientist George Schoneveld interviews jatropha farmers in Zambia reportedly abandoned by an overly-ambitious outgrower scheme. Photo by Jeff Walker/CIFOR.

BOGOR, Indonesia (28 June, 2011)_When is the right time to talk publicly about new findings from a research project? This is subject I have been thinking about a lot recently. This past month, I have participated in a number of events designed to disseminate research results to relevant policy arenas and practitioner communities.

These included side events at the UNFCCC SBSTA in Bonn (one on agriculture as a driver of deforestation, and one specifically on bioenergy crops), a conference in London to launch preliminary findings of the Poverty and Environment Network (PEN) study on the contribution of forests to rural livelihoods, and the Oslo REDD Exchange to share early results of CIFOR’s Global Comparative Study on REDD+ (GCS-REDD).

In several cases, the research findings being shared at these events were based on partial data and/or preliminary analysis, and had not yet been published or subjected to peer review.  In the wake of Climategate (the 2009 controversy surrounding charges that climate scientists had manipulated data), this would appear to be risky.  On the other hand, the significant social and environmental impacts of forest loss underscore an urgency to share findings as soon as possible – especially those that could help avert harm to vulnerable communities and ecosystems. What is the right balance between prudence and responsiveness?

This is but one aspect of a larger set of challenges facing “research for development” organizations such as mine, the Center for International Forestry Research (CIFOR), and other member centers of the Consortium for International Agricultural Research (CGIAR).  Most of our funding comes from development agencies, which are in turn accountable to their taxpayers for showing results. Our funders often tell us as we negotiate new grants, “We want the answers to all the questions you pose by yesterday.”

In recent years, the pressure to demonstrate impact has intensified. Proposals for new CGIAR Research Programs (CRPs) must include detailed – and if possible, quantified — “impact pathways” that map out precisely how research findings will be translated into on-the-ground results, such as higher rural incomes or reduced deforestation. The CIFOR-led CRP on Forests, Trees and Agroforestry, approved in April, suggests some rather ambitious targets.

Although scientists tend to resist these exercises – for fear of being held accountable for outcomes beyond their control — mapping out such impact pathways is a good discipline.  It forces us to be explicit in our assumptions about who will use our research results, and how such results could plausibly change behavior in ways that could lead to ultimate impacts.  And yet the pressure to show impact could have unintended negative consequences on the research enterprise itself.

At the PEN conference in London earlier this month, several panelists engaged in a lively exchange on this dilemma.  One stressed the importance of each study being clear about its intended impact from the outset, while another warned that this could lead to an inappropriate shortening of the time horizons and narrowing of the scope of research agendas.  Indeed, tightly focused studies targeted to particular decision points are easier to “sell” in term of impact.  But might the pressure for impact push research organizations inappropriately far into the consulting business?

Scientists often rightly argue that an important feature of research is that you don’t know what the outcome is going to be when you start, and are quick to offer anecdotes describing a research project that had impact, but in a completely different area than was initially envisioned.

Contributing to this set of challenges is the difficulty of attributing impact, especially when it comes to policy research, and research designed to influence natural resources management (NRM) rather than (for example) to increase the yield of a particular grain crop.  The CGIAR’s Independent Science and Partnership Council has recently initiated a study of the impacts of NRM research – which have proven difficult to document in the past — with the hunch that different methods may be necessary to capture those impacts.

At CIFOR, we are attempting to position our research agenda so that it optimizes between investing in and publishing the results of rigorous, long-term, globally-comparative studies (such as PEN and GCS-REDD), and spinning off interim results and conducting the one-off analyses necessary to respond to the shorter-term needs of our stakeholders.  As demand for our “product” has increased with the recent spike in global interest in forests and climate change, maintaining the right balance has gotten harder.

It’s a problem that’s never solved, only managed, but it’s a good problem to have.


  • http://www.unep.org/eou Michael Spilsbury

    I enjoyed this discussion. It hasn’t really changed since my days as a CIFOR scientist!

    “Although scientists tend to resist these exercises – for fear of being held accountable for outcomes beyond their control — mapping out such impact pathways is a good discipline.”

    I find that the whole debate about the level to which staff are ‘held accountable’ is often confused.

    As you point out, it is not possible to hold staff accountable for outcomes (changes in the behaviour of others) it is, however, reasonable to hold staff to account for doing the things that are most likely to move outputs on towards outcomes and further along the pathway(s) (ie. activities at a level beyond outputs).

    This means that impact pathways must articulate the activites that help to effect changes between the different ‘steps’ in the causal chain (we call them impact drivers). Whereas ‘assumptions’ for each step in the pathway capture the risks that may jeopardise / constrain that change.

    I am trying to get these ideas adopted in UNEP to ensure that work planning is more results-oriented and management of staff can be more results-based.

    Some 15 years after we managed it at CIFOR, UNEP might soon start to routinely use the concept of impact pathways in all of its work planning!

    best regards
    Mike

  • Joseph Opio-Odongo

    While the pressure for impact appears to compromise research, I find it beneficial, especially to the policy-cum-capacity oriented research projects, in terms of forcing the scientists to be clear on the theory of change that drives the scientific enterprise.

    As you mention in the blog, it is essential to be clear on “who will use our research results, and how such results could plausibly change behavior in ways that could lead to ultimate impacts.” That research scientists, therefore, have to clear on how the research process and ultimate findings are expected to contribute to the sought change in behavior of well-defined entities; including changes in the behaviors of the prospective consumers of the research results.

    In my view, sharing preliminary research findings can do more good that harm if it is based on a well-crafted information strategy that uses emerging lessons and stakeholder/partner feedback for purposes of steering the research project in pursuit of expected outcomes. It also provides an opportunity to know if there are other projects poised to making similar contributions and their unfolding experiences.

    Bearing in mind the timeliness of some crucial policy decisions, sharing preliminary, but credible, findings, with policy makers can avert the problem of a “missed opportunity”. To hold on until one has the solid research results may translate into missing the opportunity to influence the policy-making process.

  • Diji Chandrasekharan

    I agree with Joseph about averting the problem of a “missed opportunity”. I also think that research pieces that aim to provide quantitative or qualitative analysis but do not clearly articulate their assumptions and shortcomings in terms of data (i.e., not provide a sense of the margin of error) are doing an injustice to the end user. So when we are trying to inform processes with prelimiary information, an element to making it credible is sharing the assumptions and shortcomings rather than just a “bumper sticker message” that can divert attention and resources from the real issue. Another element that provides credibility to “preliminary findings” is when the research process is adopting an informal “peer review” process throughout – e.g., when the methodology is peer reviewed by other experts, the initial findings are discussed with other experts before being released etc. To keep science and research relevant in policy dialogue we need to be answering questions that people are asking in a timely manner. Of course, easier said then done and as Francis says it requires finding that right balance.

  • Rolf Czeskleba-Dupont

    As Joseph Opio-Odongo says, a good information strategy contains the two elements of (a) using emerging lessons and (b) stakeholder/partner feedback. When Diji Chandrasekharan in connection with that recommends an ‘informal “peer review” process’, this discussion points clearly into the direction of, what has been called ‘post-normal science’ (developed by Funtowicz and Ravetz in the 1990s as extension of both science and consultancy). It is by definition transcending the limits of normal science from the Kuhn tradition. And wants to extend this approach by (a) extending the notion of facts in a given realm, see ‘emerging lessons’ above; and (b) extending the notion of ‘peers’ to include relevant knowledge of others than appointed peers e.g. in editorial boards. If the experience of the Copenhagen climate stall-mate has shown anything, it must be that the old-timer use of peer-reviewed articles as the sole criterion of acceptance of scientific truth is not enough to make the transfer of critical knowledge from science to practice in questions of accelerating climate change. This was e.g. the case with a critical session on IUFROs findings upon the climate consequences of a more and more probable scenario of +2,5 degrees C of global warming (Fischlin et al. in IPCC, working group 2 report, ch.4), namely that the CO2 sink function of forests globally may tip into a source. The session held at FOREST DAY 3, the official side event to COP15, has never got its message through. So, there are research findings of importance for climate policy which are left out of the picture, which the political-administrative apparatuses want to regard at all – and consequently are made to realize. The whole gamut of giving CO2-credits in Europe (e.g. Vattenfalls projects in Berlin) for the combustion of fuelwood from African countries (or from North America or Russia) would collapse if Fischlin’s truth was heard! At any rate, compliance with formal peer-reviews is no guarantee for the ‘necessary’ transfer of new knowledge form science into practice – necessary because of the urgency of the issues involved (high stakes) and its inherent uncertainty, when decades or centuries have to be dealt with (re. forest growth and old forests as sinks). This new knowledge is, however, better than wrong security given by myths (such as that the wood burnt is only releasing the same amount of CO2 as it has bound in-before; this truth applies also to fossil fuels!). As Immanuel Kant stated: Sapere aude (have the courage to know).

  • Leodegardo M. Pruna

    The professional way to treat the issue of prudence against responsiveness in the sharing the output of research work is to adhere to some moral and legal guidelines, to wit:
    1. A research which is in progress should be kept confidential.
    2. A completed research should immediately be registered for a patent or copyright, which ever is applicable, to avoid false claims on the research.
    3. If two or more entities are known to be engaged in the same research work, said parties could be encouraged to come to an agreement to facilitate conclusion of the research by way of staff meetings and sharing of information not contrary to some confidentiality clause agreed upon.
    4. It is presumed that only completed research would be beneficial to users of the research. The prospective users of the work should not be deprived from the use of the research especially when the same is for the uplift of human welfare.
    5. Some other conditions which the parties may agree which are not contrary to law and ethics.