Achieving better outcomes from environmental policy is more than just good science
Decision science is only one of many factors that may be relevant in the environmental-policy process.
Decision science has enormous potential to make policies more effective at delivering outcomes that are highly valued by the community.
There are a number of challenges for decision science if it is to meet its potential.
There are many examples where decision science has met these challenges and made important contributions to environmental policy.
Key success factors for decision scientists to contribute to environmental policy include good communication, pragmatism, patience, persistence, a capacity to develop meaningful relationships with policy makers, and choosing the right decision-science projects in the first place – ones that are likely to make a difference.
The creation and design of an environmental policy involves a myriad of decisions. Here are just a few of them:
What’s the need?
Should there be a policy at all? Which specific environmental issues should be covered by the policy? What goals or targets should be set for the policy? Over what spatial scale should the policy operate? Over what time frame should it operate?
Where’s the emphasis?
Should effort and funding be targeted to key priority issues or locations, or spread relatively evenly across issues or locations?
What’s the approach?
Should the policy involve public education? If so, who needs educating, what about, and how? Should the policy involve regulation of the actions of individuals or businesses? If so, how large should the fines for non-compliance be? How much should be invested in inspection and testing of compliance?
Should the policy involve payments to reward positive environmental behaviour? If so, how large and how frequent should the payments be? Should people be rewarded for approved actions, or only if those actions result in positive environmental changes? Is research or a pilot study needed prior to full implementation? What approach to monitoring the results of the policy should be used?
And, of course, each question (of which the above is just a logical beginning) spawns its own set of sub-questions, which in turn generate even more, and so on.
The relevance of decision science
While easy to pose, most of these questions are not straightforward to answer. The alternative options to each decision may have multiple consequences, some of which are hard to predict. In most cases, there is high uncertainty about the environmental consequences of the various policy options. Relevant considerations include social, administrative, political and financial issues, not just environmental consequences. These different issues need to be drawn together and weighed up, but the best way to do that may not be apparent.
Decision science is definitely relevant to this. It provides a set of concepts, models and tools that have been developed to help address complex decisions under uncertainty. It aims to formulate each decision problem into a coherent and logical structure, so that it is easier to think about the options. In so doing, the consequences of these options become more transparent and easier to compare.
This helps decision makers compare and rank policy options in ways that are logically sound. While this seems like a sensible and rational thing to do, it’s something that environmental managers and policy makers often fail to practice.
For example, we know a lot about what are good approaches to prioritisation and yet this knowledge is frequently not applied (eg, consider how prioritisation metrics are often poorly constructed, see Decision Point #82). Or, as another example, a lot of decision science has gone into how biodiversity offsets should be estimated (see Decision Point #69, p10,11) but these lessons are sometimes not well accounted for in policy.
If done well, decision science should also help decision makers avoid some of the biases that plague most decision making (see the box on bias and NRM and Decision Point #93). On the face of it, decision science should be highly useful to environmental policy makers.
Nevertheless, for a range of reasons (some of which are discussed here), most environmental policy decisions are made without help from decision science. There are many excellent examples where it has played a helpful role, or even a pivotal role, but they are the exceptions. The more common situation is that prioritisations are done poorly or not at all, that decisions are made without thorough and systematic analysis of the options, and that ongoing monitoring of the outcomes of our environmental decisions is done inadequately.
In this article I will reflect on the nexus between environmental policy and decision science. What aspects of the policy process make it difficult for decision science to play a bigger role? I’ll briefly present some examples of environmental policy development with varying levels of use of tools developed by decision scientists. And, for those decision scientists who aspire to play a bigger role in environmental policy, I’ll discuss strategies they may want to consider.
Challenges for decision scientists
In my time working in the field of environmental decision science, working with researchers from a variety of disciplines, I have identified a range of challenges that have increased the difficulty of decision science influencing policy outcomes. These include the following:
Non-scientific considerations matter: Sometimes scientific information may be known to policy makers, but the decision reached may still appear to be inconsistent with the science. This may or may not be a concern. It may simply reflect that policy makers and managers must consider additional factors, such as legal mandates, societal desires, economic benefits and costs, rights, distributional equity and procedural fairness.
Hidden agendas: There may be political or bureaucratic objectives unrelated to the public interest, so that research that seeks to advance the public interest is not given the priority it deserves.
Policy fashions and crises: Policy attention tends to be directed to certain issues with high currency, often issues where there is a perceived crisis, and this may leave little scope for research (or any other input) to influence policy in an area that is not currently high on the agenda.
On the other hand, in certain circumstances, it may be easier to foster policy change in an area that is not in the public spotlight, at least from the perspective of avoiding controversy. This depends on convincing policy makers that a change is justified, which is likely to be difficult for non-topical issues.
Difficulty getting access: Policy makers often prefer local trusted sources for information. This is understandable, given the flood of information that policy makers can face on some issues, but it does not ensure that the most appropriate information is used.
Distrust: There may be suspicion about motivations of researchers, so that they are treated as just another interest or lobby group. Even without overt advocacy, some environmental scientists tend to intertwine facts and values, and this affects the perceived independence of their scientific advice.
Lack of expertise: In some policy agencies there is rapid turnover or movement of staff leading to a lack of expertise in responsible staff and a lack of knowledge of relevant research and researchers. A culture or belief may develop in government agencies that detailed subject knowledge is not necessary.
Another set of challenges arises from differences between the circumstances, cultures and reward systems of researchers and policy makers.
Different outcomes sought: Researchers place a high value on knowledge and innovation while policy officers seek to advance the public interest, to capture resources, and to please their political masters, who are primarily concerned about being reelected.
Source of recognition: The sources of recognition for practitioners are different: from an administrative or political master in the case of policy, and from peers in the case of research.
Achievements rewarded: Policy officers tend to be promoted based on their ability to successfully implement desired policy programs, while researchers are promoted according to their productivity of research outputs, especially of those judged to be high in scientific quality.
Controversy vs compromise: Policy officers aim to resolve management or political problems with minimal controversy, making pragmatic compromises wherever necessary, whereas a healthy scientific discipline thrives on open debate, and will not compromise the truth.
Communication: Researchers and policy makers speak different languages, with different acronyms and jargon and different hidden assumptions. Scientific communication can be hard to understand even between different scientific disciplines. Policy officers deal mainly with very brief, simply written and highly interpreted/synthesised material conveying only essentials with a focus on practical implications and recommendations. On the other hand, even brief scientific writing is considerably more detailed and qualified and much research deals with practical implications as an afterthought, if at all.
Time frame: Once the decision has been made to develop or change a policy, the time frame for the process is usually short. This can place participants under great pressure, with little time for careful consideration or collection of information. Research is generally slow and unresponsive to urgent policy needs, although it can be responsive in the longer term.
Supply vs demand: Policy usually addresses a problem identified by someone else (demand-driven), while the directions of research are often selected by researchers (supply-driven), particularly in a university environment. In a government research agency, research directions may be a mix of supply- and demand-driven.
Complexity vs simplicity: Policy officers prefer simple, straightforward advice with few, if any, caveats, whereas researchers tend to enjoy unravelling the full complexity of an issue, with all caveats highlighted.
Specialisation vs breadth: University training and the academic reward system encourages narrow specialisation, whereas policy officers need to consider a broad range of factors.
People focus: Policy involves intensive interaction among diverse groups of people, requiring highly developed social skills, while for some scientists working intensively with people is not comfortable. Of course, good interpersonal skills are valuable in both realms, but I believe that there is a difference in the degree of their importance.
Given these many challenges, it might seem remarkable that research ever plays a meaningful role in the development of policy. Often it doesn’t, but sometimes it does. Consider the following case studies.
Case studies of impact
Fiona Gibson (working with a range of colleagues, including me) recently analysed the factors that facilitated or inhibited the use of rigorous decision-support tools in Australian environmental policy and management (Gibson et al, 2017). We selected seven case studies of policy development and conducted personal interviews with managers and policy makers who had been directly involved.
The case studies we reviewed involved:
The Southern and Eastern Scalefish and Shark Fishery policy (perceived by interviewees to be an example of where decision-support tools were much used = high);
The Representative Areas Program (high);
The South West Marine Reserve Network (low);
The National Reserve System (none); and
Three examples of Threatened Species Protection in different jurisdictions, Australian national (none), New Zealand (moderate) and New South Wales (moderate).
The selection of case studies was not intended to be representative of all possible conservation policies. However, they were a diverse selection and provided insights that are likely to be transferable to other case studies and policies.
Drawing from the existing literature, the researchers identified a range of factors that are likely to promote or prevent the uptake of decision support tools in environmental and conservation programs. I’d suggest that these can be interpreted as factors influencing the uptake of decision science more generally, not just decision-support tools.
- Presence of a champion for the tool within the agency
- Presence of an advocate for the tool outside of the agency
- Existence of a relationship between agency staff and tool experts
- Presence of large numbers of stakeholder groups affected by the policy outcome
- Ability of the tool to deal with missing information
- Whether the tool can be applied quickly
- Whether the policy process allows adequate time for tool use
- Whether the tool capabilities align with policy objectives and policy operation
These factors were used to develop the questions used in the policy-maker interviews. Those factors rated as most important by the interviewed policy officers and managers were the alignment of a decision support tool with policy objectives and operation, and its ability to be useful even when there is missing data.
The issue of alignment with objectives arose because in some cases a tool was perceived to be focused on environmental outcomes and not able to adequately represent non-environmental policy objectives.
Lack of complete data is almost always a challenge when considering policy options, and the tools that were viewed favourably and utilised more extensively tended to be viewed as being amenable to strategies such as expert elicitation to fill data gaps.
The least important factors, at least for these interviewees, were the presence of a champion of the decision-support tool within the management agency, and the time required to apply the tool. The finding regarding an internal champion conflicts directly with some previous studies where the presence of a champion was critical to effective use of the tool (eg, consider Richard Maloney’s discussion on success factors behind the implementation of the Project Prioritisation Protocol in which he cites champions as being of critical importance, see Decision Point #76).
Interestingly, the interviews revealed additional important factors that had not previously been identified in the literature. These included: the existence of multiple (potentially unstated) policy objectives, and the autonomy of the agency. Where there were two conflicting objectives (eg, conservation and sustainable economic use of the resource), some interviewees lacked confidence that the tool could handle this, or would give the competing objectives weights consistent with those of the policy makers. It was felt by some interviewees that relatively autonomous government agencies are less prone to intervention by a government minister concerned with the politics of an issue. This lower risk of interference made it easier for transparent and systematic decision processes to operate. The particular autonomous bodies in these case studies also had a greater emphasis on day-to-day engagement with stakeholders, such that the potential benefits of a decision support tool in enhancing engagement may have been more apparent.
The findings of Gibson et al (2017) reflect the perceptions of the interviewees. Notably, it appeared that some of the most negative perceptions about particular tools may have arisen from misunderstandings about the tools, or a lack of capability to use the tool appropriately. There were examples where a tool that was rated negatively by one agency for a particular criterion (eg, ability to cope with missing data), was rated positively by another agency for the same criterion, and the other agency had actually used it successfully. This highlights the importance of clear communication channels and the provision of training and support to allow the decision tools to deliver their potential.
So, if you are a decision scientist, how might you make a bigger impact when it comes to the development of policy? In the box on ‘decision scientists in the policy arena’ I outline a set of strategies you might consider. It is is not a prescriptive or complete list but these 13 strategies are important considerations for any researcher wanting to influence policy.
Bias and natural resource management
Like all people, environmental managers and natural resource managers are subject to bias. Sayed Iftekhar and David Pannell recently explored the influence of bias in natural resource management (NRM) and found that it’s possible to improve our performance if we recognise these biases and work to reduce them (Iftekhar & Pannell, 2015).
The take home message from their study is that NRM agencies need to be aware of the influence of biases when management decisions are undertaken. There are many things they can do that will help imit the impact of bias.
First, agencies need to promote a culture of learning. It needs to be recognized that both successful and failed projects generate valuable information about the management interventions. This could be done by providing appropriate incentives (tangible and intangible) for the managers and decision makers to consider the full range of options before making any decision, or asking managers to justify their decisions to external parties.
Second, adoption of a decision support system could facilitate retention and storing of relevant information. It may also make learning from past projects easier and help in systematic evidence-based decision making. Of course, relevant staff should be adequately trained and properly incentivized to use such systems.
Third, conducting benefit-cost analyses of planned options would help to refine and prioritize the options during the design phase of an adaptive management cycle. Benefitcost analysis provides a systematic framework to include all relevant costs and benefits (both market and nonmarket goods and services) related to a project.
Fourth, involvement of external third-party reviewers may also help in designing more realistic and feasible projects.
And, finally, scenario analysis should be conducted as part of the assessment and design phase to anticipate the expected outcomes of different decision options.
It is advisable to consider the likely impacts of different types of biases, and the effectiveness of potential remedial measures before making any final recommendation for use in decision making for natural resources.
To inform policy, speak to policy makers
How can researchers and policy-makers work together more effectively to narrow the gap between science and policy? Influencing policy is not a process as easily definable as, say, publishing a scientific paper. Policy outputs appear in a number of forms over a variety of timeframes, and are rarely tracked back to single meetings or workshops. The very process of influencing policy makers is difficult to define. Anyone who thinks it’s a rational, linear process probably hasn’t tried it.
About a decade ago a group of decision scientists from the Applied Environmental Decisions Analysis Hub (the precursor of CEED) ran a workshop with policy makers from the Department of the Environment to discuss: How can we communicate research discoveries to policy makers and managers at minimal cost? How can we find out what research questions may deliver answers needed by policy makers and managers? The answers emerging from that discussion can be read in Gibbons et al, 2008 or read about the discussion in Decision Point #25, p8.
It was agreed that personal relationships and networks were key to effectively influencing the development of policy. Activities that would serve to help foster effective relationships and networks include the creation of policy buddies (ie, researchers nominating policy people they need to interact with on specific topics), having research staff sit in government departments and vice-versa, reviewing rewards to researchers for making the extra effort to influence policy (currently there are few), creating mechanisms by which policy makers can alert researchers to their specific concerns, and contact mapping (ie, figuring out just who is in whose network).
Decision scientists in the policy arena
The following strategies, adapted from Pannell and Roberts (2009) and Gibbons et al (2008), are provided as food for thought for decision scientists. They are not in order of priority and they shouldn’t be considered as a prescriptive recipe. Rather, think of them more as ingredients. It depends on how you combine them as to what type of cake you create.
- Develop relationships with policy makers: Attempt to establish a high level of mutual understanding and trust. Information needs to flow in both directions. Only that way can you understand their perspectives and needs, and they understand your contribution.
- Research is not enough: Appreciate that good science is needed but is not sufficient for decision makers. In considering policy options, policy makers will probably be more concerned with social, economic, political or administrative aspects than with technical aspects of science.
- Practice excellent communication: In communications, recognise the lack of time that policy makers have. Be very brief, focus on clear messages, use simple language, free of jargon, using a mixture of approaches. Written material is useful but is not sufficient. Even more important is effective verbal communication.
- Simplicity is essential: As far as possible, the solutions one offers need to be simple, transparent and understandable. Policy makers are likely to be suspicious of solutions that rely on complex and opaque computer models (although the success of climate modellers in influencing the debate about climate change shows that models can be used if their message is sufficiently clear and simple).
- Work with intended users: This will help to ensure that the solution being proposed is in fact practical and sufficiently simple. It will help to make sure that their issue of concern is addressed in a way that is relevant to them. When attempting to convince policy makers, it helps to be able to demonstrate that the solutions being proposed are already in use in the real world.
- Distinguish between knowledge and values: Be clear that the values that policy attempts to enhance are based on the desires of the community, not those of researchers. It is acceptable for research to deal with values (eg, studies of the non-market values of environmental outcomes) but one must be clear that policy makers will have their own views about the values. Traditionally, science deals primarily with knowledge rather than values, but of course scientists are influenced by their own values.
- Be pragmatic: One has to accept compromise, and it may be necessary to make conscious decisions about where you can and cannot afford to give ground.
- Be patient and persistent: Your work may not be influential at first, but its acceptance could grow over time if you are persistent. Establish networks and build support for your ideas over time. Repetition is essential, even to people who are already on your side.
- Be resilient: Numerous problems, frustration and setbacks will arise. People with vested interests in the status quo will actively resist proposals for change. These people may be insiders to the policy organisation and so have better access to decision makers than outside researchers do.
- Timeliness is important: Be prepared to respond quickly to requests for information. Policy makers cannot wait for additional research.
- Find a champion: Elsewhere I’ve noted mixed evidence on this, but at least in some cases it is likely to be worth cultivating a champion for your work within the policy organisation.
- Avoid any appearance of vested interest: Do not present findings and seek funds at the same time.
- Choose your research/analysis topics well: It doesn’t matter how good your communication is or how strong your policy networks are if the topics you are researching or analysing are not important to policy makers, or are not providing information that can help them. This seems obvious, but is sometimes not considered sufficiently when decision scientists are planning their own work.
Measuring science performance and policy impact
How effective has CEED been in influencing policy? People close to CEED are well aware that CEED researchers have made many important contributions to environmental policy and management. However, measuring these impacts is notoriously difficult.
Even if environmental policy or management have changed, it can be very difficult to know the extent to which the change can be attributed to research. Policy development is a complex and messy process, with many players involved. And the eventual impacts of policy change on environmental outcomes are often uncertain, unclear and delayed. Maybe that is why quantitative analyses of the impact of research on environmental management, environmental policy and environmental outcomes are rare.
Working with a small team from UWA, we tried to capture CEED’s impacts, as well as documenting its academic outputs, collaborations and citations (Thamo et al, 2018). We hope our approach might assist other environmental research networks and centres to measure the influence of their own research efforts.
The CEED impact evaluation collected data on 87 CEED projects and discussed nine of these in detail. It found that there was high academic performance in many of CEED’s outputs and high policy and management impact in some projects, but not all.
A number of important lessons and implications were identified in the impact analysis.
There have been many studies on the factors that underpin research impact, and most of them highlight the importance of engagement and good relationships with research users, the quality of communication (eg, see Decision Point #73 and Decision Point #74).
However, the evaluation found that just as important was what research is actually done. If research is not providing insight or tools that are actually useful to policy makers or managers, even strong relationships and excellent communications won’t lead to impact.
Therefore, developing a research culture that values impact and considers how it may be achieved prior to the selection of research projects is potentially important. The role of the centre leadership team in this is critical. Embedding impact into the culture of a centre probably happens more effectively if expertise in research evaluation is available internally, either through training or appointments.
A challenge in conducting this analysis was obtaining information related to engagement and impact. There may be merits in institutionalising the collection of impact-related data from early in the life of a new research centre.
In this analysis, there was little correlation between academic performance and impact on policy and management. It should not be presumed that the most impactful projects will be those of greatest academic performance.
Finally, there are often long time lags between commencing research and delivering impact – decades in many cases. Therefore, there is a need to allow the longest possible time lag when assessing impact. On shorter timescales, it may be possible to detect engagement, but not the full impact that will eventually result.
More info: David Pannell email@example.com
References and recommended reading
Gibbons P, C Zammit, K Youngentob, HP Possingham, DB Lindenmayer, S Bekessy, M Burgman, M Colyvan, M Considine, A Felton, RJ Hobbs, K Hurley, C McAlpine, MA McCarthy, J Moore, D Robinson, D Salt & B Wintle B (2008). Some practical suggestions for improving engagement between researchers and policy-makers in natural resource management. Ecological Management and Restoration 9: 182-186.
For a discussion on this paper see Decision Point #25, p8.
Gibson FL, AA Rogers, ADM Smith, A Roberts, HP Possingham, M McCarthy & DJ Pannell (2017). Factors influencing the use of decision support tools in the development and design of conservation policy. Environmental Science and Policy 70: 1-8.
Iftekhar MS & DJ Pannell (2015). ‘Biases’ in adaptive natural resource management. Conservation Letters 8: 388-396.
For a discussion on this paper see Decision Point #93
Kahneman D (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux, New York. Maron M, JR Rhodes & P Gibbons (2013). Calculating the benefit of conservation actions. Conservation Letters 6: 359-367.
For a discussion on this paper see Decision Point #69, p10,11.
May J, RJ Hobbs & LE Valentine (2017). Are offsets effective? An evaluation of recent environmental offsets in Western Australia. Biological Conservation 206: 249–257.
McDonald JA, J Carwardine, LN Joseph, CJ Klein, TM Rout, JEM Watson, ST Garnett, MA McCarthy & HP Possingham (2015). Improving policy efficiency and effectiveness to save more species: A case study of the mega-diverse country Australia. Biological Conservation 182: 102-108.
Pannell DJ & AM Roberts (2009). Conducting and delivering integrated research to influence land-use policy: salinity policy in Australia. Environmental Science and Policy 12: 1088-1099.
Pannell DJ & FL Gibson (2016). The environmental cost of using poor decision metrics to prioritise environmental projects. Conservation Biology 30: 382-391.
For a discussion on this paper see Decision Point #82
Schwartz MW, CN Cook, RL Pressey, AS Pullin, MC Runge, N Salafsky, WJ Sutherland & MA Williamson (2017). Decision support frameworks and tools for conservation. Conservation Letters 11 https://doi.org/10.1111/conl.12385
Clark RN, EE Meidinger, G Miller, J Rayner, M Layseca, S Monreal, J Fernandez & MA Shannon (1998). Integrating science and policy in natural resource management: lessons and opportunities from North America. Gen. Tech. Rep. PNWGTR-441. Portland, OR: U.S. Department of Agriculture, Forest Service, Pacific Northwest Research Station.
Mumford T & N Harvey (2014). Champions as influencers of science uptake into Australian coastal zone policy. Coastal Management 42: 495-511.
Rose DC, N Mukherjee, BI Simmons, ER Tew, RJ Robertson, ABM Vadrot, R Doubleday & WJ Sutherland (2017). Policy windows for the environment: Tips for improving the uptake of scientific knowledge. Environmental Science and Policy https://doi.org/10.1016/j.envsci.2017.07.013
Thamo T, T Harold, M Polyakov & D Pannell (2018). Assessment of Engagement and Impact for the ARC Centre of Excellence for Environmental Decisions. University of Western Australia.