Learning about adaptive management

Good decisions produce results AND help you learn

Fishery managers constantly have to deal with the challenge of making decisions on allowable catch quotas with incomplete information. Adaptive management is one pathway to meet this challenge. (Photo by Megan Saunders)

Fishery managers constantly have to deal with the challenge of making decisions on allowable catch quotas with incomplete information. Adaptive management is one pathway to meet this challenge. (Photo by Megan Saunders)


Key messages:
  • Ecological problems are poorly understood, but often require immediate management action.
  • Adaptive management allows action to be taken immediately, and also helps learning.
  • Though commonly cited, true adaptive management is rarely applied.
  • Changing management because of ongoing monitoring at management sites is not adaptive management.
  • Adaptive management requires long funding time-frames, a high tolerance for risk, and institutional and management flexibility (factors that are rarely found in conservation).

The world’s ecosystems and species face a wide range of threats. Managers have a range of things they could try – ‘interventions’ – but they are often uncertain how the system will respond to any particular intervention. They are also often uncertain about which of the many threats is the most important to address. While additional research could allow us to work out which actions work and which may not, we don’t have the time or the money to do all the research required for a ‘perfectly’ informed decision. We need to act now, in the face of all that we don’t know.

The harvesting of marine ecosystems (fishing) illustrates this dilemma well. Species targeted by commercial fisheries have both economic and conservation value. Managers have to regulate these fisheries to ensure the persistence of the harvested species, while also allowing vital economic activity to continue. The appropriate limitations on catch will depend on the dynamics of the ecosystem, but there are many unknowns. Managers are uncertain about how fast the population recovers from low abundance; about the strength of density-dependent feedbacks; about interactions between harvested species and other parts of the ecosystem.

Given enough time, research could resolve each of these unknowns. However, fishing – with or without regulations – will continue in the meantime. Managers are therefore faced with conundrum. Decisions need to be made immediately, but good decisions require additional information and a delay.

The theory of adaptive management, and the set of analytic tools that surround it, offer a path between these two common and contrasting demands. Adaptive management is often called ‘learning by doing’. It recommends, as its name suggests, that managers act experimentally, undertaking the conservation action that they believe will simultaneously deliver benefits and information. They watch the ecosystem respond to their actions, and then measure the consequences to better understand how the system operates – to ‘learn’ as they do.

However, adaptive management is a little more complicated than just trying something to see if it works. This approach is sometimes held up as adaptive management, and is often colloquially referred to as ‘suck-it-and-see’. This, however, is not a good representation of what adaptive management is about.

True adaptive management is active. It involves making an explicit prediction about how the system being managed will respond to a particular action, using multiple competing models of how the system works. And it ideally involves trialling multiple actions and comparing the results of these different ‘experiments’. It’s about learning about the system and then applying these lessons to subsequent management iterations, learning more with every iteration. It’s about undertaking a formal mathematical analysis of the problem in order to better understand how best to proceed.

The characteristics of adaptive management

The main aim of adaptive management in conservation is to identify and undertake actions that will deliver conservation benefits, but will also reduce our uncertainty about ecosystem dynamics. Management actions therefore have dual consequences, with explicit and implicit value to managers. Actions deliver explicit benefits when they achieve management goals. They also deliver implicit benefits when they help managers learn, because better information improves future decisions.

Achieving today’s management goals is obviously the most important, since it delivers immediate and direct benefits. Learning is ‘less’ valuable, since the benefits it offers will only occur in the future (and will therefore be time-discounted, see the box on ‘discounting active adaptive management’), and may be relatively small if the reduction of uncertainty is marginal. Nevertheless, their implicit value is real, can be quantified, and can change the optimal management action in significant ways.

Simple experimental management takes a passive stance towards learning. A manager undertakes the action that they believe is most likely to deliver benefits, and they then learn by observing the outcomes of this best-practice management. This is sometimes referred to as passive adaptive management.

A clear and succinct definition of adaptive management can be found in a guide on the topic published by the US Government (Williams et al, 2009): “Adaptive management is a systematic approach for improving resource management by learning from management outcomes.” ‘Learning’ is important but in a decision framing that importance is determined by how much it improves the decisions we make. It is never undertaken for its own sake

A clear and succinct definition of adaptive management can be found in a guide on the topic published by the US Government (Williams et al, 2009): “Adaptive management is a systematic approach for improving resource management by learning from management outcomes.” ‘Learning’ is important but in a decision framing that importance is determined by how much it improves the decisions we make. It is never undertaken for its own sake.

Active adaptive management, by contrast, recommends a strategic approach to learning. The value of each action is a combination of its expected beneficial outcomes, and also the future learning that could be gained. This combination makes adaptive managers more likely to choose actions that they actually think might be suboptimal, in the short term, to verify those beliefs. It can even encourage them to undertake actions that they know to be terrible, like collapsing a fish stock by over-harvesting, if the resulting collapse and recovery will help them to rapidly learn about the ecosystem dynamics.

These facts can be distilled into a series of features that must be present for a project to be considered formal adaptive management. These are:

  1. The identification of goals for management (which are developed in collaboration with stakeholder groups).
  2. The specification of multiple management actions that could potentially achieve these management goals.
  3. A dynamic model (or models) that predicts how the system will change in response to management interventions, coupled with a statistical process for interpreting outcomes when they are observed. This model should be dynamic and stochastic, and the variation will be only partly predictable.
  4. The implementation of at least one of the identified management actions, coupled with a monitoring program that observes how the system responds to the intervention.
  5. A statistical updating of the system model(s) following the observed post-intervention responses, and a consequent change in management actions if this is recommended by the model.

This whole process should be iterative, with reassessment of the system model and management intervention following monitoring, followed by a new phase of management.

Limitations to adaptive management

Adaptive management is a popular idea in conservation, and has been widely recommended in the applied science literature. Its popularity has reached the point where the term is routinely included in policy documents and legislation (eg, the Marine Life Protection Act in California). However, in the vast majority of cases, closer scrutiny reveals that these programs are not implementing adaptive management as described here.

Instead, decision-makers are using the term to describe management programs that are coupled with ongoing monitoring, and where the managers are uncertain about which intervention will be most effective. Most lack conceptual models of the system dynamics against which management outcomes can be compared. Thus, while they might be ‘adaptive’ in the broadest sense of the word (in that the management will ‘adapt’ as new information comes to hand), their approach to resolving management uncertainty is heuristic and informal. Consequently, these forms of pseudo-adaptive management are likely to be inefficient or ineffective. They will also definitely be suboptimal. They are better described as reactive management, or ‘trial-and-error’ management.

Adaptive management is often incorrectly applied because the term has become so broadly interpreted. However, even in those rare occasions where formal adaptive management has been applied, the approach has a low success rate. Reviews have attributed these failures to three primary issues:

  1. inadequate or absent funding was made available for the monitoring step, which is essential for assessing the outcomes of actions;
  2. decision-makers are often unwilling to admit their uncertainty, or to experimentally undertake management actions that they believe to be suboptimal; and
  3. the scientific leadership, and the financial and political capital required to update a complex, adaptive program of management is rarely available on the necessary time-frames.

It should be noted that each of these issues is institutional, rather than technical. That is, the flaws are in the political science, not the biological science. In the latter case, researchers have pointed out an inherent contradiction in the implementation of adaptive management: the technical and scientific skills required to design and execute a complex experimental management plan are rarely found alongside the logistical and political acumen needed to execute such a plan in a difficult political environment.

Because none of these limitations are technical or scientific, they can’t be addressed through better mathematical methods or decision-support tools. In broad terms, each of these constraints speaks to the inherent challenges of managing complex, stochastic ecosystems that evolve on timescales that are vastly longer than political – or even scientific – attention spans.

Adaptive management requires scientists and managers to transparently admit how little they understand about the systems they study. They must then seek funding to experimentally manage these systems over very long timescales, during which time they will have to consistently cite their own ignorance as a rationale for (potentially unpalatable) experimental management. Finally, they are required to maintain attention and focus on this problem for long periods, during which time they may be able to draw few useful conclusions.

When considering active adaptive management over short to medium time frames the optimal strategy may require the manager to be extra cautious. For example, in considering strategies for revegetation, we might actively experiment early on with different planting densities if we were in a position to consider the gain over a very long time frame. However, if we are only going to monitor the return over a medium time frame it’s likely we won’t get a decent return for our experimental effort. (See the box on discounting active adaptive management.) (Photo by David Salt)

When considering active adaptive management over short to medium time frames the optimal strategy may require the manager to be extra cautious. For example, in considering strategies for revegetation, we might actively experiment early on with different planting densities if we were in a position to consider the gain over a very long time frame. However, if we are only going to monitor the return over a medium time frame it’s likely we won’t get a decent return for our experimental effort. (See the box on discounting active adaptive management.)
(Photo by David Salt)

Successful applications of adaptive management

Despite these constraints, adaptive management can be successful, even at extensive spatial and temporal scales, and in socio-ecological systems that include large numbers of stakeholders with disparate values and goals. As evidence, there are a small number of well-documented examples where the application of formal adaptive management has delivered management benefits.

The best-known example is of waterfowl management in the United States, which has been managed with a formal adaptive management program for the past three decades. Other examples include the management of Glen Canyon, on the Colorado River, USA, and the restoration of sand-mined ecosystems in Australia.

Although each of these success stories applied the same formal adaptive management techniques, each case-study is also deeply idiosyncratic. In each example, managers found unique ways to address a series of difficult social, fiscal and political constraints. The examples therefore provide lessons, but not necessarily broad solutions, to the factors that are considered the greatest obstacles to successful adaptive management.


What is good ‘learning’?

Despite its focus on uncertainty and learning, adaptive management does not place an explicit value on learning. Gaining a better understanding of the ecological system, or of how it responds to management actions, is not a direct goal. Instead, lower uncertainty is valuable to the extent that it allows managers to better achieve the conservation goals.

Learning – with its attendant costs and delays – is never undertaken for its own sake and, as a consequence, adaptive managers don’t care what the largest uncertainties are. Instead, the analytic machinery of adaptive management incorporates a hidden value-of-information analysis. As with Value-of-information analysis, an adaptive manager would be happy to leave large sources of uncertainty unresolved, if the information would not change their decision. The result is a style of management that is highly (but specifically) tolerant of uncertainty.


The evidence on adaptive management

‘Adaptive management’ is everywhere. Google it and you’ll get over five million hits, while academic search engines return over 20,000 articles. These articles discuss a huge range of topics – from ecology and conservation biology through to epidemiology, medicine and even construction. And more are being written all the time.

Adaptive management receives so much attention because it is intuitive, broadly applicable and conceptually appealing. Its basic premise is that as management proceeds, information is collected that improves knowledge of the system being managed. This knowledge is then used to improve future management practice.

Unfortunately, this simple overview masks a large amount of controversy in the academic literature (eg, see Keith et al, 2011). For example, some proponents have advocated that adaptive management be applied to nearly all environmental problems. Conversely, detractors have argued that adaptive management is merely a corporate buzzword, sometimes used to justify the continuation of flawed policies. Despite this diversity of opinion, quantitative evidence describing the pros and cons of adaptive management is difficult to find.

Working with David Lindenmayer and Gene Likens, Martin Westgate recently assessed the current ‘state of the science’ of adaptive management in ecology, with the aim of quantifying the usefulness of the concept (Westgate et al, 2013). They did this by finding articles whose authors stated that they were part of an adaptive management project, then looked for similarities and differences between these projects.

First, they quantified just how rare the actual application of adaptive management was in the academic literature. In a series of stages, they narrowed down their original 1,336 articles to 61 that explicitly claimed to enact adaptive management. Put differently, more than 95% of articles that discuss adaptive management don’t test it.

Second, there is a notable lack of empiricism in the adaptive-management literature. The articles that we identified related to 54 separate projects, but most were only discussed in a qualitative manner, usually as a part of a broad review. Only 13 projects were supported by published monitoring data.

Finally, few projects were able to sustain the effort to complete the adaptive management cycle. Most papers that claimed to be starting adaptive-management projects were recent (within the last five years). Only four of the 13 adaptive management projects that we identified lasted longer than 10 years.

References

Keith DA, TG Martin, E McDonald-Madden & C Walters (2011). Uncertainty and adaptive management for biodiversity conservation. Biological Conservation 144: 1175-1178.

Westgate MJ, GE Likens & DB Lindenmayer (2013). Adaptive management of biological systems: a review. Biological Conservation 158: 128-139.
For a discussion on this paper see Decision Point #74


Pushing the frontiers of AM

As any reader of Decision Point would know, over its life, CEED has explored and developed many aspects of adaptive management. Here are two recent examples.

Tracy Rout and Cindy Hauser and colleagues examined if adaptive management improved decisions surrounding searches for invasive species (Rout et al, 2017). They found that adaptive search strategies consistently outperformed alternative intuitive tactics. However, when they compared active and passive adaptive approaches to the searches they found there wasn’t much difference in outcomes. As passive adaptive management is much easier to do than active forms it will often make sense to stick with passive.

Iadine Chadès and colleagues reviewed and updated what’s known about the latest optimal or near-optimal approaches for solving adaptive management problems (Chadès et al, 2017). They reviewed three mathematical concepts required to solve adaptive management problems (Markov decision processes, sufficient statistics, and Bayes’ theorem) and provided a decision tree to determine whether adaptive management is appropriate.

References

Chadès I, S Nicol, TM Rout, M Péron, Y Dujardin, J Pichancourt, A Hastings & CE Hauser (2017). Optimization methods to solve adaptive management problems. Theoretical Ecology 10: 1-20. https://link.springer.com/article/10.1007%2Fs12080-016-0313-0

Rout TM, CE Hauser, MA McCarthy & JL Moore (2017). Adaptive management improves decisions about where to search for invasive species. Biological Conservation 212: 249-255. http://dx.doi.org/10.1016/j.biocon.2017.04.009


Discounting active adaptive management

Discounting has a long history in the economic literature, and has found its way to ecology via fisheries and wildlife harvest management. The premise is that achieving my goals this year is more important to me than achieving my goals in some future year. For example, a fisher taking a 100 tonne catch this year can sell those fish and invest his profits at some interest rate. They expect that this investment will be worth more in 10 years time than the 100 tonnes of fish they’ll catch and sell in that 10th year.

The future is also less valuable because it may never arrive. In 10 years, the fisher could be dead, or working as a chauffeur.

So, achieving our goals in the future isn’t as valuable as achieving the same goals now. It follows that sacrificing our goals now for experimentation and learning may not be so worthwhile if it’s only going to improve our achievements a long way into the future. Thus, discounting diminishes the value of experimentation, and experimentation is therefore less likely to be a part of the optimal active adaptive management plan.

It makes sense that experimental management may not be worthwhile when our time frame is short. If we experiment and jeopardise our goals at the beginning, we will probably need quite some time to learn about the system and make up for our losses with superior management based on better understanding. If it’s important to achieve our goals over a short time frame then we are probably better off making do and acting on the limited information we already have, while still adjusting our actions if we see new clues along the way.

Until recently, prominent mathematical studies of active adaptive management have adopted long time frames, on the scale of decades or centuries. Mathematicians sometimes even determine the optimal management strategy over an infinite time frame! However, managers may also be faced with problems where there are only a handful of opportunities for monitoring, learning and adapting before the success of the project is evaluated.

Cindy Hauser and Hugh Possingham developed a model of adaptive management and investigated the merits of experimental actions over a variety of time frames. What they discovered was quite a surprise. Their model was consistent with prevailing thought over long time frames – when there’s substantial uncertainty about the best way to manage our ecosystem, some early experimentation is beneficial. However, over medium-length time frames (such as 10-15 years), the optimal active adaptive strategy may be extra cautious. Actions that could cause ecosystem damage are avoided, even if that risk is acceptable under the ‘balance of current evidence’ passive adaptive strategy.

Over medium time frames, there is insufficient time to learn, adapt and recover any losses incurred while learning. An action that is reliable, albeit probably inferior, is preferable. Thus, the active adaptive manager may be a model of caution rather than daring experimentation.

Reference

Hauser CE & HP Possingham (2008). Experimental or precautionary? Adaptive management over a range of time horizons. Journal of Applied Ecology 45: 72—81.
And see a discussion of this paper in Decision Point #18, p4,5.


An illustrative example

In its application, adaptive management is a process of dynamical optimisation under uncertainty. Its mathematical formulation explicitly describes the uncertainty associated with key system parameters. The problem dynamics include (1) a model of how each action is expected to change the objective function, and (2) a model of how observations of the outcome alter our understanding of the system dynamics – how we learn. Dynamical optimisation tools (eg, SDP, optimal control) are used to consider sequences of decisions, looking forward in time.

These dynamics are most easily illustrated with a simple example. Imagine a manager is attempting to conserve a threatened species. Two management actions are available to her. The first action is the current best-practice, and its probability of success is known with complete confidence. The alternative action has never been applied before, and so it is unclear whether it is better or worse than the current best-practice action. If successful, each action would deliver the same benefit to the species, and the only issue is therefore whether the new action is better or worse than the current best-practice.

If the manager were only making a single, one-off management decision, then the appropriate choice would be to implement the best-practice action, since it has a demonstrated track-record of success and, without subsequent actions, there is no value to learning about the alternative.

However, if she intends to continue managing for multiple time-steps, there may be value in exploring the alternative action, since there is some probability (shaded in red in Fig 1) that the alternative action is superior. We have therefore set up a simple adaptive management problem. Managers have two actions available to them. They need to take actions immediately, but they don’t know everything they would like to know about the system – specifically, whether the alternative management action is better or worse than the best-practice action. They therefore consider taking actions that will offer both information and benefit. Below, we explore these ideas by formulating and solving this problem as a simple, two-step adaptive management project.

Fig 1: Belief distributions of the adaptive manager in our simple example. The solid blue line indicates that the manager’s belief distribution in the success rate of the best-practice action centres around 0.53, and has very low uncertainty. The solid red line indicates the manager’s deep uncertainty about the performance of the new, alternative action. The shaded blue region is the probability that the best-practice action is superior; the shaded red region is the probability that the alternative action is better. The two other red lines show what the manager’s belief distribution would look like if her first attempt to apply the alternative action yielded a success (dotted line, showing a stronger belief in its success), or a failure (dashed line, reflecting the evidence of its failure).

Fig 1: Belief distributions of the adaptive manager in our simple example. The solid blue line indicates that the manager’s belief distribution in the success rate of the best-practice action centres around 0.53, and has very low uncertainty. The solid red line indicates the manager’s deep uncertainty about the performance of the new, alternative action. The shaded blue region is the probability that the best-practice action is superior; the shaded red region is the probability that the alternative action is better. The two other red lines show what the manager’s belief distribution would look like if her first attempt to apply the alternative action yielded a success (dotted line, showing a stronger belief in its success), or a failure (dashed line, reflecting the evidence of its failure).

In the first time-step, the manager could undertake the best-practice action, and we can easily calculate the expected value of this choice. In time-step 1, this action will deliver a known expected benefit of p(blue line in Fig 1), since the probability of success is known with (almost complete) certainty. The manager will then make the same decision in the second time-step: with no subsequent actions there is no benefit to exploring the alternative action, and the optimal choice will therefore be to again undertake the best-practice action. Starting with the best-practice action therefore has an expected benefit of B=2 p(without time-discounting).

Alternatively, the manager could undertake the alternative action in the first time-step, observe the outcome, and learn about its success rate before making her second decision. Given we’ve never seen the alternative in action, we start with a uniform belief in its success rate. Perhaps it is always successful (p2 =1); perhaps it will never work (p2 =0). We can capture this belief with a uniform distribution, which we describe using the beta function f(p2 )=B(1,1) (the solid red line in Fig 1).

In the first time-step, the expected benefit of the alternative action will reflect the manager’s informed-but-uncertain belief distribution in the method’s success rate. Given our belief distribution, this probability is p2 =1/2. The manager’s actions in the second time-step will reflect the outcome of this first attempt.

On the one hand, if the action was unsuccessful (with probability 1-p2 =1/2), she will have learned a pessimistic lesson about the alternative action. Specifically, her new belief about the action can be described by the beta distribution B(1,2) (the dashed red line in Fig 1). From the properties of the beta distribution, the expected benefit of taking the alternative action in the second time-step would then be ρ2 =1/3. On the other hand, if the action was unsuccessful, the expected benefit of taking the alternative action in the second time-step would be ρ2 =2/3 (dotted red line in Fig 1).

We note first that, if the probability of the best-practice action is lower than 1/3 then the manager will automatically take the alternative action in both time-steps, since the expected outcome will always be higher. Second, we note that if the best practice action has a probability of success higher than 2/3, there is no point to learning about the alternative action, since even a successful application would still leave the best-practice action a superior choice. Learning in this case cannot alter the management decision, and there is therefore no value to this information.

Following this adaptive approach, we can predict that the expected benefit of taking the alternative action in the first time-step is:

B2 =p2 +p2 ρ2 +(1-p2) p1 = 1.083

This equation implies that an adaptive manager faces 1 of 5 scenarios (Fig 2). They are ordered here from low p (the best practice action is awful) to high p (it usually succeeds).

Figure 2: Optimal adaptive management approach to the simple management problem. The x-axis shows the manager’s belief in the performance of the best-practice action (the value of 0.55 reflects the situation in Fig 1). The blue line shows B1, the benefit that the manager can expect if she starts by applying the best-practice action in the first timestep. The red line shows B2, the benefit she can expect if she starts with the alternative action. The optimal adaptive management decision for a given belief in the best-practice action is determined by the higher of the two lines.

Figure 2: Optimal adaptive management approach to the simple management problem. The x-axis shows the manager’s belief in the performance of the best-practice action (the value of 0.55 reflects the situation in Fig 1). The blue line shows B1, the benefit that the manager can expect if she starts by applying the best-practice action in the first time-step. The red line shows B2, the benefit she can expect if she starts with the alternative action. The optimal adaptive management decision for a given belief in the best-practice action is determined by the higher of the two lines.

p1 < 1/3 (red shaded area in Fig 2): In this case, the manager will undertake the alternative action in both time-steps. Even if the alternative action fails in the first application, it is still expected to outperform the best-practice action.

p1 > 1/3 (blue shaded area in Fig 2): In this case, the manager will never attempt the alternative action. Even if it were successful in the first application, its expected performance will still be lower than the best-practice action. Note that in this case, we are aware of the substantial probability that the alternative action is superior to the best-practice action. However, the adaptive management analysis indicates that there are too few learning opportunities for management to resolve the question.

1/3 ≤ p1 < 1/2 (green shaded area in Fig 2): In this case, the manager suspects that the alternative action will be the better option, but will keep an open mind. In the first time-step they take the alternative action, and make their second decision after observing its performance.

1/2 ≤ p1 < 0.555 (green shaded area in Fig 2): Based on the current information, the manager believes that the best-practice action is superior to the alternative action. Nonetheless, they will undertake the alternative action in the first time-step, because the value of learning about its true performance outweighs the short-term loss of expected benefits.

0.555 < p1 < 2/3 (yellow shaded area in Fig 2): In this situation, the manager is sufficiently confident about the superior performance of the best-practice action that they are not willing to learn about the alternative action. This is despite the very real possibility (perhaps as high as 45%) that the alternative action is superior. The value of obtaining this information is just not high enough to justify its short-term costs.

This problem is a conservation realisation of the ‘one-armed bandit’ problem, first solved by economists in the 1950s, and famous for its application to clinical trials of medical interventions. There are a range of methods available for calculating or approximating the optimal solution in more complex contexts, and all agree that the best way forward is a mixture of exploitation and exploration. Exploitation involves the application of the action that we currently believe to be superior (action 1, in Fig 1); exploration requires the judicious use of uncertain actions that may (or may not) turn out to be superior than the current best practice (action 2; Figure 1).

The optimal balance of exploitation and exploration will depend on a suite of factors, including the amount of uncertainty associated with each action, the length of the management horizon (ie, are we planning to manage for 2 years, or 200 years?), and the broader applicability of the actions.


 

More info: Michael Bode michael.bode@qut.edu.au

Recommended reading:

Hughes TP et al (2007). Adaptive management of the Great Barrier Reef and the Grand Canyon world heritage areas. AMBIO: A Journal of the Human Environment 36: 586-592.

Lyons JE et al (2008). Monitoring in the context of structured decision-making and adaptive management. Journal of Wildlife Management 72: 1683-1692.

McCarthy MA & HP Possingham (2007). Active adaptive management for conservation. Conservation Biology 21: 956-963.

Rumpff L et al (2011). State-and-transition modelling for adaptive management of native woodlands. Biological Conservation 144: 1224-1236.

Walters CJ & R Hilborn (1978). Ecological optimization and adaptive management. Annual review of Ecology and Systematics 9: 157-188.

Walters CJ (1986). Adaptive management of renewable resources. Macmillan Publishers Walters CJ (2007). Is adaptive management helping to solve fisheries problems? AMBIO: A Journal of the Human Environment 36: 304-307.

Williams BK, RC Szaro & CD Shapiro (2009). Adaptive Management: The U.S. Department of the Interior Technical Guide. Adaptive Management Working Group, US Department of the Interior, Washington DC.

Leave a Reply