On the Meaning of Impact in Disaster Risk Reduction

David E. Alexander 1

Abstract This article off ers a discussion of the meaning, assessment, and measurement of impact in disaster risk reduction. It begins with a historical perspective on the impact of learned work in times when orthodoxy posed severe limits on the impact of new thinking. Regarding the modern age, the article explains why impact is considered important and how it might be recognized when it occurs,including a tentative classif ication of types of impact. The question of whether impact can truly be measured remains pending, as the answer is diff use and dependent on many different circumstances. Further sections consider the relationship between impact and mainstreaming and the question of whether a piece of work should be regarded as having impact if its eff ects are negative rather than positive. Next, impact is considered in terms of whom it benef its. Given the large number of possible reservations about the concept, the question is raised as to whether too much emphasis is given to the impact of research and scholarship. Finally, some suggestions are off ered regarding how to obtain a better indication of what the impact of an academic study actually is.

Keywords Assessing impact · Disaster risk reduction ·Disruptive thinking · Impact of research · Mainstreaming disaster risk reduction · Measuring impact

1 Introduction

This article off ers some meditations on the assessment and measurement of the impact of research in disaster risk reduction. Both historical and contemporary perspectives indicate that impact is a subtle and often elusive quality that easily def ies measurement. The article considers whether it is worthwhile at all to measure or assess the impact of scholarly work. Are there perhaps too many qualif ications and impediments? It also enquires into the question of how we might def ine impact so that we can recognize it when it occurs. Finally, some potential guidelines for assessing impact are off ered,caveat lector.

For obvious reasons, disaster risk reduction is a f ield of imperatives. Therefore, theory has a special role, in that it needs to be immediately applicable and capable of generating positive results in the short term. Lives are at risk from disasters and damage needs to be avoided. Theory is the road map by which we make sense of complex situations such as those that arise in disasters so that we can reduce their human and environmental consequences. This is a powerful argument in favor of trying to ensure that research on disasters has recognizable positive impact. However, public debate does not always set priorities according to the most logically and morally justif iable set of criteria (Quintavalla and Heine 2019). This may mean that there is sometimes an element of illusion in what appears to be high impact.

We begin with some observations on the impact of major discoveries in history and the conditions of acceptance when they were made.

2 The Historical Limitations of Impact

History is replete with luminaries whose work had little or no impact on society or the advancement of humanity during their own lifetimes. In Europe through much of the recorded past, enforcement of the status quo posed strict limitations on free thought and its expression. Very often, heretics were at risk of their lives because unorthodox thinking was deemed to threaten the social order, or at least the established power structures. Perhaps it still does. The response was usually swift and devastating (Shrady 2009). Perhaps it still is in many parts of the world. In history, there was little tolerance of “disruptive thinking” (Vollmer 2013), and where it did catch on, it led to prolonged and bitter armed conf lict (Wilson 2011).

One example stands out. In the Renaissance, paper was a scarce and expensive commodity. Leonardo da Vinci was known to have covered at least 5000 sheets with sketches,drawings, plans, and notes (Pedretti 2006). Half of these documents survive after being kept for centuries in cupboards and on shelves, preserved lackadaisically for their curiosity value and the quality of his sketches, but shorn of impact. It was not until 380 years after his death that Leonardo began to be recognized widely as much more than a painter. As the child of a middle-class accountant, moreover born out of wedlock, he had no access to a Classical education, although in adulthood he taught himself Latin and Greek. With his mind untrammelled by the prevailing dogma, he was able to apply his particular brand of superbly rational, creative thinking to problems of nature, mechanics,and anatomy. He never f inished the books he intended to write (on the behavior of water, on the human body, and on the scientif ic basis of painting), but since his work has been rediscovered he has remained a constant source of study,inspiration, and citation (Kemp 2007).

It is often stated that Leonardo, who was left-handed but largely ambidextrous, wrote in mirror image to disguise his thoughts from zealots who would have taken them as evidence of punishable heresy. It is now considered more likely that he did so as a form of relaxation of his writing hand(Aff atigato 2019), but there is no doubt that ill-considered originality could lead an author to imprisonment, trial by torture, or execution (Nigg 1990), and in his early manhood Leonardo narrowly escaped such a fate.

In the present age of science, we deem ourselves to live in more enlightened circumstances. This is perhaps true,although not in all human societies in the twenty-f irst century. It brings with it a need to avoid what the historian Gordon Herries Davies and others termed “historical whiggishness” (Herries Davies 1989; Mayr 1990). This is the tendency to assume that all developments in history lead up to the current triumph of enlightenment over ignorance and obscurity. In reality, human development is part of a progression in which the achievements of the current age will sooner or later appear to be as paltry and inadequate as their predecessors. Moreover, such is the variety of people’s perceptions and opinions, that progress is marred by constant setbacks. The struggle to induce people to accept the concept of human-induced climate change should remind us of that (Whitmarsh 2011). In disaster risk reduction (a f ield that has borne that name for hardly more than two decades), there is still an on-going struggle to induce people to see disasters as generated by human actions and decisions,despite emphatic statements decades ago about the role of vulnerability in causing disaster (Hewitt 1983).

3 Impact in the Modern Age

We are like dwarfs on the shoulders of giants, so that we can see more than they, and things at a greater distance, not by virtue of any sharpness of sight on our part, or any physical distinction, but because we are carried high and raised up by their giant size. (Bernard of Chartres 1159–see Stock 1979)Nowadays, we live in the age of impact. Moreover, it usually has to be instant impact. The reasons for this are three. First,those who fund scientif ic endeavor want tangible evidence that they are receiving value for money. Secondly, there is an unwritten compact between scholars and the general public that their work should be benef icial to society. Moreover, it has to be seen to be benef icial. Thirdly, in a competitive age there is a desire to measure performance. Gauging impact is regarded as one of the ways in which this can be done for researchers and scholars.

Although the English word “impact” stems from the Latinimpactus, a variant ofimpingere(to impinge), its use in a f igurative sense apparently began only in the nineteenth century (OED). Thus it is a rather young term in the Anglophone lexicon. In the context of this discussion, “impact”can be given the following (rather arbitrary) working def inition. It is “the presence of improvements that can be measured or otherwise assessed as a direct result of the publication or sharing of one or more pieces of work.” This begs several questions. What def ines improvement? Can impact be measured or even simply gauged? When in the cycle of production and consumption of knowledge does it have to occur? How can we prove that there is a causal relationship between the work and its apparent impact? Finally, does impact have to be positive (that is, a net gain in a benef icial quality, such as peace, prosperity, safety, or harmony) or can it be appreciated if instead it is negative? In many cases, the answers to these questions are diffi cult to establish.

In disaster risk reduction the impact of academic work is often particularly difficult to gauge because of the complexity of relations between the many components of the systems involved. Not only are cascading, concurrent, and compound risks and impacts involved, but unintended consequences abound. For example, there has been a major global eff ort (albeit a rather heterogeneous one) to apply science and technology to disaster risk reduction (Aitsi-Selmi et al.2016). However, as Quarantelli ( 1997) noted, technology leads a double life: that which its makers intended, and that which they did not. In particular, the fashion for optimization algorithms (Boonmee et al. 2017) is particularly diffi cult to assess in terms of impact. In most disaster situations it is doubtful whether anything can be optimized.

Remaining in the context of the present discussion,impact can take several forms. Alternatively, it can respond to several means of classif ication:

? instantaneous, cumulative (ramped), or delayed

? slight, moderate, or substantial

? positive or negative

? tangential (that is, slight, partial, or subordinate)

? direct or indirect (that is, catalytic)

? universal or sectoral (restricted to relatively narrow themes)

This further complicates assessment and measurement.

If a signif icant change is observed in disaster mitigation, preparedness, response, or recovery, and if this can be directly and unequivocally attributed to a particular study,then the work doubtless has had impact. However, there are distinct problems with assuming such a relationship. Causality may be very diffi cult to establish. Many other factors may create change such that isolating one cause is counterproductive or even illusory. Moreover, we may be quite unable to measure delayed impact.

Such is the weight given to “impact” in the modern world of science and academia that it is often automatically gauged by, for example, bibliometric measures. This off ers us the paradox of seeing the value of science measured by essentially unscientif ic means (Casadevall and Fang 2014).

Popularity is not impact, although it is often mistaken for such. There is no robust reason for saying that, in scholarly circles, that which is most popular is that which is most important. In science, there are fashions and fads, as there are in any other walk of life (Abrahamson 2009). Particular forms of language or jargon, particular methodologies and styles of analysis, and particular ways of looking at a problem, all contribute to legitimizing studies by making them appear up-to-date and in line with present tendencies(Woodcock 2014). Rather than enhancing their impact, this often has the opposite eff ect. For example, the popularity of statistical methods such as the analytical hierarchy process(AHP) may be misleading as a measure of its true utility.No doubt when properly used, this technique is a powerful tool for resolving complexity. However, it suff ers from the weaknesses of all inductive methods: f irst it is dependent on the assumptions that underpin it, secondly it can be used uncritically without profound thought about its implications,and thirdly it may simply be based on the wrong criteria(Schenkerman 2007). Despite this, it remains extremely popular in disaster risk reduction as a problem-solving tool.One might therefore argue that AHP studies have impact simply by means of their proliferation and popularity, and only sometimes because of their superior ability to solve problems.

4 Impact and Mainstreaming

As noted above, the history of science is rich in discoveries that in their time were unpopular, unrecognized, or treated with great scepticism but that turned out to be fundamental once they had found their proper milieu. This transition is sometimes termed “mainstreaming,” and it is not quite clear whether it is a process that can be engineered or whether it must necessarily be left to occur naturally. It is a function of context. Thus, disaster risk reduction can only be“mainstreamed” (that is, propelled into the mainstream of discourse, dialogue, and decision making) when the context allows, particularly the political context. Nonetheless, the procedures and tools exist for the change to be made (Benson et al. 2007; ADPC/USAID 2010).

As Bernard of Chartres observed in AD1159 (Stock 1979), all learned discovery is derivative, or better still cumulative. This either modulates or mitigates the impact of most discoveries, but it also creates the conditions for impact to occur, as a discovery requires a certain set of conditions for it to be accepted by potential users or benef iciaries. The question hinges on how we def ine the utility of scientif ic work (Sikorav 1991). Once a concept has been discovered and presented to potential users, it is mainstreamed when it has been put to use widely enough for it to be recognized as a valuable or essential tool.

Mainstreaming occurs both outside and within science.In the former case this usually involves impact upon public policy; in the latter, it can create a paradigm, as more and more scientists and scholars follow the lead. Nevertheless,decades of debate on Thomas Kuhn’s hypothesis of scientif ic revolutions indicate that the concept of “paradigm,” though convincing, is by no means as universal as one might think,or in other words that sciences tend to be more pluralistic than Kuhnian disciples might believe (Bird 2012). For example, if resilience can be regarded as a paradigm in disaster risk reduction (Sudmeier-Rieux 2014), there are plenty of researchers who do not think it deserves that status (for example, Keating and Hanger-Kopp 2020). It is therefore likely to be very rare that the impact of a piece of work drives all before it.

5 Should We Applaud Negative Impact?

In 1996 the American political scientist Samuel P. Huntington published his bookThe Clash of Civilizations and Remaking of the World Order, which was based on ideas he f irst put forward in a lecture four years earlier (Huntington 1996). He argued that ideology would no longer be the principal source of conf lict, but instead the world would retreat into rival cultural groupings. His assessment of what these are was very broad and involved nine major domains at the world scale. Huntington’s theory has been very popular in right-wing political circles where there is a growing sense of“culture warfare.” However, the book has been intensively criticized by scholars and commentators from many diff erent backgrounds. In brief, it is regarded as a misleading oversimplif ication of the trajectory of world aff airs. Critiques and alternative models have proliferated, including a particularly dismissive one by the Nobel Prize winning economist Amartya Sen (Sen 1999).

Similar criticisms could be levelled against Francis Fukuyama’s 1992 bookThe End of History and the Last Man(Fukuyama 1992), which argued that the end of the Cold War led to the termination of humanity’s ideological evolution. Once again there was a large groundswell of criticism.The French Philosopher Jacques Derrida went so far as to label Fukuyama as devious (Derrida 1994).

As it has been widely discussed all over the world, Huntington’s work has undoubtedly had enormous impact. On balance, much of that appears to be negative in the sense that inf luential thinkers have mobilized in large numbers to contest his thesis. Should we therefore regard his work as of high or low value? One might argue that he has muddied the waters of international political science by advancing an untenable thesis. On the other hand he has stimulated debate and that has led to alternative theories of human social evolution.

In disaster risk reduction, the concept of social capital has been widely embraced (Aldrich 2012) despite considerable reservations about its robustness and utility (Haynes 2009). Again, this begs the question of whether value must lie entirely in positive developments or to what extent honest appraisal of negative elements can also be treated as a positive result.

There is no clear answer to the question of whether negative impact should be respected, but as the novelist du Maurier ( 1981) wrote, “Living as we do in an age of noise and bluster, success is now measured accordingly. We must all be seen, and heard, and on the air.” Or as Tacitus ( 2009)remarked in hisThe Histories, “Etiam sapientibus cupidoglori? novissima exuitur” (“Even for the learned, love of fame is the last thing to be given up”). On that resignedly philosophical note I shall leave the question hanging.

6 Impact in the Service of Whom?

One further question remains to be given a brief critical examination. If the impact of a study can be demonstrated,who does it benef it? In environmental matters there is a“cure-to-damage ratio” (Burton 2018). In the simplest terms,this means that the amount of money or size of resources invested in solving a problem can be measured against the amount spent on making it worse. Hence, although the world may be concentrating increasingly on “decarbonization” in the hope of containing anthropogenic climate change, it is still true that orders of magnitude more funds are invested in fossil fuel exploration, extraction, and ref inement than on promoting alternative, “greener” sources of energy (Laville 2021). Indeed, the cure to damage ratio is reckoned to be about 1:1000. Likewise, in disaster risk reduction, more is spent on creating disaster risk than reducing it, as can be seen, for instance, in unwise construction and land use(Lewis and Kelman 2012). Marie Aronsson-Storrier ( 2020)argued that, while international agreements such as the Sendai Framework for Disaster Risk Reduction 2015?2030(UNISDR 2015 ) may increase the incentive to reduce disaster risk, international investment law acts to undermine such eff orts.

As a general observation, research must first benefit the organization that funded it. This has, of course, led to numerous conf licts of interest in the medical and pharmaceutical f ields when prof it-making companies have funded research on their products (Lo and Field 2009). The same is true for the fossil fuels industries. Nevertheless, it would be unfair to condemn research simply because it fulf ils the needs of its sponsor, as this is not inherently or necessarily a violation of academic integrity or scientif ic objectivity.

7 Have We Put Too Much Emphasis on Impact?

The need to present, assess, and measure impact is a function of the desire to make scientif ic and scholarly endeavor effi cient, as well as directly responsive to people’s needs.Many works now need to be preceded by an “impact statement,” which is ironic as the impact, if any, will come after publication and any attempt to prejudge it may be highly presumptuous. At the very least, much of it is likely to be wishful thinking. No doubt this desire is laudable, but it has been overemphasized. As noted above, impact occurs along a spectrum, or rather a series of intersecting spectra, for it is a multi-dimensional phenomenon.

Is a contribution to knowledge a form of impact in its own right? The promoters of impact statements would argue that it is not unless the contribution has been recognized and in some positive way acted upon. Others might be more cautious, given the propensity for discoveries to lie dormant until conditions are right to activate them.

8 In Search of the Best Way of Judging the Impact of Academic Work in Disaster Risk Reduction

To have written a critique almost automatically puts one in the position of being expected to provide an alternative to what has been criticized. Here, the simplest approach would be to stop trying to measure and judge the impact of research studies. It is highly debatable as to whether this would lead authors and scientists astray, as the desire to “do something useful” is pretty much universal. Perhaps there would be a slight lessening in the desire “to be seen to be doing something useful.” This would be a positive development if it gave researchers more freedom to concentrate on what they personally regard as important and what they think they are best at doing. It might lessen the inf luence of fads and fashions, which would probably be a positive development,but there is no guarantee that this would happen. However,it would not help abate the clamor for value for money in research.

If we must continue trying to measure impact, I recommend that the following points be taken into account.

(1) The best way to measure impact is intuitively. This allows for the integration of diff erent factors and their prioritization. Although this approach can be a prey to bias and arbitrary judgement, that is also true of apparently objective numerical methods, which are dependant on the assumptions and choice of inputs that sustain them.

(2) Impact is easily misrepresented. It is often a hypothetical quantity and it is not a direct function of the popularity of a work.

(3) The impact of a piece of research can be dependent on sudden changes in what society deems is important,usually on the basis of the salience of particular issues in current aff airs.

(4) Impact is not directly correlated with the scientif ic quality of a piece of research. Hence, it should not be used as a surrogate for quality.

(5) It should be recognized that impact takes many forms and manifests itself according to diff erent dynamic processes, as in the classif ication given above. If it is delayed, partial or tangential, this does not necessarily diminish its value.

(6) Too much emphasis on impact risks giving way to populism to the detriment of studies of topics and problems that are diffi cult and unpopular but vitally important.

Regarding point (3), no increase in impact has been greater or more dramatic than that of years of patient work on viral disease pandemics. The year 2020 marked a turning point in attention paid to this problem, funding streams devoted to it, public debate on pandemics and viruses, and much more (Horton 2021). There are still neglected areas where impact should be greater. One of these is emergency planning (Alexander 2020). Regarding the last point, the study of existential risk (Currie and ó héigeartaigh 2018)is an example of work whose impact is diffi cult to assess.Nonetheless, there are circumstances in which it could suddenly become central to public discourse and offi cial decision making.

In conclusion, on a personal level, I look back over a career spanning the better part of half a century and I f ind it extremely diffi cult to assess whether my eff orts in the f ield of disaster risk reduction have “had impact.” I know from correspondence that a book I wrote on emergency planning was much appreciated by a few people in Latvia and the Shetland Islands, but I have no idea of its inf luence more generally. It is, of course, even harder to assess whether one’s eff orts will have any impact in the future. At the least we should continue to play devil’s advocate and ask whether the emphasis on the impact of scholarly work is a mere distraction or whether it does indeed play a useful role.

Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source,provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.