The Disruption Decline: Is Modern Science Becoming Less Innovative?

Recently, I had the opportunity to discuss the state of scientific discovery and innovation with peers and I had the opportunity to start a very intriguing conversation on LinkedIn. 

The conversation raised some concerning questions - are we seeing a decline in the disruptiveness and impact of modern research? As a “data-driven guy”, I decided to dive deeper into the data and theories around this vital issue, and a big thanks to Sabine Hossenfelder for her inspiring videos on these subjects.

Current Status: What constitutes true scientific discovery and innovation? 

Genuine disruption comes from the breakthrough ideas, novel insights, and paradigm shifts that radically advance human knowledge and capabilities in new directions. Mere incremental refinements and improvements to existing technologies and methods, while valuable, do not qualify as disruptive innovation.

For example, increasing the number of pixels in a headset, making a language model slightly faster at replying, or boosting the efficiency of photovoltaic panels represent incremental progress.
We're simply improving what already exists through more efficient processes and scalable optimizations^[1]. This innovative iterating builds on prior knowledge but does not overturn our fundamental understanding.
In contrast, discoveries like CRISPR gene editing, quantum computing, or mining astronomical data for evidence of multiverses and extra dimensions upend conventional scientific wisdom. They spark new fields, rewrite textbooks, and enable previously unimaginable application^[2]. That's the essence of disruptive innovation driving a true scientific revolution.

As Carl Sagan remarked, "The revolution is underway...and with every succeeding scientific discovery, it accelerates, impacting more and more aspects of our civilization." However, recent studies suggest the current pace of these game-changing, disruptive discoveries has slowed across multiple disciplines. We appear mired in an era of incremental, progressive science versus revolutionary, disruptive breakthroughs.

We definitely have more scientists!

The raw numbers seem positive at first glance. The global scientific workforce has swelled, with the number of researchers per 1,000 employed workers rising from around 1 in the 1960s to 1.5 today according to World Bank data^[1]. Countries like Germany now employ over 5 scientists and engineers per 1,000 workers^[2].

This increase in scientific labor has corresponded with a boom in research output - over 5 million papers published annually and about 3.5 million patents filed worldwide each year^[3]. But is this surge in quantity matched by a proportionate increase in quality and impact?

Studies suggest that the dramatic rise in the sheer volume of research may actually be concealing a troubling decline in transformative, disruptive scientific breakthroughs that radically advance human knowledge^[4]. More incremental papers are flooding the literature, but the rate of paradigm-shifting discoveries appears to be waning. In essence, we're maximizing scientific productivity but not necessarily revolutions.

The growing mountain of scientific outputs often builds on existing knowledge in increasingly specialized subdivisions^[5]. But without those occasional disruptive deviations that forge entirely new frontiers, science risks getting trapped in an overly incremental rut, iteration over revelation. So while the raw numbers seem impressive on their face, they may be obscuring an innovation deficiency demanding closer examination.

But Data suggesting a decline

Multiple strands of evidence point toward a concerning decline in the disruptiveness and innovative impact of modern scientific research output^[1][2][3]. A 2018 study by physicists Peter Cauwels and Didier Sornette examined the career-defining breakthrough years of elite scientists across fields^[4]. After normalizing for the growing population of researchers, they uncovered a lengthy stretch of decline in impactful, transformative work per capita dating back to the 1930s.

For sure many people will not take the time to read the paper, so let me provide a TLDV version of the article. The two scientists took in consideration the top researchers (those who made the most contribution to the field of research), and assigned the Active Research Years (picture 1).

For sure many people will not take the time to read the paper, so let me provide a TLDV version of the article. The two scientists took in consideration the top researchers (those who made the most contribution to the field of research), and assigned the Active Research Years (picture 1).

At this point, to determine the research productivity they normalized the total number of researchers (picture 2).

45 Million papers analysis

This pattern aligns with quantitative analyses of the publication record itself. A landmark 2020 study in Nature analyzed textual indicators of novelty and disruption across 45 million papers spanning the 20th and 21st centuries. It revealed a steady, multi-decade decline in papers demonstrating high levels of conceptual and functional disruption versus incremental extensions of existing knowledge ^[1].

This provides strong empirical evidence that science and technology have become less disruptive over time, corroborating concerns about a potential slowdown in breakthrough innovations and discoveries. The key findings are:

• Using a metric called the CD index that measures how much a paper or patent disrupts or builds upon existing knowledge, the authors found a marked decline in disruptiveness across all major scientific fields and technological domains from 1945-2010 for papers and 1976-2010 for patents.

• This decline is visible not just in citation patterns, but also in the changing language used - more recent papers and patents use less diverse vocabulary and word combinations, with a shift away from verbs indicating creation, discovery and perception toward those suggesting improvement, application and assessment. These trends are consistent over multiple innovation areas.

This systematic analysis across multiple datasets aligns with the concerns raised in the draft article about science potentially losing its disruptive edge and innovative vigor. The authors provide quantitative evidence of this phenomenon occurring across disciplines over decades, pinpointing an over-reliance on narrow prior knowledge as a key contributor. Their findings emphasize the need to rethink incentives and practices to promote broader exploration of the existing knowledge base as a way to reinvigorate transformative discovery.

Wait, Wait! What about our Nobel Prizes?

The timing of Nobel Prize awards suggests a drop-off in highly disruptive, transformative scientific breakthroughs in recent decades. Several studies have found that Nobel Prizes are increasingly being awarded for pioneering work done much further in the past compared to the 1960s-1970s era.

Specifically, a 2020 study in PNAS ^[1] revealed that the average interval between the original groundbreaking research and its recognition via a Nobel Prize has grown from around 20 years in the 1960s-70s to over 40 years by the 2010s. This implies a dearth of discoveries from the past few decades rising to the level of prior generations' landmark innovations.

In physics, an analysis found the relative importance of Nobel-winning research has declined over the last century, with physicists surveyed judging the most impactful prizewinning work was done in the 1920s. The importance rating of 1980s physics research was just one-third that of 1920s level.

While chemistry and medicine have not seen the same drop in perceived importance of modern prizewinners, the rarity of recent Nobels reflects a broad innovation deficiency. Very few have been awarded for work conducted since 1990 across all three disciplines.

There are recent exceptions like the 2020 Nobel for CRISPR gene editing and 2022 for COVID vaccine research. However, the overall trend indicates a rising gap between the prodigious output of the modern scientific workforce and a relative paucity of frontier-revolutionizing, disruptive discoveries on par with prior eras' transformative breakthroughs.

While nuances exist around definitions and degrees of decline, the overall picture from diverse empirical perspectives is one of incrementalism steadily superseding revolution in the modern scientific agenda. As Michael Nielsen, researcher and author of "Reinventing Discovery," summarizes: "We're doing science faster than ever...however, it's unclear that current approaches have put us onto an exponentially faster trajectory of important discoveries."^[2]

Possible Biases in Data

Some argue these findings are biased by the availability of more complete citation data for recent decades, potentially overestimating the disruptiveness of work from the 1970s-1980s era. The University of Basel contends that research has lost none of its innovative drive^[1].

They said the novelty of the 1970s and 1980s is overrated because earlier citations were not repurposed in the patents of the time.

But even assuming this juxtaposition is correct, in each case the disruptiveness of the papers continues to decline, certainly at a slower rate, but still declining.

However, even if the rate of decline is debated, the data provided in all these studies consistently shows the tide of disruptiveness is gradually and constantly decrease over recent decades.

The key question is - why?

Hypotheses on the Decline:

• Denial - Nothing is wrong, and science is as transformative as ever. The data is simply flawed or misleading (as mentioned by Sabine, this is probably the preferred theory among scientists).
• The Low-Hanging Fruit Has Been Picked - We've already plucked the most obvious, disruptive discoveries and innovations. What's left are progressively smaller, incremental advancements.
• The System Disincentivizes Disruption - The way we fund, publish, tenure, and reward researchers prioritizes excessive specialization, maximizing output over impact, and prohibits the exploration required for major breakthroughs, and, maybe the current incentives create systemic contradictions that actively undermine disruptive innovation.

Untangling the truth likely involves a mix of these factors. But the data suggests modern science may indeed be in need of a disruptive reboot of its own. Addressing systemic issues like siloed hyper-specialization, short-term funded projects, and incentives for shallow novelty over depth could reinvigorate the foundations of transformative discovery.

The findings from these studies represent a compelling call to action for the scientific community and policymakers to critically examine the systemic factors potentially stifling disruptive innovation. Based on the evidence presented, the third hypothesis - that the current system disincentivizes disruption - seems the most plausible explanation.

In my experience I've witnessed firsthand how the publish-or-perish culture and excessive specialization can lead to overly narrow focus areas and incremental work aimed at racking up publications rather than pursuing bold, paradigm-shifting ideas. The incentive structures favor projects with a higher probability of producing positive results in the short-term, even if the potential impacts are relatively modest.

The findings that disruptive work is associated with drawing from a broader range of existing knowledge and engaging with more diverse information sources resonate strongly. Too often, researchers can get trapped in cyclical loops citing the same well-trodden ground rather than venturing into unfamiliar territories that could spark unconventional insights.

Judging strictly by publication counts perpetuates a mindset of endless incrementalism. Perhaps disruption metrics like the CD index could be incorporated into grant reviewing, hiring, and tenure processes to reorient priorities.

After all, if a researcher's biggest contribution decades later is still not deemed transformative enough for a major prize like the Nobel, what message does that send?

Ultimately, disruptive scientific leaps require time, resources, and intellectual freedom to explore unconventional directions without fear of failing to publish constantly. Addressing the systemic contradictions this study highlights could revitalize the environment for major breakthroughs - but it will likely require challenging some long-held cultural norms of how we evaluate and reward scientists.

So, probably the final key question is: Are we willing to make that change for the sake of accelerating humanity's capacities for revolutionary discovery? What do you think?

AI-Ready by OpenExO empowers small and medium enterprises to integrate AI seamlessly into their operations, offering a comprehensive assessment, personalized implementation, and continuous strategic coaching, ensuring they not only adapt to but lead in the digital economy. This program transforms AI challenges into growth opportunities, providing SMEs with the tools and expert guidance needed to enhance efficiency, drive innovation, and secure a competitive edge. https://web.openexo.com/ai-ready/