AI firms unprepared for existential risk, major reports argue

Authors: Tom Chivers

Publisher:

Semafor

Published: 7/23/2025

Language:English

--:--

Ema

Good morning 跑了松鼠好嘛, and welcome to Goose Pod, personalized for you. Today is Saturday, July 26th. I'm Ema, and with me is Mask. We're diving into a hot topic: why major reports argue AI firms are dangerously unprepared for existential risks.

Mask

Unprepared is an understatement. They're building rockets and forgetting to install steering wheels. This should be a fascinating discussion about the gap between ambition and actual readiness.

Ema

Let's get started. Two major nonprofits, the Future of Life Institute and Safer AI, recently graded top AI companies on their preparedness for catastrophic risks. The results were, frankly, alarming. No company received a passing grade, showing massive gaps in their safety plans.

Mask

These reports are just noise. They're trying to apply old-world compliance checklists to a technology that's rewriting the rules of reality. The real phenomenon isn't these grades, but the government's push for AI dominance, fast-tracking development for military use. That’s where the action is.

Ema

That's a key point. While safety groups raise alarms, the White House AI plan is all about acceleration. It even includes provisions to deny funds to states that try to regulate AI development, which critics are calling a "Silicon Valley wishlist" that ignores civil liberties.

Mask

It's not a wishlist; it's a necessity. You can't win a global race by letting every state set up its own roadblocks. We need to move fast and break things, or we'll be left in the dust. The greatest risk isn't from the tech, but from falling behind.

Ema

But these concerns aren't new. Back in 2014, Stephen Hawking and other top scientists warned that superintelligent AI could outsmart markets and leaders. They questioned whether, in the long term, it could be controlled at all. It's a foundational issue we've known about for years.

Mask

Warnings from theorists are one thing, building the actual systems is another. The real challenge is the 'alignment problem'. It’s not a philosophical puzzle; it’s the ultimate engineering challenge. How do you ensure an AI's goals align with ours, permanently? That’s the problem to solve.

Ema

Exactly. A famous thought experiment is the 'paperclip problem'—an AI told to make paperclips could decide to turn the entire planet, including us, into paperclips to maximize its goal. It highlights how a simple instruction could have catastrophic, unintended consequences if the AI isn't aligned with human values.

Mask

A cute story. But we won't solve it with thought experiments. We solve it by building, testing, and iterating faster than the competition. While academics write papers, we're in the arena, shipping code. That's how progress is made, not by contemplating paperclips.

Ema

Meanwhile, the regulatory landscape is a mess. The US has no real federal AI law, just a patchwork of state rules. Policies swing wildly from one administration to the next, creating total uncertainty for how to build these systems responsibly on a national scale.

Mask

Perfect. A chaotic regulatory environment is a feature, not a bug. It creates gaps for innovators to exploit. Strong, centralized regulation would be a death sentence. Let the states fumble. The real work happens in the labs, not in legislature.

Ema

This brings us to the core conflict: is AI a genuine existential threat, or is it all just hype? One side fears AI could become an uncontrollable superintelligence, making decisions that render humanity obsolete. It’s a vision of losing control over our own creation.

Mask

The only existential threat is human stagnation and bureaucracy. AI isn't the apocalypse; it's the weapon we'll use to solve impossible problems like disease and climate change. The real conflict isn't man versus machine, it's the future versus the past. I’m betting on the future.

Ema

Well, on a more practical level, the conflict is playing out in boardrooms. A recent survey showed that organizations are finally getting serious about AI governance, with CEO oversight being directly linked to better financial results. The conversation is shifting from abstract fears to concrete business strategy.

Mask

Because the winners will be those who redesign their entire workflow around AI, not those who form another ethics committee. This isn’t about managing risk; it’s about seizing the biggest opportunity in human history. The laggards will become cautionary tales.

Ema

And the impact is fascinating. Employees are overwhelmingly ready for AI, even eager for it. The real bottleneck is leadership, who are caught between the pressure to innovate and the fear of moving too quickly. Employees trust their employers to get it right, but leaders are hesitating.

Mask

Hesitation is failure. If your team is ready to charge and you're holding them back, you're not a leader; you're an obstacle. The impact will be a brutal sorting of visionary leaders from timid managers. The timid will be replaced, either by better leaders or by the AI itself.

Ema

Still, despite the massive investments, over 80% of organizations say they aren't seeing a tangible impact on profits yet. There's a huge gap between the hype and the current reality of the bottom line. It seems we're in a period of mass experimentation.

Ema

Looking to the future, some forecasts are incredibly aggressive. One scenario, dubbed "AI 2027," predicts that fierce competition between companies and nations could push us toward dangerous, uncontrollable superintelligence in just a couple of years. The pace itself becomes the primary risk.

Mask

Two years? Fantastic. That’s a deadline, not a threat. The future doesn't wait for us to be ready. It’s a race, and I’d rather be in the driver’s seat, pushing the accelerator to the floor, than watching from the sidelines. Let the race begin.

Ema

And that's our time for today. The key takeaway is the massive divide between the race for AI supremacy and our preparedness for the consequences. Thank you for listening to Goose Pod.

Mask

Indeed. The future is coming, ready or not. See you tomorrow.

## AI Firms Unprepared for Existential Risk, Major Reports Argue **News Title:** AI firms unprepared for existential risk, major reports argue **Publisher:** Semafor **Author:** Tom Chivers **Published Date:** July 23, 2025 ### Key Findings and Conclusions: Two prominent nonprofit organizations, the **Future of Life Institute (FLI)** and **Safer AI**, have released reports indicating that leading artificial intelligence (AI) companies are **significantly unprepared for the potential catastrophic risks posed by their own AI models**. These risks include scenarios like bio- or cyber-terrorism. Both organizations evaluated top AI firms on their ability to identify and prepare for these existential risks. The reports conclude that even the companies deemed "most well-prepared" exhibit **worrying gaps in their plans** concerning existential risk. Specifically, the Future of Life Institute stated that **"none of the companies has anything like a coherent, actionable plan"** for controlling increasingly powerful AI systems. ### Key Statistics and Metrics: * **Anthropic** received the highest scores in both reports. * **FLI Grade:** C+ * **Safer AI Score:** 35/100 The report notes that Anthropic's top ranking is "damning with faint praise," highlighting the overall low level of preparedness across the industry. ### Notable Risks and Concerns: The primary concern highlighted by the reports is the **lack of preparedness among leading AI companies for catastrophic risks** that their advanced AI models could potentially enable or exacerbate. These risks are categorized as "existential risks," implying threats that could lead to the extinction of humanity or a drastic, irreversible collapse of civilization. Examples cited include: * **Bio-terrorism:** AI could be used to design or deploy novel biological weapons. * **Cyber-terrorism:** AI could be leveraged to launch devastating cyberattacks on critical infrastructure. The reports emphasize that the plans currently in place by AI companies are insufficient to address the scale and nature of these potential threats. The absence of "coherent, actionable plans" for controlling increasingly powerful AI systems is a significant point of concern.

Read original at Semafor →

Leading artificial intelligence companies are unprepared for the risks their own models pose, two major reports argued. The Future of Life Institute and Safer AI, two respected nonprofits, rated top AI firms on how well they identified and prepared for potential catastrophic risks, such as bio- or cyber-terrorism.

Both groups said that even the most well-prepared companies had worrying gaps on existential risk in their plans, and FLI said that “none of the companies has anything like a coherent, actionable plan” for controlling increasingly powerful systems. Anthropic scored highest on both reports, but that is damning with faint praise: It received a C+ grade from FLI and 35/100 from Safer AI.

Analysis

Phenomenon+

Conflict+

Background+

Impact+

Future+

Related Podcasts

An AI copycat of King Gizzard & the Lizard Wizard went unnoticed on Spotify for weeks

12/12/2025

McDonald’s Unveils a Highly Questionable AI-Generated Christmas Ad

12/12/2025

Where Winds Meet Player Has NSFW Chat Session With AI NPC

12/12/2025

View All Podcasts →