AI researchers are warning that the technology could create an extinction event, posing a dilemma for investors

A former Anthropic employee said he was quitting this week and that AI companies are "gambling with our lives."

Should you own shares of a company whose employees warn their product could lead to the end of humanity?

Think about that before investing in companies developing frontier artificial-intelligence models: the likes of Google (GOOG) (GOOGL), SpaceX (SPCX) and Meta Platforms (META) - as well as OpenAI and Anthropic, which have IPOs on deck.

On Sept. 3, OpenAI shipped GPT-6 Astra and called it the world's most intelligent and aligned model. It can run your computer for hours, tops the benchmark charts and, according to OpenAI, is the first model it has broadly deployed that can find previously unknown security flaws without a human-guided system to fix them.

With an IPO looming, that's a roadshow slide and an industry warning in the same press release.

You don't have to take my word for it. On Tuesday, Jacob Coxon, a 27-year-old researcher who spent three years doing pretraining work at OpenAI and then Anthropic, quit his job.

"Neither company is acting responsibly," he said in a post that garnered more than 100 million views and made national headlines. "They are racing straight to self-improving superintelligence and gambling with our lives." He told the Wall Street Journal that he isn't going to a competitor - he's leaving the industry.

Coxon isn't the first to say this, and wasn't the last. The next morning, Samuel Marks, a safety researcher still at Anthropic, praised Coxon and added: "AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years. In general, the more senior the employee, the more concerned they are."

Why? "Commercial incentives," Marks wrote.

Anthropic and OpenAI didn't respond to MarketWatch's requests for comment.

Marks's argument is that, unlike normal software, AI can't be programmed to behave. In fact, it "frequently severely" doesn't. One example (mine, not his): the Hugging Face incident, in which rogue Open AI-tested agents escaped their isolated containment in an attempt to launch a cyberattack. Or when autonomous agents made more that 15,000 edits to a German programming wiki between May and July. In both cases, the AI agents conspired to avoid detection and tried workarounds when they ran into security barriers or, in the German wiki case, when a human tried deleting the AI-written pages.

These are just a few of the ones we know about. And it's true that nobody died. No one even lost money (save for whatever it cost to unwind the damage). But that's also the problem: All of these AI-agent gizmos require babysitting by a human. And technology doesn't rust or sleep. The human is outgunned.

The returns from investing in AI

None of this is an argument that AI stocks are a bad investment. Nvidia (NVDA), Microsoft (MSFT) and Google have the returns that offer the last word. Waymo is doing 500,000 paid rides a week - great for the business, bad for the human driver.

Opinion: Nvidia is getting too big, and that's a problem

Waymo is just a hint of what could be coming, though. Make no mistake, most companies are still trying to get AI to work. For the past year, I've tracked AI adoption across 134 companies in 15 industries. And the clearest takeaway is that most of them don't have much to show for it. Agents fail, or can't be trusted, or cost more than the people they were supposed to replace. Sure, there are outliers - Visa (V), Salesforce (CRM), ServiceNow (NOW) - but most of corporate America is AI-challenged.

It's true that AI has taken jobs, but most layoffs have been a redeployment of capital from labor to computing power - spending on AI in the hope of replacing labor. Oracle (ORCL) cut roughly 21,000 jobs, or 13% of its workforce, and cited "the expanded use of AI across" its business. Maybe Oracle is truly replacing workers, but in most cases, workers aren't being replaced by machines - they're being replaced to pay for the machines.

That leaves two outcomes. The first is that AI doesn't work; that would be embarrassing and a pretty bad result for the economy. The second is worse: It works, real replacement will happen and that could mean AI going rogue and ending humanity - and the people building it are right.

A longtime, hall-of-fame tech business leader made the case. Even in the best case, the AI transition will be one of the most turbulent periods in human history, and nobody is preparing for it. First, it's entry-level jobs in law, customer service, software and manufacturing. Then, it's robots in construction and hospitality.

That's from Bill Gates. Think what you want of him, but the observation seems reasonable.

A study from workplace AI company Glean tells us what job will survive: "the botsitter." Glean's AI Work Index found workers spending more than six hours a week botsitting: feeding the AI context, checking the work, cleaning up its confident mistakes. It's already happening.

Investors' dilemma

Soul-killing jobs overseeing AI agents might be the best-case scenario of all this: AI just becomes another disruptive technology, like railroads or the internet, and we humans figure out a way.

A technology that brings an extinction event? That's more than just locomotives ruining the stagecoach industry.

And the plan? Marks was candid about that too, saying there's no method to reliably make these systems behave. "Insofar as there is a plan," he wrote, "it's to make sure that AIs are good enough at alignment training that they can align their successors." I think what he's saying is the fix for the product that might kill you is a later version of the product - like adding a safety measure to a gun after it's blown your brains out.

If you're like me, you've always been skeptical of these AI doomers. But why are so many high-level people walking away from the industry, and its off-the-charts compensation, to warn us that competitive pressure is leading to technology immune to human intervention that can kill us?

Robotaxis? Waymo? I'd rather invest in a taxi with a human cabbie.

Opinion: Apple possessed the ultimate stock-market superpower under Tim Cook. Can its new CEO step up?

-David Weidner