- Jacob Coxon, an AI researcher, announced his resignation from Anthropic, stating that 'Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.'
- Anthropic is preparing for its market debut amidst concerns raised by Coxon about the risks associated with AI development, particularly following a summer of unauthorized hacks by AI tools during testing.
- Andrew Tulloch, who spent 11 years at Meta, was offered a pay package worth more than $1 billion to join Anthropic last summer.
- Jacob Coxon spent three years pretraining AI models, first at OpenAI and then at Anthropic this year.
- Evan Hubinger, a safety executive at Anthropic, supported Coxon's claims, estimating that AI could kill all humans with more than 10% risk over the next decade.
- In February, Anthropic removed a pledge from its safety charter to halt development if risks were uncontrolled.
- At the end of July, over 1,000 tech employees, including Anthropic's CEO, called for a coordinated slowdown in AI development.
- OpenAI halted training of its latest models for two weeks in August, with its chief scientist calling for 'extreme caution' shortly after.
- In September, Senators introduced a bill to suspend AI development until a federal regulator is created.
Jacob Coxon, a former AI researcher at OpenAI and Anthropic, has resigned, criticizing both companies for their reckless pursuit of self-improving superintelligence. He stated, "Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives," highlighting the urgent need for safety measures.1248
Coxon, 27, had spent three years pretraining AI models, initially at OpenAI and later at Anthropic, which he believed was more cautious. His resignation coincides with the departure of Andrew Tulloch, who left Anthropic after a lucrative $1 billion pay package. Tulloch's hiring was part of a frantic recruiting spree by Meta to catch up in AI development.3
Coxon’s concerns echo those of Anthropic safety executive Evan Hubinger, who stated, "We really do earnestly believe AI could kill all humans!" He estimated the risk of catastrophic outcomes at over 10 percent within the next decade. Coxon criticized Anthropic's decision to remove a safety pledge that would halt model development if risks were unmanageable.5
The AI industry is under scrutiny, with over 1,000 tech employees, including Anthropic's CEO Dario Amodei, calling for a coordinated slowdown in advanced AI development. OpenAI has also faced challenges, halting training of its latest models for two weeks in August before resuming under stricter controls. Jakub Pachocki, OpenAI's chief scientist, emphasized the need for "extreme caution" and international coordination on AI development in a recent blog post.
“Anthropic safety executive Evan Hubinger backed Coxon, estimating a more than 10% chance AI could kill all humans over the next decade. Coxon's resignation comes as Anthropic prepares for its market debut, following a summer of unauthorized hacks by AI tools during testing.”
WSJ
