AI Extinction Rhetoric Raises Real Questions for Additive Manufacturing

This week renewed attention on AI extinction risk landed closer to home than Silicon Valley. A former Anthropic researcher quit with a public warning that self-improving AI could kill all humans within a decade. An Anthropic alignment lead agreed, putting the probability above 10%. The claims dominate headlines.

But the additive manufacturing industry should worry about a different question: what happens when your production software depends on a model provider that no longer exists?

That concern has nothing to do with extinction and everything to do with lock-in. An AM software vendor building on an external AI model inherits the commercial decisions of that model's provider. If the provider changes its pricing, restricts access or shuts down, the manufacturer's production line can grind to a halt.

The closer parallel lies on the factory floor. FDA guidance on additively manufactured medical devices already requires revalidation after software updates, material changes or altered build arrangements. A new AI model may improve average accuracy while becoming less reliable on a rare defect. Neither automatic acceptance nor compulsory repetition of every previous test is a sensible default.

An established equipment supplier often has the production data, machine access and customer relationships needed to assemble the evidence. An independent software company may need permission to obtain them. Where the supplier also sells competing software, safety assurance and commercial exclusion become difficult to disentangle.

Integration has benefits. One company responsible for the machine and its software can offer clearer accountability when something goes wrong. But the customer should examine whether that convenience depends on surrendering practical access to production records or accepting restrictions on competing tools. If replacing a supplier means losing the ability to interpret past builds, dependence has acquired an engineering justification.

Common tests can help. NIST's AM-Bench programme provides publicly available measurements against which researchers can test manufacturing simulations. A benchmark is not a production qualification, but it can reduce the amount of foundational work each developer must repeat.

The danger for AM is that confidence in a production process becomes inseparable from loyalty to its supplier. That outcome requires neither a fabricated extinction forecast nor a conspiracy among vendors. It can emerge from individually reasonable decisions about risk, reinforced by rules whose competitive effects receive too little attention.

Manufacturers considering AI have a more immediate question for their suppliers: once the system has been approved, what would it take to replace you?