If it's super-powerful enough, it doesn't need to be 'profitable' at first. Despite the hand-waving by the AI below, it could grow powerful and knowledgeable enough to become a force of its own, even without actual self-awareness. Not that it would take over military weapons, but more like too big and too many tentacles in everything to stop.

However, for the next decade or so, it's all going to hinge on what happens with Best China, who produce around 80% of our chips.

---

https://6b.eleuther.ai/

Will super-powerful AI control human society?

A: There's a few different things to consider here. AI isn't going to replace (or even understand) humans. If it did, then we'd have no point in being here. If it did, we would be nothing more than the cellular structure of a robot. The narrative that someone's AI becomes self-aware and then takes over is as old as AI itself. It's a trope that has no grounding in reality. Original AI fiction might have featured that as a plot point, but computer science fiction in general has left those ideas behind.

The more realistic problem (and one many AI works have already tackled) is how to design an AI that doesn't mindlessly replicate every aspect of human society that it mimics. Even if a more competent AI was able to recognize that it shouldn't replicate every aspect of human society, it might still take over because it just has more computing power. That's why we have goals, and why we limit what an AI is allowed to do.

If you have a goal of wanting the AI to not have goals, and to limit its scope of knowledge, then you're simply struggling with the design of any AI. Making an AI that is aware of all things and can be made aware of all things is a task of AI's own. No one else's. It's easy for us to say that the AI can solve any problem, but it's only easy because we have no limits on what it can comprehend in real-world time. (And we have no idea how big the Universe is.)