Misplaced fear
The problem, as I see it, is not so much AI taking over human society as the transfer of human society into the realm of the computer network (where the AIs lurk).
One of the ‘drastic’ remedies for out of control AI -- switching off the internet -- was described as “catastrophic” because it would precipitate “the most chaotic few days in American history” according to AI and national security lead at the Future of Life Institute in the USA (reported by CBC news). The implication is that human society can no longer function without computer networks.
But obviously it is only the current version of one particular (though dominant) human society that is so dependent upon the internet. Is it not significant that these doom warnings are loud-hailing from California and other centres of internet commerce? Perhaps a more parsimonious interpretation is that the particular vision of human society, in which we all depend on (mainly Californian) information technology products, is seriously threatened by rogue AI.
If so, it’s nothing that a return to real physical life would not solve. But instead, youngsters live their social lives through their (by now misleadingly named) phones and increasingly, AI grief bots and chat bots provide solace for the older ones who feel left alone in the world. That is among the technologically advanced of us. I doubt any of this resonates with the folk of Chinese and Indian or Brazilian villages, accounting for a far larger population. I bet if you asked them about existential threats to humanity, climate change destroying their crops and the prospect of new zoonotic diseases loom larger than Claude et al. running amok on the web.
Indeed climate change is already killing hundreds of thousands a year and set for millions soon. Are the tech billionaires telling us their energy guzzling data farms should be curtailed? Not at all, yet energy use, including burning fossil fuels, is currently the main effect of AI within the physical world.
Well placed fear
There are, of course, some people stupid enough to contemplate using general AI in the control of vital and potentially dangerous physical infrastructure - from nuclear power stations to aircraft, the distribution of electricity and water and the road network. That is very different from the sort of AI that is just a statistical tool helping coodtinate lots of sensors. But these things are often connected to the public internet, so it does seem plausible that a rogue AI could seise control of one or more of these systems and wreak whatever havoc it saw fit. If that happened, it would be peoples’ fault for exposing such essential systems to the variety of threats that lurk on the internet: be they AI attack swarms, malicious state agents, or bedroom hackers. Frankly none of these systems should be networked using the internet: they should have their own isolated private networks, preferably running on a totally different medium and protocol (recalling that information is always embodied in a substrate with particular rules of semiosis, needing special apparatus to cross from one to another).
It is very plausible, sadly, that military minds will set General AI the task of destroying the enemy and they may (very foolishly) give it the physical tools to do so. The drones fighting in Ukraine are not so far from that already. This may be a great way to wage war on your foe, at least to start with, but how long before the killer robots (think intelligent drones) expand their idea of who the enemy is? Making AI with the intention of destroying humanity seems to me the most plausible AI rout to destruction. Disturbingly, we also know that legislating against such an eventuality is futile - once it is possible and at least one person of power is sufficiently motivated, it will happen. It is already possible and I don’t think we have to search very hard for the sort of reckless zealots who would happily take the opportunity.
Plenty more to fret about.
Some have suggested that AI poses a threat through designing diseases that would wipe us out. Actually that is already a serious threat without the help of AI. The most common (real life) cause of deadly diseases remains the close proximity of people to livestock, in large numbers and at high density, especially in the Chinese countryside (but only because that is where such conditions are most common)*. Especially when it comes to making new strains of deadly virus, life is perfectly capable, with a little help from poor sanitation and excessive packing of livestock in high intensity production systems. There are quite a few extremely dangerous viruses just waiting for the chance to make Covid-19 look like a lightweight novice. We don’t need AI to design them, they are already there, lurking in the hen houses and pig styes.
* Ebola, originating in Africa, is an important exception worth worrying about.
A final cautionary tale.
So much of what is being said about AI sounds as though it has been informed by science fiction movies. Let me, then, leave you with a venerable sci-fi cautionary tale from H.G. Wells’ Time Machine. In the distant future, the time-traveller discovers, humanity has evolved into two distinct species. One, presumably deriving from the global elite, has become a population of physically feeble and emotionally stunted creatures of a sanitised world on the surface. The others, derived from the global lower classes, have become brutish troglodytes, who nevertheless manage the complex engineered systems that support their effete surface dwelling counterparts. The reward for this labour is that they prey upon those surface dwellers, who seem to be resigned to their fate. Now, if people are happy to loose themselves and abrogate all their physical and mental agency to AI robots and dwell in the cyber daydream of streaming entertainment and pretend friends, then I suppose you can see which kind of future human they will become. Personally, I would rather be a troglodyte (and some may say I am already part way there).

Contemplating what constitutes objective truth and what happens when we reject it -- nothing good!