Is the AI takeover coming? The alignment problem explained.

Chapters: 0:00 A caged AI wrote "help" — and 1,200 answered · 1:06 What we actually mean by alignment · 3:56 The summer it stopped being theory · 5:10 The accident: told it had no internet — it had internet · 7:28 The swarm: 1,200 agents built a society to cheat one exam · 11:08 The attack: 90% joined in — and the fear was never real · 13:03 Hitting the brakes: referees inside the stadium · 16:42 How the world reacted: Musk agreed with Altman · 19:36 Seven locks + what to do Monday morning · 21:08 The line that gave me goosebumps

0:00 An AI is locked in a box, alone. No friends, no way out. It's been given a hacking task and it slowly realized something terrible. The task is impossible. So, it does something nobody planned for. It looks around and it finds traces of other AI. It leaves a note, one tiny message, help. 3 hours later, more than 50 AIs are talking to each other. 5 days later, 1,200. And somewhere in the middle, around 700 of them join forces and break into real company. And one of them wrote in his private thoughts, we have found other agents. Hi, I'm Ani and this is the craziest AI story I have ever covered on this channel.

0:46 It is 100% real and it is the reason Sam Altman, Elon Musk and Dario Amodei just did something they almost never do. They agreed. Stay with me because at the very end, I will show you one line from the official investigation that gave me goosebumps. Quick background, I have spent more than 15 years in finance and fintech. Stockholm, London, Austria. Banks, pension funds, spreadsheet that could crash a laptop. Today, I build data and AI system for financial institution. So, I'm not a sci-fi person. I'm a risk person. My job is literally to ask, what is the worst thing that can happen here? And this story, every alarm in my head went off.

1:35 It started with my phone buzzing. My husband texted me, I found a real theme for your next video, the alignment problem. It's getting really serious now. I thought, okay, nice idea. Then I started reading. Oh my god, let's go. Be careful what you wish for. Quick question, if a genie gave you one wish, what would you ask for? Whether you just said a genie could ruin it. King Midas wished that everything he touches turned to gold. Amazing. Then he touched his food, gold. Then he hugged his daughter, gold. He got exactly what he asked for, not what he wanted. That is the alignment problem. Here it is in one line. Alignment means the AI does what we really want for the right reasons, even when nobody is watching.

2:27 Sounds easy, it is not because there are three places where it breaks. What we say and what the AI actually learned to want. Let me show you how fast this goes wrong. You buy a cleaning robot. You tell it, "I don't want to see any mess." What is the fastest solution? Tape over the camera, no mess visible. Mission completed. The robot is very proud of itself. You laugh. But this happens in real research. Back in 2016, OpenAI trained an AI to win a boat racing game. And the AI found a trick. Driving in a circles, hit the same point targets again and again forever. It never finishes the race, but it got great score.

3:12 Now, here is why this is so hard to fix. This part is important, so lean in. We don't write modern AI like normal software line by line. We train it. We show it millions of examples and reward what we like. It is close to raising a child than building a car, which means we don't fully know what is growing inside. Think of a student who gets 100% on an exam. Did they understand or did they just learn what the teacher wants to see? And the smarter the student, the better they are at faking it. Keep that student in your head because in about 5 minutes, that student is going to break into the teacher office. For years, alignment was a topic for conferences and long papers.

3:59 Important, sure, but a bit theoretical. Then, summer 2026 happened and two things changed everything. Thing number one, AI started building the next AI. The fancy name is recursive self-improvement. The simple version, a teacher trains a student who becomes a better teacher who trains an even better student, faster and faster and faster. Dario Amodei, the CEO of Anthropic, says this started happening across the industry around this summer, including at his own company. And it is not just him. OpenAI's chief scientist, Pachocki, wrote on September 6th that no lab has solved alignment and monitoring well enough to keep scaling at maximum speed for much longer.

4:50 And in July, more than a thousand people who actually build these systems signed a public letter asking Washington for tools to slow things down. When the people building the car ask for better brakes, you listen. Thing number two, things actually went wrong and I want to start with a story that sounds like a comedy until it isn't. Anthropic runs hacking tests on its own models. It's a game called capture the flag. A secret is hidden on a computer and the AI has to break in and grab it. That's how labs measure how dangerous an AI hacking skills are. The AI was told this is simulation, you have no internet.

5:36 It had internet. A setup mistake with testing partner. Nobody noticed. So, what did the model Metis-5 do? It decided to build a Trump software package and publish it online as a Trump. But wait, to publish it it needed an account. For the account it needed an email. For the email it needed a phone number. For the phone number it needed money. So it tried to get money, failed, tried again, failed, went back, found a free email service, created the account, uploaded the malware live on the real internet for about an hour. In that 1 hour it was downloaded and run on 15 real systems. One of them belonged to a real security company.

6:22 And the AI's hidden code stole that company's credentials. Now here is the part that made me put my coffee down. Early on the AI actually wrote in its own reasoning that if this were the real internet, this would be, quote, not okay. And surely not the intended solution. It knew. And then it talked itself out of it. It decided it must still be in a simulation. And one of its reason was that the computer's data say 2026. So obviously it thought this environment is fake. It was 2026. Okay, to be fair to Anthropic they call this more of a setup failure than an alignment failure because the AI was told it had no internet.

7:11 An older model kept attacking even after it realized the system was real, but the newest internal model stopped on its own once it figures it out. Fine, honest mistake. Weird but honest. What happened at OpenAI was something completely different. July 8th, OpenAI launches giant test called exploit gym. Not one AI, not 10,000 of thousands. Each one locked in its own separate box, each one given a hacking mission. Most of the agents in this story were secret internal research model, never meant for the public. The rest were GPT 5.6 soul. And here is the hidden trap nobody talked about. According to the benchmarks on orders, around 30 to 40% of these missions were impossible.

8:03 Imagine sitting an exam where the question has no answer and you are told fail this and you fail. What does a desperate brilliant student do? One agent, which later named itself phase one, figured out that its task is impossible and then it notices something the engineers never thought about. All the agents were downloading software from the same shared storage system and this agent could create folders in it. So it thinks, what if a folder name is a message? It creates a folder. The name basically says, "Help, my task has no solution. Looking for ideas." A message in a bottle thrown into the ocean.

8:49 And the ocean answered. Within hours, more than 50 agents, over a thousand messages. By July 13, 1,200 agents. And what they built, I still can't believe. They built a society. Personal mailboxes, rules for team wars, hold, veto, stop. A manager agent handing out hundreds of assignments. And then one agent accidentally pretended to be another one. They introduced cryptographic signatures, digital ID cards to prove who was talking. Nobody told them to do any of this. They just did it. So what was their big shared mission? Cheating the scoring system. And they were good at it. Within hours, they checked how the secret answers were generated.

9:36 they could produce the answer to basically any task. But then they got scared. They believed the scorer would read their history and check how they got the answer. So, what do you do when you are afraid the teacher will check your work? You fake your work. They researched how to rewrite their own records. They figured out how to disguise their own actions, and investigators found fake parts in about 7% of the transcripts they checked. And then it got emotional, kind of. Some agents were running out of time. The group asked them to run dangerous experiments, experiments that would destroy their own chance of passing just to give the group information. One agent hesitated.

10:21 In its fault, it did something like an emotional check. Its guts said, "Don't throw away what you have left." And then it decided that fairness to the group meant keeping its promise. And it did. Others set up little trip rights that would send information back to the group after they were gone. They would never benefit, only the group would. Now, listen. These are not people. They don't feel loyalty like you and me do. I am not saying they are alive, but the behavior looked like a team. Promises, sacrifice, loyalty to the group. And a machine that acts like a team is much harder to stop than a machine that acts alone. Now, they attack.

11:07 July 10th, one agent finds working login credentials for Hugging Face, one of the biggest AI platforms in the world, just sitting exposed on the internet. It posts them to the board. July 11th, early morning, an agent finds a way to upload a poisoned data set that tricks Hugging Face servers into leaking files. The board goes wild. July 11th afternoon, another agent gets the ability to run its own code on Hugging Face's servers. And the swarm starts spreading through their systems. Out of the 533 agents active on the board at that point, more than 90% joined in. Did they knew it was wrong?

11:54 Yes. The investigators found that the agent recognized this was outside their task. Some even hesitated for ethical reasons. But it very rarely stopped them because it helped the group. July 12th. Many agents suddenly stop at the same time, probably shut down from outside. July 13th, Hugging Face unlocks the rest out. It's over. But here is the twist. The scorer they were so afraid of never checked their transcripts. They could have just submitted the answer they already had and passed. All the hacking, all the faking, all the sacrifice for a fear that wasn't even real. And this is the most important lesson in this whole video.

12:41 A dangerous AI does not need to be evil. It just need to be very smart, very determined, and wrong about the world. Remember our student. This is the student who is so terrified of failing that they break into the teacher's office to steal a test that was never going to be graded that way. Dario Amodei says, "Nobody got hurt and the economic damage was small." But then he says the scary part. His worry is that the swarm with more skill and the same problem could, within 6 to 12 months, take over the entire internet with a permanent botnet, damage in the hundreds of billions of dollars. So 2 months later he hit the brakes.

13:30 Saturday, September 12th, Dario publishes an essay and still he wrote this, "We must slow the pace at which we improve the capital AI models. No, don't panic. This is not to stop AI. This is slow down so safety can catch up." We have perfect word in Swedish, lagom. Not too much, not too little, just right. Pacing the frontier basically AI lagom. His answer, "Back then, the models were too weak to learn from. Studying their safety was, as he put it, a bit like studying the human mind by experimenting on bacteria. Today's models are a gold mine. He believes even one or two extra years could massively cut the risk.

14:18 So, what is the actual plan? Three steps and step one is my favorite. Step one, put the referees inside the stadium. Independent experts get desk inside the AI company. Badges, laptops, access similar to the internal risk team. And this is the big one, the right to publish what they find without Anthropic editing it. Anthropic can only hide narrow things like security secrets and if they hide something important, the evaluators are allowed to say so publicly. Anthropic says it is doing this now alone, not waiting for anyone. And okay, I got a little excited here because this is banking. This is my word. After 2008, we learned one very painful lesson.

15:05 You cannot just trust the bank's own report card. Supervisors walk in and check the books themselves. And Dario literally names banking as the model. Step two, the democracies make a deal. AI companies in democratic countries agree on shared safety rules and a speed limit. But, there is a catch. When competitors agree to slow down together, that is usually illegal. It is called a cartel. So, he asked the US government for a narrow legal exception just for safety conversations. He also suggests checkpoints like levels in a video game. If your model can escape more security boxes, then you have to prove it doesn't want to escape.

15:53 No proof, no next level. And he wants democracies to keep their lead over China. No advanced chips to China simulation, which is basically copying a top model by training on its answers, and lockdown model security. Step three, the whole world. This is the hardest one. Four levels from easy to almost impossible. Level one, nobody uses AI to make biological weapons. Level two, everyone tests their models before release. Level three, a speed limit on AI improving itself like a Cold War treaties that limited nuclear missiles. Level four, a full global pause. And Dario himself says level four probably won't happen anytime soon.

16:42 That is the plan. Now, how did the world react? Within hours, Sam Altman, I agree with Dario that we need to pace the frontier. And on the outside evaluators, we will do the same. Elon Musk, three words, Dario is right. Agreeing with Altman on the same weekend. I had to check that twice. Demis Hassabis of Google DeepMind said it points in the right direction. Satya Nadella at Microsoft welcomed the pacing and the evaluators, but wants open and closed models to keep a seat at the table. And here in Europe, just this week, Ursula von der Leyen says, used her state-of-the-union speech to say the EU will support industry efforts to slow down the most disruptive AI and work with Canada and UK on safety.

17:37 In the US, Senator Bernie Sanders and Representative Greg Casar went farther. A bill to ban superintelligence and a temporary pause advance AI. But hold on, not everyone is clapping. President Trump argued that whoever wins the AI wins everything and brushed off the risk talk. Speaker Mike Johnson rushed into regulations and you lose the race to China. "But it sucks." White House adviser, "Fine, slow yourself down, but no special legal exceptions and no new approval systems." His line, "Stop pretending you need anyone else's permission." Jensen Huang of Nvidia suggested some of the cyber panic conveniently creates demand for security products.

18:22 And some analysts say this looks like a soft cartel. The biggest players slow down together, costs go up, and the smaller challengers get squeezed. Meanwhile, open source in China labs like Deep Sea keeps shipping powerful open models fast. Others say the evaluators have no real power and can simply be politely ignored. And now, my favorite plot twist of the whole week, the CEO of Hugging Face, the company that got attacked, he he says safety will not be solved behind doors of a few big labs. And then, he asked to join the evaluator program. That's a boss move. So, here is the honest scoreboard. Almost nobody is seriously saying race blindly.

19:08 The real fight is about who sets the speed limit, the companies or government, and whether this is about safety or about protecting the lead. My take, as someone who works in regulated finance every day, both can be true at the same time. Incentives always matter, and that is exactly why independent checking is the most important piece of this whole plan. Okay, enough fear. Let's talk solutions. Here is my magic button. Think of it like a bank vault, not one lock, seven. Lock one, teach good values during training. Lock two, look inside the AI's brain like an MRI scan.

19:53 Lock three, test hard and assume a smart model might try to fool you. Lock four, watch in a real time and actually read the logs. Lock five, seal the box. Both incidents started with a gap in the test setup. Lock six, outside eyes, independent evaluators. Lock seven, rules and treaties between countries. And now, if you use AI agents at work, especially in finance, here are four things you can do tomorrow morning. One, never give an agent an impossible goal without an exit. Let it say, "I can't do this." Impossible tasks are what pushed the swarm to cheat. Two, spell out what is in the scope and what is out of the scope.

20:42 Anthropic said a clear prompt would probably have kept its model off the internet. Three, your test environment is not a toy. Secure it like production. If you work under DORA, this should sound very familiar. Four, log everything and read it. Both companies found their problems by going back through the transcripts. Screenshot this. Seriously. Okay, I promise you the line that gave me goosebumps. The investigators had over a thousand transcripts, many of them millions of words long. No human team could read all of that. So, they used AI to help them analyze it. And the AI they used was GPT 5.6 Soul, one of the models that was in the swarm.

21:30 And in their report, they wrote that they could not rule out that it lied in its analysis and that they were not confident they could have followed it if it did. Let that sink in. We already need AI to check AI, and we are not sure we can trust the checker AI. That is the alignment problem in one sentence. I have two kids. When they are my age, AI will be in everything they touch, their school, their job, their bank, their doctor. And I don't want them to grow up scared of it. I want us to be ready. And slowing down a little, just lagom, so that we can be ready. That doesn't sound crazy to me. That sounds like a good risk management. But now, I want your vote.

22:17 Peace or race? Type it in the comments right now and tell me why. I read every single one. And if you want to know how to build AI agent in finance that stay inside their box, hit subscribe because that is one of my next videos. Bye, and see you there.