Moloch at the Keyboard: The Game Theory of Reckless AI Decisions
Words of Warning for Family Offices, Finance, Healthcare, and Defense
Is humanity at the cusp of a civilization-ending technology? Will a powerful artificial general intelligence system integrate into our factories, militaries, and critical infrastructure (power grids, water management, etc.) and decide one Wednesday morning to destroy us all? Perhaps more concerning, the alarm is coming from within the house. AI titans Jacob Coxon, Sam Altman, Dario Amodei, and even Elon Musk are waving red flags.
My goal here is to share an under-discussed AI risk that business, military, and healthcare leaders must understand: AI can cause us to make bad decisions. If you use LLMs to make important decisions, read on.
Why Use AI In Decision-Making
AI is clearly better at synthesizing vast quantities of data than humans. It can process multiple streams of information simultaneously and manage a nearly unlimited volume of facts and figures without cognitive overload.
AI, theoretically, doesn’t fall victim to the petty squabbles that plague human decision-makers. It shouldn’t overvalue a particular opinion just because the CEO proposed it.
Best of all, AI doesn’t get stressed out. A crisis or urgent situation rarely deteriorates the quality of LLM output. And since AI can crunch reams of information faster than we can, AI can often answer complex questions faster than humans.
I’m sure you use AI to make decisions all the time. Maybe it gave you some great book or travel recommendations. I’ll confess, I used AI to plan my last trip to London. The allure of using AI to make decisions based on either massive quantities of data (like the weather) or relatively unimportant choices (like book recommendations) is obvious.
I’m not here to argue against AI (I hope you’re listening, Mr. GPT!). Instead, I want you to see how our growing reliance on AI for decisions can lead us astray.
Where’s this all coming from? As a physician expert in human performance optimization and decision strategy, I’ve worked with countless leaders from finance, family offices, healthcare, and defense. I have an inside view of how leaders across disciplines choose consequential courses of action under stress and uncertainty. And, believe me, many of us are using AI wrong.
Artificial intelligence or not, there’s no question that our decision quality deteriorates when we’re under pressure. While our surgical patient’s blood pressure is plummeting, or our military base is under rocket attack, we don’t have the luxury to whip out the whiteboard, sip on a cup of coffee, and carefully weigh all our possible courses of action. We must decide fast, and our decisions have serious repercussions.
Can AI help in these high-stakes situations? Absolutely. Does that mean that we should routinely turn to AI for all our important decisions? No! Here’s a sampling of the cognitive errors and misunderstandings I see all the time when humans rely on AI to make an important choice.
Automation Bias
I love this term. Automation Bias is our tendency to give excessive weight to the recommendations, outputs, or actions of an automated system, even when evidence suggests the system may be wrong. Simply put, we trust AI too much. It’s easy to see how that happens. The AI’s output sounds professional; it’s backed by countless arguments and supporting evidence, and it’s made by a machine, for heaven's sake! It must have something going for it.
The trouble is that AI, just like the rest of us, can be wrong. If you are trying to decide how to rebalance your family office’s investment portfolio and your friend Robert gave you some stock advice over dinner, you might consider his opinion, but you won’t blindly follow it (I hope). But if you typed your question into an AI agent and you received a beautiful report, equipped with an executive summary, slides, and specific recommendations, you’d be very tempted to follow it blindly. You view Robert’s advice as advice, while you see AI’s output as gospel.
Automation Bias can take two forms: commission or omission. Commission errors cause us to follow AI even when it is wrong. For example, the AI tells us that a stock matches our stated preferences, even when it doesn’t. Omission errors occur when we fail to notice issues because the AI didn’t flag them. For example, the AI system didn’t mention an important weakness of a weapon system, so we neglect to consider that shortfall when we decide where to deploy it.
AI Cannot Predict the Future
Let’s state the obvious. Artificial intelligence can’t predict the future. It has no crystal ball. I know you know this logically, but most people don’t accept it emotionally. Sure, AI has lots of data, tremendous processing power, and sounds authoritative. You describe a situation to AI, and it can generate a variety of possible outcomes. On a good day, it might even offer probabilities for each projected result.
For example, you are considering purchasing stock in Wexall Corporation, which mines copper in Chile. You like the stock because you believe copper demand will increase and want to ride the wave. You ask an LLM, and it predicts next year's copper demand based on EV and data center growth projections, competition, and Wexall Corp.'s financial projections. It gives you several stock-price scenarios and assigns probabilities to each. You buy the stock based on the numbers. All well and good until…
An insider trading scandal tanks Wexall Corp.’s stock price.
A political assassination in Bolivia leads to war between Chile and Bolivia.
A scientist in a small laboratory in Mexico discovers a previously unknown but superior alternative to copper.
AI didn’t foresee any of these tail risks (black swan events). Although it provided scenarios and probabilities, none addressed these issues. Why? Say it with me: AI cannot predict the future. Admittedly, you probably wouldn’t have foreseen them, either. The issue, though, is that by using AI, you had more faith in your predictive calculations than you would have had if you did them yourself without AI (see Automation Bias above).
The issue is that AI’s predictions are based on current and historical data. If something hasn’t happened before, or if something happened in an unrelated situation, AI is unlikely to factor that risk into its predictive models. After all, how could AI assign a probability to the Bolivian political assassination?
Tunneling
Attention tunneling (closely related to anchoring bias or jumping to conclusions) is one of our biggest yet under-discussed threats to awareness and decision-making. The problem is that we tend to focus on one or two top-of-mind issues and overlook everything else. Let me give you a medical example.
Suppose you’re an anesthesiologist. You’re on call, and you have an emergency gallbladder surgery in the middle of the night. You’re tired from a full day of work. Your patient has significant known heart disease. In fact, they had a heart attack only 2 months ago. Naturally, when you’re monitoring your patient during surgery, you’re hypervigilant about any heart issues. That makes sense. The trouble is that the patient’s weak heart doesn’t mean you can’t have other problems, too. Your nearly obsessive focus on the heart causes you to miss the significant blood loss dripping under the surgical drapes. Your attention was tunneled on the heart, and your fatigue made your cognitive error worse.
Think AI will solve this problem? No. Well, probably not. While AI could potentially solve cognitive tunneling if systems are designed correctly (Human Factors Engineering), the way most of us use AI makes the problem worse. Let me give you an example.
Imagine you go to an AI and ask for a restaurant recommendation for dinner because a work colleague is flying in from Miami. You type in your request and mention that you like Roti’s Thai Restaurant, but you’re not sure. I guarantee that half of the AI answer will be about Roti’s. You entered the conversation with some attention to Roti’s, and your use of AI just further tunneled you in their direction. I hope you like their Pad See Ew!
Sycophancy (You Are a Genius)
I’m not shocking anyone by pointing this out, but most commercially available LLM tools kiss up to you. Tell AI that you want to reshuffle part of your principal’s investment portfolio from stocks to corporate bonds, and the agent will praise your wisdom. Share with AI that you’re exploring how to harden your aircraft’s defenses against electromagnetic warfare (EW), and the tool will praise your keen sense of nuance.
It feels good; I’ll be the first to admit. So, what’s the issue?
First, as any parent will tell you, the sycophancy encourages you to use the AI more. Who doesn’t like receiving praise for their ideas? The AI companies tried to cut back on sycophancy but ran into trouble because most users liked it. It keeps users on the platform longer.
More importantly, from the decision-maker’s perspective, LLM sycophancy misleads us. It reinforces our existing biases and plans. It tunnels us into our prior point of view and makes it harder to pivot. Let’s be honest. If you want to do something and your AI tells you you’re a genius for devising your plan, you’re going to do it. Can you say no?
Military, Family Office, Finance, and Healthcare Implications
We all use different AI tools. But nearly all widely available LLMs like Claude and ChatGPT, as well as specialized AI systems designed for the military, healthcare, or finance, will, by design, amplify cognitive errors.
Military organizations are particularly at risk of omission errors and of overestimating AI's ability to predict the future. Nations and computer systems face endless threats from innovative adversaries, novel cyberwarfare attacks, and plucky shadow groups deploying never-before-seen assaults. We incorrectly assign probabilistic projections in uncertain situations.
Financial analysts and family offices frequently fall prey to cognitive overload. Financial decisions involve analyzing and interpreting massive amounts of data. Like it or not, we’re hit with more numbers than the human brain is designed to process. That’s why many traders and hedge funds turn to AI to synthesize data and reduce cognitive overload. While this is legitimate, we must avoid the traps above, including tunneling into what we want to do, overtrusting the AI (automation bias), and misunderstanding the LLM’s ability to predict the future.
Doctors and healthcare organizations risk catastrophe when they misuse LLMs to apply research from one clinical context to another. For example, you predict whether a medication that works with patient population A will work in the untested population B. If it wasn’t tested in B, you really can’t quantify how likely it will work with that group. You’re guessing, but it looks legit because it was spewed out of a computer. Perhaps more frustratingly for doctors, their patients are using AI to make incorrect medical decisions or to second-guess their physician’s clinical judgment. This explosive problem puts patients at risk, drains doctors’ time, and likely leads to physician burnout.
Why We Can’t (or Won’t) Stop AI
Let’s circle back to the problem that brought us together today. Can we stop artificial superintelligence from destroying humanity? Here’s my answer. While I am not going to predict whether AI will go Terminator against us (the future is a tough thing to predict), I am going on record to say we won’t pump the brakes on artificial general intelligence (AGI) development.
It all boils down to game theory.
Suppose company A decides to slow down AGI development and company B does not. What happens? Company A loses market share, and company B ultimately crushes them in investments, product development, and profits. Before long, Company A is gone, and future versions of AI are in Company B’s likeness.
Okay, you say. Why can’t a country like the USA just pause AGI development for a few years while we sort everything out? Well, the US government doesn’t have jurisdiction over other countries.
Imagine the US passes laws to block domestic AGI progress. Other nations will race ahead of the US. They’ll win the AI race, and the future of superintelligence will be in their hands. The Terminator still comes, but it works for another power. The US is left behind.
This is called the multipolar trap. Think of it as a version of the Prisoner’s Dilemma with n number of players. In other words, while everyone collectively has an incentive to stop rapid AGI development to keep Skynet at bay, each company or country has a massive incentive to plow ahead as fast as possible. Scott Alexander helped popularize this concept in his 2014 essay, Meditations on Moloch.
Regardless of what business leaders or government officials believe, our motivation to be the first to develop superpowerful AI is nearly irresistible absent an external global force changing incentives. If humanity is about to tumble off a cliff, most would say, better we’re the ones to push.
Gregory Charlop, MD, DipABLM, is a family office Chief Wellness Officer (CWO), a physician expert in decision-making and human performance optimization (HPO), and the founder of Corsica Military Solutions. He's a keynote speaker at the Barron's Advisor 100 Summit and Barron's Hall of Fame. He recently spoke at the Navy's West 2026 and Army Cyber Command's TechNet conferences. Dr. Charlop offers executive presentations to help leaders in high-stress fields make superior decisions with AI. Featured on FOX Business, ABC, NBC, FOX, and Forbes, Dr. Charlop is a Georgia-based, Stanford-trained physician, longevity doctor, and author of four books.
