Trusted #003 - The Promise and The Peril of Agentic AI
Consider these three events in the past month.
Auto-GPT was released and took the AI Twitter community by storm. This open-source product, which is an experimental attempt to make GPT fully autonomous, hit #1 on GitHub trending and is currently the #63rd most-starred project on GitHub of all-time.1 The current trending as of this writing includes Auto-GPT and three alternatives/derivatives, autogpt-gui, babyagi, and AgentGPT. (Note that AutoGPT-type applications is currently mostly useless; it’s interesting as a proof-of-concept, but way too unreliable yet to do anything. Any examples you see are likely cherry-picked from hundreds of failures.)
Stanford researchers built a Sims-like simulation, with 25 simulated humans all “controlled” by an LLM and interacting with each other (paper and canned demo). The characters were “preloaded” with a starting set of information (identity, goals, relationships, rough plans for the day) and then freely allowed to interact in a modeled sandbox town. Over time, emergent relationships and information transfer developed; one character started with a plan to host a party the next day, and by day 2, twelve of the characters has learned of the party, and five had decided to attend.
In late January, the U.S. Department of Defense issued a revised directive on the use of autonomous weapons, and there was a large international conference in February. The issue is attracting attention as the use of drones in Ukraine is accelerating rapidly.
The dawn of widespread agentic AI’s is coming. (Definitions clarification: a “tool” AI is one that calculates which action will maximize a parameter, based on a dataset, and then outputs this calculation in some manner; an “agent” AI is one that calculates the same thing, but executes an action.) Are we ready?
Agentic AI’s: The Promise
There are already some narrow “agentic AI’s” in use today, for use cases in which a human’s reaction time is too slow or the problem set is too broad. Algorithmic/high-frequency trading companies spend millions of dollars on hiring the best data scientists to designing algorithms that can automatically make stock trades based on changing market conditions.2 Banks and social media companies run complicated fraud and spam detection algorithms that automatically block a small % of transactions or messages without human intervention. The Phalanx/Centurion point defense weapons system automatically engages incoming targets without human targeting (for which I was very thankful, once upon a time in Iraq). In general, though, humans stay in the loop on decisions…but there’s no reason they have to.
Let’s look at calculators. For centuries, bankers, scientists, etc. were trained on how to do arithmetic at speed due to it being an essential part of every transaction. Schickard, Pascal, Leibniz, and Hahn all developed bespoke calculating devices in the 17th and 18th centuries, but those devices were not economically feasible. It took de Colmar’s arithmometer in 1820 to make a device that could perform mechanical calculations reliable enough to bring to the masses. This quote from The Gentlemen’s Magazine in 1857 struck me for its applicability to today:
Instead of simply reproducing the operations of man's intelligence, the arithmometer relieves that intelligence from the necessity of making the operations. Instead of repeating responses dictated to it, this instrument instantaneously dictates the proper answer to the man who asks it a question…It will soon be considered as indispensable, and be as generally used as a clock, which was formerly only to be seen in palaces, and is now in every cottage.3
While simple arithmetic quickly became the domain of the domain of the arithmometer and later the calculator, more complicated computing was still clerical human work. Human “computers,” many of them women, would receive instructions, do the work, and return a result. (Hidden Figures is a decent movie about this and worth watching.)
In the end, what humans really want is stuff done. We don’t want to know what stock trades to make, we want them made. We don’t want to know what missiles are coming in, we want them shot down. We don’t want someone to teach us how to do multiple-digit multiplication, we just want the product. It will take time for agent AI’s to become reliable, and cultural considerations may mandate a human staying in the loop even if ineffective; but that’s all. The only thing holding them back previously was the need to develop and train an AI system for each individual use case. Now that we have systems that is trained on language and images generally, it’s a lot easier to do that.
That leads to some problems, though.
Agentic AI: The Peril
Agentic AI is the first step to, potentially, some very bad outcomes, though this is highly controversial. (For much more on bad outcomes from AI, see Trusted #001.) Pretty much all the disaster scenarios about AI start with it amassing power in some way, and one way of that happening is just…giving it to them, because they are seen as indispensable, to borrow from the quote above. This, perhaps, is the type of AI I most fear; not the type that destroys humanity in a single cataclysmic swoop, but the one that becomes indispensable to society, and then supplants it.
I remember reading a discussion about AI safety in the mid-2010’s, and it went something like this:
Researcher: I’ve got a foolproof plan. If the AI’s seem dangerous, we’ll just put them in a closed system where they can’t affect the environment. We’ll just ask them questions, like an oracle.
Critic: That seems like a good idea, but what if the oracle convinces you to let it out? Or what if nobody believes that it’s a good idea to lock them up to begin with?
Researcher: Oh, I’m certain people would take the signs seriously.
Now in reality…ten years pass
Half of Twitter: Hey, wouldn’t it be funny if we created an AI Agent and gave it the goal to destroy the world. What could go wrong?
Oops.
This is a great example of Hasbrouck’s Paradox in action, which I’ve thought about quite often in the context of computer security but never actually defined before now.
Hasbrouck’s Security Paradox: The greater the treasure is to the kingdom, the heavier the lock required; the heavier the lock, the harder to turn the key; the harder to turn the key, the more the king hesitates to lock it.
As AI capabilities become better and better, they will become more and more desired. The hype we’re seeing now is nothing compared to the hype we’ll see as more capabilities get rolled into wider use, like the upcoming Microsoft 365 Copilot for Office or Google’s Workspace AI for Gmail. Consider this hypothetical progression of AI functionality:
“auto-set your out-of-office message based on your current calendar”
“auto-generate an out-of-office message based on previous messages”
“auto-generate thoughtfully crafted replies to messages based on previous messages”
“auto-send thoughtfully crafted replies to messages"
This will likely take a few years, but once we get to that point, you think anyone is willingly going to stop and say “oh wait, this is dangerous, let’s think about this?” Nah. The most likely scenario is a series of “accidents” from agents that get explained away as poor configuration or user error. I fully expect “hackers” to be blamed the first few times an agent “auto-emails” proprietary company information out, for example.
(Given this, debating “should we pause AI development or not” seems like a silly debate because we clearly won’t. instead, let’s prepare for the things that can be done, like increasing funding for “reliable” and “trusted” AI. See Trusted #002 for more on this).
Where do we go from here?
I don’t really have a good concluding point beyond “AI agents are going to be super-powerful AND super-dangerous,” so I’ll wrap this up with a prediction of how I see things playing out (US based). This will almost certainly be wrong, but it’s fun.
2023: Hype continues to slowly build around AI, with big bumps when AI tools are made more widely available to the public. Lots of new “powered by GPT-4” apps launch. General-purpose agents never really get past any proof-of-concept stages.
2024: AI hype fades as the pace of developments slow; turns out GPT-4 is good, but not good enough to do most of the things people promised out of the box. A lot of companies are working very hard behind the scenes to build in error-checking to LLM outputs to build “GPT-4 for X” tools. Most of the discourse around AI shifts back to disinformation in the runup to the U.S. political election. Stories return about social media algorithms and their biases in what they choose to/not to promote, but chatbots get added into the mix as well. The first general-purpose agents actually begin showing some promise, but are unreliable enough for most use cases. Lots of businesses begin using “tool” AI’s. Games start experimenting with LLM-driven characters; at least one game has to patch out LLM functionality after trolls make the character say horrible things .05 seconds after the game is released.
2025: The behind-the-scenes work bears fruit and agents are now broadly available! General agents that say they will aggregate everything are still not very good, but you have system-level agents; you have a Twitter agent that will summarize posts for you and generate appropriate replies; you have a New York Times agent that will customize a front page for you and surface stories as requested; you have a Gmail agent, etc. All of these agents incorporate chat and voice features; lots of thinkpieces about “the new Siri.” The warnings will come off the “reliable things” and go on the “unreliable things,” meaning actual emails/transactions/etc. Plenty of early adopters will ignore the warnings and much chaos will occur. The lines between “chatbots” and “assistive agents” start blurring later in 2025, as companies start adding personalities to their agents to increase engagement. At least one major game comes out that is near-fully LLM-driven.
~2026: “GPT-5” level tech comes out, and all those mediocre general agents start getting good, very quickly. Companies get protectionist with their data at first, but eventually “agent-compatibility” overrides that, as a few of the best general agents take over the market. These companies compete on feature availability and “persona” availability. Celebrities begin licensing their “persona” and pursuing legal action against non-licensed agents, though a thriving black market exists. Agents become better at “understanding” their limitations and knowing when to return to the user for input; this increases trust and usage. “Have your agent contact my agent” becomes an actual thing that normal people can do.
Outside the “personal assistant” market, businesses deploy general agents rapidly. Large businesses limit the use of the general agent based on their existing culture and resistance to let the AI make decisions; they slowly start losing ground to businesses that are more AI-integrated, though this takes much more time than people expect. Militaries, generally even more bureaucratic than businesses, deploy “decision support” tool AI’s at multiple levels but resist doing more than that.
~2030: “GPT-6” and/or some other breakthrough comes out, and things get weird. AI “incidents” begin happening as humans turn power/oversight over to agents too quickly; most are small and explained away (“oops, the AI agent controlling traffic flow didn’t recognize that pedestrian, his fault for not wearing normal clothes,” some are moderate and inspire consternation (“oops, our agent zeroed out our vaccine production due to the ROI not being high enough, we’ll have to ration” and a few are big (“oops, our AI decided that making itself more powerful was the most effective way to do business”) We’re way too culturally enmeshed in AI to stop using it; hopefully the existentialists are wrong about the whole “superintelligence” thing!
Conclusion
The rapid development of agentic AI presents both potential and peril. The potential benefits are clear, but the risks are much less understood. Understanding and preparing for the various outcomes of AI integration is essential in ensuring a future where technology serves humanity, rather than supplanting it.
(Further Reading: See this great roundup on AutoGPT by Zvi Mowshowitz, and Gwern from 2016! on tool vs agent AI’s.)
Standard disclaimer: All views presented are those of the author and do not represent the views of the U.S. government or any of its components.
GitHub stars is not a great way to measure impact, admittedly.
It used to be that the best-paying CS/DS jobs are at trading firms like Two Sigma, Jane Street, or Citadel (not a comprehensive list) then the big tech companies, though that may have changed recently given all the money flowing into AI.