The Rise of Algorithmic Authority: A Xenobathite Threat?
“…given two intelligent species with differing requirements on one planet, it is inevitable that, sooner or later, one will exterminate the other.” Wyndham, The Kraken Wakes (1953)
In John Wyndham’s 1953 Sci-Fi novel, The Kraken Wakes humanity’s failure to recognize an impending alien threat leads to catastrophic consequences. Wyndham’s depiction of a world in denial, grasping at conspiracy theories to avoid uncomfortable truths, feels eerily relevant in today’s discourse around autonomous AI agents. As AI systems grow in sophistication, their rise brings forth a modern parallel to Wyndham’s narrative. In the novel, a global refusal to face facts about an alien invasion leads to environmental disaster. Today, societal reluctance or plain dismissal of AI’s potential dangers such as algorithmic decision-making, driven by opaque corporate interests, risks undermining democratic processes. This blog questions whether we are prepared to confront the xenobathite-like threats emerging from the unchecked rise of autonomous AI agents.
Autonomous AI Agents vs Robotic Process Automation
‘It’s been the reluctance to postulate an intelligence that’s been the chief stumbling block,’ Wyndham, The Kraken Wakes (1953)
The concept of autonomous AI agents was introduced in early 2023 with the publication of Yohei Nakajima’s paper, “Task-Driven Autonomous Agent.” In his paper, Nakajima states that:
“In the context of our task-driven autonomous agent, there is a risk that the system might prioritize and optimize certain tasks or objectives without considering the broader implications or their impact on human values. This could lead to unintended consequences, potentially causing harm or depleting valuable resources.”
Nakajima, “Task-Driven Autonomous Agent.”
Autonomous AI agents are designed to exhibit high levels of flexibility, autonomy, and complex reasoning. They adapt to unforeseen situations, learn from new data, and make real-time context-based decisions. Unlike traditional automation systems – Robotic Process Automation (RPA) that follow predefined rules and sequences, AI agents operate with dynamic state-based approaches, maintaining an internal understanding of their environment. Best examples: Dynamic pricing – Have you ever wondered why you might get different hotels or ticket prices to take that holiday flight? The list of industries that are already using these agents is long. It will get longer as a virtual workers economy or labor reinvention opportunity is ready to be employed by E-Commerce, Sales and Marketing, Customer Support, Hospitality, Financial Trading, Autonomous Vehicles, Home Automation Systems, etc.
Cobus Greyling (2024) outlines several key differences between AI agents and traditional chaining/RPA emphasizing that AI agents can:
- Exhibit high flexibility and autonomy;
- Operate with granular, state-based approaches;
- Incorporate adaptive learning capabilities;
- Handle unstructured data;
- Make real-time decisions and dynamically adapt to new scenarios;
- Risks of Developing Autonomous AI Agents Without Oversight;
- Lack of Transparency and Accountability;
The Labor Reinvention Opportunity and Work in 2040
‘The first of those questions is the more significant. It implies
specialized purpose. The things are used, you see, but not like weapons in
the ordinary sense, not just to destroy, that is. They are more like snares.’ Wyndham, The Kraken Wakes (1953)
Let’s say again: labor reinvention opportunity and think about what such ever-present virtual 24-hour workers will contribute to federal and state systems. This new workforce, operating tirelessly and without the constraints of human needs, could fundamentally shift how productivity is measured, potentially decoupling economic growth from traditional employment metrics. Such virtual workers will primarily create value for corporations, tech conglomerates, and the elites who own the AI technologies and control the data. As AI agents drive down labor costs and operate with unparalleled efficiency, they will generate immense profits for these stakeholders, while the broader workforce and public institutions, which traditionally rely on labor-driven tax revenue and economic participation, will face the social and economic consequences of widespread job displacement.
The Bot did it!
‘No details are available as yet, but it is feared that the list of the missing may prove to be very heavy indeed.’ Wyndham, The Kraken Wakes (1953)
When large companies develop autonomous AI agents without oversight, we should be concerned about transparency. Companies often consider their algorithms proprietary, making it difficult for external parties to understand how decisions are made. This opacity can lead to situations where AI agents make unethical or harmful decisions without clear avenues for accountability. Take the case of Twitter user Jeffry van der Goot’s bot @jeffrybooks tweet saying “I seriously want to kill people” at a fashion and cosmetics convention happening in Amsterdam. The Dutch police took the threat seriously. What happened? The bot operated as a Markov chain generator, which used a simple algorithm to create sentences by analyzing patterns in a given corpus of text. A Markov chain is a mathematical concept used to describe a system that transitions from one state to another based on certain probabilities. It is defined by the “memoryless” property, meaning the next state depends only on the current state, not on the sequence of events that preceded it. This makes it a stochastic (random) process where each step forward is determined by a probability distribution. Yet, in the future might such a bot ‘graduate’ to become an independent AI agent and follow through with such a threat?
Your Health Crushed by Code
“Public interest in fireballs was down to zero, and few people troubled to send in reports of them any more. I still kept my files going though they were now so unrepresentative that I could not tell how far the apparently low incidence was real.” Wyndham, The Kraken Wakes (1953)
Investigative journalists at ProPublica, including Maya Miller, reported that Cigna is using algorithms and automated systems to assess and deny health insurance claims in bulk. In a discussion with PBS NewsHour’s Ali Rogin, it was revealed that federal data shows more than 49 million claims were denied by health insurance companies in 2021, yet customers appealed less than 0.2 percent of them. Shouldn’t the use of AI agents and undisclosed algorithms in the decision-making processes raise concerns about endangering individuals by unjustly denying necessary medical coverage without providing adequate recourse or transparency into how these decisions are made?
One of the primary concerns is the accountability of actions taken by these AI agents. If an autonomous agent makes a decision that leads to harm to persons, environment, animals, or physical assets, determining responsibility becomes a complex issue. Is it the developer, the corporation, or the AI itself that is to be held accountable?
The Age of ‘Surveillance Capitalism’
“…we were in the process, while almost unaware of it, of undergoing a species of interplanetary invasion…” Wyndham, The Kraken Wakes (1953)
Elon Musk’s X, formerly known as Twitter, recently updated its privacy policy, revealing that, as of November 15, user data – including posts, likes, bookmarks, and reposts – will be shared with third-party companies. These platforms can then use this data to train their own AI models. Users are automatically opted into this arrangement unless they actively opt out, shifting the landscape of data privacy and control. As social media platforms like X and Reddit begin selling vast amounts of user data to AI firms, it is clear that this represents the next major data race in the tech world, with companies using their customer data to win AI dominance. The increasing commodification of user data, paired with minimal transparency and corporate-driven AI agendas, raises concerns about democratic oversight and accountability. Much like in Wyndham’s The Kraken Wakes, where society’s denial of an alien Xenobathite threat leads to disaster, today’s failure to confront the unchecked development of AI could result in unforeseen consequences. Just as the novel’s characters are blindsided by the gradual takeover of Earth, we too face the risk of losing control as powerful AI systems exploit data without proper ethical safeguards. Corporations are feeding consumer data into AI to influence consumer behavior and collect extensive personal data. Shoshana Zuboff in her book “The Age of Surveillance Capital” states that ‘the goal is to automate us’: and describes this phenomenon as “surveillance capitalism,” where companies monetize personal data extracted through opaque AI systems (Zuboff, 2019).
Perhaps Wyndham foretold it best: “In war, you have at least a rough idea of the way your enemy must be thinking so you can put up appropriate countermeasures, but with these brutes it’s nearly always some slant we haven’t explored… We’ve only got the weapons we know – and they’re not the right ones for the job.” Wyndham, The Kraken Wakes (1953)
This article was written by Dr. Jasmin (Bey) Cowin, Associate Professor and U.S. Department of State English Language Specialist (2024). As a columnist for Stankevicius, she writes on Nicomachean Ethics: Insights at the Intersection of AI and Education. Connect with her on LinkedIn.
