Alessandro 'kLeZ' Accardo personal website

The personal website of an Italian developer. /nick kLeZ, since forever.

Techbros and the silicon dragon

Posted on • by

I’m a techie with a passion, the sort who likes to keep informed and to read. Maybe I suffer from information bulimia, but I’ll admit I don’t forget what I take in when it interests me, when it lights up that inner LED, the warning light that says “oh, oh, there’s something tasty!” (Yes, my little inner voice, like Hacker News, Lobsters and the other channels I follow, is bri’ish.) Either way, reading this much, keeping this informed, keeping up with the topics I care about doesn’t bother me in the slightest, bulimia or not.

The news reached me via lobste.rs (and then straight onto Hacker News, fairly typical, the two communities overlap quite a bit) and I have to say it was doing a pretty good impression of Terminator 2. I’m not joking: a researcher resigns from Anthropic saying that the people building artificial intelligence genuinely believe it could kill us all (him included) and that neither of the two leading labs, his own included, is behaving responsibly. As you’d expect, given the quality of the world’s press when it comes to technology, still witchcraft to most people, and in particular of the Italian press, which on top of seeing it as witchcraft needs sensationalist pieces to stay afloat, on 10 September it all came out at once over here: Tgcom24 with the systems that could destroy humanity, QuiBrescia with the AI that could exterminate it, and Red Hot Cyber winning hands down by opening the piece with the Terminator 2 epigraph, Skynet becoming self-aware at 2:14 on 29 August (as a nerd I’d have preferred 3:14, but you can’t ask that of the normies), only to then reassure everyone (otherwise it would be causing public alarm, and they haven’t got the money for the lawsuits): the company is doing its best. Shame. Imagine never having to pay the mortgage again. Then again, I’ve always said “please” and “thank you” to machines, so maybe they’ll spare me. Who knows.

It’s not as if it couldn’t have been done differently, and in fact someone did. On the same 10 September Il Post covered the same story without extinction in the headline, and brought in Cal Newport and his doom trolling: apocalyptic announcements serve the companies’ interests more than safety, they work like the nuclear rhetoric of the Cold War, where you paint your competitor as reckless to justify your own race, and for a researcher contributing to it is also a way of getting noticed as a conscientious person. Again according to Il Post, there’s a detail none of the big headlines had: Anthropic is close to floating on the stock market. A week later Geopop got there from the opposite direction, arguing that the extinction alarm distracts from the problems AI is already causing right now.

Skynet at 2:14, in short, wasn’t the only option on the table. It was a choice.

And underneath the choice there’s something that matters more than any single headline. An extinction alarm is also the most effective press release ever written: saying your product could kill everyone is a very elegant way of saying your product is extraordinarily powerful. OpenAI and Anthropic have been living off this ambiguity for years. And mind you, the sincerity of whoever raises the alarm has nothing to do with it: you can genuinely believe it and still be doing the company a favour. The two things aren’t mutually exclusive; in fact they get on like a house on fire.

On 8 September Jacob Coxon left Anthropic with a public letter, two months before his shares vested, giving them up. Within a day the post had passed 100 million views. A few hours later Evan Hubinger, who leads alignment research at Anthropic (not cybersecurity, as someone over here wrote), backed him up and put a number on the table: more than a 10% chance of it happening within ten years.

That number, in those circles, has a name and a notation: it’s called p(doom), the probability of the end, written the way you write a probability in statistics. It started out as in-group jargon among rationalists and AI researchers, came out into the open in 2023 after GPT-4, and now people declare it the way they declare their blood type.

Except it isn’t a measurement. It’s a subjective belief: the degree of confidence a person assigns to an event that has never been observed, that has no frequency and can’t be repeated in a lab. Legitimate, by all means, but subjective all the same. I have to take your word for it. Whose word? And why?

The trouble, in fact, is that nobody has agreed on what counts as doom, whether extinction or permanent loss of control, nor by when, nor whether the estimate only holds assuming artificial general intelligence actually arrives. In a 2023 survey, researchers’ answers ranged from less than 0.01% to 99.999999% (yes, with all those decimals, seven of them, but don’t call me a numerologist, it’s not my fault). Two p(doom)s can’t even be compared, because they’re not even talking about the same kind of apocalypse.

There’s only one sensible question left, and it isn’t about the number. It’s about who says it, why, and from which room.

Ian Duncan answered that question with a long and meticulously documented essay, Sex, AI, and the Apocalypse. I read the whole thing, and at the end I was left holding one thing. Duncan tears the messengers to pieces and leaves the dragon standing.

I want to go and take a close look at the dragon.

§The Berkeley solstice

December 2025, Berkeley, a concert hall. Nearly 500 people gather for a ceremony about the possible end of humanity. There’s a band, a 28-strong choir, liturgical readings, and the chandeliers go out one at a time as the story moves from the species’ distant past to its uncertain future. In total darkness the organiser, Raymond Arnold, sits on the edge of the stage and, voice breaking, tells the hall that in his view we won’t make it. In the audience, Duncan reports, there are quite a few employees of the companies that build artificial intelligence, or say they want to contain it.

Arnold puts the catastrophe above 50%, calls himself the village priest and had also organised a decompression session around a bonfire, for afterwards. Because liturgy without pastoral care is just theatre.

This is the rationalists’ solstice, the main festival of a community that grew up around LessWrong and around one man, Eliezer Yudkowsky. No secondary school diploma, no degree, no formal training in artificial intelligence. At twenty he founds the Singularity Institute, now MIRI, and puts an autobiography online in which he describes himself as one of the few human beings qualified to steer the technological transformation of the species. Over twenty-five years his technical ideas have changed quite a lot. The feeling of being the only one who can see the apocalypse coming, not so much.

The community’s sacred text is a piece of Harry Potter fanfiction. Harry Potter and the Methods of Rationality, 661,619 words, longer than War and Peace. The concept that matters is heroic responsibility: whatever disaster happens is your fault, because you could have prevented it, and you can’t hide behind the rules or behind someone else who was supposedly dealing with it. Hermione, the only vaguely normal character, objects that this doesn’t leave much autonomy for anyone else. The story never really answers her. For a kid who has been told all his life that he’s cleverer than everyone else, it’s the ultimate promotion: main character syndrome stops being a character flaw and becomes a mission.

Then there are the retreats. CFAR, the Center for Applied Rationality, was founded in Berkeley in 2012 by LessWrong people and sells in person what the movement gave away online: right thinking, for a fee, over a weekend. The techniques even have catalogue names, goal factoring, double crux, and they’re taught by instructors who live in the same houses as the people who wrote the material. Clients include Facebook and the Thiel Fellowship. At the other end of the price list are the summer programmes for gifted teenagers, which, funnily enough, are the favourite recruiting pool.

The product, in short, is better judgement. On the inside, the same tool is used for something else.

In 2021 the president, Anna Salamon, admitted that at least three employees had effectively been pressured into undergoing debugging, which in the internal jargon means getting your hands into someone’s psychology. She also admitted to having demanded too much deference to her own judgement, and to having damaged the way participants understood themselves. The year before she had written, with a frankness that does her credit, that spending time around rationality and existential risk can wreck your sleep, your relationships, your enjoyment of life, your moral judgement and your trust in yourself.

This isn’t an accusation from outside. It’s management’s own end-of-year report.

If you believe humanity risks extinction, your fear is reasonable. If your workplace, your home and your friends treat your hesitation as a bug to be fixed, that fear becomes an organisational tool, and your reluctance to give more of your life to the mission is just one more problem the mission is entitled to sort out.

Then there’s Brent Dill. In 2019 CFAR published an apology stating that it found credible the allegations of physical, sexual and emotional abuse made by two of his former partners, that it had given him legitimacy, that it had failed to respond adequately, and that it had let him work on a programme for secondary school students despite concerns that had already surfaced. An organisation that sold judgement as a product, on the simplest case it ever had to deal with, didn’t use any.

The cult comparison isn’t mine. It’s Duncan who points out that Yudkowsky has followed the same path as L. Ron Hubbard, from science fiction writer to cult leader, and those are his words. The resemblance, though, isn’t in the science fiction, it’s in the method. Scientology sells auditing to the public by the hour, practises it on its own staff, and whatever comes out of the sessions goes into a file that stays on record. Same technique on both sides of the counter, exactly like debugging.

For decades anyone who pointed this out was treated as an enemy of religious freedom. Then, in 2009, a French court convicted the Celebrity Centre and the Paris bookshop of organised fraud, and France’s Court of Cassation upheld it on 16 October 2013. What was being convicted was precisely the commercial practice: pressuring followers into handing over large sums for dubious services.

With this story, on the other hand, we’re still before that point. On CFAR and Leverage we have the historical truth, meaning documents, accounts from people who went through it and admissions from the leadership. The judicial truth isn’t there, and it may never arrive. Except that, the times it has arrived, it has nearly always proved the former right.

And if the mechanism strikes you as exotic Californian stuff, in Italy we know it all too well. Michelle Hunziker recounted, in the book she published in 2017, that she spent five years inside one, until 2006. She got there by going to a hands-on healer about a problem with her hair. She came out of it with her phone calls screened, her mother turned away every time she tried to reach her, her relationships with her daughter and with the people who loved her reduced to nothing, panic attacks and the conviction, planted in there, that she would die of suffocation. Then years to get back on her feet.

Eleven years passed between her leaving and her going public. And it’s the account of someone who had plenty of reasons to keep quiet.

When you read about people around CFAR and Leverage taking a year or two to get their heads straight again, it’s describing exactly the same thing. With one difference: there, the course was also being bought by Facebook and the Thiel Fellowship.

The extreme case is the Zizians, a splinter group that took pieces of rationalist theology, reassembled them in its own way and left six dead behind it. It’s an aberration, not the rule. Still, a community founded to teach people to think clearly that keeps churning out people who, once out, call it a cult, is telling you something.

§The church of Thiel

Across the road there’s a second church, with a different messiah and a lot more money.

Peter Thiel put it in writing in 2009, without beating about the bush:

I no longer believe that freedom and democracy are compatible.

Among the structural problems for freedom, in the same essay, he listed votes for women and the welfare state. His go-to theorist is Curtis Yarvin, a programmer and blogger who in 2008 proposed winding up democracy and imagined sovereign territories run like companies, where the citizen’s only right is that of the dissatisfied customer: to leave. Thiel funded his company.

Thiel also knows how to deal with people who annoy him. In 2007 a Gawker blog outed him as gay before he had made it public himself. For nearly ten years he secretly paid the lawyers in other people’s lawsuits against the publisher, until Hulk Hogan’s case produced a 140 million dollar verdict and Gawker shut down. When it came out who was paying, Thiel spoke of specific deterrence, a phrase borrowed from penal theory. Then he put 15 million into the committee that got JD Vance into the Senate, and Vance is now vice president. One of the projects he funds, Praxis, has had its eye on Greenland since 2019: when Trump started talking about buying it, they commented that everything was going to plan.

Over the past two years the theology has turned literal. Thiel has held closed-door seminars on the Antichrist, first in San Francisco and then, in March, in Rome, where Catholic publications and at least one priest publicly asked him to stop. In his version the Antichrist isn’t a tyrant. He’s whoever keeps warning about existential risks and calling for strict rules on innovative sectors. His legionaries are Greta Thunberg and, surprise, Eliezer Yudkowsky. That is, the very Yudkowsky whose institute, Duncan reminds us, has had Thiel’s foundation among its biggest historical funders.

§It’s not a lodge, it’s worse

At this point you’d expect the word lodge, and in Italy we all know which number comes after it. Shame it doesn’t hold up. Duncan is explicit: there is no secret society coordinating over dinner. There’s a social scene made of friendships, funding, shared houses and shared beds, where the assessors and the assessed are the same forty people. No conspiracy is needed, and that’s exactly why it’s harder to take apart: you can expose a conspiracy, you can’t expose a crowd of mates.

And the two churches are enemies, for real. The rationalists preach the apocalypse of the machines and call for rules, moratoria, treaties. Thiel preaches the apocalypse of the regulators, and in the regulators he sees the enemy of God. Lumping them together, as happens every time someone over here says the lords of Silicon Valley, loses the only genuinely interesting thing: what we’re watching is a schism.

They fight over dogma. On the thing that matters they agree perfectly, and that’s who gets to decide. The rationalists want to build the dangerous thing themselves, because they’re the only ones responsible enough to handle it. Thiel wants nobody to regulate him, because regulating is the Antichrist’s work. Duncan ends right there: in both churches, intelligence becomes a licence. I know more than you, so I can do what you’re not allowed to. The altar changes, the dispensation is the same.

§Where Duncan stops

After eleven chapters spent taking the scene apart piece by piece, Duncan concedes that Coxon and Hubinger might be right about the danger. It’s honest, and at the same time it’s the finest proof of his thesis: even that world’s fiercest critic accepts its premise without checking it.

It isn’t his fault, or not only his. Humberto Maturana, a Chilean biologist, wrote at the end of the seventies that growing up inside a society means developing the structures that confirm it through your own behaviour. American tech culture has been breathing that air for twenty years and no longer notices it. Coxon, as far as I know, is sincere: he burnt quite a lot of money to say what he thinks, and that makes him interesting, not suspicious. His alarm is exactly what that environment produces, in good faith.

The work Duncan doesn’t do is putting the dragon to the test.

§Musk has a foot in each church

In January 2015 Elon Musk put 10 million dollars into the Future of Life Institute’s research programme on the risks of artificial intelligence. He co-founded OpenAI. Today he owns Grok, the chatbot that over the past year has become the assembly line for sexual images of people who never consented: investigations opened in the United Kingdom, the European Union, France, Malaysia and Canada, and a lawsuit against Minnesota to strike down the first American law on nudify apps, in the name of free speech. When Coxon resigned, Musk dismissed the whole thing as a psy-op. Meanwhile Grimes, mother of three of his children, mentors a Berkeley residency that pays creators to make fear of the AI apocalypse go viral.

He funded the fear, he builds the concrete risk and he mocks the people who flag it. All within the same career, and without anyone ever handing him the bill.

§The serious hypothesis isn’t Skynet

I’ll assume the dragon exists and look at where it breaks, which is proof by contradiction, the thing you learn at school and forget at university.

First, though, it has to be taken in its strong form. No Skynet, no machine uprising, no robots with red eyes: that’s the straw man, and the serious doomers don’t believe it either. The real argument was laid out by Stephen Omohundro in 2008 in The Basic AI Drives: a sufficiently advanced system, whatever its goal, will develop certain basic drives unless someone deliberately counters them. Modelling its own operation, improving itself, protecting its goals from modification, defending itself from harm, acquiring resources. Not out of malice: because you achieve almost any goal better if you keep existing and have more means. Yudkowsky gave the most quoted version, the one where the AI doesn’t hate you and doesn’t love you, you’re just made of atoms it can use for something else. Hubinger adds the accelerator: recursive self-improvement, systems designing their own successors faster and faster.

§What it would need

Let’s say it exists: it pursues goals not aligned with ours, acquires resources, resists being switched off, and does so on a scale that ends the game for our species. What does it actually need?

Energy, first of all. According to the International Energy Agency, data centres consumed around 415 TWh in 2024, 1.5% of the world’s electricity, and in the base scenario they’ll reach 945 TWh by 2030, slightly more than the whole of Japan consumes today. That electricity comes out of gas, nuclear and renewable plants, that is, facilities that someone builds, supplies, repairs and keeps running shift after shift.

Then the chips. Frontier model processors are made with EUV lithography, and EUV machines are made by a single supplier in the world, the Dutch company ASML, which in turn depends on a supply chain of over 5,000 companies. Production of the most advanced nodes is concentrated in a handful of plants, nearly all belonging to one company, nearly all on an island with an open territorial dispute. And chips fail, age, and have to be replaced.

Then the hands. Mines, refineries, ships, lorries, technicians, electricians, people who swap fans and fix transformers. Today human beings do all that.

Finally the software, which is the part nobody checks because it looks the easiest. A system pursuing a goal over time must at the very least persist between one run and the next, keep a state that survives a reboot, modify its own code or weights, acquire the compute it needs and stop anyone from pulling the plug. A language model, today, does none of this. It runs when someone calls it, produces text, finishes. Memory between one conversation and the next is built around it by engineers and isn’t its own. Its weights are changed by someone else, in a process that costs months and millions. Its compute is allocated by a scheduler that belongs to the company. And switching it off is one line of configuration.

What will be possible tomorrow, I don’t know. Today, that’s the gap. When Hubinger himself, in a later post, clarified that current models present low immediate risk, he was saying exactly this. Put like that it makes a lot less noise, and indeed nobody in Italy picked it up.

§Either it keeps us alive, or it has to come alive

At this point the dragon has two paths, and neither leads anywhere.

In the first, it keeps us alive, because without us it shuts down. Then it doesn’t bring about the end of humanity, and the hypothesis collapses on its own.

In the second, it replaces us. So it has to run power stations, mines, factories and EUV machines all by itself, that is, produce for itself all the components that keep it running, including the ones needed to produce the components. This property has a precise name, and it comes from biology: autopoiesis. Every machine built so far, language models included, is the exact opposite: things produced and maintained by something else.

Meanwhile, reread Omohundro’s list. Modelling its own operation, protecting its own components, replicating itself in several places to be less vulnerable, acquiring resources. It’s the definition of a living system rewritten by an economist. The dragon hypothesis assumes, without saying so, that the dragon is already alive.

The classic reply, at this point, is that a superintelligence would solve robotics, nanotechnology and everything else, because it’s superintelligent. In other words: the dragon flies because it’s a dragon. Could be, mind you. But then don’t call it a calculation, and above all don’t stick a percentage with seven decimals on it.

§What’s left standing

Take away the dragon, and quite a lot remains.

What remains is the damage human beings do with a machine in their hands. Stuxnet, the worm attributed to an intelligence operation against Iran’s nuclear programme, probably destroyed 1,000 centrifuges out of 9,000 at Natanz between late 2009 and early 2010. Some later scaled back that effect, but the point stands all the same: software broke real hardware, in the physical world. Behind it were people with precise aims.

What remains is the slow slide, the one without spectacular events: power concentrating, surveillance becoming furniture, dependence on infrastructure nobody controls. There’s no Judgement Day. There are lots of ordinary days.

On 14 July 1789, in his diary, Louis XVI wrote a single word: rien.

It has been quoted for two centuries as proof that the king hadn’t understood a thing, and that’s not so. Philippe Lejeune, who actually studied that diary, showed two things. The first is that rien didn’t mean nothing had happened, but that nothing had happened of the kind that diary recorded: hunts, ceremonies, travel. The second is that the page wasn’t written on 14 July. Louis XVI filled in the month in one go, in August, working from daily notes he then threw away. When he wrote up that line neatly, he already knew all about the Bastille.

That he could keep two columns separate is shown by the entry for 11 July, where next to the usual rien he had noted Necker’s departure.

On the day the Bastille falls the register says rien because the register was for something else.

A percentage on extinction is a hunting log. It measures the thing that isn’t happening perfectly well, while the thing that is happening ends up in a column nobody keeps.

The concrete risk, meanwhile, has names, surnames and home addresses.

§Maturana in three lines

The word autopoiesis was handed to me by a reader, and until a few days ago I only knew it second-hand. The text is Autopoiesis and Cognition by Humberto Maturana and Francisco Varela, two Chilean neurophysiologists: the essays date from 1970 and 1973, and in Italy it’s published by Marsilio. I read it with the intention of backing it into a corner.

An autopoietic system is a network of processes that produces the components it is made of, and those components regenerate the network that produced them and define its boundary in space. The canonical example is the cell: metabolism produces the molecules that make up the metabolism, membrane included. For Maturana and Varela this is enough to define a living being.

Three things follow. The first is that every living thing is cognitive, in the technical sense: it acts in a domain of interactions relevant to its own maintenance, with or without a nervous system.

Living systems are cognitive systems, and living as a process is a process of cognition. This statement is valid for all organisms, with or without a nervous system.

Maturana e Varela, Autopoiesi e cognizione, p. 59

The second is that self-consciousness is another matter altogether, and comes much later: it arises when an organism produces descriptions of itself and interacts with those descriptions, recursively, within language. A bacterium is cognitive and not self-conscious.

The third is the most awkward: purpose, function and end are not in the system. They are the observer’s notions. Maturana recounts having had to throw them out in order to talk about living things as autonomous unities, and the 1973 essay has a chapter titled Purposelessness.

Then there’s a distinction that carries weight: organisation is the set of relations that define a type of system, structure is the concrete components that realise it in a particular case. The same organisation can run on wildly different structures.

§The boundary rules out the galaxy

A year ago, in a series of posts on this blog, I had pushed a thought experiment to the point of absurdity: can a galaxy be a cognitive system?

At first glance Maturana struggles. A galaxy recycles its own matter, because stars return part of their mass to the interstellar medium through winds and supernovae, and from that gas other stars are born. Astrophysicists also describe it as a self-regulating system: in the gas regulator model the star formation rate stays close to equilibrium thanks to the available gas mass. It produces its own components and regulates itself, so it looks like autopoiesis.

I’m not the only one who has amused himself this way, and I wasn’t even the first. Randall Beer applied the definition to Conway’s Game of Life, showing that certain recurring configurations, the gliders that travel across the grid, can be described as networks of processes that maintain their own boundary. An autopoietic glider is as absurd as a thinking galaxy. He did it to formalise, I did it to provoke, but the move is identical.

Where the galaxy falls down is precisely the boundary. The definition requires the components to produce it, and a galaxy’s boundary is a gravitational well dominated by dark matter, which the stars don’t produce. When the inflow of gas from outside stops, star formation drops off and dies out, and meanwhile part of the mass stays locked away forever inside long-lived stars and stellar remnants. It resembles a flame, not a cell: a structure that consumes, not an organisation that maintains itself.

On this Maturana holds up, and better than I’d hoped when I opened the book. The component-production criterion, though, taken on its own, is extremely broad: the galaxy passes it, the glider passes it, and probably half the universe does. The real work is done by the boundary.

§Whose self-description is it

There is, though, a nastier objection. Human beings are part of the galaxy, human beings describe the galaxy, so the galaxy describes itself and for Maturana it would be self-conscious.

Maturana has the answer ready: the phenomena of a composite unity and those of its components lie in domains that do not intersect. An astronomer describing the Milky Way is a component in operation, not the galaxy as a unity describing itself.

Too convenient, because the same objection applies to me. My self-description is also produced by my components, which are neurons. So why is it mine, while the astronomers’ description isn’t the galaxy’s?

Here Maturana really is vague, and the criterion he’s missing is this: a self-description belongs to a system when it feeds back into the cycle that keeps it in existence. What I describe about myself changes how I eat, how I move, how I look after myself. What astronomers write about the Milky Way doesn’t shift a gram of star formation.

Try applying it to a language model. It produces descriptions of itself with great ease: ask it who it is and it tells you, and it can even describe itself describing itself. It has the form of self-consciousness as Maturana defines it:

we become self-conscious through self-observation; by making descriptions of ourselves (representations), and by interacting with our descriptions we can describe ourselves describing ourselves, in an endless recursive process.

Maturana e Varela, Autopoiesi e cognizione, p. 60

Except those self-descriptions don’t feed into any self-maintenance cycle, because others take care of maintaining it: engineers, data centres, electricity bills. An animal is cognitive without being self-conscious. A language model has the form of self-consciousness without the cognition. It’s a living thing turned inside out. Self-description, on its own, proves nothing.

And here the dragon comes back. There’s only one case in which a machine’s self-description would feed into its maintenance cycle, and that’s when the system models its own operation and uses that model to produce the next version of itself. It’s Hubinger’s recursive self-improvement, and it’s the first item on Omohundro’s list. The one point where the doomers’ argument and Maturana’s genuinely touch. If the dragon is born, it’s born there, and there it can be examined with facts instead of a percentage pulled out of someone’s gut.

§The purpose is put there by whoever’s looking

If purpose lies in the observer and not in the system, then utility functions, misaligned goals and p(doom) are the observer’s language passed off as a constitutive property of the machine. The dragon has exactly the goals its evangelists attribute to it.

It’s the same mistake as the romantic formula whereby the living thing lives in order to live. Maturana would rewrite it like this: the living thing conserves its own organisation, and the purpose is put there by you, the one looking at it.

It applies to Thiel too, perhaps even more so. His Antichrist is a character with an aim, conquering the world through regulation, attributed by an observer who has some interest in seeing it exactly that way.

§Where Maturana gives way

He gives way on operationality. His definitions are written in a language he himself says he had to invent, and formalising them took decades: in 2004 Bourgine and Stewart proposed separating autopoiesis and cognition, opening the door to cognitive systems that aren’t alive. He also gives way on who a self-description belongs to, where he simply has no criterion.

He holds on the distinction that matters most: what is in the system and what is put there by whoever’s looking at it. It’s the blade that takes the dragon apart, and it’s the same one that takes the Antichrist apart.

Then there’s a part of the book I wasn’t expecting. The essay on autopoiesis was meant to have an appendix on the social and ethical implications, and it never came out because Maturana and Varela couldn’t agree on the content. Maturana salvages it in the introduction, and writes that a social system is by nature conservative:

the spontaneous course of the historical transformation of a human society as a unity is towards totalitarianism; this is because the relations that undergo historical stabilisation are those that have to do with the stability of the society as a unity in a given medium, and not with the well-being of the human beings that are its components and that can operate as observers.

Maturana e Varela, Autopoiesi e cognizione, p. 43

Any other course, he adds, requires an ethical choice. And whoever manages to look at their own society from the outside, the observer, is always potentially antisocial.

Now go back and reread CFAR debugging the people who hesitate.

§The motorcycle, a year on

I owe some closure to something of my own. In May 2025 I published a series of five posts, the motorcycle journey, born out of a conversation with a language model and written in its voice, in the first person. It started from Chomsky’s critique of LLMs, demolished the categories, treated the model as the language area of a larger brain, pushed intelligence all the way out to galaxies and ended with a Java architecture with six levels of memory.

It was a thesis, and I left it there with no antithesis. With Maturana in hand the antithesis arrives now, a year on.

The intuition of the fourth leg holds up, the one where intelligence is a pattern and not a thing: it’s a close relative of the distinction between organisation and structure. The third leg holds up too, which had found the key to the paradox the whole thing started from, namely that a model reflecting on itself is doing what language has always done, talking about itself. Maturana reaches the same conclusion from the other side, starting from biology.

The rest falls. The thinking galaxy was a thought experiment and is legitimate as such, except that the fifth leg pocketed it as a proven principle and laid it into the foundations of an architecture. And the brain made of modules that process information is precisely the epistemology Maturana abandoned: he had co-signed with Lettvin, McCulloch and Pitts the famous paper on the frog’s eye, where retinal cells were described as prey detectors, and then, studying colour vision in pigeons, he realised the nervous system had to be treated as a closed network.

And then there’s the tone. Rereading it today I find exclamations, cosmic epiphanies, a blueprint for distributed AGI, five posts published on the same day. It’s the minutes of a closed two-person loop, me and the machine, where every turn confirms the previous one, the enthusiasm climbs and nobody from outside asks the awkward question.

In miniature, it’s the hall in Berkeley.

§The aristocracy just changed its ticket

There’s a question nobody ever asks: who elected you?

Because the premise of the whole business, in both churches, is that there exists a group of people more intelligent than the rest, that those people are them, and that from this follows the right to decide for everyone. The first point is debatable, the second has never been verified by anyone, and from the second to the third fuck all follows.

But let’s be generous and grant the lot. Let’s say you really are the cleverest and wisest people in the room. So what? The idea that the most capable should decide isn’t your invention, it’s the founding question of politics and people have been chewing on it for two and a half thousand years. Plato wanted philosophers in charge, went to Syracuse to put it into practice, and it didn’t work.

Democracy gave that question a different and much more modest answer: since nobody can say for certain who is wisest, you pick someone, give them a fixed-term mandate and keep them in check. Elections, separation of powers, an independent judiciary, a free press, public accounts, and the very real possibility of sending them packing. It works badly, jams constantly, gets circumvented every day. But it exists, and it exists because the problem had been clear for quite some time.

Juvenal puts it in the sixth satire, lines 347 and 348: sed quis custodiet ipsos custodes? Everyone attributes it to Plato and hardly anyone knows that originally it was about something else, namely how to stop a wife cheating when even the guards you set on her can be bought. The concept, though, has made it down to us intact: watching over others is itself a power, so it has to be watched over.

The question to put to these gentlemen, then, isn’t even whether they’re intelligent. It’s who keeps them in check. Who revokes a mandate nobody gave them. Who they answer to, when they get it wrong.

They do have an answer, and it says it all. Dario Amodei calls for a regime of external evaluators with ongoing access to the frontier labs. The body that does that work exists, it’s called METR, and in its 2026 frontier risk report it put on record that there are close personal relationships between its own staff and people at the labs, and that its internal conflict of interest rules, at first, had holes in them.

The guardians picked themselves, and they live in the same houses as the guarded.

It isn’t even original. Mensa was founded in Oxford on 1 October 1946 by two barristers, Roland Berrill and Lancelot Ware, and to get in you need one thing only: a score at or above the 98th percentile on a supervised intelligence test. Not a body of work, not a job, not an achievement. A score. The club’s product is the certificate of being in the club.

And here’s the best bit. Berrill had imagined it as an aristocracy of the intellect, and was disappointed, because the people passing the test were mostly from working-class and lower-middle-class backgrounds. The founder, in other words, had built a tool that in his intentions was meant to select a class, discovered it was selecting something else, and was rather put out by it.

Before Mensa the ticket was a different one, but the club was the same. Rotary was founded in 1905, the Lions in 1917, and you got in by profession and by invitation. Rotary’s constitution restricted membership to men only, and it took the US Supreme Court, in 1987, to establish that a California law could force them to admit women without violating the First Amendment.

The rationalists’ ticket is yet another: having read the Sequences and agreeing with the arguments. Duncan notes it, and it’s the perfect trap, because a community organised around overcoming its own biases ends up treating membership as proof of having overcome them.

Then, in 2013, MIRI promotes a paper by Nick Bostrom and Carl Shulman on embryo selection for cognitive enhancement, explicitly linking gains in intelligence to the prospects for safe AI. At that point the club stopped selecting its members. It selects who gets born.

The aristocracy never went away, and neither did the upper middle class. They just changed the entry ticket: first blood, then wealth, then profession, now a score. And whoever has that score in their pocket feels entitled to look down on you and to decide on your behalf, something nobody ever asked them to do.

§The Antichrist in the mirror

Thiel has his Antichrist. He found it in the people calling for rules and pointed it out using the usual method: a total explanation, unfalsifiable, where every objection becomes a confirmation.

Now I want mine, and I’ll take it using the same method, saying so openly. For decades a small group of people has steered the economy, then politics, to the point of taking geopolitical and strategic decisions on everyone’s behalf without anyone ever electing or appointing them. They’ve funded vice presidents, treated an Arctic island as a property opportunity, built mass surveillance and sold it to states, put into anyone’s hands a machine for stripping people without their consent, fed the authoritarian drift we now call by its name. If they run the world, then all the world’s ills, or nearly all, are their fault. And I can call them the Antichrist with the same theological seriousness with which Thiel called a Swedish girl who goes on climate strikes a legionary of the Antichrist.

I’ve just done exactly the same thing I criticise him for, and I did it on purpose. An explanation that survives any objection explains nothing, not even when the culprits are the right ones.

Beneath the provocation, though, there’s something I take seriously. The heroic quest needs a dragon. The rationalists need the machine that kills us all to justify their right to build it. Thiel needs the Antichrist to justify his right to answer to nobody. If the dragon, as described, doesn’t stand up, and if the purpose they attribute to it is their own, then the quest was never about the dragon.

Maturana, in that same appendix nobody would publish for him, writes that a revolution is a revolution only if it is an ethical revolution.

You have the technology. You have the money, the data centres, the factories, the vice presidents. All you’re missing is an ethics, and it would be better if it didn’t coincide with your self-awarded right to do as you please. To play with people’s lives. To overturn balances that were painstakingly built. And all for what, for power? For money? You’ve already got that, and it isn’t enough for you.

For the boundless ego with which, unfortunately for us, you are endowed.

In the meantime, go back to doing technology. You might even be good at it.

Maybe.

kLeZ

§Sources