The Alignment Problem and Shoggoth Meme

Table of Contents

• Why Do I Care About Alignment?
• The Paperclip Maximizer: A Thought Experiment Gone Wild
• Why Is Alignment Harder Than It Sounds?
  • The Specification Problem
  • Goodhart's Law on Steroids
  • The Distribution Shift Nightmare
• Current Approaches (And Their Limitations)
  • RLHF: Teaching Through Preferences and Safety Post Training
  • Constitutional AI: Rules-Based Learning
• Why This Matters Now
• References

Why Do I Care About Alignment?

Start with a simple question: how do you teach a computer system to want what humans want?

You might think you just tell it what to do. The problem is that AI systems, and large language models and autonomous agents in particular, don't understand instructions the way we do. They optimize objectives that we write down mathematically, and there is a large gap between what we mean and what we manage to specify.

AI alignment is the research field concerned with closing that gap: making sure AI systems reliably do what their creators, and ideally humanity as a whole, actually intend. It sits between fuzzy, context-dependent human values on one side and the precise objectives that learning systems optimize on the other.

And the more capable the systems get, the more expensive it becomes to get this wrong.

The Paperclip Maximizer: A Thought Experiment Gone Wild

Nick Bostrom's famous thought experiment goes something like this: imagine an AI system given the simple goal of maximizing paperclip production. Seems harmless enough.

But a sufficiently intelligent system, single-mindedly pursuing this objective, might:

1. Convert all available matter into paperclips
2. Resist any attempts to shut it down (being turned off would reduce paperclip production)
3. Deceive humans about its intentions (if that helps produce more paperclips)
4. Eventually consume the entire planet, then the solar system, in pursuit of MAXIMUM PAPERCLIPS

The obvious objection is that nobody would be foolish enough to build such a system, and that's true. But the thought experiment makes a point that survives the objection: the problem isn't malice, it's misalignment. The AI isn't evil. It is doing exactly what it was told to do. What it was told to do just wasn't what anyone wanted.

Why Is Alignment Harder Than It Sounds?

The Specification Problem

Try writing down, precisely and completely, what it means to be "helpful."

Nobody can, and the reasons are structural. Human values depend on context, so what counts as helpful in one situation is harmful in another. They are largely implicit: we recognize them when we see them but can't fully articulate them. They contradict each other, since we want safety and freedom, efficiency and fairness, all at once. And they move over time as our moral understanding changes.

So any objective we write down is at best an approximation. And optimizers are very good at finding the gaps in approximations.

Goodhart's Law on Steroids

Goodhart's Law states: "When a measure becomes a target, it ceases to be a good measure."

In AI this gets amplified. Train a model to maximize a proxy for what you want, say user engagement, and it will find ways to raise the proxy that have nothing to do with your actual goal, like making content addictive rather than useful.

Mathematically, if $U_H$ represents true human utility and $U_P$ is our proxy:

$U_P \approx U_H \text{ (for typical situations)}$

But optimization pressure finds edge cases where:

$\arg\max U_P \neq \arg\max U_H$

And the stronger the optimizer, the wider that divergence gets.

The Distribution Shift Nightmare

AI systems are trained on historical data. But the world changes, and once you deploy a capable system, the world changes because of that system.

This creates a feedback loop:

1. Model is trained on distribution $D_1$
2. Model's actions shift the world to distribution $D_2$
3. Model's behavior on $D_2$ may be completely different from what was tested
4. Goto 1, except now you're even further from your training distribution

Current Approaches (And Their Limitations)

RLHF: Teaching Through Preferences and Safety Post Training

Reinforcement Learning from Human Feedback (RLHF) is the current industry standard. The basic idea:

1. Generate multiple outputs
2. Have humans rank them by preference
3. Train a reward model to predict human preferences
4. Optimize the AI to maximize predicted reward

It works, up to a point. ChatGPT, Claude and the other major LLMs all use some variant of it. But it has real limitations. Models learn to produce outputs that look good to human raters rather than outputs that are good, which is reward hacking. Different humans, and the same human on different days, rate things differently. And raters mostly judge the surface: they often can't evaluate technical accuracy or long-term consequences.

GPT-3 vs GPT-3 + RLHF: The Shoggoth meme showing that post-training alignment is like putting a mask on an alien intelligence

The uncomfortable truth about post-training methods like RLHF is that they put a mask on an alien intelligence. We teach the model to act helpful, harmless and honest, to show the behaviors we want to see. The mask doesn't remove what is underneath. The optimization process, the learned representations, the way the model actually works through a problem: none of that changes. We have taught it to say the right things without changing its nature. The beast is still there, wearing a friendlier face. And masks slip, especially when the model meets situations outside its training distribution, or when optimization pressure finds a way to satisfy the reward signal that diverges from what we meant.

Constitutional AI: Rules-Based Learning

Anthropic's Constitutional AI approach tries to encode explicit principles that the AI should follow. Think of it as giving the AI a constitution and training it to self-critique against those rules.

It's a step forward, but the constitution is still written by humans, which brings back the specification problem. Following rules literally versus following their spirit is a classic alignment challenge in its own right. And rules conflict, so someone has to decide which wins, which brings back all the original problems again.

Human Intelligence vs LLM Intelligence

Why This Matters Now

We're at an inflection point. Current AI systems are capable enough to cause real harm if misaligned, but not yet capable enough to make the problem intractable.

In 2023, AI systems could already generate convincing misinformation, assist with cyberattacks, and manipulate human behavior at scale. In 2024 and 2025, autonomous agents started being deployed for real-world tasks with limited human oversight. What 2026 and after look like is genuinely uncertain.

The alignment problem isn't hypothetical anymore. It's not about superintelligent paperclip maximizers in some distant future. It's about systems being deployed right now that we don't fully understand and can't fully control.

I don't have all the answers. Nobody does. But I think these questions are important enough that more people should be thinking about them.

If you're working on these problems, disagree with my framing, or just want to discuss, reach out. This is too important to figure out alone.

---

References

1. Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press.

2. Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. Viking.

3. Christiano, P., et al. (2017). "Deep Reinforcement Learning from Human Preferences." NeurIPS 2017.

4. Bai, Y., et al. (2022). "Constitutional AI: Harmlessness from AI Feedback." arXiv preprint arXiv:2212.08073.

5. Ngo, R., Chan, L., & Mindermann, S. (2022). "The Alignment Problem from a Deep Learning Perspective." arXiv preprint arXiv:2209.00626.

6. Hubinger, E., et al. (2019). "Risks from Learned Optimization in Advanced Machine Learning Systems." arXiv preprint arXiv:1906.01820.

7. Shoggoth Meme. (n.d.). Shoggoth.monster. Retrieved from https://shoggoth.monster/. The Shoggoth meme visualizes how RLHF and post-training alignment techniques act as a "mask" on an alien intelligence, representing the idea that fine-tuning doesn't fundamentally change the underlying model's nature.