10ペンスからロボットChipへ
こんにちは、Jonnyです。ブラックプールのDickson Roadで小さな店を営み、暇な時間にAIをいじっています。
子供の頃はゲームセンターで過ごし、音と光と10ペンス硬貨に囲まれて育ちました。大人になっても技術とAIへの興味は変わりません。
2015年、画面とスコアだけからAtariゲームを自学するコンピュータの記事を読み、自分も試したくなりました。当時はUSB起動のLinuxやデュアルブートが必要で、環境構築に最も時間を取られました。
その後、別の趣味に数年を費やしました。記憶があいまいで、古いファイルを上書きする癖もあり、初期の記録はかなり曖昧です。
2026年9月に再開しました。古い方法のまま進めたため時間を浪費しました。失敗の末に、自分のキットを作り上げました。
最初はWindows内のLinuxで動いていましたが、誰でも試せるポータブル版にしました。Linuxもインストールも不要。解凍してダブルクリックし、ゲームを選ぶだけです。
技術やAIに詳しくても、セットアップに尻込みしている人のために作りました。一人でも夢中になってくれれば本望です。
娘は大学で機械学習を学び、現在もAIの仕事をしています。彼女に説明したいことと、毎月集まるレトロアーケードクラブに見せたいことが、このプロジェクトのきっかけです。
Jonnyより
FOR THE CURIOUS: HOW CHIP ACTUALLY LEARNSNo equations, promise. Just how it works, the way I'd explain it over the counter.
He looks at the screen, like you do
Chip doesn't get any secret information from the game. He sees the same picture you'd see, shrunk down, plus the score. He looks at the last few pictures together, like a flick-book, so he can tell which way the ball's going and where the bullets are heading.
He has a go, then looks back
He plays for a little while, then pauses and has a think: did that go better or worse than I expected? Whatever went better, he does a bit more of. Whatever went worse, a bit less. Never too much at once, so he doesn't forget what he already knows. Then he plays again. Millions of times.
It's like a boxer watching the tape of their last fight. Same boxer next time, just a better one.
He learns to be patient
My early bots only cared about points, so they grabbed the quickest points they could. In Space Invaders that meant sitting in the corner and firing at nothing. The fix was making a lost life really count. Camping in the corner gets you shot more, so he learns the only way to keep his lives is to move, dodge and get good. Points now are nice. Still being alive later is better.
Get the balance wrong the other way and he turns coward. One night I paid him a little for every second he stayed alive and went to bed. By morning he was hiding under the bases, not firing at all.
He gets curious
In some games the points are miles away. A bot that only chases points never finds them, so it just wanders about. Pick the exploration setting and Chip gets a little reward every time he finds a screen he's never seen before. Curiosity, basically. It's his best chance in something like Pitfall.
Before the breakthrough: breeding bots
I bred them first. An Atari has 128 bytes of memory, so I made each bot a list of 12,800 numbers (12,806 if you're counting). I liked the symmetry. Each bot picked one of six moves at a time. All 896 of them played three games each, the best survived, and the next lot were copies of the winners with every number nudged slightly at random. They learned fast, then hit a wall. Well, a corner.
Evolution mode: a family of Chips
The kit's second way to learn is the grown-up version of my old breeding idea. A small family of Chips take turns practising. After each round, the weakest gets swapped for a copy of the best, with a few of its learning settings nudged up or down at random. I call those nudges mutations. No idea if that's the right word. The hope is a monster in a surprisingly short time; I haven't raced it against the one-brain version yet.
He can learn by watching
Both ways of learning can learn by watching you play. Show him how, and next time he trains he copies you first, then learns to beat you. You can even let one bot show a fresh Chip how it plays. When I tried that, he seemed to fly ahead, and the teacher's style stuck around long after. Part of that's by design (the pull towards the teacher fades slowly, over 5 million moves), so I want to measure it properly. It still baffles me.
What's next: playing in his head
The cleverest AIs now work out how a game works, then practise inside their own imagination, like a snooker player lining up shots in their head the night before a match. Then there's learning from videos of experts playing. That's where I'm heading next. No promises on dates.
THANK YOU: THE CLEVER PEOPLE AND PAPERS BEHIND ITI'm self-taught and I learn as I go. These are the people whose work I've read, borrowed from or bumped into along the way.
- DeepMind · 2015Human-level control through deep reinforcement learning
The one that started it all for me. A computer learning dozens of Atari games from nothing but the screen and the score.
- OpenAI · 2017Proximal Policy Optimization
Chip's way of practising: have a go, look back, do a bit more of what worked, but never change too much at once.
- DeepMind · 2018IMPALA
Where Chip's eyes come from: the part of his brain that looks at the screen.
- Uber AI · 2017Deep Neuroevolution
Proof that breeding the best bots can hold its own on Atari games. Read loads on it.
- Uber AI · 2019Go-Explore
Clever ideas for games where the points are miles away and you have to go exploring first.
- OpenAI · 2018Exploration by Random Network Distillation
Where Chip's curiosity comes from: a little reward for finding screens he's never seen.
- NVIDIA · 2019CuLE: Atari games on a graphics card
Running thousands of Atari games at once on a graphics card. Blew my mind.
- DeepMind · 2018Playing hard exploration games by watching YouTube
An AI that learns from videos of people playing. Next on my list.
- DeepMind · 2019MuZero
Works out the rules for itself, then plans its moves in its head.
- Danijar Hafner and friends · 2023DreamerV3
Dreams up the game in its head and practises in its dreams. One day, Chip.
- The Farama Foundation and friendsArcade Learning Environment
The Atari 2600 playground AI researchers use. The kit uses it for the Atari games.
- The emulator teamsFinalBurn Neo and MAME 2003-Plus
Years of volunteers' work keeping arcade machines alive. They run the arcade games inside the kit, under their own licences.
- Everyone behindPyTorch and Python
Free tools that do the heavy lifting under the bonnet.
And thanks to my retro arcade club, for listening to me go on about it.
