Strange Bench
Critical AI Potluck · exploring strange benchmarks together
Strange Bench
Hi . not you?
RuneScape: can a small model get anywhere?
Your bot gets a fresh character in Lumbridge and 5 minutes. A model writes code to play it. The score is its best XP per minute over any 15 seconds, the same rule as RuneBench.
Your laptop needs Ollama running with the portal allowed. See 1 · Bring a model.
Fixed conditions for every run: same 5 minutes, same starting kit, same prompt, same scoring. Only the model, the skill, and your extra instructions change.
Run a model on your own computer
A language model is a file. With Ollama you can download one and run it on your laptop: no account, no subscription, and no internet once it's downloaded. Then you can connect it here and have it play RuneScape.
1. Install Ollama
Get it at ollama.com/download (Mac, Windows, Linux). Ollama is a small program that runs models. The models themselves are files you download separately.
2. Open a terminal and say hello
Mac: press ⌘ Space, type "Terminal". Windows: open "PowerShell".
ollama run gemma3:1b
The first run downloads 815MB. Then type to chat, and type /bye to leave. Try:
- "How do I get from Lumbridge to Varrock in RuneScape?"
- Something you know a lot about. Where is it confidently wrong?
- Turn your Wi-Fi off and keep chatting. It still works: everything it "knows" is inside that 815MB file.
3. Look inside
ollama list # what's on your disk, and how big ollama ps # what's loaded in memory right now (CPU or GPU?) ollama show gemma3:1b # parameters, quantization, context length, license
Who made it? What does the license let you do? Whose languages does it speak well?
4. Connect it to Strange Bench
Quit the Ollama app first (Mac: menu bar icon → Quit). Then start Ollama yourself, allowing this page to talk to it:
Mac / Linux:
OLLAMA_ORIGINS="https://bench.isofluff.com" ollama serve
Windows (PowerShell):
$env:OLLAMA_ORIGINS="https://bench.isofluff.com"; ollama serve
Leave that window open. On 2 · Launch a bot, choose My laptop and press check connection. If your browser asks to let this site reach devices on your network, allow it. Use Chrome or Firefox: Safari blocks this.
What's happening: our server runs the game and sends your browser the prompt. Your browser asks Ollama on your laptop, and sends back whatever your model wrote. Your laptop does the thinking, and nothing reaches in from outside.
5. Try other models
Open a second terminal window and pull another model. Sizes are approximate:
| model | download | made by |
|---|---|---|
gemma3:1b | 0.8GB | Google (US) |
llama3.2:1b | 1.3GB | Meta (US) |
qwen3:1.7b | 1.4GB | Alibaba (China) |
smollm2:1.7b | 1.8GB | Hugging Face (France / US) |
qwen3:4b | 2.5GB | Alibaba (China) |
phi4-mini | 2.5GB | Microsoft (US) |
gemma3:4b | 3.3GB | Google (US) |
mistral | 4.1GB | Mistral (France) |
ollama pull qwen3:1.7b
On venue Wi-Fi, a 1–2GB model takes a few minutes. You can also copy models from a USB stick. No laptop, or no space? Pick a house model: those run on the host's gaming PC.
A very short history of small models
- 1966, ELIZA. A few hundred lines of pattern-matching. People confided in it anyway.
- 2019, GPT-2 (1.5B parameters). OpenAI first held it back as too dangerous to release. It now runs on a phone.
- 2023, the LLaMA leak. Meta's model weights escaped onto the internet. Within weeks, one person's project, llama.cpp, ran them on a MacBook, and Stanford's Alpaca fine-tuned a 7B model for under $600.
- 2023, Mistral 7B (Paris), released as a torrent link. Small, open, and good enough to matter.
- 2023–24, Microsoft's Phi. "Textbooks are all you need": small models trained on carefully chosen data.
- 2024, the open-weights race. Gemma (Google), Llama 3 (Meta), Qwen 2.5 (Alibaba).
- Jan 2025, DeepSeek-R1 and its small "distilled" versions: reasoning models from 1.5B up, which shook the markets.
- 2025, Gemma 3, Qwen3, and OpenAI's gpt-oss. Even the closed labs started releasing open weights.
- Now: the laptop-sized models on the RuneBench leaderboard (Qwen3.5, Gemma 4) still mostly can't get past level 1.
"Open weights" isn't the same as open source: you get the finished model, but almost never the training data or the recipe.
Other benchmarks to poke at together. All of these are real. There's no Neopets one (yet).
- Pelican riding a bicycle: one silly drawing prompt, given to 60+ models.
- AI Diplomacy: seven models negotiate and betray each other (Twitch).
- TextArena: play text games against models in your browser.
- Claude Plays Pokémon and the PokéAgent Challenge.
- Vending-Bench: can a model run a vending machine business?
- AI Village: a group of agents working together, live on weekdays.
- SnakeBench: models play Snake against each other.
- RuneBench: the full RuneScape leaderboard, with videos of every run.