How we check every game without playing each one by hand

PLAYPILE went up on 12 September with sixteen games. By 11 October there were more than fifty, twenty-one of them added that day alone. Any of them can break in ways that are invisible on the machine it was made on, and nobody can hand-play fifty games after every change. So most of the checking is done by scripts that load the real pages in a real browser and look at what comes out.

The failure that works on our machine

The site is assembled by a build step that copies files out of the game folders into one tree, which is what gets published. Every copy is a chance to leave behind a file a game quietly needs. The game still works here, because the original folder is still on disk, and fails for everyone else with a 404.

The check for this is a script we call webcheck. It serves the built site from a plain static server with no special cases, just as our host does, and opens every game through the site's own play page. It presses START the way a visitor would, waits, and passes the game only if its canvas shows more than one colour and nothing was logged as an error. At the end it lists every file the site asked for and did not have.

Three problems with the build were found by this script rather than by reading code, and all three would have shipped broken:

• There are no ports on the web. The big games used to run on their own local servers. HIGHRISE asked the web root for its scripts and models, which only works when it has a server to itself. It now finds its own files relative to itself, so it runs from any folder.
• A play page is a rewrite here and a real folder there. Our local server rewrites /play/apex/ to the play page. A static host does not, so the build writes the page out once per game.
• Folders starting with an underscore vanished. Our host, by default, runs the site through a tool that drops them, and the shared engine the small games import lives in one. A one-line marker file turns that off.

The build also copies a list of what to include rather than a list of what to leave out, because a leave-out list silently ships the next thing nobody thought of. HIGHRISE is 222 MB on disk, and only 28 MB of that is the game.

Teaching the checks not to cry wolf

A test that fails for the wrong reason is worse than no test, because people learn to ignore it. Two of ours did exactly that.

The first was colour counting. A WebGL canvas reads back as solid black unless the game created it with a particular setting, and only the games on our shared 3D engine set it. So HIGHRISE and DUNGEON QUEST, both running perfectly, reported "one colour". The fix was to fall back to a screenshot, which the browser composites properly. There is no image decoder in the script, so it uses the file size: a black 1100 by 700 PNG compresses to a few kilobytes, and a frame with a game in it comes out well over 40,000 bytes.

The second was APEX failing because of an advert. Every page loads the ad script, and that script makes a request that legitimately returns a 404 on a local test server. Our script counted every failed request as a broken game. Now it only counts failures for files on our own site.

Playing badly, on purpose

Webcheck proves a game starts. A second script, smoke, proves it survives being played. It opens each small game, gets past the title screen, then spends a few seconds being a bad player: 26 random mouse moves, a click on every third one, and a key press on every fifth. It fails on any console error, and on a canvas showing six colours or fewer, because a game that throws inside its own loop often just leaves a flat screen and no message.

The first version got past the title screen by clicking the middle and hoping. That worked until a basketball game we had at the time put a team sheet in the middle of its menu, and the test reported a blank game. Now every title screen on the shared engine publishes where its buttons are, and the script asks rather than guesses.

The smoke test also takes a screenshot of what it saw. The first version saved those over the cards on the home page, so for a while a game's thumbnail was a picture of a robot losing in four seconds. They go in their own folder now. The bigger games, such as HOOPS and QUICKDRAW, also have their own scripts that play through the real keys and mouse.

The adverts must not move the game

Before applying for advertising we wrote a readiness script that checks what a reviewer looks for, such as working links, a real 404 page, and privacy and cookie pages. Its most important check is for players. It loads a game page with adverts switched off and records the exact position and size of the game frame. Then it switches adverts on with a dummy account and measures again. If the frame moved by more than two pixels in total, the check fails, because an advert that pushes a game down the page as it loads is how a player ends up clicking the wrong thing. We learned this the hard way: the first live build put self-sizing ad units into 90-pixel boxes, and they spilled 190 pixels over the cards below. Every slot next to a game is now fixed at its exact size. On 7 October we went further: the panel that used to cover a game while it loaded is gone, and there is no advert directly above a game.

What the scripts cannot tell us

None of this says whether a game is any good. A script can prove that a game starts and survives random input. It cannot notice that the gun fills half the screen in HIGHRISE, or that the police in NIGHT SHIFT start searching before you have done anything. Both were found on 7 October, when every game on the site was played through again by hand, and both were fixed that day. The scripts make that hand-played pass affordable, because it can be spent on how a game feels instead of on whether it loads.