HOOPS: when the measurement disagrees with the player

HOOPS became three games on 17 September: half-court streetball, a full court with two rings and a clock, and the original stand-still shootout. In the week since, most of our work on it has gone into one question: is it fair? This post is about how we tried to answer that with tests, and how the tests got it wrong before they got it right.

"It's impossible not to get blocked"

The first serious complaint from playtesting was that shooting felt hopeless. Almost every shot seemed to be blocked. We had a test for exactly this. It took shots from 2.5, 5, 8 and 11 metres and failed if more than a quarter were blocked. It said 8 per cent. The test and the player disagreed, and when that happens the test is usually answering a different question.

It was. The test fired through a debugging shortcut that gave every shot the perfect power and the perfect release. Nobody plays like that. A player holds the mouse button, drags back and lets go, and everything that can go wrong with a shot happens on that path: the gather, the travelling rule, letting go early or late, a hand in your face.

The old test had a second flaw. A half-court game is first to 11, so the match could end partway through the test, and every attempt after the final whistle counted as blocked. The same build measured 21 per cent blocked over twelve attempts and 68 per cent over thirty. It now restarts the match whenever the score reaches 8.

Shooting the way a player shoots

So we wrote a new test that plays a possession properly. It takes the ball, lets the defender close out, drags the real mouse towards the real power and lets go. It also sorts what happens into three outcomes, because three different things all feel like "blocked" from the sofa, and they need three different fixes:

• Taken: the ball never left your hands.
• Blocked: it left, and somebody got it in the air.
• Missed: it reached the ring on its own and did not go in.

A fourth bucket, short, catches shots that never came within 2.2 metres of the ring. The targets are fewer than 6 per cent taken, fewer than 15 per cent blocked, and more than a third made.

The first version of this test was a bad shooter. It read the ideal power once and dragged to it, and only one release in nineteen landed in the green band of the power meter. It made 10 per cent. People watch and correct, so the test now makes up to five corrections while it drags.

What the honest test found

With a test that played like a person, the problems were easy to see, and most of them were in the defence.

• Defenders blocked without jumping. A ball on its way to the ring could be caught by the man standing between you and it, even with his feet on the floor. Now a shot in flight can only be taken by someone who is off the ground, and only within 0.9 metres of his hands.
• Computer players had huge reach in the air. Their reach was cut to 0.38, while yours stays at 1.25. Before that, the bots made around twenty blocks in ten minutes.
• Defenders contested from too far away. The contest radius was 3.2 metres, and it is now the last 1.5. At the old size, the bots threw 66 passes and took 3 shots in 14 seconds, because everyone was always contested.
• Steals came from reaching through. Standing still with the ball, a separate test measured 5 steals in 25 seconds. Defenders now lean on the ball handler with a push instead of swiping at the ball. The push moves your position rather than your speed, because a speed change would have set off the travelling rule. Our first version pushed every frame, and nobody got a shot off in 40 seconds. Now it is once every 0.8 seconds.
• Passes were picked off too often. Interception was a chance per frame, so the longer a pass was in the air the more certain it was to be stolen. Six of fifteen passes were intercepted. It is now a chance per second.

The last change of the week is the simplest. For the first seven seconds of each possession nobody can block or swipe at you, and a bar over your player's legs counts that time down. We made it a clock rather than a lower chance of being blocked, because a clock is something you can see and plan around.

Keeping the computer players honest

The bots have rules too, and some of them came from tests that only watch. One script plays bots against each other and sorts every shot by distance, which made one problem obvious: the bots were spending whole possessions throwing the ball from their own end. Now how keen a bot is to shoot depends on distance. It is most keen under 3.6 metres and barely interested more than 2.5 metres beyond the arc. It will not take a shot from more than 5 metres past the arc unless the shot clock is under 3.5 seconds.

Bots also miss like people. Their release error grows with distance, with how closely they are guarded and with how tired they are. As the code puts it, a bot's hands are not a calculator. An earlier bug was worse: they worked out their power for one arc and shot at another, and made 0 of 30.

The referee

Two rules only happen when somebody does something wrong, so nothing else would check them. Travelling: once you start pulling back to shoot, the ball is gathered, and moving faster than 1.8 metres a second for more than 0.2 seconds after that is a turnover. The bots have to plant their feet too. A foul: landing on top of a player who is on the floor earns him a free throw. The first version whistled any contest jump that landed near a shooter and gave away about two dozen free throws in ten minutes. A dedicated test holds the real keys and drags the real mouse to make sure both calls are made.

Every number in this post came from a script, but the reason we went looking was a person saying the game felt wrong. The scripts were only useful once they played the way that person did.