Google, Apple and Mozilla join forces to create a new benchmark for the web

For the longest time, we’ve used the same handful of web-based benchmarks to test the overall speed of Chromebooks. Being largely web-based in their functionality, Chromebooks can generally be judged by their performance on web-based benchmark tools. While more comprehensive benchmarks like Geekbench 5 can do a good job at comparing one Chromebook to another, those that compare the speed of web browsing tend to give us the best look at a Chromebook’s actual day-to-day speed.

The go-to for many years on this front was Google’s own Octane benchmark. As a matter of fact, we still tend to reference Octane scores for new devices even though it has been retired for a couple years at this point. Because we’d developed such a strong sense of what sort of Octane score made for a great Chromebook experience, it has always been an easy, quick reference point.

A new benchmark arises

There’s some news that may finally have us shifting gears on that, however, as Google, Apple, and Mozilla are teaming up to bring out the 3rd version of the Speedometer benchmark. We’ve used Speedometer 2.0 for a few years at this point and I feel like it gives us a generally-solid idea of Chromebook performance, but this new news has me very excited that Speedometer 3.0 will be the new go-to for testing Chromebooks.

[1/3] We’re joining an important collab with Apple @WebKit & @firefox to work on the next Speedometer benchmark to measure real-world browser performance. https://t.co/GOxM6pIrkX

— Chrome (@googlechrome) December 15, 2022

The importance of the 3 largest entities in web browser software joining forces to build a better benchmark cannot be overstated. With much of the mobile web rendered with Apple’s Webkit and a huge bulk of desktop traffic routing through Chromium (the basis for Chrome, Edge, etc.) and Firefox (from Mozilla), this collaboration will cover much of the scope of web browsing devices out there.

That means we should have a reliable benchmark that not only works well for Chromium-based browsers, but for mobile and for Firefox as well. With all those bases covered, I could see Speedometer 3.0 becoming the absolute de facto benchmark for browsing speed and a go-to benchmark for testing the real-world speed of Chromebooks across the board.

Speedometer is a benchmark for web browsers that measures Web application responsiveness by timing simulated user interactions on various workloads. Our primary goal is to make it reflect the real-world Web as much as possible. When a browser improves its score on the benchmark, actual users should benefit. In order to achieve this, it should:

• Test end-to-end user journeys instead of testing specific features in a tight loop. Each test should exercise the full set of what’s needed from the engine in order for a user to accomplish a task.
• Evolve over time, adapting to the present Web on a regular basis. This should be informed by current usage data, and by consensus about features which are important for engines to optimize to provide a consistent experience for users and site authors.
• Be accessible to the public and useful to browser engineers. It should run in every modern browser by visiting a normal web page. It should run relatively quickly, while providing enough test coverage to be reflective of the real-world Web.

via the Speedometer GIT

While there are some solid benchmarks out there, seeing Googe, Apple and Mozilla combine to put in work on a single solution makes me very hopeful that – like the days when Octane was the go-to benchmark – we’ll once again have a sort of central testing source once again. That doesn’t mean other benchmarks won’t continue to exist, but it does mean we’ll likely be shifting away from making anything but Speedometer 3.0 the benchmark standard for our own testing when it arrives.

According to the tweet, this effort will take months, so this isn’t something you can use right away. Hopefully, in a few months from now, however, we’ll have this new Speedometer 3.0 benchmark to begin testing with. I can’t overstate how sorely I’ve missed having a go-to benchmark, and I really think this will be it. As we find more updates, we’ll let you know.