Support our independent tech coverage. Chrome Unboxed is written by real people, for real people—not search algorithms. Join Chrome Unboxed Plus for just $2 a month to get an ad-free experience, access to our private Discord, and more. Learn more about membership here.
START FREE TRIAL (MONTHLY)START FREE TRIAL (ANNUAL)
Octane, the benchmark we’ve used to compare Chromebook performance since the very early days, is no longer being maintained.
There are good reasons and we’ll get into those, but I’d be lying if I said I wasn’t upset that this long-standing Chrome OS benchmark test is going away.
What’s A Benchmark?
To start, for those wondering, a benchmark is:
a standardized problem or test that serves as a basis for evaluation or comparison
— Merriam-Webster Dictionary
And there are tons of them. Benchmarks can span many computational categories like 3D, graphics, rendering, JavaScript, clock speed, memory speed, and more. There are benchmark tests for just about everything.
Given Chrome and Chrome OS’ heavy leaning on JavaScript for many parts of the browser and OS, the Octane Benchmark developed by the V8 team (the V8 Engine is Google’s high-performance JS engine used in Chromium) was really the gold standard for benchmarking a Chromebook.
Why It Is Being Retired
There’s a lot of technical language in the post from the V8 team explaining why this benchmark suite is going away, but I’ll do my best to simplify it here.
First, benchmarks are synthetic by their nature. These tests are obviously not real people testing real websites and apps. They are simple, repeatable tests.
This leads to issues over time. The first among those is the ability of JS engines to ‘game the system’ by tweaking their code to score better on a benchmark. However, in this tweaking, these platforms can actually become poorer in real-world testing. Thought in the earlier days, benchmarks like Octane provided useful feedback and allowed much forward movement in the JS ecosystem, it has more recently become a hinderance to actual site rendering.
If any JS engine (V8 included) is making adjustments to make a synthetic test score better while ignoring real-world results, this is a problem.
It is the same issue with students studying for the test but not having a conceptual understanding of what they learned. They test great but fumble in the real world.
What’s Next, Then?
The V8 team says that it is now focused on more realistic, generic use-case scenarios. As V8 and other JS engines exist in many places like websites, apps and servers, finding ways to test these real results is the aim of the team moving forward.
It looks like Octane will stay around for the time being, but it will no longer be updated. Like other benchmarks before it, over time Octane will cease to be relevant.
We will still use it for the time being, but it looks like we may have to come up with our own set of tests to compare Chromebook performance. Larger, established and complex sites like Facebook, Google+, Twitter or Wikipedia can serve as real-world testing grounds to see how a Chromebook performs.
We hope to see another benchmark rise up and take Octane’s place, but the bigger question might need answering: are benchmarks still relevant?
I’m not sure at this point. Maybe the silver lining in all this is an eventual disregard for them and more focus on how a device performs on the things you and I do on a daily basis. In the end, isn’t that all that really matters anyway?
