Octane, the benchmark we’ve used to compare Chromebook performance since the very early days, is no longer being maintained.
There are good reasons and we’ll get into those, but I’d be lying if I said I wasn’t upset that this long-standing Chrome OS benchmark test is going away.
What’s A Benchmark?
To start, for those wondering, a benchmark is:
a standardized problem or test that serves as a basis for evaluation or comparison
— Merriam-Webster Dictionary
Why It Is Being Retired
There’s a lot of technical language in the post from the V8 team explaining why this benchmark suite is going away, but I’ll do my best to simplify it here.
First, benchmarks are synthetic by their nature. These tests are obviously not real people testing real websites and apps. They are simple, repeatable tests.
This leads to issues over time. The first among those is the ability of JS engines to ‘game the system’ by tweaking their code to score better on a benchmark. However, in this tweaking, these platforms can actually become poorer in real-world testing. Thought in the earlier days, benchmarks like Octane provided useful feedback and allowed much forward movement in the JS ecosystem, it has more recently become a hinderance to actual site rendering.
If any JS engine (V8 included) is making adjustments to make a synthetic test score better while ignoring real-world results, this is a problem.
It is the same issue with students studying for the test but not having a conceptual understanding of what they learned. They test great but fumble in the real world.
What’s Next, Then?
The V8 team says that it is now focused on more realistic, generic use-case scenarios. As V8 and other JS engines exist in many places like websites, apps and servers, finding ways to test these real results is the aim of the team moving forward.
It looks like Octane will stay around for the time being, but it will no longer be updated. Like other benchmarks before it, over time Octane will cease to be relevant.
We will still use it for the time being, but it looks like we may have to come up with our own set of tests to compare Chromebook performance. Larger, established and complex sites like Facebook, Google+, Twitter or Wikipedia can serve as real-world testing grounds to see how a Chromebook performs.
We hope to see another benchmark rise up and take Octane’s place, but the bigger question might need answering: are benchmarks still relevant?
I’m not sure at this point. Maybe the silver lining in all this is an eventual disregard for them and more focus on how a device performs on the things you and I do on a daily basis. In the end, isn’t that all that really matters anyway?