Synthetic VS Real World Benchmarks
Why the Debate?
You may have noticed that many hardware and software review websites make sure to note when they use real or synthetic benchmarks in a particular review. Some sites concentrate solely on synthetic benchmarks while others concentrate on the so called "real world" benchmarks. The important thing is not to know which one is right or wrong but to understand what the difference is and what you can personally do with the data that you get from these benchmarks you find online. So lets first start with the difference between real world and synthetic in terms of benchmarking.
An often overlooked fact is that 99% of all the people that will read a review are unable to attain the same results on their personal machines as in the review. This is because almost all reviews are done on a machine with a fresh install of the operating system with as little running in the background as possible. In many cases reviewers are able to artificially inflate scores by disabling hardware devices such as audio, USB, and IDE which free up the CPU and system to run faster. While this practice does not reflect an accurate picture of system performance it is a perfect example of why "real world" benchmarks are so important. In the real world a particular system may be used for several years and have a lot of different software loaded into it. These utilities and applications are largely responsible for system slowdown and are why your particular system may score so much lower than what you see online.
Traditionally the benchmarks you find in 90% of the reviews online are what is known as synthetic in nature. This means that the benchmarks itself does not actually perform a function outside of producing numbers and data used to provide a comparison. An example would be Sisoft Sandra and 3DMark06 and 3DMark Vantage. These programs exist so you can compare your machine against other sets of machines already benchmarked. The number provided by 3DMark06 for example gives relative performance and in and of itself does not give you a useful idea of the performance of the system or component. For example lets say we used benchmark A on a test system and received a score of 20,000 points. What does that score tell you? It tells you nothing unless you have another system, hopefully several, to compare it against. It is only when this data is compared against other systems the hardware for which is known does the data begin to provide a sense of positioning or performance.
Synthetic benchmarks certainly have their usefulness since the data and perofrmance is capable of being replicated on several different machines. It is also useful since in the world of reviews the reader very rarely has a system identical to that in the review. In such cases the synthetic benchmark allows the reader to test their system against the numbers provided in the review to gain an idea once again of relative performance.
Examples of Synthetic benchmarks are:
We have provided links to each of the software listed above so that you can compare your system results to those found online.
Real World Benchmarks
On the opposite end of the spectrun are real world benchmarks. These are traditionally programs that began life offering functionality not aimed at providing scores or numbers. Examples of which are audio encoding programs, video and 3D rendering, office applications, and video games. These programs when used as benchmarks are known as real world benchmarks because they are applications that the traditional users will actually run in every day life. Few people outside of game, benchmark developers, and hardware manufacturers, create a system solely to be a benchmark platform and deliver scores on. The average consumer buys or builds a machine to run office applications, play games, create video and audio, and to access information both online and locally. So for the average consumer the real world benchmarks carry much more meaning but are often overlooked.
Real World benchmarks are much more difficult to interpret for the average consumer as they are much more subjective. Variables are introduced into running these benchmarks which can wildly affect the results. Most websites do not provide enough information when reporting real world benchmarks to enable someone to accurately read the results and place them into perspective. Websites like HardOCP have been big proponents for real world benchmarking and trying to educate people on the need for such results.
Examples of "Real World" Benchmarks:
So what is more important, Real or Synthetic benchmarks?
Now that you know the difference between these types of benchmarks and what to look for we hope that you will have an easier time reading reviews online.
REAL TIME PRICING
:: Copyright © 2002-2008 Techware Labs, LLC :: All Rights Reserved