How to do a user review!

Often times I see a forum user doing a “mini” review of a product and doing some thermal testing and showing a result that is worthless because the methodology was poor.  Even some “professional” reviewers need to learn a thing or two about how to do thermal testing.  I won’t claim to be the best, but this is what I’ve learned along the way.  Let’s take a cpu water block test as an example.

Science Projects

If in school you had to do a decent science project, or any kind of write up after working in the lab, you’d know that you are expected to state a hypothesis, list your apparatus and testing methods and write up your results.  A proper thermal test should be conducted the same way so that it can be attempted to be repeated.  List what’s in your setup, what was used to record data and how you did the testing.

Idle temps

No one cares about idle temps, they are meaningless because they should track coolant temperatures with a couple of degrees tacked on.  The power level is so low that the margin of error will be huge relatively.  Look up signal to noise and think about how it effects your test results.  Listing idle temps is the sure sign of someone who doesn’t understand thermal testing.  If it’s demanded by readers then the readers should be educated by the reviewer as to why the data are useless rather than appeased.

Coolant temps

I see a lot of users who don’t realize that measuring just cpu temperature without a reference or using ambient as a reference is a bad idea.  Instead of measuring the performance of the block you’re now measuring the performance of the block and *your* cooling system setup.  Ideally when you’re testing something you want to test *only* that something.  It also increases errors as there are now more parts that can introduce error that can vary over time.  For example the performance of the cooling system vs ambient temperature – is it constant or not?  Does it change over time as your radiator gets more dust?  Measuring coolant temperatures and subtracting from load temperatures isolates the measurements to the CPU block itself and makes for a better quality of result.  Ideally you would have a good temperature probe in your system connected to a meter that you’ve calibrated.  Realistically you may not be able to afford that.  However any end user should be able to buy a $20 probe with an LCD screen and manually log the temperature into a notebook every minute for example.

Loading

Do research as to what software gives a good stable over time loading for the system is and what the settings for that are.  Do not run Prime 95 in blend mode for example, but use custom small FFTs.

Throttling

CPUs and GPUs can both throttle due to power or temperatures.  Make sure your bios or load doesn’t cause this and if necessary make changes to the bios to prevent it.  Monitor frequency and load with your logging tools to spot throttling.  If you have one block running with a 99% load and one with a 98% load because one throttled more than the other then the test results are worthless.

Be realistic and if in doubt test again

If your results don’t make sense, repeat them.  Change variables if need be to debug them.  You shouldn’t expect to see more than a few degrees between different blocks, the days of easy performance gains are pretty much over.

TIM TIM TIM, MOUNT MOUNT MOUNT

TIM cures over time.  That means that the thermal properties of TIM change as it ages.  Some TIMs are better than others at this.  I use MX2 because skineelabs shows it to cure pretty quickly and it’s also easy to spread and more tolerant of poor mounts.  In addition it’s cheap which is good when you are using a lot of it.  Curing time means that it is critical to maintain a strict time between the TIM being applied and the block mounted and the start of the data logging.  Don’t mount the block the night before and do the testing in the morning unless you are planning on doing every mount that way.

Getting a good mount is key and learning how to mount blocks and avoid a bad mount can take time.  Different CPUs, TIMs and blocks can like different amounts of TIM and different application methods.  Learn to look at the TIM spread pattern and analyze where it’s thicker and where it’s thinner.  Doing multiple mounts therefore is important and looking at the standard deviation can be a good indicator of how good you are at mounting that particular block, however some blocks are just harder and less reliable in their mounting system.

5 mounts is really considered the minimum for testing and is typically averaged, however sometimes we run less (on GPUs for example) when mount to mount variation can be lower and for purposes of time, but with the knowledge that the end result is therefore less reliable.

Ambients

If you’re measuring coolant temperatures with a good temperature logging system then ambients shouldn’t really matter.  However most likely they still do.  As much as possible you should try and keep ambients the same during testing, and make a note of what they are (or datalog them) to help debug your results.

Error Sources

Even with a perfect data logger and probe, even with infinite time to do an infinite number of mounts on the block and cpu there will still be error.

Sadly we are mainly limited in the way we measure the temperature of the CPU or GPU.  Most temperatures from the part are only given to 1C accuracy and are not calibrated.  The 1C itself gives a large degree of error that can be mitigated by averaging the data over time.  In addition ideally the temperature sensor of the part should be calibrated but that is extremely hard to do.  Instead understand that these are real errors that effect your results and don’t get cocky about how good your results are.

There is mount to mount variation simply because each  mount as a slightly different amount of TIM in a slightly different shape and spread.  On top of that your results are one sample out of a possible myriad.  You’re testing one cpu with one motherboard with one socket and one block.  Each of those variables when changed will lead to changes in performance.  Some of them small and some of them larger.  Always be wary of reading too much into the results.  For example if one block is 0.2C better than the other, that is not a convincing lead.  Even if they were 1C better in your test that does not really mean that across every situation that block will be 1C better than the competition.

Hype

Don’t believe the manufacturer hype, but do make sure of your methods.  Don’t claim a manufacturer’s claims are bogus (or support them) if your testing doesn’t back it up within the margin of error.  Some manufacturer’s do good testing work, and some don’t.  Some will try and intimidate you and claim that your methods are poor and theirs are better.  Understand that they can be protective and defensive, afterall this is their design and their livelihood.  However there is nothing wrong with challenging them in a friendly manner and working together to improve your setup or indeed theirs.  Now every company is willing to do this however.

Debug

Always think with a scientific mindset.  How can I make this test better and more accurate.  What am I comparing to?  Do I have a good control?  What other variables are changing and can I control them better?  All of these lead to better results and a better quality of test.  It sounds extreme but these days performance is so close that to differentiate it.

If you do all of this then not only will your testing be better, but your understanding of other’s testing and the watercooling system as a whole will be better!