BMC: Change Temperature Thresholds

This post shows how to update a server's Baseboard Management Controller (or iDRAC or maybe an iLO or something else) to power the server off at a different temperature threshold than the manufacturer default.  This is done using ipmitool and freeipmi commands.  We use it to lower the set points for some of our servers in a less-capable room that we have.  The servers will then do a hard shutdown if the thermal threshold we set is reached.

Performance Problems Resolved

The performance problems that we were having have been resolved.  Without making you read this whole thing to get to the conclusion:

• The problem was solved with an iDRAC firmware update provided by Dell (contact Dell to get the right version)
• The odds that you, the reader, are affected are extremely low (unless you have a PowerEdge M610 with dual Intel Xeon 5650 Westmere processors, six 4GB quad-ranked 1066 MHz DDR3 RDIMMs, no mezzanine cards, and only one 10K SAS hard disk. Even then, there may be another mitigating factor that ensures you aren't affected.)
• If you think you are affected, Dell support should be able to quickly tell you if that is the case or not
• We consider the issue to be resolved

Extrapolating Benchmark Scores Using MSRs on Intel CPUs

It turns out that you can get a very accurate estimate of Linpack benchmark scores by simply reading MSRs and comparing against a baseline score.  I have only tried it on Westmere and Nehalem so far but achieved a minimum of 98.9% accuracy using this method.  It's not good enough to submit to TOP500, but it can be useful when diagnosing problems on your systems.

I had hoped that this could be run in the background (properly niced) in a way that shouldn't impact any other users or processes on the system you are benchmarking.  However, the problem with this method is that it only seems to be valid for whatever benchmark you are using.  In this case I was using Linpack.  I had hoped to try spiking all CPUs with something else for a few minutes and have it return the same numbers.  Alas, it doesn't work as expected.  My guess is that the memory also needs to be hammered at the same time, but that is something that should probably not be done while legitimate processing is happening.

Quantifying Our Westmere Turbo Mode Problems

MYSTERY SOLVED! (Jan 21, 2011): The throttling turned out not to be a processor flaw, though some minor bugs in the CPU or its documentation (not sure which) did contribute to a mis-diagnosis.  I can't state what was causing it, but it was not a flaw with Westmere or turbo mode. The issue was fixed by a simple iDRAC firmware update. More info.

This is a continuation of our search to find the cause of slowness with our Dell M610 blades using dual Intel X5650 Westmere processors.  Please see the other articles I have written, especially Flaws with Intel Westmere X5650?  The other relevant articles are Diagnosing Throttled or "Slow" Systems (Processors to be Precise) and Diagnosing Throttled Processors - Part 2.

During a maintenance window we were able to benchmark all of our systems.  We benchmarked our nodes multiple times with Intel's Turbo mode both enabled and disabled on the processors.  The results showed that 18% of our blades had worse performance with Turbo mode enabled than with it disabled.  That's 93 out of 515 blades included in the benchmark. These tests were done on Dell PowerEdge M610 blades with dual Intel X5650 Westmere processors (2.66 GHz) with 1066 MHz DDR3 DIMMs.  The graph below shows the result of the benchmark run with turbo mode enabled.

xhpl Linpack benchmark results in GFLOPS (rounded to the nearest integer) with turbo mode enabled.  Y-axis is number of blades. Each system is a Dell PowerEdge M610 with dual 2.66 GHz Intel Xeon X5650 processors with 24 GB 1066 MHz DDR3 RAM