BMC: Change Temperature Thresholds

This post shows how to update a server's Baseboard Management Controller (or iDRAC or maybe an iLO or something else) to power the server off at a different temperature threshold than the manufacturer default.  This is done using ipmitool and freeipmi commands.  We use it to lower the set points for some of our servers in a less-capable room that we have.  The servers will then do a hard shutdown if the thermal threshold we set is reached.


Background

A smaller server room we have is shared with a college on campus.  We have compute nodes (part of our HPC lab) in that room and they have critical college infrastructure in there.  We don't like having our equipment go down, but we do like to help them keep their servers up.  As a a result, we will shut our equipment down first in the event of server room problems.

All of the servers in the room are on UPS power (unlike our main server room where no compute nodes are on UPS).  Unfortunately, the chillers are on wall power and are off for the duration of any power outages.  Thus, if the power goes out for an hour or a chiller decides to stop chilling at 3 AM, we end up in a situation where there is still a normal heat load from the servers but no more cooling.

We used to take care of everything manually by watching the temperatures climb, hope the power would come back or the chiller get fixed ASAP (the latter of which usually takes an hour our two if we're very lucky), and finally power off our servers in the room.  Sometimes the effect of us powering off our compute nodes, which are normally operating around 100% utilization, would be enough to save the college from having their operations interrupted.

Since I don't really like getting woken up to an "it's hot in here" or "on battery" notification from our room sensors when an automated response is just as good, I decided to do something about it.  I already automated the cleanup of systems after a power outage, so that wasn't a concern.  What was a concern, however, was helping out the college we share the room with.  That normally required manual intervention (again, not the best use of anyone's time at 3 AM).  Since we already have a process we follow manually, why not automate it?

Code

Notes:

• All IPMI commands (ipmitool and freeipmi) listed below can be done locally or remotely.  See the man pages for details (it's really quite simple if the servers are setup correctly).
• These were only tested on some models of Dell PowerEdge and HP ProLiant servers. Double-check the fields or you may end up telling the server to power off based on the wrong temperature sensor.
• Be sure to test this first. An easy way to do so is to set a limit 1-2 degrees above the current temperature and then add some heat in front of the server.  Change it to the intended thresholds after testing is complete.

Fetch current ambient temperature sensor value:
Dell iDRAC/BMC

ipmitool sensor get 'Ambient Temp' |grep 'Sensor Reading' | awk '{print $4}' #usually faster on Dell than:
ipmitool sdr entity 7.1 | grep Ambient | sort | awk -F '|' '{print $5}' | awk '{print $1;}'

HP iLO/BMC (only tested on one model of server)

ipmitool sdr entity 39.1 | grep Temp | sort | awk -F '|' '{print $5}' | awk '{print $1;}' #usually faster on HP than:
ipmitool sensor get 'Temp 9' |grep 'Sensor Reading' | awk '{print $4}' #double check that this corresponds to ambient

Others. After finding the right entity, use that for faster access:

ipmitool sdr type Temperature # figure out which sensor corresponds to ambient (planar is probably not what you want)

Find which values to change 

ipmi-sensors-config -L |grep Ambient

ipmi-sensors-config -L |grep Temp #may be necessary on non-Dell

This will return something like "3_Ambient_Temp" on all Dell systems I have tried (but not HP).  This is the section to use when modifying the temperature values.

You can also use this dynamically if you have a variety of Dell servers (and maybe other non-HP servers).  Instead of specifying "7_Ambient_Temp" or another value, replace it with "$(ipmi-sensors-config -L |grep Ambient)".

Example:

ipmi-sensors-config -o -e "$(ipmi-sensors-config -L |grep Ambient):Upper_Critical_Threshold"

Fetch current thresholds
ipmi-sensors-config -o -S "$(ipmi-sensors-config -L |grep Ambient)" #grab all set points for the ambient temperature sensor
ipmi-sensors-config -o -S 3_Ambient_Temp #same as above, but specify the section (assuming 3 is correct)
ipmi-sensors-config -o -S "$(ipmi-sensors-config -L |grep Ambient):Upper_Critical_Threshold" #fetch the upper critical threshold value
ipmitool sensor list | grep Ambient
ipmitool sensor get "Ambient Temp"

Set temperature thresholds (Dell and possibly other non-HP servers):
ipmi-sensors-config -c -e "$(ipmi-sensors-config -L |grep Ambient):Upper_Non_Critical_Threshold=30" #warning

ipmi-sensors-config -c -e "$(ipmi-sensors-config -L |grep Ambient):Upper_Critical_Threshold=33" #critical

Set the server to actually power off at the critical threshold instead of just alerting (Dell):
I don't have a  great way of querying the correct value programmatically, so you'll just have to find this information manually using "ipmi-pef-config".  On Dells, the value you want (in my experience) has always been "Event_Filter_9".  Then set "$SECTION:Event_Filter_Action_Power_Off=Yes"

The way to find it manually is to use "ipmi-pef-config -o" and look for an Event_Filter section that has "Event_Severity Critical" and "Sensor_Type Temperature".  Assuming the section is Event_Filter_9, run the following:

ipmi-pef-config -c -e 'Event_Filter_9:Event_Filter_Action_Power_Off=Yes'

(Theoretically) set temperature thresholds on HP ProLiant BL460c G1... BROKEN:
ipmi-sensors-config -c -e '13_Temp_9:Upper_Critical_Threshold=30' #warning
ipmi-sensors-config -c -e '13_Temp_9:Upper_Non_Recoverable_Threshold=33' #critical

This will return "ERROR: `13_Temp_9:Upper_Non_Recoverable_Threshold' is not writeable".  There's probably a non-industry standard, HP-specific way to do this, but we have so few HP servers it isn't worth looking up how to do it.  HP only recently added the ability to do IPMI over LAN, so that doesn't surprise me.  Meanwhile, Dell has had great BMC/IPMI support on every one of their servers I have ever managed.

Before IPMI over LAN support was added, I called HP to ask why that functionality wasn't available.  The excuse was something like "vendors implement it so differently that it's hard to implement."  Of course, my response was "if you're the vendor who implements it, why can't you implement it in an industry-standard way?"  They finally did get around to it but apparently don't have a complete implementation yet.  Maybe newer servers do, but I don't have access to any.

Comment below if you have any questions. I might be able to answer them. If you figure out how to make this work with HP or other vendors (if it doesn't work using the methods above), please let me know and I'll update this post.