Server Room and Three Phase Power for Systems Administrators

There doesn't seem to be much educational material about server room power that is comprehensible to systems administrators.  I don't think there is a "typical" sysadmin type out there but I'm guessing that most have had little to no formal training about server room power.  Three phase power may seem like black magic and lots of incorrect assumptions are made, thus I decided to write this post.  Hopefully this will be useful to some sysadmins out there.

Per-user /tmp and /dev/shm directories

Updated Oct 7, 2013: Tons of updates
Updated: March 19, 2014: The recommended configuration has been in production for months now and works great

I recently discovered a great feature in Linux that allows for per-process namespaces (aka polyinstantiation).  Different processes on the same machine can have different views of a filesystem, such as where /tmp and /dev/shm are.  You can easily make it so that each user on a shared system has a different /tmp that, to each of them, really looks like (and is) /tmp.  This isn't done by setting an environment variable; this redefines mount points on a per-process basis such that each users' processes are using their own directory as /tmp.

Installing a Xeon Phi (MIC) Card in a Dell PowerEdge R720

We got an early release of Dell's Phi installation kit with installation instructions that weren't all that great (to say the least). Dell told me that they are working on better instructions.  In case you're confused, here you go.

A few things to note:

• We have dual 95W CPUs.  These instructions might be different (correct?) for higher wattage CPUs (larger heat sinks, different plastic baffles?)
• The extra heat sinks are for the CPUs, not the Phi.  Our 95W CPUs did not need them.
• The 2.5" and 3.5" mounting brackets are not necessary in our configuration
• We used a different bracket that was provided

Here are some pictures of what it should look like:

RHEL 6.2 - Linux Kernel Problem?

We experienced several problems when we upgraded to Red Hat Enterprise Linux 6.2 from CentOS 5.4.  A user of ours started reporting slowness on some of his larger HPC jobs.  We looked at tons of things then started noticing that one or more nodes would start swapping for no reason.  His job would only use about 60-70% of the memory on each node but some nodes would inexplicably swap (diagnosed with vmstat).  I talked to people at other universities and HPC sites and verified that a similar problem was occurring on their RHEL 6.2 installations.

BMC: Enable SNMP Traps for Hardware Failures

You can enable SNMP traps to be sent from a BMC/iDRAC/Whatever using a few IPMI commands.  This works on every Dell server I have ever tested but didn't work on HP systems I have tried.  I imagine it would work on servers from most other vendors.  I know there are vendor-specific tools to do this, but I prefer using industry standard protocols to administer systems and really don't like having to install a lot of extra vendor tools and daemons.

Enable IPMI Over LAN from the OS using FreeIPMI

Maybe you just heard about how wonderful it is to control your hardware remotely.  Maybe you forgot to configure IPMI Over LAN on a production system's BMC and you don't want to reboot.  Fear not!  Enabling IPMI Over LAN can (usually) be done from the OS using freeipmi.

freeipmi *-config tools primer

Basic usage of bmc-config, pef-config, and other freeipmi *-config tools

The documentation below should work for all freeipmi *-config commands.  It only discusses how to connect to a BMC (including iDRAC, iLO, etc) then read from and modify its configuration.  I picked bmc-config to demonstrate program usage, though the same parameters will work for all the related commands.

In some places I reference the "-f" parameter which allows you to specify a filename.  In newer versions it appears to still work but have been deprecated by "-n".

BMC: Change Temperature Thresholds

This post shows how to update a server's Baseboard Management Controller (or iDRAC or maybe an iLO or something else) to power the server off at a different temperature threshold than the manufacturer default.  This is done using ipmitool and freeipmi commands.  We use it to lower the set points for some of our servers in a less-capable room that we have.  The servers will then do a hard shutdown if the thermal threshold we set is reached.

Performance Problems Resolved

The performance problems that we were having have been resolved.  Without making you read this whole thing to get to the conclusion:

• The problem was solved with an iDRAC firmware update provided by Dell (contact Dell to get the right version)
• The odds that you, the reader, are affected are extremely low (unless you have a PowerEdge M610 with dual Intel Xeon 5650 Westmere processors, six 4GB quad-ranked 1066 MHz DDR3 RDIMMs, no mezzanine cards, and only one 10K SAS hard disk. Even then, there may be another mitigating factor that ensures you aren't affected.)
• If you think you are affected, Dell support should be able to quickly tell you if that is the case or not
• We consider the issue to be resolved

Extrapolating Benchmark Scores Using MSRs on Intel CPUs

It turns out that you can get a very accurate estimate of Linpack benchmark scores by simply reading MSRs and comparing against a baseline score.  I have only tried it on Westmere and Nehalem so far but achieved a minimum of 98.9% accuracy using this method.  It's not good enough to submit to TOP500, but it can be useful when diagnosing problems on your systems.

I had hoped that this could be run in the background (properly niced) in a way that shouldn't impact any other users or processes on the system you are benchmarking.  However, the problem with this method is that it only seems to be valid for whatever benchmark you are using.  In this case I was using Linpack.  I had hoped to try spiking all CPUs with something else for a few minutes and have it return the same numbers.  Alas, it doesn't work as expected.  My guess is that the memory also needs to be hammered at the same time, but that is something that should probably not be done while legitimate processing is happening.