Ask the expert: How do hypervisors control server power consumption?

How should I configure a server's BIOS to allow power management through the hypervisor?

Virtualization has vastly improved server efficiency, but it does not alleviate the need to manage system power. Hypervisors like VMware ESXi can control a server's power management instead of the operating system, which ensures "virtualization-aware" power management that will only throttle back system power use when the host server is not fully utilized by guest workloads. In most cases, enabling hypervisor control requires a careful review of the server's BIOS settings.

Defer to the operating system. As a rule, set the power profile or power management option in BIOS to allow "operating system" control. This setting is usually located under the BIOS Power Management Options menu. Since a bare-metal hypervisor will effectively fill the role of the OS, it can control the server's power conservation features directly. If the BIOS is configured to use "BIOS-controlled" power management, neither the hypervisor nor a traditional operating system will be able to control system power. However, the BIOS is not virtualization-aware, and BIOS-controlled power conservation may have undesired impacts on system and workload performance.

Enable all available C-states. Modern processors provide a variety of idle states, called C-states, which define how much energy the processor is saving while it is idle. The exact number of C-states is defined by the particular processor model. This setting is often under an advanced power management options sub-menu. It's a good idea to configure the server BIOS to enable all available C-states, which allow the hypervisor to place the idle processor into a C1 state, and the processor itself can decide to use a deeper C1E (halt) state. If even deeper C-states are available, such as C3 and C6, it's worth enabling software control of those states too, because idle processors can essentially stop and use virtually no power if they're not performing any useful work, but it's best to let the hypervisor make that decision rather than the BIOS.

Avoid indirect power control. Some servers provide technologies that allow the hypervisor to control server power indirectly through the BIOS. This is often called processor clocking control or collaborative power control. When this feature is enabled, the hypervisor is supposed to cooperate with the BIOS to determine the best operating state (such as the optimum clock rate) for the processor. Some combinations of hypervisors and firmware versions may not behave properly with this feature enabled, and you can disable it in the BIOS. If you wish to allow this kind of feature, it's important to evaluate the server's long-term stability and ensure the hypervisor and firmware are up to date.

Remember to change only one setting at a time and document any changes made to BIOS settings. This approach allows IT staff to quickly correlate system stability or performance problems to the last BIOS change, and to reverse questionable changes with minimal troubleshooting or downtime.