Rethinking Data Center Design to Maximize Energy Efficiency and Sustainability

Gordon Johnson, Senior CFD Engineer, Subzero Engineering
William Frantz, Senior Principal Scientist, ASHRAE Member

Most legacy computer rooms and data centers are equipped with raised floors where underfloor cold supply air distribution is provided to the IT equipment via perforated tiles. Additionally, the raised floor has also been used as a space for data cables and power wiring. This design has been around for decades, and no doubt will continue to be deployed in data center construction projects.

However, data centers continue to evolve, and recently more and more data centers are being designed with slab floors and overhead cabling. Rather than install expensive and non-flexible ducting to supply cooling from overhead diffuser vents, engineers are seeing high efficiency and sustainability by flooding the room with cold supply air from either perimeter cooling units, computer room cooling air conditioner/computer room air handling (CRAC/CRAH) galleries, or other cooling methods (rooftop cooling units, fan walls, etc.). Hot aisle containment (HAC) separates the cold supply air from the hot exhaust air, and a plenum ceiling returns the exhaust air back to the cooling units. This design is also gaining popularity due to its simplicity and flexibility.

Why are both large hyperscale and smaller data centers moving toward this layout? What makes this design simpler and more versatile? How does design affect airflow and cooling optimization? What makes it energy efficient and sustainable? What about equipment reliability?

This question, and the debate surrounding it, has been around for as long as we’ve had data centers. There’s also an endless amount of reading material on this subject, including countless white papers to support each side. For some, the choice for one or the other simply comes down to “that’s the way we’ve always designed our data centers.”

But as mentioned in the introduction, data centers and their cooling methodologies are evolving, as is the need to make them energy efficient and sustainable. Just because the data center industry has done something or built data centers a certain way for the last 20 plus years doesn’t mean it’s the best or most suitable way going forward.

For example, most early data centers were equipped with raised floors, which are used for underfloor air distribution via a pressurized raised floor system that delivers cold supply air to the IT equipment via perforated tiles. In addition, this raised floor area is often used as an area to run both data cabling and power wiring.

Slab floors also have been around for decades but haven’t enjoyed the same popularity as raised floors. Until recently, slab floor designs have meant that the cold supply air was delivered to the IT equipment via overhead ducts and diffusers. One issue with this design has been the lack of flexibility with moving and adding racks to the floor. Once the duct system is installed and in place, the racks must be located such that the diffusers are sitting directly over the cold aisles, which unfortunately isn’t always the case. This results in difficulties of getting the cold supply air to where it’s needed — at the server inlets. As rack densities change and increase, increasing the cubic feet per minute (CFM) of airflow to specific locations is also challenging, as is determining how much airflow is supplied from each individual diffuser.

In addition, getting the hot exhaust air back to the cooling units can also be problematic since the returns are located on the front/bottom of the CRAC/CRAHs, and hot exhaust air naturally wants to rise towards the ceiling. Cold aisle containment (CAC) can and should be used for these layouts but can be difficult to install due to the location and height of the existing ducts.

What about airflow optimization and cooling efficiency? It depends who you ask. Some say from a layout flexibility perspective, a raised floor is better since it’s easier to add and rearrange perforated tiles as needed. Others argue that overhead cooling tends to produce fewer issues with air leakage or bypass air (wasted air). So when should a raised floor be used, and what are some advantages to the raised floor design?

When to Use Raised Floors

One situation where the raised floor might still be the better choice is with data centers that serve as multi-tenant (MTDC) facilities or colocations. The reason the raised floor is advantageous is because it can offer flexibility for adding cooling and electrical when needed, such as when new customers move in or need to expand their layout. Often, it’s easier to run new conduit and piping under the raised floor than overhead. Another scenario where raised floors might be preferred is when the data center has low power density racks. This might mean a lot of rack movement and changes, especially in caged hosting spaces, so a raised floor is often still the best choice. ¹

In addition, distributing the cold supply air efficiently and where needed via overhead ducts in a room with a slab floor can be difficult. With a raised floor, rearranging the perforated tiles is all that may be needed to correctly change the airflow distribution to the IT equipment. ²

Regardless of these and other advantages, there are many reasons why more and more new data center designs, both large and small, are moving away from the traditional raised floor and using the slab floor design instead. When should a slab floor be used, and what are some advantages to the slab floor design?

Raised Floor Data Center

When to Use Slab Floors

When it comes to equipment installation and movement, as well as maintenance costs, the slab floor is the better choice. There are no concerns about placing and moving newer, heavier equipment on areas of the floor that aren’t designed for the weight. For regions prone to earthquakes, seismic performance is also important, again making the slab floor the preferred choice. Not only does it cost less than a raised floor, but equipment anchoring is easier, and the added lateral bracing and reinforcement associated with a raised floor system is no longer an issue either. 

Many large hyperscale data centers, with their unprecedented demand for storage, are choosing the slab floor over the raised floor. Even smaller data centers, with space at a premium, are moving in this direction since they save space vertically by not installing the raised floor and associated ramps that usually are part of this design. Are we recommending that new data center designs should follow the traditional slab floor design with upflow cooling via overhead ducts and diffusers? No. This design, similar to raised floors, has been around for decades but presents unique challenges to efficient airflow management and cooling of the IT equipment. What we are recommending is a design that more and more data centers are using for efficiency and sustainability, as well as lower installation and maintenance costs. 

Slab Floor Data Center

Slab Floors Simplify Cooling IT Equipment

With cooling technology developments, the need to provide underfloor supply air plenum has been reduced, as has providing supply air via overhead ducts and diffusers. Instead, new data center designs are opting to use slab floors and the simple approach of flooding the room with cold supply air while exhausting the hot air through containment into the plenum ceiling return. This design is being used in both smaller sites and is becoming the standard in large hyperscale deployments due to its many advantages.

The cooling used to flood the room can be via CRAC/CRAH units located on perimeter walls or outside the data center in CRAC/CRAH galleries, indirect evaporative cooling (IDEC) units, and both airside and waterside economizers just to name a few. Additional cooling technologies including fan walls, consisting of an outside wall with cooling units, and a fan wall inside the data center that floods the room with cold supply air are also excellent candidates for this design.

Another benefit is that hot aisle containment (HAC) is used to remove the hot exhaust air out of the data center instead of cold aisle containment (CAC).

From a thermodynamic standpoint, both types of containment have similar results since they’re both preventing cold supply and hot exhaust air from mixing in the data center. Since HAC encloses the hot aisles and turns the rest of the data center into one large cold aisle, data center operators and facility managers tend to run higher supply air temperatures with HAC compared to CAC.

The result is the IT equipment receives the cold supply air needed to meet demand airflow, and the process is simplified since the data center can be viewed as one large cold aisle that’s being flooded with cold supply air.

One important note is that CAC cannot be deployed for this design since it would prevent the cold supply air from cooling the IT equipment. Instead, HAC should always be used to contain the hot exhaust air, and a plenum ceiling return should also be installed to remove the exhaust air from the data center. In addition, when it comes to HAC, it’s important to choose a containment solution that takes a simplistic approach to a typically complex design.

The AisleFrame containment system from Subzero Engineering is designed to provide a complete solution for not only aisle containment, but also provides a sleek structure that serves as the infrastructure carrier for busway, cable tray, and fiber. AisleFrame is completely ground supported and a simple flat slab floor is all that is needed.

The goal of any containment system is to improve the intake air temperatures at the IT equipment and to create an environment where changes can be made that will lower operating costs and increase cooling capacity. What makes the AisleFrame containment system unique is that it easily accomplishes this while allowing both existing and new facilities, including large hyperscale data centers, to build and scale data centers quickly and efficiently.

Traditional methods for supporting data center infrastructure such as containment, power distribution, and cable routing can be costly and time consuming. They require multiple trades working on top of each other to accomplish their work. The AisleFrame structure provides a method for the rapid deployment of infrastructure support and aisle containment. For example, all cable pathways and the busway can be installed when the AisleFrame containment is installed, allowing the electricians to come back when needed to energize the busway as the IT equipment gets installed and in the future as the IT footprint expands. AisleFrame also gives the end user the ability to deploy in small, replicable pods. This helps to limit the amount of upfront capital spent compared with building out entire data halls by providing all the infrastructure necessary at the pod level, while allowing for almost limitless scaling should the situation require it. ³

When selecting a containment solution, whether it’s AisleFrame containment or a typical HAC system consisting of end of aisle doors and containment up to the plenum ceiling return, the seal or leakage performance (typically a percentage) of the containment system is essential. It’s often stated that leakage is the nemesis of all containment systems. The AisleFrame system and all Subzero containment solutions have approximately 2% leakage. This reduces and practically eliminates both bypass air and hot recirculation air that raise server inlet temperatures on IT equipment. The result is superior efficiency of the cooling system.

There’s another important element to this design, and that’s the plenum ceiling return that’s deployed.

Contributed by Bill Frantz

Much attention is given to reduce leakage in containment aisles and raised floors. These components are known sources of fugitive leaks and distributed leaks that contribute to bypass losses. The discipline given to sealing raised floors and racks should also be applied to ceilings. Ceilings form a return air plenum and, more importantly, separate the warm return air from the conditioned room. Gaps around the tiles and drop rod penetrations allow conditioned air to bypass the IT equipment. The bypassed air must be made up by increasing cold air supply from the cooling units, thereby consuming more energy.

Typical ceiling systems have leakage rates far larger than raised floor systems. At a differential pressure of 0.02 [in WC], a raised floor system may leak about 0.05 to 0.30 [cfm/ft²]. A non-gasketed ceiling with typical drop rod penetrations will leak about 1.44 [cfm/ft²]. When Armstrong ULTIMA AIRASSURE ceiling tile is combined with DYNAMAX structural grid, leakage is reduced to about 0.19 [cfm/ft²] and becomes comparable to raised floor and containment wall systems.

ULTIMA AIRASSURE ceiling tiles with DYNAMAX structural grid is an engineered solution designed for data centers. ULTIMA AIRASSURE tiles feature a perimeter gasket that reduces ceiling leakage at the tile edges. DYNAMAX structural grid eliminates drop rod ceiling penetrations. Paired together, the system reduces bypass leakage down to levels similar to the rest of the containment system. The Armstrong ceiling solution delivers accessibility and containment that improves cooling efficiency and saves fan energy.

Heat balance and airflow calculations were solved to satisfy the governing equations for 4,000 random combinations of parameters.

The resulting chart clearly shows the effects of changing from a typical (high leakage) ceiling to the (low leakage) ULTIMA AIRASSURE with DYNAMAX grid system.

Ceiling and grid systems form return air plenums in data centers and separate cold supply air from warm return air. Leaks across this containment plane contribute to bypass losses, increased fan energy, and increased operating cost. The use of a ULTIMA AIRASSURE gasketed ceiling tiles with DYNAMAX structural grid reduces leakage rates to levels similar to the rest of the containment system.

We’ve mentioned the importance of maximizing energy efficiency and sustainability. Flooding the data center with cold supply air for the IT equipment and containing the hot aisles so that hot exhaust air returns to the cooling units (or is rejected by some other method) is a simple, easy, and flexible design. Designers of all new data centers should consider these options for future deployments. Another benefit of this (and most HAC designs) is it’s easier to achieve airflow and cooling optimization. In a perfect world, we would simply match our total cooling capacity (supply airflow) to our IT load (demand airflow) and increase cooling unit set points as high as possible. However, there’s inherent leakage in any design, including within the IT racks. The goal is to minimize this leakage as much as possible which is why the containment and ceiling structure chosen is crucial.

The lower the overall leakage, the less cold supply air needed. To maximize energy efficiency, we want to use as little cold supply air as possible while still maintaining positive pressure from the cold aisle(s) to the hot aisle(s) throughout the entire data center. When this is achieved, there will be consistent supply temperatures across the server inlets on all racks throughout the data center. Because HAC is used with this design, the data center is essentially one large cold aisle, so the total sum of cold supply airflow should only be slightly higher than the total sum of demand airflow (10% - 15% should be the goal). This percentage is easily attainable if leakage is kept to a minimum when using a quality containment and ceiling solution, along with good airflow management practices such as installing rack blanking panels, sealing the rack rails, etc.

Next, raise the cooling set points while maintaining server inlet temperatures at or below the American Society of Heating, Refrigeration, and Air-Conditioning Engineers (ASHRAE) recommended specifications for cooling IT equipment (80.6°F/27°C), and let the energy savings begin! This also results in higher equipment reliability and lower mean time between failures (MTBF). It’s been said that the best energy saved is the energy we don’t consume in the first place, and that’s especially true in the data center industry as we continue to progress and evolve in our goal to become more sustainable and lower our carbon footprint.

The data center industry is constantly evolving, and so should our designs. Energy efficiency should continue to be a top concern for data centers in the future. Data center designers and owners should carefully evaluate all options rather than just relying on, or selecting from, old projects. The saying “that’s just the way it is because it’s always been that way and there’s no reason to question it” has no place in the industry.

The Subzero Engineering AisleFrame containment system - in combination with Armstrong ULTIMA AIRASSURE ceiling tiles and the DYNAMAX structural grid system - delivers a flexible, energy effiecient, and sustainable data center. Built for today and tomorrow.