Evaluating the Reliability of Passive Server Components for Single-Phase Immersion Cooling

Nontraditional methods like immersion cooling of high-power information technology (IT) equipment have been adopted by operators to push power densities. It proves to be an efficient and cost-effective technique as it eliminates the need for server fans, elevated flooring, and in particular a computer room air conditioning unit, which constitutes a significant percentage of the traditional cooling cost. In single-phase immersion cooling, servers are immersed in a dielectric coolant, and single-phase forced convection is used to circulate the dielectric liquid across hot electronic components and ultimately rejecting the heat using a heat exchanger. In this kind of system, the hardware enjoys a few benefits like having lower junction temperature, relatively little temperature swings or hot spots, no electrochemical migration and it generally runs more reliably with the improved performance [1–3]. Thus, overall, single-phase immersion-cooled data centers would result in lower maintenance and operating cost; and a noise reduction compared to air-cooled data centers [4,5].

Single-phase liquid immersion cooling (Sp-LIC) offers significant advantages when compared to forced convection air cooling such as higher thermal mass, a high percentage of heat dissipation due to direct contact of dielectric fluids with every component, improved reliability due to as the information technology equipment is shielded from the impact of contaminants and harsh environment, reduction in CapEx and energy costs as fans and computer room air handler/computer room air conditioning units might not be required [6,7]. At the same time, Sp-LIC also does not require complex liquid distribution manifold infrastructure when compared to direct-to-chip liquid cooling, thus, proving to be a better option for edge and modular data center applications. Higher compaction and simplified data center layouts are possible with a tank-based single-phase immersion cooling, which can further reduce the CapEx. Opportunity for waste heat recovery, reduction in carbon footprint, and water usage reduction are some other added benefits of Sp-LIC [8]. It should however be noted that the quality of the waste heat recovered is higher for both pumped and open bath two-phase systems. Sp-LIC also leapfrogs traditional and more contemporary cooling technologies in terms of cooling efficiency, as shown in Fig. 1 [9]. Figure 1 shows the rack density as a function of Power Usage Effectiveness at the facility level for different system-level cooling technologies. The graph shows that even the most efficient air-cooling-based technologies are limited to 20–30 kW per rack. With its ease of implementation, especially for edge data centers, Sp-LIC can provide 300% more rack power density than air-cooling and almost twice as much as cold-plate-based single-phase direct-to-chip liquid cooling with higher efficiency [9].

Submerging most of the servers' heat-generating components in a hydrocarbon dielectric fluid causes reliability issues at the device level [10–12]. Most of the fluid suppliers and vendors perform soak tests to determine the material compatibility with hydrocarbon fluids. But they do not consider operational reliability such as mechanical and electrical properties of the components specific to the application work environment. This paper reports on the authors' attempts to explore these issues and ascertain whether they warrant special preventative care measures. There is a need for a framework for testing IT equipment cooled by single-phase immersion, as there is a lack of an established methodology or industry standard for the same. This paper is a first attempt at addressing this challenge by determining the design of experiments to test the reliability of wetted materials for single-phase immersion cooling. Accelerated thermal cycle (ATC) testing, especially the “ATC JEDEC” standard, which is utilized for air-cooling, has a crucial limitation. Due to vastly different heat capacities, there is a considerable difference in the ramp rates and corresponding thermal profile of air and dielectric fluid, making the current JEDEC standard not applicable in its current form for single-phase immersion cooling. ASTM D3455 is an industry standard available for testing the compatibility of various materials with mineral oil. These test methods cover screening for the compatibility of materials of construction with electrical insulating oil for use in electrical equipment [13]. These standards exist for the transformer insulating oils. The oil and winding temperatures of the transformer are much higher than single-phase immersion cooling operating temperatures [2]. With necessary modifications made in ASTM D3455 for immersion-cooled data center environment, it is possible to devise a method for testing material compatibility with different dielectric fluids. For this study, accelerated thermal degradation of the printed circuit board (PCB), passive components, and fiber optic cables submerged in air, white mineral oil, and synthetic fluid at a hoisted temperature of 45 °C and 35% humidity is undertaken.

Since the main constituents of a PCB are polymeric and metallic, a good test for mechanical performance would be one based on the change in elastic modulus of the PCB sample with time. To gage if any aging phenomena are triggered, in addition to the mechanical analysis, a close inspection of the formation of cracks and/or ingress of dielectric fluid in the PCB is necessary. In the case of passive components, a careful inspection of any microstructural cracks, which determines the mechanical and electrical execution of key server functions, is of prime importance.

Optical cables are employed in systems that require higher data transmission rates, thus widely in rack and data center-level connection. Maintaining the integrity of the transmitted signal is critical, and to this end, it is necessary to test if there is a reduction in the mechanical and optical performance of the cables over time when comparing immersed and traditionally air-cooled interconnections. Extensive ingress of dielectric fluid can lead to structural or optical changes at the core-cladding interface, leading to increased attenuation. Existing and even the most recent literature on mechanical testing of water-immersed optical fibers reveal that the effect of diffusion of water into the cable structure is the dominant factor in mechanical degradation, as attested by Glaesemann [14] and Dwivedi [15]. However, there does not exist any documented testing of optical cables by high-temperature aging upon immersion in mineral or synthetic oil. The analysis done in this research serves to fill that gap in knowledge by testing certain assumptions about the extent of change in thermomechanical behavior of thermally aged IT hardware components immersed in single-phase dielectric fluid.

The primary objective of this work is to quantitatively ascertain and characterize the material compatibility of single-phase oil-immersed components. This paper reviews the changes brought about by single-phase immersion in the physical and electrical properties of three types of essential data center server components: PCBs, passive electrical components, and fiberoptic data cables. The changes in the electrical properties of passive components like capacitors, resistors, and transistors are analyzed for single-phase immersion cooling. The electrical properties such as resistance and capacitance were recorded before and after thermal aging in air and hydrocarbon fluids using a multimeter. Scanning electron microscope (SEM) analysis was used to check the structural integrity and fluid ingress in the aged samples. Variation in Young's modulus was determined using dynamic mechanical analyzer (DMA) to quantify the change in mechanical properties of thermally aged samples of PCB and optical cables. It is an exercise in elucidating the reliability concerns of active and passive server components, and optical data cables due to the possible physical or chemical interaction with a dielectric fluid. This paper outlines a comprehensive methodology for testing these components and develops an understanding of the impact of single-phase immersion cooling using hydrocarbon fluids on the reliability of data center components.

Samples of PCB were prepared using an Isomet 1000 precision cutter. The Iso-Met 1000 is a precision segmenting saw intended for cutting different kinds of materials with negligible deformation. This accurate cutting machine provides seamless cutting for fragile parts by just utilizing gravity-fed force. Its extraordinary flexibility in chucking allows into consideration holding a wide range of test shapes and configurations, which makes it an excellent precision cutting machine fit for sectioning both fragile and ductile metals, composites, cement, laminates, plastics, electronic gadgets, and biomaterials. The cut samples prepared have dimensions of 50 mm × 10 mm as per the DMA requirements. The samples were cut in such a way that the copper distribution in all samples was identical.

The passive components were procured based on the surface mount components commonly used in the servers. The list of components used for this experiment includes thick film resistors, polymer capacitors, electrolytic capacitors, and transistors. Thick film resistors are widely used in the electrical and electronic industry and are readily available and cheap. The thick film resistor used in this experiment has a resistance of 1 kΩ and can be operated at a maximum temperature of 105 °C. An electrolytic capacitor is a form of a polarized capacitor where the anode is formed using pure aluminum and a nonsolid covering electrolyte acts as the cathode. A thin insulating layer of aluminum oxide acts as the dielectric of such capacitors. A paper sheet soaked in the electrolytic solution is placed between them and then the plates are wound around one another and placed into a can. The electrolytic capacitors used in this experiment have a capacitance of 100 μF and 470 μF, respectively, and can operate in a temperature range of −55 °C to 105 °C. The polymer capacitor is like the aluminum electrolytic capacitor in construction: it has an anode foil, a cathode foil, and different paper separators in between. Instead of a liquid electrolyte, the polymer capacitor has a solid polymer. The polymer capacitors used in this experiment have a capacitance of 270 μF and 560 μF, respectively, and can operate in a temperature range of −55 °C to 105 °C. A transistor is a miniature electronic component that can act both as an amplifier and as a switch. Depending on the material used for doping in transistors, we can classify the terminals as N-type and P-type. The Negative-Positive-Negative (NPN) and Positive-Negative-Positive (PNP) junction transistors are quite commonly used in electronic appliances. In this study, both NPN and PNP transistors are used and can operate reliably up to a temperature of 150 °C.

Active optical cables (AOCs), a relatively new choice for data transmission networks, are enjoying growing use in data centers. They are a variant of standard multimode fiber that is preferable for short-reach interconnections. The AOC tested in this work was a standard OM2 category Silica-based two-circular-core multimode fiber (MMF) cable. Three 60 cm long sections of the optical cable were cut from the original 3 m long cable as obtained (pristine cable). The samples were then placed in an environmental chamber for thermal aging in different fluid mediums as shown in Fig. 2. To understand the effect of single-phase immersion on the thermal aging of fiber coating, the ends of cut sections were kept open and immersed to enable penetration of hydrocarbon oils into the cables. This can be understood as a possible real-world scenario where there would be physical contact of oil with the cladding coating that protects the core of an optical cable immersed in a single-phase dielectric fluid. As will be explained in the Experimental Setup and Procedure section, the experiment was divided into four aging cycles. At the end of each cycle, 15 cm of wire sections were cut for mechanical testing from the aged 60 cm sections of the cables from each jar. After this, three specimens of length 5 cm each and an average outside diameter of about 2.9 mm, were carefully cut from the extracted 15 cm sections and tested in the DMA probe. The length of 5 cm for the final specimens was selected based on the maximum length that could fit in the DMA probe. This was done to obtain enough (three) data points for the measurement of properties after each cycle, thus increasing the confidence level of the obtained results.

Accelerated thermal aging conditions were produced by immersing all three types of samples in borosilicate jars containing mineral oil, synthetic fluid (EC-100), and exposed to air. The samples were kept at an elevated temperature and relative humidity (RH) setpoint in the environmental chamber for a total duration of 288 h, as shown in Fig. 2. The environmental chamber used for this experiment was a Thermotron SE-600-10-10 (Thermotron Industries, Holland, MI). The samples were prepared and baked per ASTM D3455 standard with some modifications done to suit the standard as per single-phase immersion of IT equipment with dielectric fluids as stated earlier. Initially, the environmental chamber underwent a dry run, immediately followed by the sample baking cycle as described in the standard but with reduced predry time. During the baking cycle, all the components placed inside the chamber had minimal contact with each other.

There were five jars used in this experiment: two jars filled with mineral oil having PCB and passive components in one jar and optical fiber cable in the other. Similarly, two jars were filled with EC-100 containing PCB and passive components in one jar and optical fiber cable in the other. The last jar was exposed to air and contained all three types of samples. A comparison of the physical properties of these fluids, water, and air are shown in Table 1. Optical cable samples were kept in a separate jar for the case of the hydrocarbon fluids to avoid the impact of the plasticizer leaching out from the poly vinyl chloride (PVC) cable on other samples. After the baking cycle, the jars were each filled with dielectric fluids up to the 800 ml mark with keeping one empty for air. All five jars were placed in the environmental chamber and the first 72-h cycle was initiated. Upon completion of the first 72 h, a set of samples from all five jars was taken out using tongs and stored in airtight aluminum packets after which the next cycle for the period of another 72 h was started. This was done for four cycles, thus totaling an aging duration of 288 h. This entailed a ramp time of approximately 1 h and a dwell time of between 70 and 71 h for each cycle. The choice of 45 °C temperature and 35% RH setpoints closely resembled the choice of aging conditions in literatures [1,2]. 35% RH corresponding to a 30 °C dew point in 45 °C dry bulb chamber temperature is low enough to remove moisture as a variable and is relatively close to realistic air-cooled data center operating conditions.

Once all the samples were extracted from the jars, they were put through the relevant mechanical and electrical tests. For PCB samples, Young's modulus was measured using DMA7100 for various temperature and frequency conditions. Young's modulus was chosen since it is an ideal indicator of changes in the mechanical properties of polymeric materials. The temperature range was kept between 30 and 50 °C at varying frequencies of 1, 2, 5, and 10 hertz, and a runtime of 30 min per sample. SEM analysis was then carried out on the cross section of the components to determine the structural integrity of the samples. For passive components, after the thermal cycling was completed, the components were taken out of the jars using a pair of tongs. They were then placed in an aluminum cover carefully to prevent any air contact. Changes in electrical properties such as capacitance, resistance, and barrier voltage were recorded using a multimeter. The material characteristics were recorded by performing SEM imaging of the cut cross section of the components. A summary of the entire procedure followed during the experiments is shown in Fig. 3.

For AOC, the temperature was ramped at a rate of 10 °C per minute ranging from 20 to 80 °C with data sampled for every 3 s interval. The choice of frequency was based on the reports from the literature on viscoelastic properties of data cables tested. A frequency scan required at least an additional frequency of 5 Hz would allow studying molecular weight and distribution trends of the molecular weight. A small sinusoidal force amplitude of 100 mN was applied by the machine on the sample with a strain amplitude of 20 μm (0.1% strain). Data generated by the software for elastic modulus facilitated the characterization of frequency and temperature-dependent viscoelastic properties of the polymeric materials in the optical cable and under sinusoidal excitations. SEM micrograph of the aged samples was analyzed to check the structural defects due to leaching out of plasticizers.

The brittleness of a component is known to increase with an increase in Young's Modulus. Figure 4 shows the comparison of variation in the Young's modulus of PCB samples for four thermal cycles. Young's modulus values for an air-exposed sample show a consistent trend. The values for mineral and synthetic oils-immersed samples closely follow the same trend as air-exposed samples but show a little increase after the fourth cycle. This is attributed to the fact that prolonged immersion at high temperatures may have little effect on the mechanical and viscoelastic properties of the PCB.

Figure 5 shows the images obtained from SEM analysis for mineral oil and synthetic fluid immersed PCB samples. This analysis was performed on the samples after the fourth cycle (288 h) of thermal aging. As seen in the images, there was no sign of any crack or oil penetration in the PCB samples. Also, there was no deformation observed and it can be sufficiently concluded that the structural integrity of the samples is unaffected post-thermal cycling for both the oils.

The electrical properties of the passive components from each of the fluid medium are measured after every thermal cycle of 72 h. Using the trend data of the electrical properties measured, a comparative variation in the electrical properties of the samples exposed to air, immersed in mineral oil or synthetic fluid is obtained. The change in resistance of the thick film resistor tested in three different mediums is represented below in Fig. 6.

Figure 6 shows the plot of the variation of the resistance of thick film resistors versus the number of thermal aging cycles. It is evident from the plots that the resistance is maintained constant at the reference value (measured before thermal cycling) for all three mediums during the thermal cycling. It can, thus, be concluded that the thick film resistors are as reliable to use in immersion cooling as they are in air cooling. The authors took two different electrolytic capacitors for the experiment and the variation in the capacitance after thermal cycling is illustrated in Figs. 7(a) and 7(b). The value of capacitance reduces slightly for mineral oil but is almost constant for the synthetic fluid for three cycles in one case. For the second capacitor, the value of capacitance also does not vary much past the reference value for two cycles and then shows a slight dip for both mineral and synthetic fluid. The allowable tolerance of the electrolytic capacitor is 20% of its original or reference value. As seen the variation in the capacitance is well within the allowable range and hence it is safe and reliable to use in immersion cooling.

For polymer capacitors, the allowable tolerance in capacitance of the polymer capacitor is 20% of its original or reference value. As shown in Figs. 8(a) and 8(b), the value of capacitance varies rather erratically in the air but is consistent for both mineral oil and synthetic fluid for one capacitor. The variation in the capacitance is well within the allowable range and hence it is safe and reliable to use in immersion cooling. It can also be seen that the variation in the capacitance for either type of the capacitors tested is approximately 3% from the base value. Thus, from an operational point of view, both polymer and electrolytic capacitors show good compatibility. Further studies on the impact of prolonged immersion on coolant chemistry due to these types of capacitors should be conducted to assess other reliability issues.

Tables 2 and 3 show the comparison of voltage across the junctions of PNP and NPN transistors for all three fluid mediums, where “OL” indicates an “over limit” case. As seen in the tables, for both the PNP and NPN transistors, the values of the voltage across the terminals for all the thermal cycles are within the range specified which was 0.45–0.9 volts. Hence, the transistors can safely operate in oil even at elevated temperatures without any effect on their reliability.

From the SEM images shown in Fig. 9, it can be inferred that there were no observed changes in microscopic structure in any of the passive components. There is no formation of cracks or penetration of fluid on the surface of the components. The rubber sealing used in the Electrolytic capacitor is synthetic rubber EPDMS. These results show that there is no effect of dielectric fluids on the material properties of rubber sealing.

For active optic cables, the DMA test results of three specimens after each of the four thermal cycles were recorded and studied. The obtained values of each of the three samples after every cycle were averaged and plotted against increasing sample aging time. For convenience and consistency, the results reported are the ones observed for the 1 Hz frequency. The results of the DMA tests at the DMA furnace temperatures of 40 °C and 22 °C have been presented in Figs. 10 and 11, respectively. Changes in the viscoelastic properties indicate changes in the macromolecular polymer chain structure of either the jacket or the cladding coating or both.

To better understand the underlying mechanism of the elastic modulus changes in the thermoplastic PVC jacket due to aging by immersion in high temperature, it was deemed that Young's modulus (E) versus time results may prove enough. The rate of change of elastic modulus, which is a strong indicator of thermomechanical behavior, undergoes a reduction by the third cycle. Logarithmic trends fitted to the data in the graphs bolster the hypothesis that the effect of aging is self-limiting and slows down after the first few cycles. PVC jacket hardness is increasing due to plasticizers leaching out and/or ingress through the diffusion of oil. Coating toughness rose in magnitude and may contribute to an increase in the fiber mechanical strength. Stiffness and Young's modulus of cable increased with aging time, as hypothesized, and this agreed with recent and extant literature [16,17]. Reduction in elastic modulus change over aging time is proposed to be due to two competing phenomena:

1. Increased initial diffusion of oil molecules through PVC jacket, and loss of plasticizers due to elevated aging temperature increasing diffusion coefficient, and aggregation of the PVC molecules increasing microvoids. The loss of plasticizer is causing PVC molecules to aggregate and allow for more voids for the oil ingress. High temperatures damage acrylate coatings, which might deteriorate its adhesion to the fiber, causing delamination and exposing the bare fiber to undergo cracking/fractures.

2. Increased thermal degradation of fiber coating entails stiffening, reduced ductility, and increased net strength resulting in a reduction in the diffusion of oil molecules, as evident from tanδ peak shifts.

The upward trend in tanδ peaks, as seen in Fig. 12 is indicative of elevation in glass transition temperature (T_g). Since T_g is strongly dependent on the content of the plasticizer and general polymer chemical state, it is reasonable to conclude that some plasticizers leached out into the oil because of aging.

An inspection of the SEM micrographs shown in Fig. 13 reveals small inorganic filler molecules, plasticizers, and microvoids on the PVC surface. When we compare micrographs of the mineral-oil immersed sample surface morphology with that of the air-exposed, increased microvoids and aggregation of PVC molecules are readily visible, which are potentially causing the coating to toughen. Nevertheless, the coating becoming stiffer is rather a positive outcome as it minimizes transmission losses due to microbending. However, this needs corroboration by optical loss testing.

This study provides a cogent methodology to design experiments for determining the material compatibility and evaluating the material compatibility of common PCBs, passive surface mount components, and active optic cables in single-phase immersion cooling. Evaluation of the material compatibility from this study will provide essential literature and reference data for data center administrators to successfully implement this cooling technology.

Many uncertainties and concerns persist regarding the effects of mineral oil immersion cooling on the reliability of IT equipment both at the component and chassis levels [18]. The issues associated with the mechanical reliability and operational service impact of adopting a single-phase immersion cooling strategy must be addressed before the technology will see widespread adoption. The above-mentioned issues were successfully investigated by utilizing a modified benchmark ASTM standard for the accelerated thermal aging test. The evaluation of mechanical and electrical properties of the samples was conducted post-thermal cycling exposed to air and two hydrocarbon oils used. Images from SEM analysis prove that there is no alteration in the structural integrity or diffusion of oils at a microscopic level in PCBs and passive components. The increase in the Young's modulus and stiffness of AOCs can be attributed to the plasticizer leaching and molecule aggregation in the cables. The electrical integrity tests sufficiently concluded that the variation in the electrical characteristics of the passive components was well within allowable ranges.

It can be reiterated that the field of reliability for single-phase immersion cooling has a lot of scope for future work. Comparable tests need to be repeated at higher aging temperatures and different humidity levels, to determine a fluid-sample interaction-dependent phenomenological exponential Arrhenius model, based on the chemical reaction kinetics of the interaction. The numerical value of the aging factor should be determined for long-term reliability evaluation. Since the present research was performed at a single aging temperature of 45 °C, there are not enough data points to develop a first-order Arrhenius-type model to the temperature dependence of cable performance. Multiphysics simulations can be done using commercially available finite element analysis (FEA) codes to validate the theoretical model. In the future, research should also be carried out to understand the changes in the properties of the fluid due to plasticizer contamination and its impact on thermal performance. Nonetheless, owing to the plateauing of elastic modulus values after longer aging durations as observed for AOC and no significant changes in mechanical and electrical properties of PCB and passive component samples, proves that single-phase immersion cooling is a reliable choice for electronics cooling. It also has huge upsides to address future packaging technologies that utilize heterogeneous integration whether it is in 2.5D or 3D form factor since the fluid can freely be in touch with all components as opposed to cold plate cooling, which requires approaches such as cold rails to cool the entire system. It is also important to extend the study to include widely used and reliable Fluorinert^TM and Novec^TM fluids.

This work is supported by NSF IUCRC Award No. IIP-1738811. The authors would like to thank Dr. David Sundin, Chief Scientist at Engineered Fluids, LLC, for providing the synthetic hydrocarbon oil EC-100 for this research.

• National Science Foundation (NSF) IUCRC (Award No. IIP-1738811; Funder ID: 10.13039/100000001).

AOC =

active optical cables

ASTM =

American Society for Testing and Materials

ATC =

accelerated thermal cycling

DMA =

dynamic mechanical analyzer

E =

Young's modulus

EPDMS =

ethylene propylene diene monomer

FEA =

finite element analysis

GPa =

giga pascal

JEDEC =

Joint Electron Device Engineering Council

MMF =

multimode fiber

PCB =

printed circuit board

PVC =

poly vinyl chloride

RH =

relative humidity

SEM =

scanning electron microscope

SpLIC =

single-phase liquid immersion cooling

μF =

microFarad