Article
`Evaluation and Optimization of a Two-Phase Liquid-Immersion
`Cooling System for Data Centers
`
`Cheng Liu 1,2 and Hang Yu 1,*
`
`1
`
`School of Mechanical and Energy Engineering, Tongji University, Shanghai 201804, China;
`cliuglobal@163.com
`2 China Mobile Group Shanghai Co., Ltd., Shanghai 200060, China
`* Correspondence: yuhang@tongji.edu.cn
`
`Abstract: An efficient cooling system for data centers can boost the working efficiency of servers and
`promote energy savings. In this study, a laboratory experiment and computational fluid dynamics
`(CFD) simulation were performed to explore the performance of a two-phase cooling system. The
`coefficient of performance (COP) and partial power usage effectiveness (pPUE) of the proposed
`system was evaluated under various IT (Information Technology) loads. The relationship between
`the interval of the two submerged servers and their surface temperatures was evaluated by CFD
`analysis, and the minimum intervals that could maintain the temperature of the server surfaces
`below 85 ◦C were obtained. Experimental results show that as server power increases, COP increases
`pPUE decreases. In one experiment, the COP increased from 19.0 to 26.7, whereas pPUE decreased
`from 1.053 to 1.037. The exergy efficiency of this system ranges from 12.65% to 18.96%, and the tank
`side accounts for most of the exergy destruction. The minimum intervals between servers are 15 mm
`under 1000 W of power, 20 mm under 1500 W, and more than 30 mm under 2000 W and above. The
`observations and conclusions in this study can be valuable references for the study of cooling systems
`in data centers.
`
`Keywords: two-phase cooling; data center; CFD; immersion; optimization
`
`1. Introduction
`The boosting of digital technology (e.g., Internet of Things, artificial intelligence, big
`data, 5G, cloud computing) and its extensive applications in many industries, such as
`transportation [1], communication [2], manufacturing [3], medicine [4], and education [5],
`demonstrate an increasing need for data processing, storage, and transmission. A data
`center can be a building or part of a building where data are gathered, processed, and
`stored [6]. According to a report by the Synergy Research Group, by the end of the third
`quarter of 2019, there were 504 hyperscale data centers worldwide [7], and another report
`predicted that this number would increase by 12–14% annually over the next five years [8].
`Of all the data centers in 2019, approximately 40% reside in the US, and China, Japan, the
`UK, Germany, and Australia collectively contain 32%. Data centers typically involve high
`energy consumption. In the UK, data centers accounted for 1.5% of electricity usage in
`2016, with power consumption projected to increase by 20% in 2020 [9]. In China, data
`centers used 160.8 billion kWh of electricity in 2018, which exceeded the total electricity
`consumption of all of Shanghai [10].
`In addition to power equipment and accessory components, data centers consist of two
`major energy-consuming parts, IT and heating, ventilation, and air conditioning (HVAC)
`equipment, which account for approximately 90% of the total energy usage [11]. The
`HVAC equipment itself has been reported to be responsible for 34% of this total energy [12],
`because of the need to cool the all-day operation of high-power-density IT equipment and
`maintain the indoor thermal environment within the appropriate temperature zones [13].
`The HVAC system needs to be in operation for approximately 24 h per day. To improve
`
`Citation: Liu, C.; Yu, H. Evaluation
`and Optimization of a Two-Phase
`Liquid-Immersion Cooling System
`for Data Centers. Energies 2021, 14,
`1395. https://doi.org/10.3390/
`en14051395
`
`Academic Editor:
`Guglielmo Lomonaco
`
`Received: 11 February 2021
`Accepted: 24 February 2021
`Published: 3 March 2021
`
`Publisher’s Note: MDPI stays neutral
`with regard to jurisdictional claims in
`published maps and institutional affil-
`iations.
`
`Copyright: © 2021 by the authors.
`Licensee MDPI, Basel, Switzerland.
`This article is an open access article
`distributed under
`the terms and
`conditions of the Creative Commons
`Attribution (CC BY) license (https://
`creativecommons.org/licenses/by/
`4.0/).
`
`Energies 2021, 14, 1395. https://doi.org/10.3390/en14051395
`
`https://www.mdpi.com/journal/energies
`
`energies
`
`# & !*-
`0+/ .
`
`Immersion Systems LLC – Ex. 1015
`PGR 2021-00104 (U.S. 10,820,446 B2)
`1 of 21
`
`

`

`Energies 2021, 14, 1395
`
`2 of 21
`
`the energy efficiency of a data center, optimizing the HVAC system is one of the key
`steps. Power usage effectiveness (PUE) is an industry-preferred index for evaluating the
`infrastructure energy efficiency of data centers [14]. A small PUE approaching 1.0 indicates
`that the data center approaches optimal energy efficiency.
`Data center cooling systems can be classified into two types—air-cooled systems [15,16]
`and liquid-cooled systems [17]. The first type sends cold air to racks on which servers
`are placed and then removes the heat generated by the servers. The advantages of this
`type of system include the following: (1) this traditional air-cooling system is relatively
`mature and easy to construct and (2) this system has promising potential for the use of
`free cooling resources, such as natural cold air and water [18], which could substantially
`benefit the reduction of power usage, decreasing the PUE from a value of 2.01 for a typical
`computer room air conditioner system to a value as low as 1.1 with free cooling [19,20]. The
`drawbacks of the air-cooled system are significant, such as a low heat transfer coefficient,
`difficulty in the design and control of cold and hot air flow, asymmetrical cooling in dif-
`ferent servers owing to the geometrical layouts of data centers, large power consumption
`through the operation of chillers, fans, and pumps, and unstable cooling capacity caused
`by the outdoor thermal environments [12,17,21,22].
`The liquid-cooled system requires the removal of high-density heat, which is too
`high to be removed by the air-cooled system; This heat is dissipated by high-performance
`information and communication technology (ICT) devices. Based on the direct contact of
`the liquid with the heat sources, the liquid-cooled system can be further categorized as
`direct cooling and indirect cooling. For direct liquid cooling, the dielectric liquid absorbs
`heat from the electronic components directly; this type of heat transfer can be highly
`effective [17,23]. Passive two-phase cooling is a fluid-cooled approach in which electronic
`components are submerged in a phase-changeable liquid bath in a closed box. When the
`surface temperature of the electronic components exceeds the evaporation temperature of
`the liquid, the process of boiling heat transfer is triggered, removing the extra heat in the
`form of vapor bubbles. The produced vapor rises up and further condenses through a water-
`based condenser installed above the bath. Finally, the heat is absorbed and removed by the
`coolant; the vapor is then liquefied and drips back to the bath, driven by gravity [24,25].
`Compared with the air-cooled system, the two-phase cooling system could reduce the use
`of accessory equipment, such as chiller pumps and fans, potentially improving the energy
`efficiency of data centers. In addition, this system could avoid the surface temperature
`symmetry of electronic devices and guarantee their working environment.
`Dashtebayaz and Namanlo [26] studied an air-based cooling system that performs
`waste heat recovery and reported a coefficient of performance (COP) that varied in the
`3–5 range with a PUE of approximately 2.5. Chen et al. [27] applied spray-cooling tech-
`nology to the cooling of computer centers and suggested that the system COP is highly
`dependent on the inlet water temperature, with COP possibly changing from 3 to 15 and
`PUE ranging from 1.45–1.52. Cho et al. [28] proposed many green technologies that can be
`applied to data center cooling systems under different climatic contexts and determined by
`applying suitable strategies. The target PUE was in the range of 1.2–1.6. Dong et al. [29]
`explored the effectiveness of using free-cooling resources to cool data centers and sug-
`gested that natural cooling could increase the system COP by 23.7%, rising from 5.9 to 7.3.
`Lu et al. [30] reported the PUE of data centers in Finland, highlighting its range of variation
`of 1.2–1.5.
`In a two-phase cooling system, the ICT equipment is completely submerged in the
`dielectric liquid bath; nucleate boiling occurs on the server surface when the server tem-
`perature reaches the boiling temperature of the dielectric liquid. Wu et al. [31] evaluated
`a full-scale two-phase liquid-immersion DC cooling system in a tropical environment.
`They found that the exergetic efficiency experienced a small rate of decrease, while the
`supplied power rate increased. The highest efficiency of 69.9% was obtained at zero load,
`whereas the lowest efficiency of 65.9% was observed at full load. Kanbur et al. [32] stud-
`ied a two-phase liquid-immersion data center cooling system through experimental and
`
`Immersion Systems LLC – Ex. 1015
`PGR 2021-00104 (U.S. 10,820,446 B2)
`2 of 21
`
`

`

`Energies 2021, 14, 1395
`
`3 of 21
`
`thermo-economic analyses. They found that the optimal COP and PUE values occurred at
`maximum operation loads of 6.67 and 1.15, respectively, whereas the minimum COP and
`highest PUE were observed at the minimum operation loads of 2.5 and 1.4, respectively.
`Choi et al. [33] used two-phase cooling (HFE-7100) for a polymer electrolyte membrane
`fuel cell. They found that the two-phase boiling heat transfer coefficients of the HFE-7100
`in mini-channels were strongly dependent on heat flux and vapor quality but less sensitive
`to mass flux. However, at present, there are fewer research reports on the mining machine
`motherboard of the two-phase liquid-immersion cooling system.
`Computational fluid dynamics (CFD) simulations have been widely applied to vali-
`date the performance of specific cooling systems for data centers and to present optimal
`designs based on the detailed results of the simulations. Ahmadi et al. [34] verified the
`energy-saving potential, optimal designs, and operation conditions for the computer
`room air handling bypass method in cooling data centers by using CFD simulations.
`Hassan et al. [35] performed CFD simulations in ANSYS Fluent and obtained the temper-
`ature, airflow, and pressure distributions for a data center. The prediction provided a
`three-dimensional (3D) thermal map of the data center and helped to optimize the cooling
`system. Fulpagare et al. [36] explored transient CFD models to simulate the dynamic
`requirements of cooling systems to achieve smart control in data centers, and the model
`performance was validated based on experiments. Using CFD simulations, Nada et al. [37]
`studied the performance of data centers under different configurations and summarized
`the effects of the computer room air conditioning unit layout on the thermal performance
`of the racks. Most previous CFD studies have focused on air-cooled systems; however,
`studies that focused on two-phase cooling systems are scant. Cheng et al. [38] studied a
`single-phase immersion cooling system (using 3 M Novec 7100) for single CPU cooling via
`3D numerical analysis using ANSYS Fluent. The results of the simulations showed that
`there was an unbalanced heat distribution around the CPU, and higher flowing speeds of
`the liquid coolant led to the removal of more heat, resulting in lower CPU temperatures
`and more balanced heat distribution around the CPU. Ali et al. [39] numerically inves-
`tigated the thermal performance and stress analysis of enhanced copper spreaders for
`nucleate boiling immersion cooling of high-power electronic chips. An et al. [40] developed
`a 3D numerical model using ANSYS Fluent to study two-phase immersion cooling for
`electronic components. They found that Novec 7000 can support cooling a 5 cm × 5 cm
`heat source in a vertical orientation with power as high as 225 W (heat flux of 9 W/cm2).
`However, less research was reported using CFD simulation to study the arrangement of
`the server motherboards.
`This study consisted of experiments and a series of CFD simulations. First, an inno-
`vative cooling structure and procedure for a two-phase immersion cooling system were
`developed. Six different cases were investigated to analyze the thermal management
`performance in the operational load range of 1127–1577 W. The energy efficiency of the
`system was evaluated based on the partial power usage effectiveness (pPUE) and COP
`indices, and exergy analysis was performed. Finally, an arrangement of submerged servers
`was proposed by CFD simulation. This study aimed to formulate an innovative two-phase
`immersion cooling system for data centers, which includes an optimal arrangement of the
`servers. Data from the two-phase cooling system were collected; the results and conclusions
`presented herein may potentially serve as valuable references for researchers and engineers.
`
`Immersion Systems LLC – Ex. 1015
`PGR 2021-00104 (U.S. 10,820,446 B2)
`3 of 21
`
`

`

`Energies 2021, 14, 1395
`
`4 of 21
`
`2. Methodology
`2.1. Experimental Study
`A novel, innovative cooling structure and procedure of a two-phase immersion cooling
`system were established for the cooling of ICT devices. The scheme of the proposed two-
`phase cooling system is shown in Figure 1, which includes a primary heat exchanger and
`a tank with coolant and server boards as ICT devices. The ambient air temperature was
`maintained at 21–23 ◦C.
`
`Figure 1. Scheme of the proposed cooling system.
`
`Figure 2a shows the tank where real servers were submerged in the dielectric liquid
`Novec 7100 supplied by 3M (Minnesota Mining and Manufacturing, Saint Paul, MN,
`USA) [41]. The dielectric liquid, being expensive, was filled in the tank only to submerge
`all the servers (Figure 2c) during the experiments. For each experiment, the server power
`output was maintained at a constant level. As the surface temperature of the servers rose
`and reached the boiling point of Novec 7100, the surrounding liquid formed bubbles and
`absorbed heat through evaporation. The produced vapor gathered on the top of the tank,
`where condenser coils were placed around the wall, as shown in Figure 2c. The heat was
`transferred from the vapor to the cold coolant circulating inside the condenser coils and
`was removed from the tank. Thereafter, the vapor condensed into the fluid phase and
`dripped back onto the liquid bath because of gravity. The heated coolant inside the coil
`released heat to the environment by a primary heat exchanger, as shown in Figure 2d.
`Table 1 lists the thermal characteristics of Novec 7100 at 25 ◦C, and the specific data are
`listed in Table A1 in Appendix A. The dimensions of the tank were 650 × 450 × 1050 mm3
`(length × width × height), and the tank was insulated to prevent the influence of the sur-
`rounding thermal environment. The servers used in this study were application-specific inte-
`grated circuit (ASIC) miners designed to “mine” a specific cryptocurrency. The type of server
`board was T2T-25T, and the dimensions of each board were 235 × 182 × 8 mm3. There were
`140 T2T CPUs on each board, and the dimensions of the CPU were 8 × 8 × 1 mm3, with an
`average CPU TDP of 8.37 W/cm2. The server boards were placed at 100 mm intervals. This
`type of server has several modes with rated power outputs ranging from 1000 to 2000 W.
`The mode of each server can be controlled remotely through a computer program.
`
`
`
`Immersion Systems LLC – Ex. 1015
`PGR 2021-00104 (U.S. 10,820,446 B2)
`4 of 21
`
`

`

`Energies 2021, 14, 1395
`
`5 of 21
`
`Figure 2. Two-phase cooling tank: (a) the tank, (b) servers, (c) condenser coil, and (d) primary heat exchanger.
`
`Table 1. Properties of Novec 7100 at 25 ◦C.
`
`Boiling Point
`(◦C)
`
`Vapor Pressure
`(kPa)
`
`Molecular Weight
`(g/mol)
`
`Density Liquid
`(kg/m3)
`
`61
`
`27
`
`250
`
`1510
`
`Dynamic
`Viscosity
`(cSt)
`0.38
`
`Specific Heat
`(J/kgK)
`
`1183
`
`During the experiment, all the data were measured when the system was under a
`stable status, characterized by a lack of fluctuations in the CPU surface temperature. Under
`each condition, the system was kept running for 60 min before the data were recorded.
`The temperatures of the server surface, liquid bath, and inlet and outlet coolant inside
`the coils were recorded using a digital temperature controller (E5CC, OMRON, Kyoto,
`Japan). The temperature was measured using K-type thermocouples (TT-K-24, OMEGA,
`Norwalk, CT, USA), which were calibrated using a mercury thermometer with an accuracy
`of ±0.1 ◦C. The core temperature of the servers was measured and recorded automatically
`by the aforementioned computer program. The pressure in the tank was monitored by
`an automatic pressure relief valve with a pressure sensor (ZSE40AF-01-T, SMC, Kyoto,
`Japan), with the pressure set to 1.2 kg/cm2. The heat from the coolant was removed by a
`heat exchanger (ERM-3K3UC Liquid Cooling System, Koolance, Auburn, AL, USA), and
`the cooling capacity of the cooling system was 2600 W (8872 BTU/h). The power of the
`cooling unit (including the pump and the fan) was monitored using a power meter that
`can monitor and record real-time power consumption. The experiment consisted of six
`conditions, and the stable data are listed in Table 2.
`
`
`Figure 2. Two-phase cooling tank: (a) the tank, (b) servers, (c) condenser coil, and (d) primary heat exchanger.
`
`Immersion Systems LLC – Ex. 1015
`PGR 2021-00104 (U.S. 10,820,446 B2)
`5 of 21
`
`

`

`Energies 2021, 14, 1395
`
`6 of 21
`
`Table 2. Operating conditions and experimental results.
`
`Case
`
`Room
`Temperature (◦C)
`
`CPU Power(W)
`
`1
`2
`3
`4
`5
`6
`
`21.9
`21.6
`22.4
`23.0
`23.3
`23.4
`
`1127
`1396
`1332
`1494
`1516
`1577
`
`Heat Exchanger
`Coolant Temperature (◦C)
`Inlet
`Outlet
`26.5
`25.1
`27.1
`25.8
`27.9
`26.5
`29.2
`27.6
`29.6
`28.1
`30.1
`28.4
`
`Temperature of CPU (◦C)
`
`Surface
`66.0
`68.8
`68.5
`70.5
`71.7
`71.9
`
`Core
`67.4
`71.2
`70.7
`72.3
`72.9
`73.3
`
`2.2. CFD Simulation
`2.2.1. Physical Model and Boundary Conditions
`In this study, a commercial CFD tool, ANSYS Fluent, was used to complete the
`simulation [42]. CFD simulations were intended to explore the effects of the interval
`of adjacent servers on the performance of a two-phase cooling system and propose an
`appropriate server arrangement. The assumption is that for a given power of servers and a
`specific cooling setting, there is the shortest interval between submerged servers for which
`each server should run in an ideal environment with core temperatures lower than the
`requirements. The purpose of searching for the shortest interval is that the submerged space
`in a two-phase cooling tank is limited; thus, a well-arranged matrix of servers facilitates
`efficient usage of space, thereby providing savings in investment and running costs.
`Building physical models is the first step in running a CFD simulation. The dimensions
`of the simulated two-phase cooling tank are the same as those used in the experimental
`study, which is 650 × 450 × 1050 mm3 (L × W × H), as demonstrated in Figure 3. Figure 3b
`shows the mesh distribution near the server board, which was locally encrypted. The walls
`of the tank were set as adiabatic, and three server boards were placed under a 50 mm
`liquid bath, with the dimensions of each board at 235 × 182 × 8 mm3 (L × W × H).
`There are 140 CPUs on each board, and each CPU has dimensions of 8 × 8 × 1 mm3.
`During the simulation, the boundary conditions of the servers were set as constant heat
`flow, which was consistent with the server power output. There are some assumptions
`and simplifications in this model. The server boards, including printed circuit boards
`and CPUs (covered with phenolic epoxy resin), were treated as an entire heating plate
`covered with phenolic epoxy resin (thermal conductivity of 2.2 W/m K). The coolant-side
`cooling equipment and heat transfer process were removed from the simulation, and the
`top of the tank was considered as a pressure outlet. The assumption was that all the heat
`from the vapor could be fully released to the environment by the heat exchanger. These
`simplifications were applied because this simulation focused on the heat transfer between
`the submerged servers and the liquid but not on the entire system. The grid chosen in this
`study was “Tet/Hybrid,” which consisted of tetrahedrons and other hybrid elements. One
`of the limitations of the model is that the detail of the CPU was ignored.
`
`Immersion Systems LLC – Ex. 1015
`PGR 2021-00104 (U.S. 10,820,446 B2)
`6 of 21
`
`

`

`Energies 2021, 14, 1395
`
`7 of 21
`
`Figure 3. Models for the tank and servers. (a) Mesh of the model (b) Grid section.
`
`2.2.2. CFD Modeling of Boiling Flow
`In this study, a volume of the fluid model was used to simulate the boiling heat
`transfer (two-phase vapor–liquid flow regimes) in the tank. In this multiphase flow model,
`an Eulerian–Eulerian approach is applied, where both fluids are assumed to behave as
`continuous media [43]. The mass and energy transfer between both phases during the
`evaporation process, as shown in Equations (1)–(3) (Reynolds-averaged Navier–Stokes
`(RANS) equations), and further details on these equations were provided by Versteeg g [44].
`
`(ρui) = SM
`
`(1)
`
`3∑
`
`j=1
`
`∂
`∂xi
`
`(ρ) +
`
`∂ ∂
`
`t
`
`∂ui
`∂xj
`
`+
`
`∂uj
`∂xi
`
`− 2
`3 δij
`
`·(cid:0)ρuiuj
`
`(cid:1) = − ∂p
`
`∂xi
`
`+
`
`3∑
`
`j=1
`
`∂
`∂xj
`
`(cid:34)
`
`(cid:32)
`
`µ
`
`3∑
`
`j=1
`
`∂
`∂xj
`
`(ρui) +
`
`∂ ∂
`
`t
`
`(cid:33)(cid:35)
`
`+
`
`j=1
`
`3∑
`(cid:17)
`
`∂
`∂xj
`
`(cid:33)
`
`(cid:16)−ρu(cid:48)
`
`iu(cid:48)
`
`j
`
`(cid:17)
`
`+ SF,i
`
`(2)
`
`− 3∑
`
`j=1
`
`∂
`∂xj
`
`qj + SE
`
`(3)
`
`τij − ρu(cid:48)
`iu(cid:48)
`
`j
`
`ui
`
`(cid:32)
`
`∂ul
`∂xl
`
`l=1
`
`3∑
`(cid:16)
`
`∂
`∂xj
`
`(cid:0)ρEuj
`
`(cid:1) =
`
`3∑
`
`3∑
`
`i=1
`
`j=1
`
`3∑
`
`j=1
`
`∂
`∂xj
`
`(ρE) +
`
`∂ ∂
`
`t
`
`where SM is the source term in the mass conservation equation (kg/m3 s), SF is the source
`term in the momentum conservation equation (kg/m2 s2), and SE is the source term in the
`energy equation (J/m3 s).
`The RANS equations describe the transport of the averaged flow quantities in which
`the entire range of turbulence scales is modeled. Because of the averaging, these equations
`contain additional unknown variables called Reynolds stresses. By applying the Boussinesq
`hypothesis [44], the Reynolds stresses are related to the mean velocity gradients. The
`expression for the calculation of these Reynolds stresses corresponds to Equation (4).
`
`(cid:32)
`
`− ρu(cid:48)
`iu(cid:48)j = µturb
`
`
`
`∂ui
`∂xj
`
`+
`
`∂uj
`∂xi
`
`(cid:33)
`
`(cid:32)
`
`− 2
`3
`
`ρk + µturb
`
`3∑
`
`l=1
`
`∂ul
`∂xl
`
`(cid:33)
`
`δij
`
`(4)
`
`To solve this equation, an expression for the turbulent viscosity (µturb) is required.
`The calculation of turbulent viscosity using the standard k-ε model is computationally
`expensive. In the first step, two additional transport equations for the turbulent kinetic
`energy k and the viscous dissipation of the turbulent kinetic energy ε are solved, which
`
`
`
`Immersion Systems LLC – Ex. 1015
`PGR 2021-00104 (U.S. 10,820,446 B2)
`7 of 21
`
`

`

`Energies 2021, 14, 1395
`
`8 of 21
`
`correspond with Equations (5) and (6). Finally, the turbulent viscosity is calculated as a
`function of k and ε (Equation (8)).
`
`(cid:34)(cid:18)
`
`µ +
`
`µturb
`σk
`
`∂xj
`
`(cid:35)
`
`+ Pk − ρε
`
`3∑
`
`j=1
`
`∂
`∂xj
`
`(cid:34)(cid:18)
`
`εk
`
`+ C1ε
`
`∂xj
`
`µ +
`
`µturb
`σε
`
`∂
`∂xj
`
`j=1
`
`l=1
`k2
`ε
`Many expressions describing the mass transfer during evaporation are based on
`gas kinetic theory. A well-known equation for the net mass flux over the vapor–liquid
`interphase during the evaporation process is the following Hertz–Knudsen equation [45]:
`√
`M√
`2πR
`
`ε2
`k
`
`(5)
`
`(6)
`
`(7)
`
`(8)
`
`(9)
`
`(cid:19) ∂k
`(cid:35)
`(cid:19) ∂ε
`(cid:33)
`
`Pk − C2ερ
`(cid:35)
`(cid:33)
`
`∂uj
`∂xi
`
`− 2
`3 ρkδij
`
`(cid:0)ρkuj
`(cid:0)ρεuj
`(cid:1) =
`3∑
`(cid:34)(cid:32)
`(cid:32)
`
`(cid:1) =
`
`3∑
`
`j=1
`
`∂
`∂xj
`
`(ρk) +
`
`∂ ∂
`
`t
`
`3∑
`
`j=1
`
`3∑
`
`∂
`∂xj
`3∑
`
`i=1
`
`j=1
`
`(ρε) +
`
`∂ ∂
`
`t
`
`Pk =
`
`∂ul
`∂xl
`
`2 3
`
`µturb
`
`∂ui
`∂xj
`
`+
`
`∂uj
`∂xi
`
`− δij
`
`3∑
`
`µturb = ρCµ
`
`(cid:18) p√
`
`Tv
`
`− psat(Tl)√
`T1
`
`(cid:19)
`
`J(cid:48) = αc
`
`where J is the net mass flux over the vapor–liquid interface (kg/m2 s), αc is the accommo-
`dation coefficient, M is the molecular weight (kg/kmol), Tv is the temperature of the vapor
`phase (K), Tl is the temperature of the liquid phase (K), and psat is the saturation pressure
`(Pa). In the case of condensation, αc is defined as the ratio of the experimentally observed
`condensation velocity to the maximal theoretical condensation velocity. Meanwhile, in the
`case of evaporation, this parameter is defined as the ratio of the experimentally observed
`evaporation velocity to the maximal theoretical evaporation velocity [43]. The specific
`calculation procedure for the mass and energy transfer during the evaporation process was
`based on the study by Schepper et al. [43].
`The semi-implicit method for pressure-linked equations (SIMPLE) algorithm was
`used for pressure–velocity coupling. A body force-weighted scheme was used to discretize
`the pressure terms. The momentum, energy transport, and turbulence equations were
`solved using the quadratic upstream interpolation for the convective kinematics (QUICK)
`scheme. The convergence criteria of the whole solution were defined with respect to the
`residuals for mass, momentum, turbulence, and energy—-10−3 for the mass, momentum,
`and turbulence residuals and 10−6 for energy residual. Table 3 shows a summary of the
`applied CFD model settings.
`
`2.2.3. Material Properties
`The thermodynamic properties of the liquid and vapor were obtained from the Novec
`7100 manual [41]. Each thermodynamic parameter was approximated by temperature-
`dependent polynomial functions. Figures 4 and 5 provide two examples that show the
`correlations between the temperature and the liquid dynamic viscosity/vapor conductive
`coefficient. Table A2 in Appendix A lists all the temperature-dependent properties of the
`thermodynamic parameters used in Fluent.
`
`Immersion Systems LLC – Ex. 1015
`PGR 2021-00104 (U.S. 10,820,446 B2)
`8 of 21
`
`

`

`Energies 2021, 14, 1395
`
`9 of 21
`
`Table 3. Settings of the computational fluid dynamics (CFD) Model.
`
`Setting Parameters
`Solver type
`Turbulence model
`Near-wall treatment
`Pressure-velocity coupling scheme
`
`Type
`
`Spatial discretization
`
`Residuals
`
`Gradient
`Pressure
`Momentum
`Turbulent kinetic energy (k)
`Turbulent dissipation rate
`(ε)
`Energy
`Continuity
`X, Y, Z-Velocity
`Energy
`k, ε
`
`Settings/Options
`Pressure-based
`k–ε realizable model
`Standard wall functions
`SIMPLE
`Least-squares cell-based
`Body force-weighted
`QUICK
`QUICK
`
`QUICK
`
`QUICK
`0.001
`0.001
`10−6
`0.001
`
`Figure 4. Correlation between liquid dynamic viscosity and temperature.
`
`Figure 5. Correlation between vapor conductive coefficient and temperature.
`
`
`
`
`
`Immersion Systems LLC – Ex. 1015
`PGR 2021-00104 (U.S. 10,820,446 B2)
`9 of 21
`
`

`

`Energies 2021, 14, 1395
`
`10 of 21
`
`2.2.4. Grid Independence
`The analysis of the grid independence was conducted by running simulations with
`different numbers of grid cells, as shown in Table 4. The time step (∆t) was 0.01 s, and
`the maximum number of iterations was 20. The total number of time steps was 5000
`(total physical time 50 s), which was sufficient to obtain steady-state conditions, and the
`calculation time was nearly 23 h for one case. The initialization temperature was 331 K,
`and the vapor volume fraction was 0. A computer (Thinkpad S3, i5-4210U, two cores,
`four threads, Lenovo, Peking, China) with Windows 7 Home Edition was used for the
`simulations. The mean surface temperature of the servers was taken as a comparative
`index, and the results showed that the predicted surface temperature with 510,438 cells
`changed by only 0.9%, with the corresponding temperature predicted with 121,471 cells. In
`the following simulations, the grid setting of 121,471 cells was selected.
`
`Table 4. Analysis of grid independence.
`
`Cell Number
`
`Time Step (s)
`
`121,471
`246,680
`510,438
`
`0.01
`0.01
`0.01
`
`Mean Surface
`Temperature (◦C)
`72.65
`72.28
`71.63
`
`Temperature
`Difference (%)
`-
`−0.5%
`−0.9%
`
`2.2.5. CFD Model Validation
`To begin the study, model validation was performed by comparing the experimental
`results, as displayed in Table 5. In summary, the maximum error was found to be less than
`5%, which implies that the CFD simulation precision reached the level of the experiment.
`The heat transfer coefficients of HFE-7100 were assessed for the interval between every
`two servers ∆l = 100 mm under different powers. The results showed that the heat transfer
`coefficients in our study were within a reasonable range compared with those demonstrated
`in other studies [46].
`
`Table 5. Verification of the proposed CFD settings.
`
`Power (W)
`
`1127
`1332
`1577
`Ref. [43]
`
`Experimental
`Average
`Surface
`Temperature
`(◦C)
`66.0
`68.5
`71.9
`-
`
`Numerical
`Average
`Surface
`Temperature
`(◦C)
`69.43
`71.56
`74.67
`-
`
`Error (%)
`
`4.95
`4.28
`3.71
`-
`
`Numerical
`Heat Transfer
`Coefficients
`(kW/m2 K)
`
`7.98
`10.16
`10.46
`6.1−18.5
`(∆T = 10 ◦C)
`
`Experimental
`Average
`Liquid
`Temperature
`(◦C)
`58.5
`61.5
`63.8
`
`Numerical
`Average
`Liquid
`Temperature
`(◦C)
`61.40
`64.13
`66.17
`
`2.3. COP and pPUE
`COP and PUE are two important indices used to evaluate the system energy perfor-
`mance. COP is defined as the ratio of cooling energy removed by a cooling system to the
`consumed power [47]. With the assumption of all the input server power transferring to
`heat, in this case, COP can be represented by Equation (10).
`
`COP =
`
`Wserver
`Wtotal
`
`(10)
`
`where Wserver is the server power and Wtotal is the total cooling power consumption in units
`of W.
`
`Immersion Systems LLC – Ex. 1015
`PGR 2021-00104 (U.S. 10,820,446 B2)
`10 of 21
`
`

`

`Energies 2021, 14, 1395
`
`11 of 21
`
`PUE is used to assess the thermal performance of cooling systems in data centers
`and is recommended by ASHRAE (American Society of Heating, Refrigerating and Air-
`Conditioning Engineers). In this study, owing to the two-phase immersion cooling system,
`a component of the real ICT equipment cooling system. Therefore, the pPUE was used
`to assess the efficiency of the two-phase immersion cooling system. It is defined as the
`total energy within a boundary divided by the IT equipment energy within that boundary
`(pPUE can only be calculated for zones with IT) [48]. Theoretically, pPUE should be larger
`than 1.0, and a value close to 1.0 implies high energy efficiency. The pPUE, in this case, can
`be calculated using Equation (11).
`
`pPUE =
`
`Wtotal + Wserver
`Wserver
`
`(11)
`
`2.4. Exergy Analysis
`Exergy analysis is based on the second law of thermodynamics and is an important
`method to evaluate the efficiency of a cooling system [49–51]. The specific exergy flow in
`any state can be expressed as
`
`e f = h − h0 − T0(s − s0)
`where h is the specific enthalpy, kJ/kg; s is the specific entropy, kJ/(kg K); and “0” represents
`the reference state of T = 293.15 K.
`The exergy efficiency can be defined as the ratio, as shown in Equation (13). For each
`part of the exergy destruction (
`D,i), its percentage can be calculated using Equation (14).
`
`(12)
`
`.E
`
`.E
`
`D
`Wtotal
`
`η = 1 −
`
`D,i
`
`.E
`
`.E
`
`pd,i =
`
`(cid:16)
`
`(cid:16)
`
`Ed,2 = mg
`
`D
`The transfer of heat in the proposed two-phase cooling system is accompanied by
`exergy destruction owing to an increase in entropy. On the tank side, the exergy destruction
`results from the evaporation of the liquid bath by absorbing heat from the servers, Ed,1 and
`condensation of the vapor through the loss of heat to the coolant inside the coils, Ed,2. The
`heat received by the coolant from the condensed vapor, Ed,3, is released to the environment
`through a heat exchanger, Ed,4.
`When the liquid bath is heated and evaporated to gas, the exergy destruction Ed,1 is
`calculated as
`e f ,l − e f ,g
`Ed,1 = Wserver + ml
`where ml is the mass flow of the evaporated liquid (kg/s), ef,l is the specific exergy rate of
`the liquid (kJ/kg), and ef,g is the specific exergy rate of the vapor, kJ/kg.
`When the evaporated gas is condensed to the liquid phase by the condenser, the
`exergy destruction Ed,2 is calculated as
`e f ,g − e f ,l
`
`(13)
`
`(14)
`
`(15)
`
`(16)
`
`(cid:17)
`
`(cid:19)
`
`Q1
`
`(cid:18)
`(cid:17) −
`(cid:0)hg − hl
`
`1 − T0
`(cid:1)
`Tl
`
`Q1 = mg
`(17)
`where T0 is the setpoint temperature. In this study, the temperature was set as the ambient
`temperature 293.15 K, Tl is the temperature of the condensed liquid near the servers, K;
`Q1 is the resealed heat of vaporization under relatively stable conditions, W; and mg is the
`mass flow of condensed gas (kg/s).
`
`Immersion Systems LLC – Ex. 1015
`PGR 2021-00104 (U.S. 10,820,446 B2)
`11 of 21
`
`

`

`Energies 2021, 14, 1395
`
`12 of 21
`
`(cid:19)
`
`Q2 + Wpump
`
`(18)
`
`(cid:18)
`When the coolant is heated by the evaporated gas, the exergy destruction Ed,3 is
`calculated as
`1 − T0
`Ed,3 = mw(ef,w,1 − ef,w,2) +
`Tw,l,2
`Q2 = mw(hw,l,2 − hw,l,1)
`(19)
`where Tw,1,2 denotes the temperature of the output coolant (K), Q2 is the absorbed heat
`of the coolant (W); mw is the mass flow of coolant (kg/s); ef,w,1 denotes the exergy rate of
`the input coolant; ef,w,2 denotes the exergy rate of the output coolant, kJ/kg; hw,l,1 is th