The Merits of Open Bath Immersion Cooling of Datacom Equipment
`
`Phillip E. Tuma
`3M Company
`3M Center, 236-2B-01
`St. Paul, MN 55144-1000
`petuma@mmm.com
`
`Abstract
`This paper discusses the economic and environmental
`merits of passive 2-phase immersion in semi-open baths of
`dielectric fluid for cooling datacom equipment such as
`servers. The technique eliminates the need for hermetic
`connectors, pressure vessels, seals and clamshells typically
`associated with immersion cooling and the connectors,
`plumping, pumps and cold plates associated with more
`traditional liquid cooling techniques. A board level power
`density of 11.7W/cm2 can be sustained with 100cm3 of fluid
`per kW. The modular 80kW baths modeled can eject 130kW
`per m2 of floor space via water-cooled condensers. It is
`estimated that 28°C water at 15gpm could maintain average
`CPU junction temperatures, Tj<60°C and 62°C water at
`30gpm could maintain Tj<85°C, maximizing the availability
`of the heat for other purposes. Alternatively, the heat can be
`transferred directly to ambient air without water as an
`intermediate. The costs and greenhouse gas emissions
`associated with conservative annual fluid emission estimates
`are found to be less than those associated with the electrical
`power required for traditional chassis fans and liquid pumps.
`Since these fugitive losses occur at one point, more efficient
`capture techniques can be easily applied.
`Keywords
`data center, datacenter, datacom, cooling, immersion, open
`bath, fluoroketone, evaporative bath, passive, 2-phase.
`Nomenclature
`a
`constant
`C
`specific heat [J/kg-K]
`heat transfer coefficient [W/m2-K] or [W/cm2-K]
`h
`k
`thermal conductivity [W/m-K]
`K
`Henry’s Law constant [Pa-mol/mol]
`m
`mass flow rate [kg/s]
`n
`moles
`kinematic viscosity [cSt]
`
`P
`pressure [Pa]
`Q
`power or heat [W]
`heat flux [W/cm2]
`Q”
`R
`ideal gas constant = 8.314 J/mol-K or
`thermal resistance [°C/W], [°C-cm3/W]
`density [kg/m3]
`temperature [°C] or [K]
`volume [m3]
`
`
`T
`V
`
`Subscripts
`a
`ambient
`air
`air
`atm
`atmospheric
`b
`boiling or boiling point
`c
`chip
`cond
`condenser or condensation
`f
`fluid
`H
`headspace
`i
`initial or inlet
`j
`junction
`o
`final or outlet
`s
`sink
`sat
`fluid saturation
`t
`cold trap
`v
`vapor
`w
`water
`1. Introduction
`1.1. Air Cooling and its Limitations
`Among the causes of inefficiency in traditional data center
`air cooling schemes are: 2nd Law irreversibility resulting from
`multiple heat transfer processes; mixing of warm and cool
`airstreams; power consumption of cooling hardware such as
`chillers, computer room air conditioners (CRACs), fans,
`blowers and pumps; and the reliance on air as a heat transfer
`media. The technologies being implemented target one or
`more of the aforementioned causes of inefficiency.
`Water-cooled rear door heat exchangers [1], ducted rack
`exhaust plena and closed forced air racks, for example, limit
`the mixing of airstreams. Raising the facility air temperature
`can, in some instances, increase overall efficiency [2]. These
`and other technologies may enable a facility to operate
`without a chiller substituting cooling-tower-only operation
`with on-demand economizers when whether permits [3, 4]
`Facilities that use full-time economizers are simpler still and
`can achieve a Power Usage Effectiveness (PUE) <1.3 [5].
`This is limited by economizer blowers, air filters [6] and fans
`within the servers that draw power, particularly when
`temperatures are high. Furthermore, facilities like these must
`be located in relatively cool climates [7].
`There are other inherent economic and environmental
`impacts. Managing airflow at the chassis, rack or facility
`level adds
`significant engineering cost during
`the
`development of each new server, datacenter or datacenter
`expansion. The number of publications on the subject and the
`
`978-1-4244-6460-9/10/$25.00 ©2010 IEEE
`
`123
`
`26th IEEE SEMI-THERM Symposium
`
`Authorized licensed use limited to: Alexander Karl. Downloaded on June 16,2021 at 17:17:42 UTC from IEEE Xplore. Restrictions apply.
`
`Immersion Systems LLC – Ex. 1013
`PGR 2021-00104 (U.S. 10,820,446 B2)
`1 of 9
`
`

`

`sensors,
`software,
`supplying
`companies
`in
`growth
`components and services aimed at refining air cooling are
`testament to its limitations. Air-cooled facilities are usually
`large buildings as required to accommodate airflow and
`evaporative cooling towers require a large volume of water [8].
`The fabrication of fans, heat sinks, CRACs, economizers,
`filters, etc. requires refined natural resources [9, 10] that may
`end up in landfills.
`Optimization of energy efficiency eventually leads beyond
`considerations of how best to eject a facility’s waste heat to
`how best to utilize it. However, the feasibility and cost of
`recovering its waste heat at any distance from an air-cooled
`datacenter are limited by the heat’s low thermodynamic
`availability or exergy and the large volumetric flow rate of
`air.
`1.2. Traditional Liquid Cooling and its Limitations
`Liquid cooling can reduce each of the aforementioned
`causes of inefficiency; facilitate waste heat recovery; and
`increase the thermodynamic availability of the heat removed
`[11]. Ellsworth and Iyengar [12] compared the facility level
`cooling performance of an air/liquid hybrid supercomputing
`cluster to its air-cooled equivalent with traditional chilled
`water and CRACs. They also predicted the efficiency gains
`achievable with an all-liquid technology and that same
`technology operating in a chiller-less or cooling-tower-only
`(water economizer) mode (Figure 1). The latter would result
`in a 90% reduction in cooling energy consumption versus the
`air-cooled cluster. While the efficiency of an all-liquid
`cluster was not shown to be dramatically higher than that of
`an air/liquid hybrid, the elimination of facility level air
`cooling
`infrastructure would
`likely
`reduce
`facility
`construction cost.
`However, implementation of traditional liquid cooling
`schemes, be they single- or two-phase, indirect-contact or
`immersion, is complicated by the number and variety of heat
`generating devices on a server and the requirement that each
`server within a rack be “hot swappable,” meaning that it can
`be removed and replaced without disturbing the operation of
`others. This makes it challenging to capture all of the heat
`generated on a printed circuit board (PCB) and move it to an
`external liquid stream. As a result, hybrid air-liquid systems
`bear inherent costs for design [13] and fabrication of cold
`plates, redundant pumps, plumbing, quick disconnects (QDs),
`controls, and heat exchangers [14] (Table 1). All-liquid
`systems (Table 2) are often more complicated, requiring
`additional or more complex cold plates, clamshells, or
`hermetic electrical connectors. Performance for many is
`limited by secondary even ternary thermal interfaces and fluid
`temperature
`glide. Hydrofluorocarbon
`(HFC)
`and
`perfluorocarbon (PFC) systems are prone to leakage and
`resultant global warming emissions at intractable sites like
`couplings.
`
`Cooling Tower
`Chiller
`Facility Pumps
`Rack Pumps
`CRAC
`
`1
`
`0.8
`
`0.6
`
`0.4
`
`0.2
`
`0
`
`Air Cooled
`
`All-Liquid All-Liquid with
`Air/Liquid
`Economizer
`Hybrid
`Figure 1: Normalized cooling power consumption of
`different components for various technologies [12]
`
`# Description
`224 Machined/brazed copper cold plates, clamps and TIMs
`168 Inter cold plate brazed copper manifold assemblies
`28 Brazed copper header assemblies
`28 Centrifugal blowers
`14 EMI gasket shield assemblies
`28 High performance quick disconnects (QDs)
`Rack Level
`2 Large brazed copper manifold assemblies
`1 Rear door heat exchanger
`2 Plate heat exchangers
`2 Reservoir tanks
`2 Proportional metering valves
`2 Pressure relief valves
`2 Check valves
`2 Solenoid isolation valves
`12 Temperature sensors
`6 Liquid level sensors
`2 Magnetically-coupled centrifugal pumps
`8 High performance large bore QDs
`1 Pressure relief valves
`TABLE 1: Partial list of cooling components in a 72kW
`air/liquid hybrid rack [14]
`
`See List
`a,b,c,d,
`(e),(f),g
`a,b,d2,
`(e)
`a,c,(e),
`(f),g
`
`Technology
`1. Pump water or HFC refrigerant onto each node and through
`cold plates12.
`2. Cold plates in contact with, not connected to server/node.
`Transfer heat to interface via heat pipes etc[15, 16]
`3. Pump dielectric fluid onto each server/node enclosed in
`direct immersion cooling clamshell [17].
`Shortcomings ()=depends on fluid technology
`a. Cost and environmental impact of cold plates, pumps, etc.
`b. Difficult to capture all heat without complex cold plate assemblies.
`c. QD leakage risk.
`d. Secondary thermal interface(s) (TIM2) impacts performance.
`e. Fluid glide effects performance (single-phase)
`f. Global warming emissions from fluid loss at intractable sites
`g. Risk of leakage at clamshell, QD or electrical vias.
`TABLE 2: Shortcomings of some commonly cited all-liquid
`cooling technologies
`
`Tuma, The Merits of Open Bath Immersion Cooling …
`
`26th IEEE SEMI-THERM Symposium
`
`Authorized licensed use limited to: Alexander Karl. Downloaded on June 16,2021 at 17:17:42 UTC from IEEE Xplore. Restrictions apply.
`
`Immersion Systems LLC – Ex. 1013
`PGR 2021-00104 (U.S. 10,820,446 B2)
`2 of 9
`
`

`

`immersed in the rinse sump where ultrasonics may be used to
`displace particulate (2). It is next moved from the rinse sump
`back to the vapor where clean condensing vapor again rinses
`the part (3). After a brief pause above the saturated vapor
`zone, during which solvent remaining on the part quickly
`evaporates (4), it leaves the machine dry.
`Vapor degreasers can clean thousands of parts per day
`with little fluid consumption despite the fact that they are
`open most of the time and only loosely lidded when not in
`use. Low loss rates are due in part to the secondary coils that
`often operate below 0°C. These set the solvent partial
`pressure that drives diffusion out of the bath through the
`freeboard region during use.
`
`Clearly a simple, compact, inexpensive liquid cooling
`technique
`is needed
`that minimizes natural
`resource
`consumption and global warming emissions. It must capture
`all heat while minimizing the junction-to-water temperature
`difference. It should be modular, scalable and should easily
`accommodate evolving hardware [18].
`1.3. Immersion Cooling History
`Passive 2-phase (evaporative) immersion cooling has been
`used for decades
`to cool high value electronics
`like
`transformers,
`traction
`inverters
`(Figure 2), specialized
`computers and klystrons. This technology is still in use
`today, being favored for its simplicity, reliability, power
`density and performance.
`These systems generally use sealed pressure vessels with
`hermetic electrical connections. They are evacuated and
`filled much like refrigeration systems and as such do not lend
`themselves to field service. It can be costly and complex to
`create
`such
`a hermetic
`enclosure
`for
`commodity
`computational electronics with their myriad of swappable
`components and electrical connections. For this reason, many
`people dismiss the idea of immersion in the context of
`datacom equipment. As will be shown, these measures are by
`no means necessary for static systems that remain at nearly
`constant temperature and power output.
`
`Figure 2: Immersion-cooled traction inverter from mining
`haul truck (photo courtesy of Siemens)
`1.4. The Open Bath Vapor Degreaser
`The management of fluid in vapor degreasers [19] is
`directly applicable to the concept discussed in this work.
`These ubiquitous machines are commonly used to clean parts
`ranging from screws and bearings to printed circuit boards,
`orthopedic implants and diesel engines. Degreasers comprise
`a tank, open at the top and fitted around its interior periphery
`with primary and secondary cooling coils (Figure 3). The
`tank is partitioned below a certain level to create two baths or
`“sumps” that are filled with a volatile solvent. The boil sump
`is heated from below causing the solvent within it to boil.
`Vapor rises to the height of the primary coils creating a
`saturated vapor zone beneath them. The condensed vapor
`flows back to the rinse sump, usually through a water
`separator. Because the sub-cooled solvent condensate is
`distilled, it is quite free of dissolved contaminants.
`The sequence of cleaning steps is also shown in Figure 3.
`The part to be cleaned is placed in a wire basket and lowered
`into the saturated vapor zone (1). Vapor condenses on the
`part beginning the cleaning process. The part is next
`Tuma, The Merits of Open Bath Immersion Cooling …
`
`Figure 3: Schematic of an open bath vapor degreaser.
`2. Open bath Immersion Cooling Concept
`The concept discussed in this work is based on the
`premise that electronics can be immersion cooled in semi-
`open baths similar in many ways to a vapor degreaser (Figure
`4). The term “semi” denotes a bath that is closed when access
`is not needed much like a chest-type food freezer. Like a
`freezer, it operates at atmospheric pressure and has no
`specialized hermetic connections for electrical inputs and
`outputs.
`In this concept, each server or node plugs into a backplane
`in the bottom of a tank (versus the back of a rack) that is
`partially filled with a volatile dielectric working fluid.
`Electrical connections from the backplane enter a conduit
`beneath the liquid level and exit the top of the tank. A vapor
`condenser integrated into the tank is cooled by tower water or
`water used at some distance for comfort heating. If desired,
`the vapor can flow passively to an outdoor natural draft
`cooling tower to transfer its heat passively to outdoor air
`without water as an intermediate. This open bath concept has
`a multitude of advantages over the other liquid cooling
`schemes that have been proposed.
`26th IEEE SEMI-THERM Symposium
`
`Authorized licensed use limited to: Alexander Karl. Downloaded on June 16,2021 at 17:17:42 UTC from IEEE Xplore. Restrictions apply.
`
`Immersion Systems LLC – Ex. 1013
`PGR 2021-00104 (U.S. 10,820,446 B2)
`3 of 9
`
`

`

`3. Thermal Performance
`The system thermal performance has two components.
`The first is at the board level and is device dependent. It is
`quantified by considering the junction-to-fluid temperature
`difference of the most difficult to cool component, the central
`processing unit (CPU). The second is the temperature
`difference from the fluid to the facility water. The fluid
`temperature, Tf, is the fluid’s atmospheric boiling point,
`
`(1)
`
`T
`
`
`
`
`
`)
`
`.
`
`P(T
`T
`sat
`atm
`b
`f
`3.1. Junction-to-Fluid Performance
`The performance capabilities of passive 2-phase heat
`transfer with dielectric coolants are well documented [20].
`Despite its simplicity and passive nature, it is not an inferior
`technology. Passive 2-phase immersion has been used to
`cool power semiconductors dissipating over Qc”=1100W/cm2
`with Tj-f=45°C, a performance level competitive with the
`best emerging pumped water technologies [21]. A typical CPU
`package configuration with its integrated heat spreader (IHS)
`is almost ideally suited for passive 2-phase immersion
`cooling. In most cases, it requires only the addition of a
`100m thick porous metallic boiling enhancement coating
`(BEC).
` These coatings produce boiling heat transfer
`coefficients, H>10 W/cm2-K at heat fluxes exceeding
`Q″=30W/cm2. Incorporating this technology directly onto the
`IHS during package assembly eliminates the secondary
`thermal interface common to many liquid cooling schemes
`without altering the package assembly process.
`
`Principal among them (Table 3) is the fact that most of the
`aforementioned air and
`liquid cooling hardware are
`eliminated as are considerations relating to their integration,
`reliability and power consumption. Power density and
`efficiency are very high and fire protection is intrinsic to the
`technology. Of course, there are other considerations, such as
`fugitive fluid emissions. Since these occur at one point rather
`than at countless seals and junctions, they are easily
`quantified and mitigated with simple techniques as will be
`discussed.
`
`Figure 4: Water-cooled open bath immersion concept
`
`Attribute
`1. No quick disconnects (QDs), clamshells, hermetic
`connectors.
`2. No pumps, fans, economizers, compressors
`3. No server/rack-level cold plates and plumbing
`4. Fluid losses at one point
`5. Intrinsic fire protection
`6 Tjf is low, no fluid glide
`7. High power density
`Advantage
`a. Less risk due to leakage
`b. Reduced power consumption
`c. Uses less natural resources
`d. Reduced cost and complexity
`
`e. Reduced greenhouse gas emission
`f. Uniform device temperatures
`g. More efficient capture of heat
`h. Reduced facility construction cost
`
`See list
`a,c,d,(e)
`
`b,c,d,e,h
`a,c,d
`a, e
`h
`g
`h
`
`Tc
`
`Ts
`
`BEC
`IHS
`Solder
`Silicon
`Junction
`
`Tf=Tsat(Patm)
`
`20x20mm chip
`Rj-c=0.008
`Rc-s=0.007
`Rs-f=0.030
`Rj-f=0.045°C/W
`
`+
`
`0
`
`10
`5
`Ac [cm2]
`
`15
`
`Tj
`
`0.15
`
`0.10
`
`0.05
`
`0.00
`
`Rs-f [°C/W]
`
`PCB
`
`TABLE 3: Advantages of open bath immersion cooling
`compared with other liquid and hybrid techniques
`
`Figure 5: Sink-to-fluid performance as a function of die size
`and
`total
`junction-to-fluid performance
`for
`20x20mm die in a hydrofluoroether (HFE) liquid
`
`Tuma, The Merits of Open Bath Immersion Cooling …
`
`26th IEEE SEMI-THERM Symposium
`
`Authorized licensed use limited to: Alexander Karl. Downloaded on June 16,2021 at 17:17:42 UTC from IEEE Xplore. Restrictions apply.
`
`Immersion Systems LLC – Ex. 1013
`PGR 2021-00104 (U.S. 10,820,446 B2)
`4 of 9
`
`

`

`The resultant sink-to-fluid resistance, Rs-f, is dependent on
`the chip size (Figure 5). For a typical 20x20mm chip, Rs-f
`=0.03°C/W. The additional resistances from sink-to-junction
`based on a 20x20mm thinned die and solder interface total
`0.015°C/W[23]. With Rj-f=0.045°C/W, a 200W processor has
`an average junction-to-fluid temperature difference,
`
`T
`
`
` 
`f
`
`j
`
`QR
`f
`j
`
`
`CPU
`
` ,
`
`(2)
`
`of about 9°C. It is likely this could be improved by
`optimizing a secondary heat transfer path through the package
`substrate.
`It is worth noting that although this passive technique
`requires a first level interface (TIM1), the resultant chip-to-
`fluid thermal resistance is lower than that achievable with
`direct-die-contact spray or jet impingement schemes based on
`dielectric coolants. These active techniques achieve heat
`transfer coefficients of H=1-3W/cm2-K [23,24] resulting in Rc-f
`>0.09°C/W, for the 20x20mm die discussed earlier.
`
`3.2. Fluid-to-Water Performance
`Previous research 21 has shown that a high density, water-
`cooled condenser can achieve a volume-specific, fluid-to-
`water inlet resistance,
`
`
`T
`cond
`Q
`
`V
`
`R
`
`
`wf
`
`
`
`T
`
`w
`
`f
`
` ,
`
`cond
`
`(3)
`
`However, it is worthwhile exploring what power density
`could be cooled. Previous research of immersion cooling of
`power electronics [21] suggests that power densities are limited
`more by the electrical bus than by the capabilities of
`immersion. The authors estimated that 100cc of fluid are
`required to cool a 1kW module if density limitations are
`exercised.
`Additional experiments were conducted with a form factor
`more consistent with a high density server. The simulated
`printed circuit board (PCB) shown in Figure 6 holds 20 heater
`assemblies comprised of 19x19mm 200W ceramic heaters
`epoxy bonded on one side to 30x30x3mm copper heat
`spreaders enhanced on the opposite side with a boiling
`enhancement coating (BEC). A thermocouple in the fluid, Tf,
`and one within each heat spreader, Ts, permit calculation of
`the individual thermal resistances and ensure that the heaters
`have not passed into the film boiling regime.
`The simulated PCB was immersed in a confined vertical
`channel of the same area as the board with 4 and 7mm gaps
`between the boiling surface and the adjacent wall. This
`assembly was able to dissipate 4kW (200W per heater
`assembly) for a 4mm gap at atmospheric pressure when the
`bath was filled with C3F7OCH3, a hydrofluoroether working
`fluid. Incipience overshoot did not exceed 7°C for any heater
`assembly. The average Rs-f are shown in Figure 6, bracketed
`by ½ standard deviation on each side. 4kW equates to a PCB
`level heat flux of Q”=11.7W/cm2 versus 1.7W/cm2 for the
`Cray X1E spray-cooled supercomputer [24]. These data
`suggest that 1kW/cm of bath normal to the PCB and 100cc of
`fluid per kW are certainly attainable, if only from a thermal
`point of view. Also, the potential for reduction of materials
`and waste associated with PCB manufacture is significant.
`
`of 1.4ºC-cm3/W under conditions of low water temperature
`glide. This number can be used in a log mean temperature
`difference (LMTD) analysis to predict condenser performance
`when the water temperature rises significantly:
`
`LMTD
`
`
`
`R
`
`Q
`cond
`V
`cond
`
`wf
`
`
`20X17X2.5cm BOARD
`
`(4)
`
`20 Heater
`Assemblies
`
`HEATER ASSEMBLY
`DETAIIL
`
`BEC
`
`Epoxy
`
`Ceramic
`Heater
`
`Copper Boiler
`30x30x3mm with
`Thermocouple
`
`Ts
`
`7mm gap
`4mm gap
`
`50
`
`25
`
`0
`
`1000xRs-f [°C/W]
`
`Bath
`
`4-7mm
`
`50
`
`100
`
`150
`
`200
`
`Q c [W]
`Figure 6: Experimental apparatus to demonstrate power
`density capabilities of immersion in a vertical PCB
`orientation
`
`Glide
`
`
`
`T
`w
`
`
`
`T
`i,w
`
`
`
`T
`o,w
`
`
`
`Q
`cond
`Cm
`
`
`w
`
`
`
`
`
`(5)
`
`The resultant inlet water temperature, Tw,i, is
`
`(6)
`
`(7)
`
`T
`i,w
`
`
`
`T
`
`f
`
`
`
`a
`
`
`1
`
`a
`
`
`
`glide
`
`.
`
`
`
`Glide
`LMTD
`
`
`
`a
`
`
`
`exp
`
`where
`
`4. Power Density
`By eliminating the heat sinks, fans and airspace normally
`required for air cooling, an immersion cooled server can, in
`principle, be quite compact. Current off-the-shelf air-cooled
`hardware, spread out as it is to facilitate air cooling, is not
`well suited for immersion. Determining how densely a server
`could be packaged is beyond the scope of this work.
`
`Tuma, The Merits of Open Bath Immersion Cooling …
`
`26th IEEE SEMI-THERM Symposium
`
`Authorized licensed use limited to: Alexander Karl. Downloaded on June 16,2021 at 17:17:42 UTC from IEEE Xplore. Restrictions apply.
`
`Immersion Systems LLC – Ex. 1013
`PGR 2021-00104 (U.S. 10,820,446 B2)
`5 of 9
`
`

`

`
`
`5. Working Fluid
`Table 4 shows one hydrofluoroether (HFE) and two
`fluoroketone (FK) [25] working fluids possessing the requisite
`thermophysical, safety and material compatibility properties.
`The dielectric properties of the FK fluids are similar to those of
`C6F14, a PFC fluid commonly used in immersion systems,
`while the HFE has significantly higher dielectric constant and
`lower resistivity. This may limit the utility of HFEs in some
`applications. The Global Warming Potential (GWP) of the
`HFE fluid is less than 60. The FK fluids have a GWP of 1,
`among the lowest of man-made compounds. The first of these
`FK fluids is already widely used globally as a Halon
`replacement for fire protection. Both of the FK fluids shown
`are being field tested in immersion applications at this time.
`
`
`Propertya
`C6F14
`Molecular Formula
`PFC
`Fluid Type
`56
`Tb [°C]
`<-100
`Tfreeze [°C]
`None
`Tflash [°C]
`12.0
` [mN/m]
`0.057
`k [W/m-K]
`1050
`C of liquid [J/kg-K]
`1680
` [kg/m3]
`0.40
` [cSt]
`88
`L. Heat @ Tb [kJ/kg]
`30.9
`Psat at 25ºC [kPa]
`350
`Psat at 100ºC [kPa]
`Resistivity [Gohm-cm] 1,000,000
`Dielect. Const.
`1.76
`DS [kV@2.54mm]
`~40
`GWPb
`9300
`EGc [ppmv]
`NA
`a. Properties at 25°C unless noted
`b. GWP based on 100 year integration time horizon
`c. 8-hour time weighted average (TWA) Exposure Guideline
`
`Working Fluid
`C6F9OH5
`C6F12O
`HFE
`FK
`76
`49
`<-100
`<-100
`None
`None
`13.6
`10.8
`0.068
`0.059
`1220
`1103
`1420
`1600
`0.41
`0.4
`119
`88
`15.7
`40.4
`206
`441
`0.1
`10,000
`7.3
`1.84
`~40
`~40
`55
`1
`200
`150
`
`C7F14O
`FK
`74
`<-100
`None
`12.3
`~0.06
`1130
`1670
`0.52
`90
`15.7
`228
`10,000
`1.85
`~40
`1
`150
`
`TABLE 4: Possible Working Fluids
`
`
`
`
`6. Fluid Losses
`The viability of open bath immersion technology depends
`primarily on the rate of fluid loss. It affects the cost of
`ownership but also the greenhouse gas emissions and the
`likely human exposure in the datacenter. The fluid loss
`mechanisms associated with purging air from low pressure
`sealed systems are well understood as are techniques for
`mitigating them [21] One simple technique is to pass the
`exiting
`air/vapor
`stream
`through
`an
`on-demand,
`thermoelectrically-cooled condenser or “trap” that condenses
`the vapor allowing it to return to the system. Similar
`techniques can be applied for open bath immersion. There are
`different loss mechanisms associated with each stage of
`operation.
`
`
`
`P
`atm
`
`
`
`
`)T(P
`a
`sat
`
`Kn
`
`l
`
`n
`v
`
`
`
`(11)
`
`The saturated vapor zone will rise until it contacts
`sufficient condenser surface area to condense the vapor being
`generated. This results in reduced headspace volume above
`the vapor zone, displacement of the air/vapor mixture within
`it and losses of
`
`
`)T(P
`t
`sat
`)T(P
`
`t
`sat
`
`P
`atm
`
`.
`
`
`
`
`
`P
`atm
`
`
`
`n
`v
`
`
`
`
`V)T(P
`a
`sat
`RT
`a
`
`
`
`V
`
`f,H
`
`i,H
`
`
`
`)T(P
`t
`sat
`)T(P
`
`t
`sat
`
`P
`atm
`
`.
`
`(12)
`
`
`
`The headspace will saturate to Tw. This too displaces air
`and results in fluid loss of
`
`
`n
`v
`
`
`
`
`
`
`V)T(P)T(P
`
`sat
`w
`sat
`a
`RT
`w
`
`f,H
`
`)T(P
`t
`sat
`)T(P
`
`sat
`t
`
`P
`atm
`
`.
`
`
`
`(13)
`
`Tuma, The Merits of Open Bath Immersion Cooling …
`
`
`
`
`
`
`26th IEEE SEMI-THERM Symposium
`
`Authorized licensed use limited to: Alexander Karl. Downloaded on June 16,2021 at 17:17:42 UTC from IEEE Xplore. Restrictions apply.
`
`
`
`6.1. During Filling
`If the liquid is added slowly from the bottom of the tank,
`the amount of vapor in the displaced air is negligible (Figure
`7a). Fluid will then evaporate until the headspace volume,
`VH,i, above it is saturated with vapor at ambient temperature,
`Ta, which is here assumed to be lower than the water
`temperature, Tw. As this occurs, air is displaced. The
`maximum amount of vapor carried with that air can be
`calculated as
`
`
`n
`v
`
`
`
`2
`
`)T(PV
`a
`sat
`i,H
`PRT
`2
`atm
`a
`
`.
`
`
`
`
`
`
`
`
`
`(8)
`
`
`The resultant losses can be mitigated by passing the
`exiting airstream through a cold trap operating at Tt so that
`
`
`.
`
`
`
`(9)
`
`
`
`)T(P
`sat
`t
`
`2
`
`21
`
`)T(P)T(P
`sat
`t
`sat
`a
`
`
`
`
`
`V
`i,H
`PRT
`atm
`a
`
`n
`v
`
`
`
`
`This analysis assumes that the headspace is well mixed as the
`fluid evaporates.
`6.2. During Start Up
`When the servers are turned on (Figure 7b), the fluid
`begins to boil. Air that was dissolved in the fluid,
`
`
`Kn
`
`l
`
`n
`
`air
`
`
`
`
`
`P
`atm
`
`
`
`)T(P
`a
`sat
`
`,
`
`
`
`
`
`
`
`(10)
`
`is first liberated, pushing some of the saturated air/vapor
`mixture through the trap. The resultant losses are
`
`
`Immersion Systems LLC – Ex. 1013
`PGR 2021-00104 (U.S. 10,820,446 B2)
`6 of 9
`
`

`

`
`
`
`Figure 7: Bath upon and after filling a.), during and after start up b.) and during load fluctuations in operation c.)
`
`
`
`
`
`
`6.3. During Steady State Operation
`During operation (Figure 7c) there are losses associated
`with fluctuating power levels that cause the vapor volume to
`rise and fall by VH. These are
`
`
`
`
`P
`atm
`
`
`
`n
`v
`
`
`
`
`V)T(P
`
`sat
`w
`RT
`w
`
`H
`
`)T(P
`t
`sat
`)T(P
`
`t
`sat
`
`P
`atm
`
`.
`
`
`
`(14)
`
`7. Results
`Table 5 summarizes results of the thermal performance
`calculations for the modular 80kW bath shown in Figure 8.
`If a commercially available fluoroketone fluid is used (FK1,
`Tf=49°C) in the bath, a 15gpm water flow rate could produce
`average Tj<60°C with a water inlet temperature, Tw,i=28°C. If
`the average Tj and the water flow rate are allowed to rise to
`83°C and 30gpm, respectively, a very useful Tw,i=62°C water
`temperature may be possible with a higher boiling fluid. The
`tank footprint power density is 130kW/m2, higher than the
`52kW/m2 level typical of air-cooled or hybrid air-liquid racks
`and beyond most long term industry projections [26] of what
`may be achievable with other air and liquid cooling
`technologies. With no real estate area and construction
`volume devoted to housing air handlers, plena, hot/cold aisles,
`etc., an all-liquid facility (Figure 9) might achieve 25kW/m2
`(2300W/ft2) versus 2.2kW/m2 (200W/ft2)[26] for most air
`cooled facilities.
`
`Processor Power [W]
`T j-f for 20x20mm CPU [°C]
`Power per node [kW]
`Number Nodes per Tank
`Total Tank Power [kW]
`Condenser Volume [cc]
`Log Mean Temp. Difference [°C]
`FK1
`Working Fluid
`T b =T sat (P atm )=T f [°C]
`49
`Resultant T j [°C]
`58.0
`30
`15
`30
`15
`10
`Water Flow [gpm]
`113.6
`56.8
`113.6
`56.8
`37.9
`Water Flow [liter/min]
`Water glide =T w [°C]
`10.1
`20.1
`10.1
`20.1
`30.2
`T w,i [°C]
`62.0
`53.4
`37.0
`28.4
`18.7
`Table 5: Water temperature calculations for 1 tank
`
`FK2
`74
`83.0
`
`
`
`200
`
`92
`
`40
`80
`22,000
`5.5
`
`
`
`
`26th IEEE SEMI-THERM Symposium
`
`
`In the calculations to follow, it was assumed that the power
`level and hence the vapor height underwent a single cycle per
`day of 25%.
`Finally, fluid will diffuse through the conduit housing the
`electrical IO. Experiments indicate that the diffusion rate is
`on the order of 100g-cm/cm2-day. If the cross sectional area
`of the diffusion path (conduit area minus wire conductors,
`etc.) is kept below 2cm2 and the length 2 meters, diffusion
`rates are near 1 g/day.
`
`
`Figure 8: Modular 80kW tank dimensions
`
`Tuma, The Merits of Open Bath Immersion Cooling …
`
`
`
`
`
`
`
`
`
`
`Authorized licensed use limited to: Alexander Karl. Downloaded on June 16,2021 at 17:17:42 UTC from IEEE Xplore. Restrictions apply.
`
`Immersion Systems LLC – Ex. 1013
`PGR 2021-00104 (U.S. 10,820,446 B2)
`7 of 9
`
`

`

`Figure 9: Conceptual aerial view and interior cross sections
`of air and water cooled immersion concepts
`
`Table 6 summarizes results of the fluid loss calculations
`for the modular 80kW bath shown in Figure 8 as a function of
`the vapor trap operating temperature, Tt, assuming a single
`25% load fluctuation per day. With Tt=10°C, annual fluid
`consumption cost at $123/yr compares favorably with the
`$184/yr cost of operating rack level pumps[12] at $0.05/kWh
`and the $2,800/yr cost of operating just the server fans in a
`typical air-cooled rack assuming they use 80W per kW of
`server power [2]. If the fluid GWP=1, the greenhouse gas
`emissions resulting from annual fluid loss are 0.2% of those
`associated with operating
`the aforementioned pumps
`assuming 7.18x10-4 metric tons CO2/kWh [27]. Use of an HFE
`with GWP=55 would raise this to 5%.
`
`Loss
`
`Tt [°C]
`Saturate empty tank [g]
`Beginning boil [g]
`Rise of vapor [g]
`Saturate headpace to P sat (T w ) [g]
`
`10
`45.9
`83.3
`59.8
`30.3
`219
`9.65
`6.7
`1
`2794
`123.0
`2.8E-03
`0.154
`
`20
`57.7
`152.6
`109.5
`55.5
`375
`16.5
`12.2
`1
`4815
`212
`4.8E-03
`0.265
`
`Eqn.
`5
`0
`-
`38.8
`32.1
`9
`62.4
`46.9
`11
`44.8
`33.6
`12
`22.7
`17.0
`13
`169
`130
`Total startup [g]
`7.42
`5.70
`Total Startup Fluid Cost at $0.044/g [$]
`5.0
`3.7
`Due to load cycle of 25% [g/day]
`10
`1
`1
`Diffusion through conduit [g/day]
`-
`2183
`1732
`Fluid lost [g/yr]
`96.0
`76.2
`Resultant cost [$/yr]
`2.2E-03
`FK fluid Metric Tons CO2 Equivalent (MTCE) 1.7E-03
`0.120
`HFE fluid MTCE
`0.095
`MTCE comparison for pumped water and air cooled
`Water pumps 5.8W/kW for 1 yr
`2.9
`Rack fans at 80W/kW for 1 yr
`40.3
`
`However, commonly cited all-liquid technologies are
`costly. They require significant engineering and hardware
`such as cold plates, pumps, plumbing, couplings, valves, and
`clamshells that consume natural resources and are prone to
`mechanical failure.
`Passive 2-phase immersion cooling has unique attributes:
`• All of the aforementioned node level and most of the rack
`level liquid cooling hardware are eliminated reducing
`environmental impact and simplifying design.
`• Low Tj-w increases the availability of heat and the ability
`to use full-time and even dry (non evaporative) cooling
`tower water.
` Alternatively, an outdoor-air-cooled
`condenser can be employed and water eliminated.
`• The tank (rack) and facility footprint power densities are
`estimated to be 2.5 and 10 times, respectively, those
`achievable with traditional air cooling.
`• A PCB level power density of ~12W/cm2 was demonstrated
`that
`only
`electronic
`and
`not
`thermal
`implying
`considerations limit node level power density
`• All devices are at same temperature (no glide).
`• The fluid cost is estimated to be $8 (100cc) per kW, less
`than the cost of copper used in two 2U heat sinks.
`• Fluid consumption on startup is estimated to be <25¢/kW.
`Annual consumption is estimated to be 1-3$/kW-year, less
`than 4% of the cost of running the server chassis fans in an
`equivalent air-cooled system at 5¢/kW-hr.
`• The resultant greenhouse gas emissions are 0.004-0.7% and
`0.1-5% (depending on fluid chemistry) those associated
`with the power draw of the server chassis fans in an air-
`cooled system and the rack pumps in a pumped water
`system, respectively.
`• Unlike most pumped r