111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US010123463B2
`
`c12) United States Patent
`Best et al.
`
`(10) Patent No.: US 10,123,463 B2
`(45) Date of Patent:
`Nov. 6, 2018
`
`(54) LIQUID SUBMERGED, HORIZONTAL
`COMPUTER SERVER RACK AND SYSTEMS
`AND METHOD OF COOLING SUCH A
`SERVER RACK
`
`(75)
`
`Inventors: Christiaan Scott Best, Austin, TX
`(US); Mark Garnett, Oklahoma City,
`OK (US)
`
`(73) Assignee: Green Revolution Cooling, Inc.,
`Austin, TX (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 971 days.
`
`(21) Appl. No.:
`
`13/057,881
`
`(22) PCT Filed:
`
`Aug. 10, 2009
`
`(86) PCTNo.:
`
`PCT /US2009/053305
`
`§ 371 (c)(l),
`(2), ( 4) Date: Feb. 7, 2011
`
`(87) PCT Pub. No.: W02010/019517
`
`PCT Pub. Date: Feb. 18, 2010
`
`(65)
`
`(60)
`
`(51)
`
`Prior Publication Data
`
`US 2011/0132579 Al
`
`Jun. 9, 2011
`
`Related U.S. Application Data
`
`Provisional application No. 61/188,589, filed on Aug.
`11, 2008, provisional application No. 61/163,443,
`(Continued)
`
`Int. Cl.
`F28D 15100
`H05K 7120
`G06F 1120
`
`(2006.01)
`(2006.01)
`(2006.01)
`
`(52) U.S. Cl.
`CPC ......... H05K 7120772 (2013.01); F28D 15100
`(2013.01); G06F 1120 (2013.01);
`(Continued)
`(58) Field of Classification Search
`CPC ........... H05K 7/20872; H05K 7/20627; H05K
`7/20636; H05K 7/20645; H05K 7/20654;
`(Continued)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,590,538 A *
`4,834,257 A *
`
`. ...................... 361/700
`5/1986 Cray, Jr.
`5/1989 Book et a!. ................... 220/646
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`CN
`JP
`
`5/2009
`101443724
`1112004
`2004319628
`(Continued)
`
`OTHER PUBLICATIONS
`
`Inetl Core 2 Duo Processor on 65 nm process for Embedded
`Applications, Aug. 2007, Intel, pp. 1, 14, and 21.*
`(Continued)
`
`Primary Examiner- Christopher R Zerphey
`(74) Attorney, Agent, or Firm- The Marbury Law
`Group, PLLC
`
`ABSTRACT
`(57)
`Apparatus, systems, and methods for efficiently cooling
`computing devices having heat-generating electronic com(cid:173)
`ponents, such as, for example, independently operable serv(cid:173)
`ers, immersed in a dielectric liquid coolant in a tank.
`
`33 Claims, 14 Drawing Sheets
`
`836
`
`HEATED LIQUID COOLANT
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`1 of 35
`
`

`

`US 10,123,463 B2
`Page 2
`
`Related U.S. Application Data
`
`filed on Mar. 25, 2009, provisional application No.
`61/165,470, filed on Mar. 31, 2009.
`
`(52) U.S. Cl.
`CPC ............... G06F 11206 (2013.01); HOSK 7120
`(2013.01); HOSK 71203 (2013.01); HOSK
`712079 (2013.01); HOSK 7120236 (2013.01);
`HOSK 7120281 (2013.01); HOSK 7120327
`(2013.01); HOSK 7120381 (2013.01); HOSK
`7120763 (2013.01); HOSK 7120781 (2013.01);
`HOSK 7120827 (2013.01); HOSK 7120836
`(2013.01); G06F 2200/201 (2013.01); HOJL
`2924/0002 (2013.01); YJOT 29/4973 (2015.01)
`(58) Field of Classification Search
`CPC ........... H05K 7/20236; H05K 7/20709; H05K
`7/20218; H05K 7/20763; H05K 7/203;
`H05K 7/20781; H05K 7/2079; G06F
`1/20; G06F 2200/201; H01L 23/473;
`F24F 1/02; F25D 17/02
`USPC ............... 62/259.2, 263, 434, 435; 361/699;
`165/104.33
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,297,621 A
`6,374,627 B1 *
`6,600,656 B1 *
`6,621,707 B2
`6,909,606 B2
`7,086,247 B2
`7,184,269 B2 *
`7,210,304 B2
`7,307,841 B2
`7,318,322 B2 *
`7,403,392 B2
`7,609,518 B2
`7,905,106 B2
`7,911,782 B2
`7,911,793 B2
`8,009,419 B2
`2002/0185262 A1
`2003/0053293 A1 *
`2003/0127240 A1
`2004/0008490 A1 *
`2004/0246683 A1 *
`2005/0083657 A1
`2005/0114876 A1
`2005/0259402 A1 *
`2006/0026610 A1
`2006/0064709 A1
`2006/0123436 A1
`2006/0135042 A1
`2006/02507 55 A1
`2006/0274501 A1 *
`2007/0006599 A1 *
`2007/0025081 A1 *
`2007/0034360 A1 *
`2007/0199340 A1 *
`2007/0213000 A1
`2007/0267741 A1 *
`2008/0002364 A1
`2008/0017355 A1 *
`2008/0026509 A1
`
`3/1994 Taraci
`4/2002 Schumacher et al ........ 62/259.2
`7/2003 Mori eta!. .................... 361/724
`9/2003 Ishimine
`6/2005 Barsum
`8/2006 Campbell
`2/2007 Campbell et al ............. 361/700
`5/2007 Nagashima
`12/2007 Berlin
`112008 Ota et a!. ..................... 62/259.2
`7/2008 Attlesey
`10/2009 Hopton
`3/2011 Attlesey
`3/2011 Attlesey
`3/2011 Attlesey
`8/20 11 Attlesey
`12/2002 Baer
`3/2003 Beitelmal et al ............. 3611687
`7/2003 Beckbissinger
`112004 Cheon ........................... 3611699
`.. 361/720
`12/2004 Honsberg-Riedl et a!.
`4/2005 Harmnan
`5/2005 Atarashi
`11/2005 Yasui et al .................... 361/716
`2/2006 Sasao
`3/2006 Throckmorto
`6/2006 Tanaka
`6/2006 Frost et al.
`1112006 Tilton
`12/2006 Miller ........................... 3611690
`112007 Kawamura eta!. ........... 62/54.1
`2/2007 Berlin et a!. .................. 3611698
`165/104.33
`2/2007 Hall
`8/2007 Knight et al.
`. .............. 62/259.2
`9/2007 Day
`.............. 257/714
`11/2007 Attlesey et a!.
`112008 Campbell et al.
`112008 Attlesey eta!.
`112008 Campbell
`
`165/104.33
`
`2008/0029250 A1 * 2/2008 Carlson ................ F24F 1110001
`165/104.33
`2008/0030945 A1 * 2/2008 Mojaver et a!. . ............. 3611685
`3/2008 Murakami
`2008/0055845 A1
`2008/0158818 A1
`7/2008 Clidaras
`2008/0196870 A1
`8/2008 Attlesey
`2009/0260777 A1
`10/2009 Attlesey
`2010/0226094 A1 * 9/2010 A ttl esey et a!.
`9/2010 Attlesey
`2010/0246118 A1
`2010/0290190 A1
`1112010 Chester et al.
`2010/0302678 A1
`12/2010 Merrow
`201110075353 A1
`3/2011 Attlesey
`201110132579 A1
`6/2011 Best eta!.
`201110240281 A1
`10/2011 Avery
`
`.............. 3611699
`
`FOREIGN PATENT DOCUMENTS
`
`JP
`wo
`wo
`wo
`wo
`wo
`
`2004363308
`wo 2007023130
`wo 2007098078
`wo 2008027931
`wo 2008089322
`2010019517
`
`12/2004
`3/2007
`8/2007
`3/2008
`7/2008
`2/2010
`
`OTHER PUBLICATIONS
`
`International Search Report and Written Opinion dated Oct. 14,
`2009, Application No. PCT/US2009/053305, 10 pages.
`Singapore Written Opinion and Search Report dated May 2, 2012,
`Application No. 201100595-6, 21 pages.
`Examination Report from Australian Application No. 2009282170,
`dated Nov. 15, 2013, pp. 1-3.
`Patent Examination Report No. 2 from Australian Application No.
`2009282170, dated Jun. 18, 2014, pp. 1-4.
`International Search Report and Written Opinion from PCT/US12/
`49668, dated Oct. 19, 2012, Green Revolution Cooling Inc., pp.
`1-10.
`Office Action from Chinese Application No. 200980131707.3, dated
`Dec. 31, 2014, English and Chinese versions, pp. 1-10.
`Office Action from Chinese Application No. 200980131707.3, dated
`Apr. 3, 2014, English translation, pp. 1-13.
`Office Action from Chinese Application No. 200980131707.3, dated
`Dec. 20, 2012, English and Chinese versions, pp. 1-17.
`Office Action from Chinese Application No. 200980131707.3, dated
`Jul. 31, 2013, English and Chinese versions, pp. 1-8.
`Office Action from Chinese Application No. 200980131707.3, dated
`Apr. 18, 2014, English and Chinese versions, pp. 1-7.
`Office Action from Chinese Application No. 200980131707.3, dated
`Jul. 31, 2013, English translation, pp. 1-3.
`U.S. Appl. No. 14/338,035, filed Jul. 22, 2014, Christiaan Scott
`Best.
`U.S. Appl. No. 14/338,013, filed Jul. 22, 2014, Christiaan Scott
`Best.
`U.S. Appl. No. 14/338,020, filed Jul. 22, 2014, Christiaan Scott
`Best.
`U.S. Appl. No. 14/338,026, filed Jul. 22, 2014, Christiaan Scott
`Best.
`U.S. Appl. No. 14/667,091, filed Mar. 24, 2015, Christiaan Scott
`Best.
`Canadian Office Action dated Nov. 8, 2016 for Canadian Patent
`Application No. 2,731,994 related to National Phase of Application
`No. PCT/US2009/053305 filed Aug. 10, 2009.
`Non-Final Office Action in U.S. Appl. No. 14/667,091, dated May
`22, 2015, 16 pages.
`Substantive Examination Adverse Report, Malaysian Application
`No. PI2011000494, dated May 15, 2015, 3 pages.
`* cited by examiner
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`2 of 35
`
`

`

`120 120 120
`
`110 120 120
`
`1[1' RAC: (~111
`~
`
`"
`" "
`" "
`" " "
`"
`" "
`"
`" "
`" " " "
`,,
`"
`"
`
`170
`
`1~0
`
`_.-r
`" "
`
`13
`
`0
`
`PUMP
`
`~
`
`FILTER
`
`60
`V1
`
`~
`
`)100
`
`e •
`
`00
`•
`~
`~
`~
`
`~ = ~
`
`z 0
`
`~
`~Cl\
`N
`
`0 ....
`
`QO
`
`------
`" "
`CONTROLLER ~/
`
`\
`
`\
`\
`\
`\
`\
`\
`\
`\
`\
`\
`\
`\
`\
`\
`\
`\
`
`HEAT
`EXCHANGER
`
`I
`¥
`
`A
`I
`I
`
`COOLING
`APPARATUS
`
`\
`\
`\
`\
`\
`\
`\
`\
`\
`\
`\
`\
`\
`
`\ '
`
`140
`
`150
`
`-------~
`
`INFORMATION
`
`DIELECTRIC LIQUID FLUID FLOW
`
`- - - -> -
`
`COOLING FLUID FLOW
`
`Fig. 1A
`
`('D
`('D
`
`rFJ =(cid:173)
`.....
`....
`0 .....
`....
`
`.j;o.
`
`d
`rJl
`-..=
`""""'
`""""' N w
`~
`
`0'1 w = N
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`3 of 35
`
`

`

`!200
`
`120 210
`
`RACK
`
`260
`
`270
`
`\
`
`.. --
`
`----
`
`~--
`
`.. --
`
`--------
`
`----
`
`CONTROLLER
`
`~ ........
`........
`........
`........
`........
`........
`............ ........ ___
`
`HEAT
`EXCHANGER
`
`~290
`_i
`
`I
`I
`_l
`
`----------I INFORMATION
`
`280
`
`- - - -1 DIELECTRIC LIQUID FLUID FLOW
`
`- - - -~ 1 COOLING FLUID FLOW
`
`COOLING
`APPARATUS
`
`1--250
`
`Fig. 18
`
`e •
`
`00
`•
`~
`~
`~
`
`~ = ~
`
`z 0
`
`~
`~Cl\
`N
`
`0 ....
`
`QO
`
`('D
`('D
`
`rFJ =(cid:173)
`.....
`N
`0 .....
`....
`
`.j;o.
`
`d
`rJl
`-..=
`""""'
`""""' N w
`~
`
`0'1 w = N
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`4 of 35
`
`

`

`U.S. Patent
`
`Nov. 6, 2018
`
`Sheet 3 of 14
`
`US 10,123,463 B2
`
`300
`
`j
`
`120
`'
`
`3 WAY
`
`VALVE
`
`REFRIGERATION
`
`v3_90
`r-._
`,------1-- -~ - :-~-"-;~-~t56 _ _!!0 --
`352i
`------ ---------:
`!
`j OUTSIDE AIR
`HEAT
`~
`j RADIATOR
`RECOVERY
`---------------+- _L ________________ ~
`! 354 ~~-----------------:1
`I
`:
`:
`- - - - - - SIGNAL
`II
`I EVAPORATIVE
`:
`c __ ______ -+i-----L..rc _______ _!_LUI~ __ _i
`!
`COOLER
`I
`'---------------------~=-~=-~r-~=-~=-~-_:-
`
`I
`
`358
`
`Fig. 2
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`5 of 35
`
`

`

`U.S. Patent
`
`Nov. 6, 2018
`
`Sheet 4 of 14
`
`US 10,123,463 B2
`
`~ .
`0')
`LL
`
`l 1
`
`M .
`0') ·-LL
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`6 of 35
`
`

`

`U.S. Patent
`
`Nov. 6, 2018
`
`Sheet 5 of 14
`
`US 10,123,463 B2
`
`.
`LO
`C') ·-
`LL
`
`!'-I
`~
`
`~' "*'
`
`,---/'
`~ "*'
`
`~
`
`~~ ~
`,.._
`""'-!
`
`~
`
`~
`~
`
`lr-II~
`
`- -
`
`_J
`
`I
`
`!'-I -
`
`~
`
`I
`
`1-
`
`~
`
`~
`
`c.... =>
`~
`t
`
`(
`
`(
`
`!l
`
`I
`I
`I
`
`-
`
`~
`!'-I
`
`_J
`
`.
`CD
`·-
`C')
`LL
`
`~
`-.:t-
`
`l
`
`I
`
`~
`
`~
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`7 of 35
`
`

`

`U.S. Patent
`
`Nov. 6, 2018
`
`Sheet 6 of 14
`
`US 10,123,463 B2
`
`540
`
`515
`
`Fig. 7
`
`510
`
`-
`
`.560
`
`500
`~
`
`562
`
`Fig. 8
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`8 of 35
`
`

`

`U.S. Patent
`
`Nov. 6, 2018
`
`Sheet 7 of 14
`
`US 10,123,463 B2
`
`674
`
`610
`
`600
`
`j
`
`620
`
`J
`
`670
`
`700
`
`j
`
`772
`
`760
`
`710
`
`Fig. 9
`
`660
`
`672
`
`720
`
`710
`
`750
`
`740
`
`Fig. 10
`
`782
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`9 of 35
`
`

`

`U.S. Patent
`
`Nov. 6, 2018
`
`Sheet 8 of 14
`
`US 10,123,463 B2
`
`,....
`,.... .
`C) ·-LL.
`
`~I
`
`~I
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`10 of 35
`
`

`

`U.S. Patent
`
`Nov. 6, 2018
`
`Sheet 9 of 14
`
`US 10,123,463 B2
`
`~
`~
`
`<C
`,.... .
`C\1
`C) ·-LL
`
`~o-~
`
`~
`
`~
`
`m
`,.... .
`C\1
`C) ·-LL
`
`~
`
`,.... .
`C) ·-LL
`
`('I)
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`11 of 35
`
`

`

`U.S. Patent
`
`Nov. 6, 2018
`
`Sheet 10 of 14
`
`US 10,123,463 B2
`
`~I
`
`\c
`~~~~========~==================~
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`12 of 35
`
`

`

`U.S. Patent
`
`Nov. 6, 2018
`
`Sheet 11 of 14
`
`US 10,123,463 B2
`
`901
`
`916
`
`_900
`
`j
`
`924
`________ L_
`
`I
`I
`I
`I
`I
`
`_925
`_ _i ______ 4
`•
`
`920
`
`---:......-
`
`-----
`
`I
`I
`I
`I
`~----+-----~~~----~~~--~~~~-----i
`_914
`914 T
`I
`I
`clc
`I
`I
`I
`
`_908
`
`T
`:
`932
`--------~---------c-J
`
`I
`
`--FLUID LINE
`------CONTROL LINE
`
`~f---+--+--....1 -------------\-------------
`912
`_928
`
`Tc R
`
`I
`
`TH R
`
`I
`
`Fig. 15
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`13 of 35
`
`

`

`FLOWING DIELECTRIC LIQUID THROUGH THE SERVERS
`
`~
`MONITORING THE TEMPERATURE OF THE DIELECTRIC LIQUID
`
`~
`DETERMINING THE OPTIMUM ELEVATED TEMPERATURE
`
`~
`DETERMINING THE ENERGY NEEDED TO COOL THE SERVERS
`
`~
`DETERMINING THE OPTIMUM SECONDARY COOLING APPARATUS TO MINIMIZE ENERGY USAGE
`
`~
`THERMALLY COUPLING THE HEATED DIELECTRIC LIQUID TO DISTALLY LOCATED HEAT EXCHANGER
`
`~
`ADJUSTING THE AMOUNT OF HEAT TO BE REJECTED THROUGH THE HEAT EXCHANGER TO
`MAINTAIN THE ELEVATED TEMPERATURE
`
`10
`
`12
`
`14
`
`16
`
`18
`
`-20
`
`-22
`
`Fig. 16
`
`e •
`
`00
`•
`~
`~
`~
`
`~ = ~
`
`z 0
`
`~
`~Cl\
`N
`
`0 ....
`
`QO
`
`('D
`('D
`
`rFJ =(cid:173)
`.....
`....
`N
`0 .....
`....
`
`.j;o.
`
`d
`rJl
`-..=
`""""'
`""""' N w
`~
`
`0'1 w = N
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`14 of 35
`
`

`

`U.S. Patent
`
`Nov. 6, 2018
`
`Sheet 13 of 14
`
`US 10,123,463 B2
`
`26
`
`r------------------
`:
`--------------
`28
`:
`......c:----
`
`32
`..,.. _______ _
`
`34
`
`CONTROLLER
`
`APPLICATIONS
`SOFTWARE
`
`I
`50
`
`~
`
`FLOWING DIELEGRIC LIQUID 24
`AT A LOWER TEMPERATURE
`INTO THE TANK WITH
`SUBMERGED SERVERS
`t
`FLOWING THE DIELEGRIC
`LIQUID THROUGH THE
`SERVERS FOR ABSORBING
`HEAT FROM THE SERVERS
`t
`QPTIONALLY ENHANCING
`FLUID VELOCITY OF
`DIELEGRIC FLUID THROUGH
`THE SERVERS
`~0
`t
`MONITORING THE TEMPERATURE OF
`THE DIELECTRIC LIQUID AND SERVER __
`ELECTRONIC COMPONENTS AND
`FLOW RATE OF THE DIELEGRIC LIQUID
`t
`SELECTING THE MORE
`EFFICIENT SECONDARY
`COOLING SYSTEM TO
`MINIMIZE ENERGY USAGE
`t
`PUMPING THE HEATED
`DIELEGRIC LIQUID TO A
`DISTALLY LOCATED HEAT
`EXCHANGER
`t
`REJECTING THE ABSORBED
`HEAT IN THE DIELEGRIC
`LIQUID AT THE HEAT
`EXCHANGER ASSOCIATED WITH
`CHOSEN SECONDARY
`COOLING APPARATUS
`
`36
`FLUIDLY COUPLING THE
`__jr-+-.-. COOLED DIELEGRIC LIQUID -
`TO THE TANK
`
`ADJUSTING THE FLUID FLOW -40
`OF THE SECONDARY
`-=------
`COOLING APPARATUS
`
`Fig. 17A
`
`'
`
`DISSIPATING THE REJEGED
`HEAT IN A SECONDARY
`COOLING APPARATUS
`
`-42
`
`RECOVERING THE REJECTED
`HEAT IN A SECONDARY
`COOLING APPARATUS
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`15 of 35
`
`

`

`U.S. Patent
`
`Nov. 6, 2018
`
`Sheet 14 of 14
`
`US 10,123,463 B2
`
`RECEIVE SIGNALS OF SYSTEM
`52
`OPERATION FROM SENSORS: f-t.
`FLUID FLOW, POWER,
`TEMPERATURE
`+
`DETERMINE OPTIMUM
`ELEVATED TEMPERATURE OF
`DIELECTRIC FLUID
`
`54
`f.'.
`
`+
`PERIODICALLY DETERMINING f..r
`56
`THE ENERGY NEEDED TO
`COOL THE SERVERS
`
`f..r 58
`
`+
`DETERMINE THE OPTIMAL
`SECONDARY COOLING
`METHOD TO MINIMIZE
`ENERGY USAGE
`+
`DETERMINE PREFERABLE
`SETTINGS FOR DIELECTRIC
`FLUID PUMP FLOW,
`SECONDARY COOLING SYSTEM,
`AND OPTIONALLY TANK FLUID
`VELOCITY
`+
`EXECUTE OUTPUT CONTROL
`62
`SIGNALS TO PUMPS, VALVES, f-''
`AND FLUID VELOCITY
`AUGMENTATION SYSTEMS
`+
`
`60
`f-r'
`
`EXECUTE NOTIFICATION OF
`FAILURE
`
`64
`f.''
`
`Fig. 178
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`16 of 35
`
`

`

`US 10,123,463 B2
`
`1
`LIQUID SUBMERGED, HORIZONTAL
`COMPUTER SERVER RACK AND SYSTEMS
`AND METHOD OF COOLING SUCH A
`SERVER RACK
`
`CROSS-REFERENCES TO RELATED
`APPLICATIONS
`
`This application claims priority pursuant to 35 U.S.C. 119
`to the following U.S. provisional patent applications:
`Ser. No. 61/188,589 entitled LIQUID SUBMERGED,
`HORIZONTAL COMPUTER SERVER RACK filed Aug.
`11, 2008;
`Ser. No. 61/163,443 entitled LIQUID SUBMERGED,
`HORIZONTAL COMPUTER SERVER RACK filed Mar.
`25, 2009; and
`Ser. No. 61/165,470 entitled LIQUID SUBMERGED,
`HORIZONTAL COMPUTER SERVER RACK filed Mar.
`31, 2009.
`
`FIELD OF INVENTION
`
`This application concerns cooling of heat-generating elec(cid:173)
`tronics such as, for example, rack mounted servers in data
`centers.
`
`BACKGROUND
`
`In 2006, data centers in the United States (U.S.) accounted
`for about 1.5% (about $4.5 billion) of the total electricity
`consumed in the U.S. This data center electricity consump(cid:173)
`tion is expected to double by 2011. More than one-third of
`data center electricity consumption is for cooling servers,
`which could equate to more than about 1% of all U.S.
`electricity consumed by 2011. Electricity, personnel, and
`construction costs continue to increase and server hardware
`costs are decreasing, making the overall cost of cooling a
`large and growing part of the total cost of operating a data
`center.
`The term "data center" (also sometime referred to as a
`"server farm") loosely refers to a physical location housing
`one or "servers." In some instances, a data center can simply
`comprise an unobtrusive comer in a small office. In other
`instances, a data center can comprise several large, ware(cid:173)
`house-sized buildings enclosing tens of thousands of square
`feet and housing thousands of servers. The term "server"
`generally refers to a computing device connected to a
`computing network and running software configured to
`receive requests (e.g., a request to access or to store a file,
`a request to provide computing resources, a request to
`connect to another client) from client computing devices,
`includes PDAs and cellular phones, also connected to the
`computing network. Such servers may also include special(cid:173)
`ized computing devices called network routers, data acqui(cid:173)
`sition equipment, movable disc drive arrays, and other
`devices commonly associated with data centers.
`Typical commercially-available
`servers have been
`designed for air cooling. Such servers usually comprise one
`or more printed circuit boards having a plurality of electri(cid:173)
`cally coupled devices mounted thereto. These printed circuit
`boards are commonly housed in an enclosure having vents
`that allow external air to flow into the enclosure, as well as
`out of the enclosure after being routed through the enclosure
`for cooling purposes. In many instances, one or more fans
`are located within the enclosure to facilitate this airflow.
`"Racks" have been used to organize several servers. For
`example, several servers can be mounted within a rack, and
`
`2
`the rack can be placed within a data center. Any of various
`computing devices, such as, for example, network routers,
`hard-drive arrays, data acquisition equipment and power
`supplies, are commonly mounted within a rack.
`Data centers housing such servers and racks of servers
`typically distribute air among the servers using a centralized
`fan (or blower). As more fully described below, air within
`the data center usually passes through a heat exchanger for
`cooling the air (e.g., an evaporator of a vapor-compression
`10 cycle refrigeration cooling system (or "vapor-cycle" refrig(cid:173)
`eration), or a chilled water coil) before entering a server. In
`some data centers, the heat exchanger has been mounted to
`the rack to provide "rack-level" cooling of air before the air
`enters a server. In other data centers, the air is cooled before
`15 entering the data center.
`In general, electronic components of higher performing
`servers dissipate correspondingly more power. However,
`power dissipation for each of the various hardware compo(cid:173)
`nents (e.g., chips, hard drives, cards) within a server can be
`20 constrained by the power being dissipated by adjacent
`heating generating components, the airflow speed and air(cid:173)
`flow path through the server and the packaging of each
`respective component, as well as a maximum allowable
`operating temperature of the respective component and a
`25 temperature of the cooling air entering the server as from a
`data center housing the server. The temperature of an air
`stream entering the server from the data center, in tum, can
`be influenced by the power dissipation and proximity of
`adjacent servers, the airflow speed and the airflow path
`30 through a region surrounding the server, as well as the
`temperature of the air entering the data center (or, con(cid:173)
`versely, the rate at which heat is being extracted from the air
`within the data center).
`In general, a lower air temperature in a data center allows
`35 each server component to dissipate a higher power, and thus
`allows each server to dissipate more power and operate at a
`level of hardware performance. Consequently, data centers
`have traditionally used sophisticated air conditioning sys(cid:173)
`tems (e.g., chillers, vapor-cycle refrigeration) to cool the air
`40 (e.g., to about 65° F.) within the data center for achieving a
`desired performance level. By some estimates, as much as
`one watt can be consumed to remove one watt of heat
`dissipated by an electronic component. Consequently, as
`energy costs and power dissipation continue to increase, the
`45 total cost of cooling a data center has also increased.
`In general, spacing heat-dissipating components from
`each other (e.g., reducing heat density) makes cooling such
`components less difficult (and less costly when considering,
`for example, the cost of cooling an individual component in
`50 a given environment) than placing the same components
`placed in close relation to each other (e.g., increasing heat
`density). Consequently, data centers have also compensated
`for increased power dissipation (corresponding to increased
`server performance) by increasing the spacing between
`55 adjacent servers.
`In addition, large-scale data centers have provided several
`cooling stages for cooling heat dissipating components. For
`example, a stream of coolant, e.g., water, can pass over an
`evaporator of a vapor-compression refrigeration cycle cool-
`60 ing system and be cooled to, for example, about 44° F.
`before being distributed through a data center for cooling air
`within the data center.
`The power consumed by a chiller can be estimated using
`information from standards (e.g., ARI 550/590-98). For
`65 example, ARI550/590-98 specifies that a new centrifugal
`compressor, an efficient and common compressor used in
`high-capacity chillers, has a seasonal average Coefficient-
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`17 of 35
`
`

`

`US 10,123,463 B2
`
`3
`of-Performance ("COP") from 5.00 to 6.10, depending on
`the cooling capacity of the chiller. This COP does not
`include power consumed by an evaporative cooling tower,
`which can be used for cooling a condenser in the refrigera(cid:173)
`tion cycle cooling system and generally has a COP of70, or
`better. The combined COP for a typical system is estimated
`to be about 4.7.
`According to some estimates, some state-of-the-art data
`centers are capable of cooling only about 150 Watts-per(cid:173)
`square-foot, as opposed to cooling the more than about
`1,200 Watts-per-square-foot that could result from arranging
`servers to more fully utilize available volume (e.g., closely
`spacing servers and racks to more fully utilize floor-to(cid:173)
`ceiling height and floor space) within existing data centers.
`Such a low cooling capacity can significantly add to the cost
`of building a data center, since data centers can cost as much
`as about $250 per-square-foot to construct.
`As the air-cooling example implies, commercially avail(cid:173)
`able methods of cooling have not kept pace with increasing
`server and data-center performance needs, or the corre(cid:173)
`sponding growth in heat density. As a consequence, adding
`new servers to existing data centers has become difficult and
`complex given the effort expended to facilitate additional
`power dissipation, such as by increasing an existing data 25
`center's air conditioning capacity.
`Various alternative approaches for cooling data centers
`and their servers, e.g., using liquid cooling systems, have
`met with limited success. For example, attempts to displace
`heat from a microprocessor (or other heat-generating semi- 30
`conductor-fabricated electronic device component, collec(cid:173)
`tively referred to herein as a "chip") for remotely cooling the
`chip have been expensive and cumbersome. In these sys(cid:173)
`tems, a heat exchanger or other cooling device, has been 35
`placed in physical contact (or close physical relation using
`a thermal-interface material) with the package containing
`the chip. These liquid-cooled heat exchangers have typically
`defined internal flow channels for circulating a liquid inter(cid:173)
`nally of a heat exchanger body. However, component loca- 40
`tions within servers can vary from server to server. Accord(cid:173)
`ingly, these liquid-cooling systems have been designed for
`particular component layouts and have been unable to
`achieve large-enough economies of scale to become com(cid:173)
`mercially viable.
`Research indicates that with state-of-the-art cooling,
`PUEs (as defined on page 10 hereinafter) of 1.4 might be
`attainable by 2011. However the costs to capitalize such
`cooling were not mentioned, and indicators suggest that
`saving electricity requires expensive equipment.
`Immersion cooling of electronic components has been
`attempted in high-performance (e.g., computer gaming)
`applications, but has not enjoyed widespread commercial
`success. Previous attempts at immersion cooling has sub(cid:173)
`merged some, and in some instances all, components 55
`mounted to a printed circuit board in a dielectric fluid using
`a hermetically sealed enclosure to contain the fluid. Such
`systems have been expensive, and offered by a limited
`number of suppliers. Large scale data centers generally
`prefer to use "commoditized" servers and tend to not rely on 60
`technologies with a limited number of suppliers.
`Control systems have been used to increase cooling rates
`for a plurality of computers in response to increased com(cid:173)
`putational demand. Even so, such control systems have
`controlled cooling systems that dissipate heat into the data 65
`center building interior air (which in turns needs to be cooled
`by air conditioning), or directly use refrigeration as a pri-
`
`4
`mary mode of heat dissipation. Refrigeration as a primary
`mode of cooling, directly or indirectly, requires significant
`amounts of energy.
`Two-phase cooling systems have been attempted, but due
`to technical complexity, they have not resulted in cost(cid:173)
`effective products or sufficiently low operating costs to
`justify investing in two-phase-cooling capital. Still other
`single- and two-phase cooling systems bring the coolant
`medium to an exterior of the computer, but reject heat to a
`10 cooling medium (e.g., air) external to the computer and
`within the data center (e.g., within a server room). Accord(cid:173)
`ingly, each method of server or computer cooling currently
`employed or previously attempted have been prohibitively
`expensive and/or insufficient to meet increasing cooling
`15 demands of computing devices.
`Indirectly, many researchers have tried to reduce the
`power of individual components such as the power supply
`and CPU. Although chips capable of delivering desirable
`performance levels while operating at a lower relative power
`20 have been offered by chip manufacturers, such chips have,
`to date, been expensive. Consequently, cooling approaches
`to date have resulted in one or more of a high level of
`electricity consumption, a large capital investment and an
`increase in hardware expense.
`Therefore, there exists the need for an effective, efficient
`and low-cost cooling alternative for cooling electronic com(cid:173)
`ponents, such as, for example, rack-mounted servers.
`
`SUMMARY OF INVENTION
`
`Briefly, the present invention provides novel apparatus,
`systems, and methods for efficiently cooling computing
`devices having heat-generating electronic components, such
`as, for example, independently operable servers immersed in
`a dielectric liquid coolant in a tank.
`The system may include at least one tank defining an
`interior volume and having a coolant inlet for receiving a
`dielectric liquid coolant within the interior volume and
`having a coolant outlet for allowing the dielectric liquid
`coolant to flow from the interior volume, the coolant inlet
`and the coolant outlet being fluidly coupled to each other;
`one or more mounting members positioned within the inte(cid:173)
`rior volume and configured to mountably receive a plurality
`of independently operable servers; a dielectric liquid cool-
`45 ant; a heat exchanger fluidly coupled to the coolant outlet of
`the at least one tank, the heat exchanger being distally
`located from the tank; a pump fluidly coupled to the heat
`exchanger and the interior volume of the at least one tank,
`the pump being configured for pumping the liquid coolant
`50 through a fluid circuit comprising a first circuit portion
`extending from the coolant inlet of the tank to each server,
`a second circuit portion extending from each respective
`server to the coolant outlet, a third circuit portion extending
`from the coolant outlet to the heat exchanger, and a fourth
`portion extending from the heat exchanger to the coolant
`inlet; a controller for monitoring the temperature of the
`dielectric liquid coolant at at least one location within the
`fluid circuit and for adjusting the flow of the dielectric liquid
`coolant through the fluid circuit in order that the dielectric
`liquid coolant is maintained at an elevated temperature as it
`exits the second circuit portion of the fluid circuit; wherein
`the at least one tank is configured for containing the dielec(cid:173)
`tric liquid coolant within the interior volume such that, when
`the plurality of servers are mountably received therein, each
`server is submerged within the dielectric liquid coolant for
`sufficiently cooling each respective server while maintaining
`the exiting heated liquid coolant at the elevated temperature
`
`Immersion Systems LLC – Ex. 1006
`PGR 2021-00104 (U.S. 10,820,446 B2)
`18 of 35
`
`

`

`US 10,123,463 B2
`
`5
`to reduce the amount of energy consumed to sufficiently cool
`each of the plurality of servers.
`Alternatively, the cooling system includes at least one
`tank defining an open interior volume; one or more mount(cid:173)
`ing members positioned within the open interior volume and
`configured to mountably receive a plurality of independently
`operable servers within the interior volume; a dielectric
`liquid coolant circulating in a first fluid circuit through the
`plurality of servers; a secondary cooling system having a
`cooling fluid flowing in a second fluid circuit wherein the
`secondary cooling system rejects heat from the cooling fluid;
`a coupler located within the at least one tank for thermally
`coupling heated dielectric coolant from the portion of the
`first fluid circuit exiting the plurality of servers within the
`tank to the cooling fluid in the second fluid circuit for
`rejecting heat from such heated dielectric coolant; a con(cid:173)
`troller for monitoring the temperature of the dielectric liquid
`coolant at at least one location within the first fluid circuit
`and for adjusting the flow of the cooling fluid through the
`second fluid circuit in order that the heated dielectric liquid
`coolant exiting the plurality of servers is maintained
`approximately at an elevated temperature wherein the
`elevated temperature is a temperature significantly higher
`than the typical comfortable room temperature for humans
`and lower than the maximum permissible temperature of the
`most sensitive heat generating electronic component in the
`plurality of servers; wherein the at least one tank is config(cid:173)
`ured for containing the dielectric liquid coolant within the
`interior volume such that, when the plurality of servers are
`mountably received therein, at least a substantial portion of
`each server is submerged within the dielectric liquid coolant
`for sufficiently cooling each respective server when the tank
`is sufficiently full of the liquid coolant maintaining the liquid
`coolant [exiting] the plurality of servers at approximately the
`elevated temperature to reduce the amount of energy con(cid:173)
`sumed to sufficiently cool each respective server.
`Alternatively, the cooling system may include at least one
`tank defining an open interior volume; one or more mount(cid:173)
`ing members positioned within the open interior volume and
`configured to mountably receive a plurality of independently
`operable servers within the interior volume; a dielectric
`liquid coolant circulating in a first fluid circuit through the
`plurality of servers; a secondary cooling system having a
`cooling fluid flowing in a second fluid circuit wherein the
`secondary cooling system rejects some of the heat from the
`cooling fluid; a coupler located within the at least one tank
`for thermally coupling heated dielectric coolant from the
`portion of the first fluid circuit exiting the plurality of servers
`within the tank to the cooling fluid in the second fluid circuit
`for rejecting some of the