Case 6:20-cv-01152-ADA Document 1-4 Filed 12/16/20 Page 1 of 18
`Case 6:20-cv-01152—ADA Document 1-4 Filed 12/16/20 Page 1 of 18
`
`EXHIBIT4
`EXHIBIT 4
`
`

`

`Case 6:20-cv-01152-ADA Document 1-4 Filed 12/16/20 Page 2 of 18
`
`I 1111111111111111 11111 lllll lllll lllll 111111111111111 11111111111 111111111111
`US008572612B2
`
`c12) United States Patent
`Kern
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,572,612 B2
`Oct. 29, 2013
`
`(54) AUTONOl\llC SCALING OF VIRTUAL
`MACHINES IN A CLOUD COMPUTING
`ENVIRONMENT
`
`(75)
`
`Inventor: Eric R. Kern, 01.apel Hill, NC (US)
`
`(73) Assignee: International Business Machines
`Corporation, Armonk, NY (US)
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 483 days.
`
`(21) Appl. No.: 12n60,141
`
`(22) Filed:
`
`Apr. 14, 2010
`
`(65)
`
`Prior Publication Data
`
`US 2011/0258621 Al
`
`Oct. 20, 2011
`
`(51)
`
`(2006.01)
`
`Int.Cl.
`G06F 9/455
`(52) U.S. Cl.
`USPC ................................ 718/1; 717/177; 717/178
`(58) Field of Classification Search
`None
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`7,093,086 Bl
`7,370,164 Bl
`7,502,962 B2 *
`7,802,248 B2
`7,831,600 B2
`7,890,613 B2
`7,917,617 Bl
`8,046,473 B2
`8,060,476 Bl
`2005/0198303 Al
`2006/0085792 Al
`2006/0184936 Al
`2007/0130566 Al
`2007/0208918 Al
`
`8/2006 Rietschote el al.
`5/2008 Nagarkar et al.
`3/2009 Yach et al ....................... 714/21
`9/20 1 0 Broquere et al.
`11/2010 Kilian
`2/2011 Tameshige et al.
`3/2011 Ponnapur et al.
`10/20 I 1 Piper et al.
`11/2011 Afonso et al.
`9/2005 Knauerhase et al.
`4/2006 Traut
`8/2006 Abels ct al.
`6/2007 van Rietschote et al,
`9/2007 Harbin et al.
`
`2007/0244938 Al
`2009/0070771 Al
`2009/0113109 Al
`2009/0138541 Al
`2009/0249284 Al
`2009/0288084 Al
`2009/0293056 Al
`2009/0300076 Al
`2009/0300151 Al
`2010/0037038 Al•
`2010/0042720 Al*
`
`10/2007 Michael et al.
`3/2009 Yuyitung et al.
`4/2009 Nelson et al.
`5/2009 Wing et al,
`10/2009 Antosz et al.
`11/2009 Astete et al.
`11/2009 Ferris
`12/2009 Friedman et al.
`12/2009 Friedman et al.
`2/2010 Bieswanger et al. ......... 712/220
`2/2010 Stienhans et al. ... ........ .. 709/226
`(Continued)
`
`OTHER PUBLICATIONS
`
`"System and apparatus for automatic health monitoring and mainte(cid:173)
`nance of virtual machines", IP.com No. IPCOM000191156D, Dec.
`18, 2009.
`
`(Continued)
`
`Primary Examiner - Camquy Truong
`(74) Attorney, Agent, or Firm - Edward J. Lenart; Cynthia
`G. Seal; Biggers & Ohanian, LLP.
`
`(57)
`
`ABSTRACT
`
`Autonomic scaling of virtual machines in a cloud computing
`environment, the cloud computing environment including
`virtual machines ('VMs'), the VMs installed upon cloud
`computers disposed within a data center, also including a
`cloud operating system and a data center administration
`server operably coupled to the VMs, including deploying, by
`the cloud operating system, an instance of a VM, flagging the
`instance of a VM for autonomic scaling; monitoring, by the
`cloud operating system, one or more operating characteristics
`of the instance of the VM; deploying, by the cloud operating
`system, an additional instance of the VM if a value of an
`operating characteristic exceeds a first predetermined thresh(cid:173)
`old value, including executing a portion of the data processing
`workload on the additional instance of the VM; and terminat(cid:173)
`ing operation of the additional instance of the VM if a value of
`an operating characteristic declines below a second predeter(cid:173)
`mined threshold value.
`
`15 Claims, 5 Drawing Sheets
`
`81st
`
`JltoeoW1.~h,_UW.._EIIIMd9f'Ae.llSt!Qfll:lllllll(l(A
`~~a,.t,m.U.9p«:iulinOIA .... .;m.
`
`...........
`
`011loM --°'""
`
`Adil'li:allll'llliru
`
`"'"'"' "'
`
`

`

`Case 6:20-cv-01152-ADA Document 1-4 Filed 12/16/20 Page 3 of 18
`
`US 8,572,612 B2
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`20I0/0106885 Al
`2010/0107158 Al
`2010/0115332 Al
`2010/0131324 Al
`2010/0205304 Al
`2010/0211829 Al
`2010/0241896 Al
`2010/0293409 Al
`2010/0306379 Al•
`2010/0306382 Al
`2011i0010515 Al
`2011/0022695 Al
`201 l/0022812 Al
`2011/0060832 Al
`2011/0072208 Al
`20 lliO 126275 Al
`20lli0179l 76 Al
`2011/0208908 Al
`201 l/0231696 Al
`
`4/2010 Gao et al.
`4/2010 Chen et al.
`5/2010 Zheng et al.
`5/2010 Ferris
`8/2010 Chaturvedi et al.
`8/2010 Ziskindctal.
`9/2010 Brown et al.
`11/2010 Machida
`12/2010 Ferris ... ......................... 709/226
`12/20 IO Cardosa et al.
`1/2011 Ranade
`1/2011 Dalal et al.
`1/2011 Van der Linden et al.
`3/2011 Govil et al.
`3/2011 Gulati et al.
`5/2011 Anderson et al.
`7/2011 Ravichandran et al.
`8/2011 Chou et al.
`9/2011 Ji et al.
`
`9/2011 Zlati et al.
`20 ll/0231698 Al
`9/2011 Kem
`2011/0238803 Al
`10/2011 Kern
`20 I 1/0246627 Al
`OTHER PUBLICATIONS
`
`Kangarlou et al., "VNsnap: taking snapshots of virtual networked
`environments with minimal downtime", IEEEIIFIP International
`Conference on Dependable Systems & Networks, 2009.
`Office Action, U.S . Appl. 121730,402, Dec. l , 2011 .
`Notice of Allowance, U.S. Appl. No. 121730,402, Apr. 16, 2012.
`Office Action, U.S. Appl. No . 12/752,762, May 31, 2012.
`Office Action, U.S. Appl. No. 12/752,322, May 16, 2012.
`Lagar-Cavilla et al., "SnowFlock: rapid virtual machine cloning for
`cloud computing", 2009, in Proceedings of the 4'" ACM European
`conference on Computer systems (EUroSys '09). ACM, New York,
`NY, USA, 1-12.
`Office Action, U.S. Appl. No. 12/759,976, Aug. 28, 2012.
`Office Action, U.S.Appl. No. 12/760,141, Jul. 10, 2012.
`Final Office Action, U.S. Appl. No. 12/785,322, Aug. 29, 2012.
`Final Office Action, U.S. Appl. No. 12/752,762, Oct. 29, 2012.
`OfficeAclion, U .S. Appl. No. 12/760,141 , Dec. 10, 2012.
`Office Action, U .S. Appl. No. 12/759,976, Mar. 20, 2013.
`
`* cited by examiner
`
`'
`
`

`

`Case 6:20-cv-01152-ADA Document 1-4 Filed 12/16/20 Page 4 of 18
`
`U.S. Patent
`
`Oct. 29, 2013
`
`Sheet 1 of 5
`
`US 8,572,612 B2
`
`-,-··----~-~---·---·---- ···- -··
`
`:(~
`
`VM Template
`11.6.
`
`' ' ' ' ' ' ' ' ' '
`
`••• . •• - -- --- -- •••• !a,. •• -- - -- --
`
`.
`
`'
`' '
`
`Deployment
`VM
`Engine
`Catalog
`I VM Record 202 l
`176
`1aQ
`......................................................•............................. •'
`
`Template,
`User Spec
`182
`
`UUID,
`Address
`184
`
`Cloud Computer .11Q
`
`RAM 168
`
`VM 102
`
`I Application m I
`I UUID 12Q I
`I Network Address izal
`IVMAgent ml
`joperaling System iM:I
`
`Hypervisor 164
`
`Callout ill
`
`CPU
`~
`
`~
`
`Bus
`Adapter
`~
`
`Comm.
`Adapter
`.1fil
`
`•
`rsonal
`mputer ::::::::: •• :
`FIG.1
`
`

`

`Case 6:20-cv-01152-ADA Document 1-4 Filed 12/16/20 Page 5 of 18
`
`U.S. Patent
`
`Oct. 29, 2013
`
`Sheet 2 of 5
`
`US 8,572,612 B2
`
`Receive, Through A User Interface Exposed By A Self Service Portal Of A
`Cloud Operating System, User Specifications Of A VM 302
`
`Deploy, By The Cloud Operating System, An Instance Of The VM 304
`
`Flag The Instance Of The VM For Autonomic Scaling 306
`
`Store, In The Cloud Operating System In Association With An
`Identifier Of The Instance Of A VM, Predetermined Threshold
`Values Of Operating Characteristics, Including Threshold Values
`For Processor Utilization And For Memory Utilization 308
`
`Execute A Data Processing Wor1doad On The Instance Of A VM llQ.
`
`Monitor, By The Cloud Operating System, One Or More
`Operating Characteristics Of The Instance Of The VM
`312
`
`VM Record 202
`
`VM Identifier 204
`Boolean Scalable 2Q§
`CPU Up Threshold 208
`CPU On Threshold 210
`Mem Up Threshold 212
`Mem On Threshold fil
`
`Value Of
`Operating
`Characteristic
`314
`
`---
`
`No:
`Continue
`- -~
`Monitoring
`326
`
`Terminate Operation OfThe
`Additional Instance Of The VM
`336
`
`First
`Predetermined
`Threshold
`Value 316
`
`Yes 322------.
`
`Second
`Predetermined
`Threshold
`Value~
`
`Deploy An Additional
`Instance OfThe VM
`332
`
`Execute A Portion
`Of The Data
`Processing
`Workload On The
`Additional Instance
`OfTheVM
`334
`
`----ontinue Monitoring 3311-.......__ ______________ __. FIG. 2
`
`

`

`Case 6:20-cv-01152-ADA Document 1-4 Filed 12/16/20 Page 6 of 18
`
`U.S. Patent
`
`Oct. 29, 2013
`
`Sheet 3 of 5
`
`US 8,572,612 B2
`
`Receive, Through A User Interface Exposed By A Self
`Service Portal Of A Cloud Operating System, User
`Specifications Of A VM 302
`
`User
`Specifications
`174
`
`Deploy, By The Cloud Operating System, An Instance Of The VM ~
`
`Flag The Instance Of The VM For Autonomic Scaling 306
`
`Execute A Data Processing Workload On The Instance Of A VM filQ,
`
`Pass User Specifications To The Deployment Engine 340
`
`Implement And Pass To The Data Center Administration Server A
`VM Template With The User Specifications 342
`
`Call A Hypervisor On A Cloud Computer To Install The VM Template
`As An Instance Of A VM On The Cloud Computer 344
`
`Monitor, By The Cloud Operating System, One Or More
`Operating Characteristics Of The Instance Of The VM
`312
`
`Threshold
`Value 316
`
`>----Yes 322---~
`
`--- No:
`
`Continue
`Monitoring
`326
`
`Threshold
`Value 328
`
`Yes 330
`
`Terminate Operation Of The
`Additional Instance Of The VM
`336
`
`Deploy An Additional
`Instance Of The VM
`332
`
`Execute A Portion
`Of The Data
`Processing
`Workload On The
`Additional Instance
`Of The VM
`334
`
`ontinue Monitoring 333_...._ _______________ _, FIG. 3
`
`

`

`Case 6:20-cv-01152-ADA Document 1-4 Filed 12/16/20 Page 7 of 18
`
`U.S. Patent
`
`Oct. 29, 2013
`
`Sheet 4 of 5
`
`US 8,572,612 B2
`
`Receive, Through A User Interface Exposed By A Self
`Service Portal Of A Cloud Operating System, User
`Specifications Of A VM 302
`
`User
`Specifications
`174
`
`Monitor, By The Cloud Operating System, One Or More
`Operating Characteristics Of The Instance Of The VM
`312
`
`Deploy, By The Cloud Operating
`System, An Instance OfThe VM
`304
`
`Operating
`Value 314
`
`---
`
`No:
`Continue
`Monitoring
`326
`
`Terminate Operation Yes 330
`Of The Additional
`Instance Of The VM
`336
`
`Threshold
`Value 316
`
`Yes 322------,
`
`Threshold
`Value 328
`
`Deploy An Additional Instance Of The VM
`332
`
`Continue Monitoring 338
`
`Execute A Portion Of The Data Processing Workload On The
`Additional Instance Of The VM 334
`
`Pass User Specifications To The Deployment Engine 346
`
`Implement And Pass To The Data Center Administration
`Server With The User Specifications A Same VM Template
`Used To Deploy A Previous Instance OfThe VM 348
`
`Call A Hypervisor On A Cloud Computer To Install The Same
`VM Template As An Additional Instance Of The VM On The
`Cloud Computer 350
`
`ontinue Monitoring 330--------------
`
`FIG.4
`
`

`

`Case 6:20-cv-01152-ADA Document 1-4 Filed 12/16/20 Page 8 of 18
`
`U.S. Patent
`
`Oct. 29, 2013
`
`Sheet 5 of 5
`
`US 8,572,612 B2
`
`Receive, Through A User lntertace Exposed By A Self
`Service Portal Of A Cloud Operating System, User
`Specifications Of A VM 302
`
`User
`Specifications
`174
`
`Monitor, By The Cloud Operating System, One Or More
`Operating Characteristics Of The Instance Of The VM
`312
`
`Deploy, By The Cloud Operating
`System, An Instance Of The VM
`304
`
`Threshold
`Value 316
`
`Yes 322'----
`
`No:
`Continue
`1------
`- -<
`Monitoring
`326
`
`Yes 330
`
`Threshold
`Value 328
`
`Deploy An Additional
`Instance OfThe VM
`332
`
`Terminate Operation Of The Additional Instance Of The VM
`336
`
`Transmit, By The Self Service Portal To The Data Center
`Administration Server, An Instruction To Terminate The
`Additional Instance OfThe VM 352
`
`I -
`
`Call, By The Data Center Administration Server, A Hypervisor
`On A Cloud Computer To Terminate Operation Of The
`Additional Instance Of The VM On The Cloud Computer 354
`
`i....
`
`-~ -ontinue Monitoring 338:---.&...--------------.....1
`FIG. 5
`
`

`

`Case 6:20-cv-01152-ADA Document 1-4 Filed 12/16/20 Page 9 of 18
`
`US 8,572,612 B2
`
`1
`AUTONOMIC SCALING OF VIRTUAL
`MACHINES IN A CWUD COMPUTING
`EWIRONMENT
`
`BACKGROUND OF THE INVENTION
`
`2
`operating characteristic exceeds a first predetermined thresh(cid:173)
`old value, including executing a portion of the data processing
`workload on the additional instance of the VM; and tenninat(cid:173)
`ing operation of the additional instance of the VMif a value of
`5 an operating characteristic declines below a second predeter(cid:173)
`mined threshold value.
`The foregoing and other objects, features and advantages
`of the invention will be apparent from the following more
`particular descriptions of exemplary embodiments of the
`10 invention as illustrated in the accompanying drawings
`wherein like reference numbers generally represent like parts
`of exemplary embodiments of the invention.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 sets forth functional block diagrams of example
`apparatus that carries out autonomic scaling of virtual
`machines in a cloud computing environment according to
`embodiments of the present invention.
`FIGS. 2-5 set forth flowcharts illustrating example meth(cid:173)
`ods of autonomic scaling of virtual machines in a cloud com(cid:173)
`puting environment according to embodiments of the present
`invention.
`
`DETAILED DESCRIPTION OF EXEMPLARY
`EMBODIMENTS
`
`1. Field of the Invention
`The field of the invention is data processing, or, more
`specifically, methods, apparatus, and products for autonomic
`scaling of virtual machines in a cloud computing environ(cid:173)
`ment.
`2. Description of Related Art
`The development of the EDVAC computer system of 1948
`is often cited as the beginning of the computer era. Since that
`time, computer systems have evolved into extremely comp Ii- 15
`cated devices. Today's computers are much more sophisti(cid:173)
`cated than early systems such as the EDVAC. Computer sys(cid:173)
`tems typically include a combination of hardware and
`software components, application programs, operating sys(cid:173)
`tems, processors, buses, memory, input/output devices, and 20
`so on. As advances in semiconductor processing and com(cid:173)
`puter architecture push the performance of the computer
`higher and higher, more sophisticated computer software has
`evolved to take advantage of the higher performance of the
`hardware, resulting in computer systems today that are much 25
`more powerful than just a few years ago.
`One of the areas of technology that has seen recent
`advancement is cloud computing. Cloud computing is
`increasingly recognized as a cost effective means of deliver(cid:173)
`ing information technology services through a virtual plat- 30
`form rather than hosting and operating the resources locally.
`Modem clouds with hundred or thousands of blade servers
`enable system administrators to build highly customized vir(cid:173)
`tual machines to meet a huge variety of end user require(cid:173)
`ments. Many virtual machines, however, can reside on a 35
`single powerful blade server. Cloud computing has enabled
`customers to build virtualii.ed servers on hardware over
`which they have absolutely no control. Deploying an appli(cid:173)
`cation in a cloud computing environment simplifies the over-
`all solution because the hardware becomes abstracted behind 40
`the cloud infrastructure. 1be end user, however, loses all
`control over the underlying hardware infrastructure including
`particularly a loss of control over scaling the number of
`virtual machines running an application. In a cloud environ(cid:173)
`ment scaling of an application is currently carried out manu- 45
`ally by a system administrator-only when end users report
`performance degradation. This technique is slow and com(cid:173)
`plex, and it inherently risks a user's experiencing a poor
`quality of service.
`
`'Scaling,' as the term is used here, refers to a desirable
`feature of a cloud computing environment in which the envi(cid:173)
`ronment gracefully handles varying workloads, either
`increasing or decreasing. The notion of grace here refers to
`the ability of the environment to adapt to varying workloads
`transparently, smoothly, and with a minimum of difficulty for
`users of the data processing service provided by such a cloud
`computing environment. 'Autonomic' refers to an ability for
`a cloud computing environment to manage its own operations
`with little or no governance by hand from any user or system
`administer. 'Autonomic scaling' therefore refers to an ability
`of a cloud computing environment to manage its own scaling
`operations with little or no governance by hand from any user
`or system administer. This paper describes autonomic scaling
`in the sense of adding and removing instances of virtual
`machines and their data processing applications as data pro(cid:173)
`cessing workload increases and decreases within a cloud
`computing environment.
`Example methods, apparatus, and products for autonomic
`scaling of virtual machines in a cloud computing environment
`according to embodiments of the present invention are
`described with reference to the accompanying drawings,
`50 beginning with FIG. 1. PIG. 1 sets forth a functional block
`diagram of apparatus that carries out autonomic scaling of
`virtual machines in a cloud computing environment accord(cid:173)
`ing to embodiments of the present invention. The apparatus in
`the example of FIG. 1 implements a cloud computing envi(cid:173)
`ronment (192) that includes virtual machine ('VMs') (102,
`104), where each VM is a module of automated computing
`machinery installed upon a cloud computer (109, 106) dis(cid:173)
`posed within a data center (128).
`The cloud computing environment (192) is a network(cid:173)
`based, distributed data processing system that provides one or
`more cloud computing services. Although shown here, for
`convenience of explanation, with only a few cloud computers
`(106, 109) in the cloud computing environment, such a cloud
`computing environment typically includes, as a practical mat-
`ter, many computers, hundreds or thousands of them, dis(cid:173)
`posed within one or more data centers, with the cloud com-
`puters typically implemented in the blade form factor. Typical
`
`SUMMARY OF THE INVENTION
`
`Methods, apparatus, and computer program products for
`autonomic scaling of virtual machines in a cloud computing
`environment, the cloud computing environment including a 55
`plurality of virtual machines ('VMs'), the VMs composed of
`modules of automated computing machinery installed upon
`cloud computers disposed within a data center, the cloud
`computing environment also including a cloud operating sys(cid:173)
`tem and a data center administration server operably coupled 60
`to the VMs, including deploying, by the cloud operating
`system, an instance of a VM, including flagging the instance
`of a VM for autonomic scaling and executing a data process(cid:173)
`ing workload on the instance of a VM; monitoring, by the
`cloud operating system, one or more operating characteristics 65
`of the instance of the VM; deploying, by tl1e cloud operating
`system, an additional instance of the VM if a value of an
`
`

`

`Case 6:20-cv-01152-ADA Document 1-4 Filed 12/16/20 Page 10 of 18
`
`US 8,572,612 B2
`
`3
`examples of cloud computing services include Software as a
`Service ('SaaS') and Platform as a Service ('PaaS'). Saas is
`a model of software deployment in which a provider licenses
`an application to customers for use as a service on demand.
`SaaS software vendors may host the application on their own 5
`clouds or download such applications from clouds to cloud
`clients, disabling the applications after use or after an on(cid:173)
`demand contract expires.
`PaaS is the delivery from a cloud computing environment
`of a computing platform and solution stack as a service. PaaS 10
`includes the provision of a software development platform
`designed for cloud computing at the top of a cloud stack. PaaS
`also includes workflow facilities for application design,
`application development, testing, deployment and hosting as
`well as application services such as team collaboration, web 15
`service integration and marshalling, database integration,
`security, scalability, stomge, persistence, state management,
`application versioning, application instrumentation and
`developer community facilitation. These services are provi(cid:173)
`sioned as an integrated solution over a network, typically the 20
`World Wide Web ('web') from a cloud computing environ(cid:173)
`ment. Taken together, Saas and PaaS are sometimes referred
`to as 'cloudwarc.'
`In adclition to SaaS and PaaS, cloud computing services
`can include many other network-based services, such as, for 25
`example, utility computing, managed services, and web ser(cid:173)
`vices. Utility computing is the practice of charging for cloud
`services like utilities, by units of time, work, or resources
`provided. A cloud utility provider can, for example, charge
`cloud clients for providing for a period oftin1e certain quan- 30
`tities of memory, I/0 support in units ofbytes transferred, or
`CPU functions in units of CPU clock cycles utilized.
`Managed services implement the transfer of all manage(cid:173)
`ment responsibility as a strategic method for improving data
`processing operations of a cloud client, person or organi7,a- 35
`tion. The person or organization that owns or has direct over(cid:173)
`sight of the organization or system being managed is referred
`to as the offerer, client, or customer. The person or organiza(cid:173)
`tion that accepts and provides the managed service from a
`cloud computing environment is regarded as a managed ser- 40
`vice provider or 'MSP.' Web services are software systems
`designed to support interoperable machine-to-machine inter(cid:173)
`action over a network of a cloud computing environment.
`Web services provide interfaces described in a machine(cid:173)
`processable format, typically the Web Services Description 45
`Language ('WSDL'). Cloud clients interact with web ser(cid:173)
`vices of a cloud computing environment as prescribed by
`WSDL descriptions using Simple Object Access Protocol
`('SOAP') messages, typically conveyed using the HyperText
`Transport Protocol ('IITTP') with an eXtensible Markup 50
`Language ('XML') serialization.
`A data center (128) is a facility that is used for housing a
`quantity of electronic equipment, particularly computers and
`communications equipment. Data centers are maintained by
`organizations for the purpose of handling the data necessary 55
`for its operations. A bank, for example, may have data centers
`where all its customers' account information is maintained
`and transactions involving the accounts are carried out. Prac(cid:173)
`tically every company that is mid-sized or larger has at least
`one data center with the larger companies often having doz- 60
`ens of data centers. A cloud computing environment imple(cid:173)
`mented with cloud computers in data centers will typically
`include many computers, although for ease of explanation,
`the cloud computing environment (192) in the example of
`FIG. l is shown with only a few (106,109). 'lbe apparatus in 65
`the example of FIG. 1 includes a data center administration
`server (118), a cloud computer (109) running a cloud operat-
`
`4
`ing system (194), several additional cloud computers (106),
`and a data communications network (100) that couples the
`computers of the cloud computing environment (192) for data
`communications.
`A 'computer' or 'cloud computer,' as the terms are used in
`this specification, refers generally to a multi-user computer
`that provides a service (e.g. database access, file transfer,
`remote access) or resources (e.g. file space) over a network
`connection. The terms 'computer' or 'cloud computer' as
`context requires, refer inclusively to the each computer's
`hardware as well as any application software, operating sys(cid:173)
`tem software, or virtual machine installed or operating on the
`computer. A computer application in this context, that is, in a
`data center or a cloud computing environment, is often an
`application program that accepts connections through a com(cid:173)
`puter network in order to service requests from users by
`sending back responses. The form factor of data center com(cid:173)
`puters is often a blade; such computers are often referred to as
`'blade servers.' Examples of application programs, often
`referred to simply as 'applications,' include file servers, data(cid:173)
`base servers, backup servers, print servers, mail servers, web
`servers, FTP servers, application servers, VPN servers,
`DHCP servers, DNS servers, WINS servers, logon servers,
`security servers, domain controllers, backup domain control(cid:173)
`lers, proxy servers, firewalls, and so on.
`The data center administration server (118) is a computer
`that is operably coupled to VMs in the cloud computing
`environment through data communications network (100).
`The data center administration server (118) provides the data
`center-level functions of communicating withhypervisors on
`cloud computers to install VMs, terminate VMs, and move
`VMs from one cloud computer to another within the data
`center. In addition, data center administration servers (in
`some embodiments support an additional module called a
`VM Manager (126) that implements direct communications
`with VMs through modules called VM agents (122) installed
`in the VMs themselves.
`The example apparatus of FIG. 1 includes a cloud operat(cid:173)
`ing system (194) implemented as a module of automated
`computing machinery installed and operating on one of the
`cloud computers (109). The cloud operating system is in turn
`composed of several submodules: a virtual machine catalog
`(180), a deployment engine (176), and a self service portal
`(172). The self service portal is so-called because it enables
`users (101) themselves to set up VMs as they wish, although
`users specifying VMs through the self service portal typically
`have no knowledge whatsoever of the actual underlying com(cid:173)
`puter hardware in the cloud computing environment-and no
`knowledge whatsoever regarding how their VMs are disposed
`upon the underlying hardware. Any particular VM can be
`installed on a cloud computer with many other VMs, all
`completely isolated from one another in operation. And all
`such VMs, from the perspective of any operating system or
`application running on a VM, can have completely diflerent
`configurntions of computer resources, CPUs, memory, I/0
`resources, and so on. Examples of cloud operating systems
`that can be adapted for use in autonomic scaling of virtual
`machines in a cloud computing environment according to
`embodinients of the present invention include VMware's
`Cloud OS™, the open-source eyeOS™ from eyeOS Forums,
`Xccrions' iCloud™, Microsoft's Windows Live Core™,
`Google's Chrome™, and gOS™ from Good OS.
`In the example cloud operating system (194) of FIG. 1, the
`self service portal (172) exposes user interface (170) for
`access by any user (101) that is authorized to install VMs in
`the cloud computing environment (192). The user may be an
`enterprise Information Technology ('IT') professional, an IT
`
`

`

`Case 6:20-cv-01152-ADA Document 1-4 Filed 12/16/20 Page 11 of 18
`
`US 8,572,612 B2
`
`5
`manager or IT administrator, setting up VMs to run applica(cid:173)
`tions to be used by dozens, hundreds, or thousands of enter(cid:173)
`prise employees. Or the user (101) may be an individual
`subscriber to cloud computing services provided through or
`from the cloud computing environment. The self service por-
`tal (172) receives through the user interface (170) user speci(cid:173)
`fications (174) ofVMs. The user specifications include for
`each VM specifications of computer resources to be provided
`as the VM, types and numbers of computer processors, quan(cid:173)
`tity of random access memory, hard disk storage, input/output 10
`resources, application programs, requirements or specifica(cid:173)
`tions for scaling, and so on. The specifications can also
`include requirements for I/0 response timing, memory bus
`speeds, Service Level Agreements ('SLAs' ), Quality Of Ser(cid:173)
`vice ('QOS') requirements, and other VM specifications as 15
`may occur to those of skill in the art.
`Having received user specifications for a VM, the cloud
`operating system (194) then deploys an instance of the now(cid:173)
`specified VM in accordance with the received user specifica(cid:173)
`tions. The self service portal (172) passes the user specifica- 20
`tion (174) to the deployment engine (176). The VM catalog
`(180) contains VM templates, standard-form descriptions
`used by hypcrvisors to define and install VMs. The deploy(cid:173)
`ment engine (176) selects a VM template (178) that matches
`the user specifications. If the user specified an Intel processor, 25
`the deployment engine selects a VM template for a VM that
`executes applications on an Jntel processor. If the user speci(cid:173)
`fied PCie I/0 functionality, the deployment engine selects a
`VM template for a VM that provides PCie bus access. And so
`on. The deployment engine fills in the selected template with 30
`the user specifications and passes the complete template
`(182) to the data center administration server (118) in the
`local data center (127). The data center administration server
`(118) then calls a hypervisor (164) on a cloud computer (110)
`to install the instance of the VM (102) specified by the 35
`selected, completed VM template. The data center adminis(cid:173)
`tration server (118) records a network address assigned to the
`new instance of the VM as well as a unique identifier for the
`new instance of the VM, here represented by a UUID, and
`returns the network address and the UUID (184) to the 40
`deployment engine (176). The deployment engine (176)
`returns the network address and the UUID (184) to the self
`service portal (172).
`In this example, the cloud operating system deploys an
`instance of a VM (102) on a particular one (110) of the cloud 45
`computers (106). For further explanation, PIG. 1 sets forth in
`a callout (111) a block diagram of some of the components of
`automated computing machinery comprised within cloud
`computer (110). The cloud computer (110) ofFIG. 1 includes
`at least one computer processor (156) or 'CPU' as well as 50
`random access memory ('RAM') (168) which is connected
`through a high speed memory bus (166) and bus adapter (158)
`to CPU (156) and to other components of the cloud computer
`(110). The example cloud computer (110) of FIG. 1 includes
`a communications adapter (167) for data communications 55
`with other computers through data communications network
`(100). Such data communications may be carried out serially
`through RS-232 connections, through external buses such as
`a Universal Serial Bus ('USB'), through data communica(cid:173)
`tions data communications networks such as IP data commu- 60
`nications networks, and in other ways as will occur to those of
`skill in the art. Communications adapters implement the hard(cid:173)
`ware level of data communications through which one com(cid:173)
`puter sends data communications to another computer,
`directly or through a data communications network.
`Examples of communications adapters useful for autonomic
`scaling of virtual machines in a cloud computing environment
`
`6
`according to embodiments of the present invention include
`modems for wired dial-up communications, Ethernet OEEE
`802.3) adapters for wired data communications network com(cid:173)
`munications, and 802 .11 adapters for wireless data commu(cid:173)
`nications network communications.
`Stored in RAM (168) in the example cloud computer (110)
`of FIG. 1 is a hypervisor (164). The hypervisor (164) is a
`mechanism of platform-virtualization, a module of auto(cid:173)
`mated computing machinery that supports multiple operating
`systems running concurrently in separate virtual machines on
`the same host computer. The hypervisor (164) in this example
`is a native or bare-metal hypervisor that is installed directly
`upon the host computer's hardware to control the hardware
`and to monitor guest operating systems (154) that execute in
`virtual machines. Each guest operating system runs on a VM
`(102) that represents another system level above the hypervi-
`sor (164) on cloud computer (110). Examples ofhypervisors
`useful or that can be improved for use in autonomic scaling of
`virtual machines in a cloud computing environment accord(cid:173)
`ing to embodiments of the present invention include IBM's
`z/VMTM, VMware's vCenter™, INTEGRlTY™ from Green
`Hills Software, LynxSecure™ from LynuxWorks, IBM's
`POWER Hypervisor (PowerVM)™, Oracle's VM Server™,
`and Sun's Logical Domains Hypervisor™.
`In the example of PIG. 1, the hypervisor (164) implements
`an instance of a VM (102) in the cloud computer (110). The
`instance of the VM (102) runs an application program (132)
`and an operating system (154). The VM (102) is