I 1111111111111111 11111 111111111111111 11111 111111111111111 111111111111111111
`
`
`
`
`US009875160B2
`
`c12) United States Patent
`Bezbaruah et al.
`
`(IO) Patent No.:
`(45) Date of Patent:
`
`US 9,875,160 B2
`*Jan. 23, 2018
`
`(54) EFFICIENTLY PROVIDING VIRTUAL
`MACHINE REFERENCE POINTS
`
`(71) Applicant: MICROSOFT TECHNOLOGY
`LICENSING, LLC, Redmond, WA
`(US)
`
`(72)
`
`Inventors: Angshuman Bezbaruah, Redmond,
`WA (US); Lars Reuther, Kirkland, WA
`(US); Taylor O'Neil Brown, Bellevue,
`WA (US); John Andrew Starks,
`Seattle, WA (US)
`
`(73) Assignee: Microsoft Technology Licensing, LLC,
`Redmond, WA (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by O days.
`
`This patent is subject to a terminal dis(cid:173)
`claimer.
`
`(21) Appl. No.: 15/219,958
`
`(22) Filed:
`
`Jul. 26, 2016
`
`(65)
`
`Prior Publication Data
`
`(58) Field of Classification Search
`CPC ........ G06F 3/0619; G06F 3/065; G06F 3/067;
`G06F 9/45558; G06F 11/1451; G06F
`2201/84
`
`(Continued)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,721,915 A
`6,411,964 Bl
`
`2/ 1998 Sockut et al.
`6/2002 Iyer et al.
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`CN
`
`103946807 A
`
`7/2014
`
`OTHER PUBLICATIONS
`
`Zhang et al., "Optimizing VM Checkpointing for Restore Perfor(cid:173)
`mance in VMware ESXi", 2013, USENIX Annual Technical Con(cid:173)
`ference, 12 pages.*
`
`(Continued)
`
`Primary Examiner - Ted T Vo
`(74) Attorney, Agent, or Firm - Workman Nydegger
`
`US 2017/0109240 Al
`
`Apr. 20, 2017
`
`(57)
`
`ABSTRACT
`
`Related U.S. Application Data
`
`(63) Continuation of application No. 14/573,976, filed on
`Dec. 17, 2014, now Pat. No. 9,430,272.
`
`(51)
`
`Int. Cl.
`G06F 9/455
`G06F 11114
`G06F 3/06
`(52) U.S. Cl.
`CPC .......... G06F 1111451 (2013.01); G06F 3/065
`(2013.01); G06F 3/067 (2013.01);
`(Continued)
`
`(2006.01)
`(2006.01)
`(2006.01)
`
`identify
`identifiers that
`A computer system maintains
`changed blocks of virtual machine (VM) storage. The com(cid:173)
`puter system accesses a stable VM checkpoint comprising a
`restorable VM image at a time, and that stores a represen(cid:173)
`tation of data of at least one block as it existed at the time.
`The computer system converts the checkpoint to a reference
`point. Reference point information is transferable with the
`VM, such that if the VM is moved to a different computing
`system, any data identified by the reference point is recov(cid:173)
`erable. The conversion includes querying the storage to
`determine an identifier corresponding to the block of the
`checkpoint at the time, storing this identifier as a part of the
`reference point, and releasing the representation of the data
`(Continued)
`
`Computer System 1Q1
`102
`103
`
`88
`I Communications Module j 104
`
`105
`
`-------
`
`---------------------
`
`StateManagemen!Module r111
`
`Replication Module
`
`r112
`
`Data Store fil
`116
`Checkpoint Data
`
`~
`~
`
`Data Storage
`Identifiers
`114
`
`' ' ' ' ' ' ,
`
`WIZ, Inc. EXHIBIT - 1071
`WIZ, Inc. v. Orca Security LTD.
`
`

`

`US 9,875,160 B2
`Page 2
`
`of the block from the checkpoint. The computer system then
`uses the reference point to identify changes in the blocks of
`the storage since the time.
`
`20 Claims, 3 Drawing Sheets
`
`(52) U.S. Cl.
`CPC ........ G06F 3/0619 (2013.01); G06F 9/45558
`(2013.01); G06F 2201/84 (2013.01)
`( 58) Field of Classification Search
`USPC
`....................................... 717/101-103; 718/1
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`6,795,966 Bl
`6,981,114 Bl
`7,162,662 Bl
`7,249,262 B2
`7,328,366 B2
`7,447,854 Bl
`7,516,286 Bl
`7,519,858 B2
`7,574,459 B2
`7,860,837 B2
`7,865,470 B2
`8,055,613 Bl
`8,145,601 B2
`8,335,902 Bl
`8,356,013 B2
`8,356,148 B2
`8,443,166 B2
`8,463,747 B2
`8,463,749 B2
`8,468,609 B2
`8,538,919 Bl
`8,560,855 B2
`8,712,970 Bl
`8,726,127 B2
`8,726,273 B2
`8,751,515 Bl
`8,782,086 B2
`8,788,773 B2
`8,805,788 B2
`8,813,204 B2
`8,966,341 B2
`9,081,714 B2
`9,081,715 B2
`9,086,994 B2
`9,130,957 B2
`9,276,980 B2
`9,430,272 B2 *
`2008/0126445 Al
`2008/0140963 Al
`2010/0049929 Al
`2010/0228913 Al
`2010/0262585 Al
`2010/0274980 Al
`2011/0167196 Al
`2012/0072659 Al
`2013/0067179 Al
`2013/0254479 Al
`2013/0290782 Al
`
`9/2004 Lim et al.
`12/2005 Wu et al.
`1/2007 Svarcas et al.
`7/2007 Hauck et al.
`2/2008 Michelman
`11/2008 Cannon
`4/2009 Dalai et al.
`4/2009 Korlepara
`8/2009 Sen et al.
`12/2010 Ali et al.
`1/2011 Fries et al.
`11/2011 Mu et al.
`3/2012 Zizys et al.
`12/2012 Feathergill
`1/2013 Fachan et al.
`1/2013 Popovski et al.
`5/2013 Czezatke et al.
`6/2013 Wahlert et al.
`6/2013 Zizys et al.
`6/2013 Leggette
`9/2013 Nielsen et al.
`10/2013 Resch
`4/2014 Sim-Tang
`5/2014 Grube et al.
`5/2014 Le
`6/2014 Xing et al.
`7/2014 Resch
`7/2014 Goodman et al.
`8/2014 Gross, IV et al.
`8/2014 Leggette
`2/2015 Grube et al.
`7/2015 Grube et al.
`7/2015 Grube et al.
`7/2015 Resch
`9/2015 Y amaura et al.
`3/2016 Chan
`8/2016 Bezbaruah
`5/2008 Michelman
`6/2008 Thomason et al.
`2/2010 Nagarkar et al.
`9/2010 Czezatke et al.
`10/2010 Rosikiewicz et al.
`10/2010 Stringham et al.
`7/2011 Scales et al.
`3/2012 Wade et al.
`3/2013 Paleologu et al.
`9/2013 Czezatke et al.
`10/2013 Chen et al.
`
`.......... G06F 9/45558
`
`2014/0025913 Al
`2014/0040572 Al
`2014/0164722 Al
`2014/0236892 Al
`2014/0250347 Al
`2014/0337666 Al
`2014/0337932 Al
`2016/0179568 Al
`2016/0203052 Al
`
`1/2014 Fuente et al.
`2/2014 Kotagiri et al.
`6/2014 Garthwaite et al.
`8/2014 Blyler
`9/2014 Grube et al.
`11/2014 Resch et al.
`11/2014 Leggette et al.
`6/2016 Bezbaruah
`7/2016 Starksetal.
`
`OTHER PUBLICATIONS
`
`"Notice of Allowance Issued in U.S. Appl. No. 14/573,976", dated
`Jun. 7, 2016, 7 Pages.
`"Notice of Allowance Issued in U.S. Appl. No. 14/595,047", dated
`Mar. 14, 2016, 11 Pages.
`"Notice of Allowance Issued in U.S. Appl. No. 14/595,047", dated
`May 16, 2016, 9 Pages.
`"International Search Report and Written Opinion Issued in PCT
`Application No. PCT/US2015/063565", dated Jun. 30, 2016, 20
`Pages.
`"International Search Report and Written Opinion Issued in PCT
`Application No. PCT/US2015/067761", dated May 24, 2016, 12
`Pages.
`in
`Ramdas, Aashish, "Resynchronization of Virtual Machines
`Hyper-V Replica", Retrieved
`from: <<https://blogs.technet.
`microsoft.com/virtualization/2013/05/ 10/resynchronization-of-vir(cid:173)
`tual-machines-in-hyper-v-replica/> >, May 10, 2013, 5 Pages.
`Garg, et al., "Checkpoint-Restart
`for a Network of Virtual
`Machines", In Proceedings of International Conference on Cluster
`Computing, IEEE, Sep. 23, 2013, 8 Pages.
`Park, et al., "Fast and Space-Efficient Virtual Machine Checkpoint(cid:173)
`ing", In Proceedings of the 7th ACM SIGPLAN/SIGOPS Interna(cid:173)
`tional Conference on Virtual Execution Environments, Mar. 9, 2011,
`pp. 75-85.
`Sinofsky, Steven, "Building the next generation file system for
`Windows: ReFS", Retrieved from: <<https://blogs.msdn.microsoft.
`com/b8/2012/0 l/ 16/building-the-next-generation-file-system-for(cid:173)
`windows-refs/> >, Jan. 17, 2012, 45 Pages.
`Werneburg, Ken, "VMware vSphere® Replication 5.5 Overview:
`Simple and Effective Virtual Machine Protection", In Technical
`White Paper, Nov. 16, 2013, 14 Pages.
`"Advanced Restore-Virtual Server Agent for VMware", Retrieved
`from:
`< <https://documentation.commvault.com/commvault/v 10/
`article?p~products/vs_ vmware/restore_adv.htrn>>, Retrieved on:
`Nov. 7, 2014, 32 Pages.
`"Changed Block Tracking (CBT) on Virtual Machines (1020128)",
`Retrieved from: < <https://kb. vmware.com/ selfservice/microsites/
`search.do?cmd~displayKC&docType~kc&externalid~ 1020128
`&sliceid~l&docTypeID~DT_KB_l_l&dialogID~l55782394
`&stateid~0 0 160353152>>, Retrieved on: Nov. 7, 2014, 2 Pages.
`"High Availability and Data Protection with EMC Isilon Scale-Out
`NAS", In White Paper, Nov. 2013, 36 Pages.
`"What's New in Hyper-V for Windows Server 2012", Retrieved
`from:
`<<https://technet.microsoft.com/en-us/library/
`hh831410(vc=ws.ll).aspx>>, Feb. 29, 2012, 6 Pages.
`"Non Final Office Action Issued in U.S. Appl. No. 14/573,976",
`dated Dec. 3, 2015, 10 Pages.
`"International Preliminary Report on Patentability Issued in PCT
`Application No. PCT/US2015/063565", dated Mar. 24, 2017, 14
`Pages.
`Notice of Allowance dated Oct. 6, 2016 cited in U.S. Appl. No.
`14/595,047.
`"SecondWritten Opinion Issued in PCT Application No. PCT/
`US2015/063565", dated Jan. 13, 2017, 12 Pages.
`
`* cited by examiner
`
`

`

`Checkpoint Accessing Module I-107 I ~ I ) Data Storage
`
`114
`___J
`Identifiers
`
`0 ....
`
`QO
`
`N
`'"~
`N
`?
`~
`~
`
`~ =
`
`~
`
`~
`~
`~
`•
`00
`
`e •
`
`Storage Data
`I
`""l. 117
`118
`I \ ~ State Data )
`Checkpoint Data
`5116
`Data Store 115
`
`I
`
`I
`
`I
`
`106
`
`Checkpoint Generating Module 1105 I
`
`1104
`
`Communications Module
`
`103
`
`102
`
`Processor B
`
`Computer System 101
`
`I
`
`~~
`100'-J..
`
`= = N
`
`"'""'
`0--,
`__ u.
`-....l
`00
`_."-0
`
`d r.,;_
`
`....
`....
`('D .....
`=- ('D
`
`~
`
`0
`
`rJJ
`
`,, /,, /
`
`I
`I
`I
`I
`I
`I
`I
`I
`
`r112
`
`Replication Module
`
`I
`
`Figure 1
`
`State Management Module 1111 I
`
`VM ~~~ence ~ ~ 1~ _____ ~ _________________
`
`Identifiers
`
`Data
`
`114
`
`VM Reference Point r109 L--1 Storage
`
`Generating Module
`
`1
`
`r108
`
`Query Generating Module
`
`

`

`U.S. Patent
`
`Jan.23,2018
`
`Sheet 2 of 3
`
`US 9,875,160 B2
`
`220,
`
`230,
`
`310,
`
`320,
`
`330,
`
`Access Stable Virtual Machine Checkpoint That Includes
`Underlying Data Stored In Data Storage
`
`1 '
`
`Query Data Storage To Determine Data Storage Identifiers That
`Reference The Point In Time Associated With The Checkpoint
`
`1 '
`
`Store Determined Data Storage Identifiers
`As Virtual Machine Reference Point
`
`Figure 2
`
`Establish Stable, Unchanging State Within Virtual Machine
`
`1 '
`
`Access Previously Generated Reference Points To Identify
`Differences In Virtual Machine State
`
`1 '
`
`Replicate Differences In Virtual Machine State Between Current Stable
`State And Selected Past Stable Point In Time
`
`Figure 3
`
`

`

`0--, = = N
`
`"'""'
`-....l __ u.
`00
`_."-0
`
`d r.,;_
`
`Point
`
`VM Reference 412
`
`0 ....
`('D .....
`rJJ =- ('D
`
`~
`
`~
`
`0 ....
`
`QO
`
`N
`'"~
`N
`?
`~
`~
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`I
`
`I
`
`403
`
`Identifiers
`409
`~
`Local Disk
`
`401.r
`~ Data
`
`Backup Generating Module r411
`
`I
`
`402
`
`405
`
`I Processor I B
`
`Computer System 420
`
`r404 <==>
`
`Figure 4
`
`r407
`
`Merged Backup
`
`r406
`
`Differential
`
`r410
`
`Full Backup
`
`)401
`
`Initial Backup
`
`~ Raw Data
`
`~ 4~~
`
`Data Store 408
`
`

`

`US 9,875,160 B2
`
`1
`EFFICIENTLY PROVIDING VIRTUAL
`MACHINE REFERENCE POINTS
`
`CROSS REFERENCE TO RELATED
`APPLICATION
`
`This application is a continuation of U.S. patent applica(cid:173)
`tion Ser. No. 14/573,976, filed Dec. 17, 2014, and entitled
`"EFFICIENTLY PROVIDING VIRTUAL MACHINE
`REFERENCE POINTS," the entire contents of which are 10
`incorporated by reference herein in their entirety.
`
`BACKGROUND
`
`2
`computer system accesses previously generated reference
`points to identify differences
`in virtual machine state
`between the current stable state and a selected past stable
`point in time. The computer system also replicates the
`5 differences in virtual machine state between the current
`stable state and a selected past stable point in time. The
`differences may be replicated to a data storage device as an
`incremental backup, or may be used for remote replication
`or disaster recovery purposes.
`This Summary is provided to introduce a selection of
`concepts in a simplified form that are further described
`below in the Detailed Description. This Summary is not
`intended to identify key features or essential features of the
`15 claimed subject matter, nor is it intended to be used as an aid
`in determining the scope of the claimed subject matter.
`Additional features and advantages will be set forth in the
`description which follows, and in part will be apparent to
`one of ordinary skill in the art from the description, or may
`20 be learned by the practice of the teachings herein. Features
`and advantages of embodiments described herein may be
`realized and obtained by means of the instruments and
`combinations particularly pointed out in the appended
`claims. Features of the embodiments described herein will
`25 become more fully apparent from the following description
`and appended claims.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`Computing systems have become ubiquitous, ranging
`from small embedded devices to phones and tablets to PCs
`and backend servers. Each of these computing systems is
`designed to process software code. The software allows
`users to perform functions, interacting with the hardware
`provided by the computing system. In some cases, these
`computing systems allow users to establish and run virtual
`machines. These virtual machines may provide functionality
`not provided by the host operating system, or may comprise
`a different operating system altogether. In this manner,
`virtual machines may be used to extend the functionality of
`the computing system. Virtual machines may be backed up
`on virtual storage devices which themselves may be backed
`up to physical or virtual storage devices. Virtual machine
`hosts may also be configured to take snapshots which
`represent point-in-time images of the virtual machine. The 30
`VM snapshots or "checkpoints" include CPU state, memory
`state, storage state and other information necessary to com(cid:173)
`pletely recreate or restore the virtual machine to that point in
`time.
`
`To further clarify the above and other features of the
`embodiments described herein, a more particular description
`will be rendered by reference to the appended drawings. It
`is appreciated that these drawings depict only examples of
`the embodiments described herein and are therefore not to
`35 be considered limiting of its scope. The embodiments will be
`described and explained with additional specificity and
`detail through the use of the accompanying drawings in
`which:
`FIG. 1 illustrates a computer architecture
`in which
`embodiments described herein may operate including estab(cid:173)
`lishing efficient virtual machine reference points.
`FIG. 2 illustrates a flowchart of an example method for
`establishing efficient virtual machine reference points.
`FIG. 3 illustrates a flowchart of an example method for
`specifying a virtual machine reference point to query incre(cid:173)
`mental changes.
`FIG. 4 illustrates a computer architecture
`in which
`embodiments may operate including specifying a virtual
`machine reference point to query incremental changes.
`
`BRIEF SUMMARY
`
`Embodiments described herein are directed to establish-
`ing efficient virtual machine reference points and to speci(cid:173)
`fying a virtual machine reference point to query incremental 40
`changes. As used herein, virtual machine reference points
`allow computer systems to identify incremental changes
`from specific points in time on. For example, in one embodi(cid:173)
`ment, a computer system accesses a stable virtual machine
`checkpoint that includes portions of underlying data stored 45
`in data storage, where the checkpoint is associated with a
`specific point in time. The computer system then queries the
`data storage to determine data storage identifiers that refer(cid:173)
`ence the point in time associated with the checkpoint and
`stores the determined data storage identifiers as a virtual 50
`machine reference point or virtual machine reference point
`artifacts, where each subsequent change to the data storage
`results in an update to the data storage identifier, so that
`virtual machine reference point is usable to identify incre(cid:173)
`mental changes from specific points in time on. Virtual 55
`machine reference point artifacts allow for cases where a
`virtual machine has two ( or more) virtual disks. Each virtual
`disk may have a different identifier for the same point in
`time, and the reference point artifact allows the computer
`system to associate both of those points in time as a common 60
`point. This will be explained further below.
`In another embodiment, a computer system performs a
`method for specifying a virtual machine reference point to
`query incremental changes. The computer system estab(cid:173)
`lishes a stable, unchanging state within a virtual machine, 65
`where the stable state is associated with a checkpoint that
`includes corresponding state data and storage data. The
`
`DETAILED DESCRIPTION
`
`Embodiments described herein are directed to establish(cid:173)
`ing efficient virtual machine reference points and to speci(cid:173)
`fying a virtual machine reference point to query incremental
`changes. In one embodiment, a computer system accesses a
`stable virtual machine checkpoint that includes portions of
`underlying data stored in data storage, where the checkpoint
`is associated with a specific point in time. The computer
`system then queries the data storage to determine data
`storage identifiers that reference the point in time associated
`with the checkpoint and stores the determined data storage
`identifiers as a virtual machine reference point, where each
`subsequent change to the data storage results in an update to
`the data storage identifier, so that virtual machine reference
`point is usable to identify incremental changes from specific
`points in time on.
`
`

`

`US 9,875,160 B2
`
`3
`In another embodiment, a computer system performs a
`method for specifying a virtual machine reference point to
`query incremental changes. The computer system estab(cid:173)
`lishes a stable, unchanging state within a virtual machine,
`where the stable state is associated with a checkpoint that 5
`includes corresponding state data and storage data. The
`computer system accesses previously generated reference
`points
`to identify differences
`in virtual machine state
`between the current stable state and a selected past stable
`point in time. The computer system also replicates the 10
`differences in virtual machine state between the current
`stable state and a selected past stable point in time. The
`differences may be replicated to a data storage device as an
`incremental backup, or may be used for remote replication
`or disaster recovery purposes.
`The following discussion now refers to a number of
`methods and method acts that may be performed. It should
`be noted, that although the method acts may be discussed in
`a certain order or illustrated in a flow chart as occurring in
`a particular order, no particular ordering is necessarily 20
`required unless specifically stated, or required because an act
`is dependent on another act being completed prior to the act
`being performed.
`Embodiments described herein may implement various
`types of computing systems. These computing systems are 25
`now increasingly taking a wide variety of forms. Computing
`systems may, for example, be handheld devices such as
`smartphones or feature phones, appliances, laptop comput-
`ers, wearable devices, desktop computers, mainframes, dis(cid:173)
`tributed computing systems, or even devices that have not 30
`conventionally been considered a computing system. In this
`description and in the claims, the term "computing system"
`is defined broadly as including any device or system ( or
`combination thereof) that includes at least one physical and
`tangible processor, and a physical and tangible memory 35
`capable of having thereon computer-executable instructions
`that may be executed by the processor. A computing system
`may be distributed over a network environment and may
`include multiple constituent computing systems.
`As illustrated in FIG. 1, a computing system 101 typically 40
`includes at least one processing unit 102 and memory 103.
`The memory 103 may be physical system memory, which
`may be volatile, non-volatile, or some combination of the
`two. The term "memory" may also be used herein to refer to
`non-volatile mass storage such as physical storage media. If 45
`the computing system is distributed, the processing, memory
`and/or storage capability may be distributed as well.
`As used herein, the term "executable module" or "execut(cid:173)
`able component" can refer to software objects, routines, or
`methods that may be executed on the computing system. The 50
`different components, modules, engines, and services
`described herein may be implemented as objects or pro(cid:173)
`cesses that execute on the computing system (e.g., as sepa(cid:173)
`rate threads).
`are 55
`follows, embodiments
`that
`In
`the description
`described with reference to acts that are performed by one or
`more computing systems. If such acts are implemented in
`software, one or more processors of the associated comput-
`ing system that performs the act direct the operation of the
`computing system in response to having executed computer-
`executable instructions. For example, such computer-ex(cid:173)
`ecutable instructions may be embodied on one or more
`computer-readable media that form a computer program
`product. An example of such an operation involves the
`manipulation of data. The computer-executable instructions 65
`(and the manipulated data) may be stored in the memory 103
`of the computing system 101. Computing system 101 may
`
`4
`also contain communication channels that allow the com(cid:173)
`puting system 101 to communicate with other message
`processors over a wired or wireless network.
`Embodiments described herein may comprise or utilize a
`special-purpose or general-purpose computer system that
`includes computer hardware, such as, for example, one or
`more processors and system memory, as discussed in greater
`detail below. The system memory may be included within
`the overall memory 103. The system memory may also be
`referred to as "main memory", and includes memory loca(cid:173)
`tions that are addressable by the at least one processing unit
`102 over a memory bus in which case the address location
`is asserted on the memory bus itself. System memory has
`15 been traditionally volatile, but the principles described
`herein also apply in circumstances in which the system
`memory is partially, or even fully, non-volatile.
`Embodiments within the scope of the present invention
`also include physical and other computer-readable media for
`carrying or storing computer-executable instructions and/or
`data structures. Such computer-readable media can be any
`available media that can be accessed by a general-purpose or
`special-purpose
`computer
`system. Computer-readable
`media that store computer-executable
`instructions and/or
`data structures are computer storage media. Computer(cid:173)
`readable media that carry computer-executable instructions
`and/or data structures are transmission media. Thus, by way
`of example, and not limitation, embodiments of the inven(cid:173)
`tion can comprise at least two distinctly different kinds of
`computer-readable media: computer storage media and
`transmission media.
`Computer storage media are physical hardware storage
`media that store computer-executable
`instructions and/or
`data structures. Physical hardware storage media include
`computer hardware, such as RAM, ROM, EEPROM, solid
`state drives ("SSDs"), flash memory, phase-change memory
`("PCM"), optical disk storage, magnetic disk storage or
`other magnetic storage devices, or any other hardware
`storage device(s) which can be used to store program code
`in the form of computer-executable
`instructions or data
`structures, which can be accessed and executed by a general-
`purpose or special-purpose computer system to implement
`the disclosed functionality of the invention.
`Transmission media can include a network and/or data
`links which can be used to carry program code in the form
`of computer-executable instructions or data structures, and
`which can be accessed by a general-purpose or special(cid:173)
`purpose computer system. A "network" is defined as one or
`more data links that enable the transport of electronic data
`between computer systems and/or modules and/or other
`electronic devices. When information is transferred or pro-
`vided over a network or another communications connection
`( either hardwired, wireless, or a combination of hardwired
`or wireless) to a computer system, the computer system may
`view the connection as transmission media. Combinations of
`the above should also be included within the scope of
`computer-readable media.
`Further, upon reaching various computer system compo(cid:173)
`nents, program code in the form of computer-executable
`instructions or data structures can be transferred automati(cid:173)
`cally from transmission media to computer storage media
`(or vice versa). For example, computer-executable instruc(cid:173)
`tions or data structures received over a network or data link
`can be buffered in RAM within a network interface module
`( e.g., a "NIC"), and then eventually transferred to computer
`system RAM and/or to less volatile computer storage media
`at a computer system. Thus, it should be understood that
`
`60
`
`

`

`US 9,875,160 B2
`
`5
`computer storage media can be included in computer system
`components that also ( or even primarily) utilize transmission
`media.
`Computer-executable instructions comprise, for example,
`instructions and data which, when executed at one or more
`processors, cause a general-purpose computer system, spe(cid:173)
`cial-purpose computer system, or special-purpose process-
`ing device to perform a certain function or group of func(cid:173)
`tions. Computer-executable
`instructions may be,
`for
`example, binaries, intermediate format instructions such as
`assembly language, or even source code.
`Those skilled in the art will appreciate that the principles
`described herein may be practiced in network computing
`environments with many types of computer system configu(cid:173)
`rations, including, personal computers, desktop computers,
`laptop computers, message processors, hand-held devices,
`multi-processor systems, microprocessor-based or program(cid:173)
`mable consumer electronics, network PCs, minicomputers,
`mainframe computers, mobile telephones, PDAs, tablets,
`pagers, routers, switches, and the like. The invention may
`also be practiced in distributed system environments where
`local and remote computer systems, which are linked ( either
`by hardwired data links, wireless data links, or by a com(cid:173)
`bination of hardwired and wireless data links) through a
`network, both perform tasks. As such, in a distributed system
`environment, a computer system may include a plurality of
`constituent computer systems. In a distributed system envi(cid:173)
`ronment, program modules may be located in both local and
`remote memory storage devices.
`Those skilled in the art will also appreciate that the
`invention may be practiced in a cloud computing environ(cid:173)
`ment. Cloud computing environments may be distributed,
`although this is not required. When distributed, cloud com(cid:173)
`puting environments may be distributed
`internationally
`within an organization and/or have components possessed 35
`across multiple organizations. In this description and the
`following claims, "cloud computing" is defined as a model
`for enabling on-demand network access to a shared pool of
`configurable computing resources ( e.g., networks, servers,
`storage, applications, and services). The definition of "cloud
`computing" is not limited to any of the other numerous
`advantages that can be obtained from such a model when
`properly deployed.
`Still further, system architectures described herein can
`include a plurality of independent components that each
`contribute to the functionality of the system as a whole. This
`modularity allows for increased flexibility when approach-
`ing issues of platform scalability and, to this end, provides
`a variety of advantages. System complexity and growth can
`be managed more easily through the use of smaller-scale
`parts with limited functional scope. Platform fault tolerance
`is enhanced through the use of these loosely coupled mod(cid:173)
`ules. Individual components can be grown incrementally as
`business needs dictate. Modular development also translates
`to decreased time to market for new functionality. New 55
`functionality can be added or subtracted without impacting
`the core system.
`FIG. 1 illustrates a computer architecture 100 in which at
`least one embodiment may be employed. Computer archi(cid:173)
`tecture 100 includes computer system 101. Computer sys(cid:173)
`tem 101 may be any type of local or distributed computer
`system, including a cloud computing system. The computer
`system 101 includes modules for performing a variety of
`different functions. For instance, the communications mod(cid:173)
`ule 104 may be configured to communicate with other
`computing systems. The communications module 104 may
`include any wired or wireless communication means that
`
`6
`can receive and/or transmit data to or from other computing
`systems. The communications module 104 may be config(cid:173)
`ured to interact with databases, mobile computing devices
`(such as mobile phones or tablets), embedded or other types
`5 of computing systems.
`The computer system 101 of FIG. 1 may further include
`a checkpoint generating module 105. The checkpoints 106
`(or "snapshots" herein) generated by module 105 may
`include various portions of corresponding checkpoint data
`10 116 which may be stored in a data store 115. The checkpoint
`data may include, for example, state data 117 and storage
`data 118. The state data 117 may include CPU state, memory
`state and device state for various computer system devices.
`The storage data 118 may include data files, application files,
`15 operating system files, backup files or other data associated
`with a checkpoint. As such, the checkpoint data 116 asso(cid:173)
`ciated with a checkpoint 106 includes sufficient data to
`perform a full restoration or backup from the data. The
`storage and state data for a given computer system or virtual
`20 machine, however, may include a significant amount of data.
`In some cases, multiple checkpoints may be stored for a
`single computer system or virtual machine (VM). Each
`checkpoint may be generated at a different point in time.
`Then, once two or more checkpoints have been generated,
`25 they can be compared to one another to determine the
`differences between them. These differences can be applied
`to a differential backup which only backs up the differences
`between
`the two checkpoints. Embodiments described
`herein introduce the concept of a "reference point", "virtual
`30 machine reference point" or "VM reference point." A VM
`reference point allows previous storage, memory and device
`state associated with a checkpoint to be deleted while still
`retaining the ability to create the differential backup. This is
`done by recording sequence numbers for state transitions.
`Incremental backups of virtual machines involve tracking
`those changes to virtual machine storage that have occurred
`since a previous specified point in time ( or points in time) of
`the VM. Traditionally, a point in time snapshot of a VM is
`represented by VM checkpoints. As mentioned above, stor-
`40 ing full checkpoints may introduce a lot of overhead on the
`I/O throughput of the VM, as copy-on-write (or similar)
`techniques are often used to maintain a point-in-time image
`of the virtual storage. Maintaining VM checkpoints just for
`the purpose of referring to previous points in time for
`45 querying
`incremental changes
`is wasteful
`in terms of
`resource usage and negatively impacts performance of appli(cid:173)
`cations running in a VM.
`The VM reference points described herein do not need to
`maintain exact point-in-time images of the VM state (e.g.
`50 storage, memory, device state). The VM reference points
`provide a representation of a previous instance in time of the
`VM. The reference points can be represented by unique
`identifiers
`(which can be globally unique
`identifiers
`(GUIDs), sequence numbers, or other identifiers.
`In one embodiment, a VM reference point (e.g. 110 of
`FIG. 1) is generated in the following manner: First, a
`point-in-time image ( checkpoint 106) is generated for a VM.
`This provides a stable copy to back up from. Along with
`creation of this image, a mechanism to track changes to the
`60 virtual storage is triggered. The triggering mechanism may
`be based on user interaction, input from an application or
`other source, or can be a manual trigger. Second, once the
`computer system or VM is backed up, the checkpoint is
`converted/demoted to a reference point (i.e. just a point-in-
`65 time representation that is not backed up by corresponding
`machine state). This frees up the overhead associated with a