(12) Unlted States Patent
`(10) Patent N0.:
`US 6,917,963 B1
`
`Hipp et al.
`(45) Date of Patent:
`Jul. 12, 2005
`
`U5006917963B1
`
`(54) SNAPSHOT IMAGE FOR THE APPLICATION
`STATE OF UNSHAREABLE AND
`SHAREABLE DATA
`
`(75)
`
`Inventors: Burton A. Hipp, Elk Grove, CA (US);
`Rajeev Bharadhwaj Los Altos CA
`’
`’
`(Us)
`
`_
`_
`.
`(73) ASSlgHeei VERITAS Operatlng COFPOFatIOII,
`Mountain View, CA (US)
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 693 days.
`
`(21) Appl. No.: 09/684,373
`.
`Flled:
`
`OCt' 5’ 2000
`
`(22)
`
`60
`
`)
`
`(
`
`,
`,
`. Related.U.S.Appllcat10n Data
`.
`11333512235331?nilcaljig’n :5);156077125?) 713196dfi1::11 816253
`5, 169%, prOVisionflpapplication.N0. 60/157:833, filed or;
`Oct. 5, 1999, provisional application No. 60/157,727, filed
`011 00t- 5, 1999, and PTOVISIOnal application NO- 60/157334;
`filed on Get 15’ 1999’
`Int. Cl.7 ................................................ G06F 15/16
`(51)
`(52) us. Cl.
`....................... 709/204, 709/205, 709/229,
`719/328
`(58) Field of Search ......................... 719/328 315—319
`719/312 313. 714/37_40 15_21 6' 707’/200_203’
`9
`9’ >10; 713/163:165; 769;204’ 205’ 201:
`203 225 227 229 245 215 216‘ 345/330.
`715/516' 711/6 100 155 163 200
`’
`’
`’
`’
`’
`References Cited
`US. PATENT DOCUMENTS
`
`(56)
`
`5,067,072
`5,088,031
`5,109,510
`5,365,606
`5,403,639
`5,530,795
`5,608,720
`5,649,152
`
`................ 717/148
`A * 11/1991 Talati et al.
`
`A *
`2/1992 Takasaki et al.
`..... 718/100
`4/1992 Baker et al. ............. 718/104
`A *
`
`A * 11/1994 Brocker et al. .......... 719/331
`............... 707/204
`A *
`4/1995 Belsan et al.
`A
`6/1996 Wan
`A *
`3/1997 Biegel et al.
`A *
`7/1997 Ohran et al.
`
`............... 370/249
`................ 711/114
`
`5,678,042 A * 10/1997 Pisello et al.
`................. 714/47
`
`5,734,865 A *
`3/1998 Yu ................
`709/250
`5/1998 Buchanan .........
`5,758,355 A *
`707/201
`
`
`......
`5,790,114 A *
`8/1998 Geaghan et al.
`345/763
`5,822,523 A * 10/1998 Rothschild et al.
`......... 709/236
`5,889,945 A
`3/1999 Porter et al.
`5,896,500 A
`4/1999 Ludwig et al.
`5,909,545 A *
`6/1999 Frese et al.
`................. 709/208
`
`6,131,148 A * 10/2000 West et al.
`.
`711/162
`1/2001 Hickman .................... 709/235
`6,173,332 B1 *
`.
`(Contmued)
`Primary Examiner—William C. Vaughn, Jr.
`(74) Attorney, Agent, or Firm—Lawrence J. Merkel;
`Meyertons, Hood, KiVlin, Kowert & Goetzel, PC.
`
`(57)
`
`ABSTRACT
`
`A system, methods, and computer program products are
`disclosed for creating Virtual application templates for the
`purpose of propagating a single application snapshot into
`multiple, distinct images. Snapshot Virtual templates allow
`multiple application instances to use the same fixed resource
`identifier by making the resource identifier Virtual, privatiz-
`ing it, and dynamically mapping it
`to a unique system
`resource identifier. When a snapshot is cloned from a Virtual
`template, the common or shared data is used exactly as is,
`whereas the non-sharable data is either copied-on-write,
`multiplexed, Virtualized, or customized-on-duplication~ The
`present invention notes access to modified resources, fixed
`system IDs/keys and unique process-related identifies and
`automatically inserts a level of abstraction between these
`resources and the application. The resources contained in a
`snapshot Virtual template can be dynamically redirected at
`restore time. Access to memory and storage is managed in
`a copy-on-write fashion. System resource handles are man-
`aged in a Virtualize-on-allocate fashion or by a multiplex-
`on-access mechanism. Process-unique resources are man-
`aged in a redirect-on-duplicate fashion. Rules may be
`defined through an application configurator that allows some
`degree of control over the creation of non-sharable data. The
`snapshot Virtual template is constructed by dividing the
`snapshot image into sharable and non-sharable data. Knowl-
`edge of which system resources can be shared is encoded in
`the application snapshot/restore framework.
`
`16 Claims, 15 Drawing Sheets
`
`
`
`
`
` 1
`
`235
`
`2
`
`m
`
`VEEAM 101 l
`
`IPR of US. Patent No. 7,093,086
`
`(ALLPROCESSES
`MOWSUSPENDED)
`254
`FOR EACH PROCESS
`
`
`
`263
`
`
`
`RocEssEsl
`
`(CHECK FOR MDRE
`
`SUS’BID PROCESS
`
`
`
`7.50
`
`v 252i
`
`
`15 STATE
`CDNSISTENTAND
`V55
`ARETHREADS
`QMESCEW
`YABNN LATE
`REMOVE REG/STEM
`
`W)
`M
`GLOBALSTATE
` 272
`
`
` RETRTEVE No (1.: SEMAPHORES)
`
`
` TERMINATE PROCESS
`
`STATE
`RESUMEANV
`
`SUSPENDED
`
`
`
`PROCESSES
`
`
` [ DUMP STATE
`
`
`TATECHANGED
`5100235551
`STNCE LAST
`“56N0
`
`
`\275
`DELTA
`
`

`

`US 6,917,963 B1
`
`Page 2
`
`US. PATENT DOCUMENTS
`
`6,205,450 B1 *
`3/2001 Kanome ..................... 707/203
`6,212,531 B1 *
`4/2001 Blea et a1.
`.................. 707/204
`6,269,431 B1 *
`. 711/162
`7/2001 Dunham .....
`
`6,324,690 B1 * 11/2001 Luu ........................... 717/177
`
`6,363,421 B2 *
`6,385,613 B1 *
`6,442,663 B1 *
`6,567,849 B2 *
`
`............... 709/223
`3/2002 Barker et a1.
`................ 707/8
`5/2002 Grewell et a1.
`8/2002 Sun et a1.
`................... 711/202
`5/2003 Ludovici et a1.
`............ 709/223
`
`* Cited by examiner
`
`

`

`US. Patent
`
`Jul. 12, 2005
`
`Sheet 1 0f 15
`
`US 6,917,963 B1
`
`VNE 1
`VN: 10.10.01
`VM: 255.255.255.255
`
`110
`
`150-1
`
`150-2
`
`“2
`
`VNE
`VN:10.10.2?0
`VM: 255.255.2550
`
`150-n
`
`VN: VIRTUAL SUBNET
`VM: VIRTUAL NETMASK
`
`10.10.22
`
`APP2
`
`FIREWALL
`
`A\
`106
`
`104
`
`100
`
`GLOBAL VIRTUAL
`ADDRESS SPACE:
`10.10.00
`
`E
`R MOTE USER
`
`1024
`
`1022
`
`REMOTE USER
`
`FIG. 1
`
`

`

`US. Patent
`
`Jul. 12, 2005
`
`Sheet 2 0f 15
`
`US 6,917,963 B1
`
`950
`
`MEMORY
`
`1 50
`
`PO! NTING
`
`DEVICE
`
`164
`
`DISK DRIVE
`
`152
`
`158
`
`"52
`
`166
`
`168
`
`KEYBOARD
`
`MONITOR
`
`CD ROM
`
`170
`
`‘L
`
`172
`
`FIG. 2
`
`

`

`US. Patent
`
`Jul. 12, 2005
`
`Sheet3 0f15
`
`US 6,917,963 B1
`
`ozzémao
`
`:2555%
`
`wmoEmEZ
`
`EE2:672
`
`$2EfimE502
`
`ammozzoz
`
`m.o_.._
`
`szmmmo
`
`5.85
`
`hzmfizomSzm
`
`#553
`
`Eém:ix
`
`
`
`zoEwonEEE#35:
`
`Ex<zm
`
`wmuzzoflm
`
`

`

`US. Patent
`
`Jul. 12, 2005
`
`Sheet 4 0f 15
`
`US 6,917,963 B1
`
`
`
`206
`
`
`
`SYSTEM
`
`OPERATING
`
`
`VIRTUAL
`
`
`ENVIRONMENT
`
`
`OPERATING
`210
`
`
`SYSTEM API
`200
`INTERFACES
`
`APPShot
`FRAMEWORK
`
`APPLICATION1
`
`' APPLICATION
`
`SHARED
`2
`RESOURCES
`
`
`
`
`
`
`PROCESS
`
`
`
`MANAGEMENT
`SUBSYSTEM
`
`
`
`214
`
`PRELOAD
`LIBRARY
`
`IMAGE OF
`PROCESS
`
`SAVED
`
`
`5'
`
`
`PROCESS STATE IS RETRIEVED USING
`SEVERAL DIFFERENT METHODS:
`a) VIA THE INTERPOSED PRELOAD LIBRARY
`
`b) PROCESS MANAGEMENT INTERFACES
`C) AppShot FRAMEWORK
`d) STANDARD API CALLS
`
`
`
`
`
`

`

`US. Patent
`
`Jul. 12, 2005
`
`Sheet 5 0f 15
`
`US 6,917,963 B1
`
`254
`
`250
`
`REQUEST SNAPSHOT
`
`PASS IN
`AID
`
`AID
`FOUND?
`
`.
`
`YES
`
`252
`
`255
`FIND A PROCESS
`WITH AID THAT
`
`MATCHES ai?
`
`
`N0
`
`(ALL PROCESSES
`NOW SUSPENDED)
`
`264
`
`FOR EACH PROCESS...
`
`
`265
`
`FOR EACH TYPE OF STATE...
`
`
`
`(NEXT
`
`PROCESSES)
`
`258
`
`SUSPEND PROCESS
`
`260
`
`280
`
`YES
`
`MORE
`PROCESSES?
`
`NO
`
`
`
`268
`
`
`
`
`
`
`
`IS STATE
`CONSISTENTAND
`
`
`ARE THREADS
`QUIESCED?
`28
`
`
`(TRY AGAIN LATER)
`REMOVE REGISTERED
`GLOBAL STATE
`
`2
`
`
`
`NO
`
`
`RESUME ANY
`SUSPENDED
`PROCESSES
`
`
`
`V‘RTUAL‘ZED?
`
`NO 278
`FETRIEVE
`
`
`RETRIEVE
`STATE
`V RTUALIZED
`
`STATE
`
`
`STATE CHANGED
`
`SINCE LAST
`
`SNAPSHOT?
`
`
`
`272
`
`274
`
`YES
`
`DUMP STATE
`DELTA
`
`275
`
`(Le. SEMAPHORES)
`
`TERMINATE PROCESS
`
`MORE
`PROCESSES?
`
`NO
`
`DONE
`
`284
`
`235
`
`FIG. 5
`
`

`

`US. Patent
`
`Jul. 12, 2005
`
`Sheet 6 0f 15
`
`US 6,917,963 B1
`
`FAILED
`
`RESTORE
`
`30°
`
`AID
`
`RECREATE GLOBAL OR
`SHARED STATE
`
`308
`
`NOT FOUND Fofi'fim
`
`302
`
`FOUND ai
`
`304
`
`FIND SNAPSHOT
`IMAGE WITH
`
`(CHECK FOR MORE
`IMAGES)
`
`FOUND I
`
`
`
`
`
`FOR EACH IMAGE...
`
`CREATE PROCESS
`(INHERITING GLOBALI
`
`SHARED STATE)
`
`ISOLATE PROCESS TO PREVENT
`INTER-PROCESS STATE CHANGES
`
`310
`
`312
`
`FOR EACH TYPE OF STATE...
`
`314
`
`NO MORE
`PROCESSES
`328
`No
`
`MORE
`IMAGES?
`
`
`
`
`
`
`FIND GLOBAL/SHARED
`STATE IN i
`
`306
`
`PUT THIS PROCESS
`IN SYNCHRONIZED
`WAIT-
`
`330
`
`REMOVE TRACES AND
`STATE INDUCED DURING
`RESTORE OF THE PROCESS N
`
`0
`
`E
`
`BIND SYSTEM STATE
`TO VIRTUAL DEFINITION
`
`332
`
`SYNCHRONIZED RESUME
`OF ALL PROCESSES
`
`FIG. 6
`
`
`
`
`
`
`
`SHARED STATE WITH
`ANOTHER PROCESS?
`
`(Le. PIPE)
`
`
`320
`
`YES
`322
`
`RECONNECT SHARED
`STATE TO OTHER PROCESS
`
`

`

`US. Patent
`
`Jul. 12, 2005
`
`Sheet 7 0f 15
`
`US 6,917,963 B1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`MMAP MEMORY INFO
`
`
`
`
`
`
`
`
`
`LIGHTWEIGHT PROCESS INFO
`
`
`FIG. 7
`
`

`

`US. Patent
`
`Jul. 12, 2005
`
`Sheet 8 0f 15
`
`US 6,917,963 B1
`
`APPLICATION
`LAUNCHED
`
`FOR EVERY RESOURCE
`
`
`
`(NEXT RESOURCE REQUEST)
`REQUEST...
`
`
`
`
`
`RECORD RULE
`WITH RESOURCE
`
`
`
`
`
`RULE FOR
`
`
`RESOURCE AND
`
`
`ACCESS TYPE
`FOUND?
`
`
`
` CHECK RESOURCE
`
`
`360
`
`3164
`
`MARK RESOURCE
`AS "SHARED"
`
`
`
`EXTERNALLY
`EXPORTED
`
`
`MARK FOR
`
`MULTIPLEX
`
`
`
`
`
`
`
`MARK FOR
`MARK FOR
`
`ALLOCATION
`NETWORK WIDE
`VIRTUALIZE
`
`UNIQUE?
`
`
`390
`
`NO
`PROCESS RESOURCE
`REQUEST
`
`FIC3.53
`
`

`

`US. Patent
`
`Jul. 12, 2005
`
`Sheet 9 0f 15
`
`US 6,917,963 B1
`
`
`
`RESTOREAPPLJCATION
`(CREATE CLONE)
`
`FOR EVERY RESOURCE...
`
`400
`
`RULE FOR
`RESOURCE AND
`
`
`
`NO RULE
`
`{NEXT RESOURCE)
`
`
`
`
`
`
`
`402
`
`RECREATE
`RESOURCE
`
`406
`
`
`
`
`COPY‘ON'WR'TE?
`
`
`
`YES
`
`ALLOCATE SYSTEM
`414
`RESOURCE
`
`
`
`COPY ORIGINAL
`RESOURCE FROM
`
`
`
`
`
`
`RRRIRTERIRGRIIG
`RESOURCE AND COPY-ON-
`WRITE IMAGE
`
`
`
`
`
`
`
`
`412
`
`YE
`
`S
`
`No
`
`418
`
`BIND SYSTEM RESOURCE
`TO SAVED RESOURCE IN
`TEMPLATE
`REGISTER CLONED
`INSTANCE WITH
`INTENDED CLIENTS
`
`416
`
`ALLOCATE NETWORK
`RESOURCE
`
`
`
`
`420
`
`426
`
`NETWORK
`ALLOCATED?
`
`YES
`
`
`
`
`
`
`N0
`422
`424
`ADD TRANSLATION TO SAVED
`
`
`NETWORK RESOURCE
`
`
`FIG. 9
`
`

`

`US. Patent
`
`Jul. 12, 2005
`
`Sheet 10 0f 15
`
`US 6,917,963 B1
`
`206
`
`
`OPERATING SYSTEM
`
`
`
`AppShot
`HARNESS
`AppShot
`(PROCESSX)
`FRAMEWORK
`
`
`
`
`
`502
`
`
`
`ai VIRTUAL RESOURCE
`TRANSLATION TABLES
`SEMAPHORES
`SHARED MEMORY
`ESE
`
`
`
`APPLICATION
`IPROCESSy)
`PROCESS ID
`
`
`l\514
`_ROCESSySTATE“11_R0CESSXSTATE
`
`
`
`
`518
`
`FIG. 10
`
`

`

`US. Patent
`
`Jul. 12, 2005
`
`Sheet 11 0f 15
`
`US 6,917,963 B1
`
`206
`
`OPERATING SYSTEM KERNEL
`
`
`(PROCESSy)
`
`
`PROCESS STATE
`
`APPLICATION
`
`
`
`AppShot FRAMEWORK
`
`
`
`SYSTEM
`SEMAPHORE
`
`POOL
`
`
`
`
`FIG. 11
`
`

`

`US. Patent
`
`Jul. 12, 2005
`
`Sheet 12 0f 15
`
`US 6,917,963 B1
`
`203
`
`206
`
`OPERATING SYSTEM KERNEL
`
`AppShot FRAMEWORK
`
`
`APPLICATION
`(PROCESSY)
`
`
`
`
`PROCESS STATE
`
`
`
`VIRTUAL RESOURCE
`
`TRANSLATION TABLES FOR 8;
`SEMAPHORES
`PROCESS ID
`
`
`
`
`
`
`OS SEMAPHORE
`IMPLEMENTATION
`
`
`FIG. 12
`
`

`

`US. Patent
`
`Jul. 12, 2005
`
`Sheet 13 0f 15
`
`US 6,917,963 B1
`
`208
`
`206
`
`OPERATING SYSTEM KERNEL
`
`AppShot FRAMEWORK
`
`
`APPLICATION
`
` (PROCESSy)
`
`
` VIRTUAL RESOURCE
`
`TRANSLATION TABLES FOR ai
`
`SEMAPHORES
`PROCESS ID
`
`
`OS SEMAPHONE
`IMPLEMENTATION
`
`
`
`
`FIG. 13
`
`

`

`US. Patent
`
`Jul. 12, 2005
`
`Sheet 14 0f 15
`
`US 6,917,963 B1
`
`Register ND
`31
`
`556
`
`
`
`
`
`
`
`
`558
`
`AID exists
`
`already?
`
`Create virtual resource translation table
`space for a1
`
`560
`
`
`Add translation tables for
` 562
`each type of resource,
`
`Link processx to a1
`i.e. semaphores.
`
`shared memory,
`etc...
`
`Create second processy and
`launch application
`
`Process inherits link
`to a1fro'm processx
`
`570
`
`Flowchart describing creation of virtual translation tables
`
`Fig. 14
`
`

`

`US. Patent
`
`Jul. 12, 2005
`
`Sheet 15 0f 15
`
`US 6,917,963 B1
`
`580
`
`
`
`
`
`ookup
`
`process
`Al D?
`
`Not found
`
`
`
`
`
`
`
`Interface
`uses semaphore lD as
`parameter?
`
`
`
`
`Operating system semaphore interface
`
`
` Was
`
`semaphore
`allocated?
`
`
`
`
`Does
`interface return
`
`semaphore?
`
`
`
`Substitute v3 for s3 Substitute v2 for 52
`
`
`
`Reverse lockup translation
`for 211:5:3
`
`(v3)
`
`Flowchart depicting translation of a virtual resource (Le. semaphore)
`Fig. 15
`
`

`

`US 6,917,963 B1
`
`1
`SNAPSHOT IMAGE FOR THE APPLICATION
`STATE OF UNSHAREABLE AND
`SHAREABLE DATA
`
`REFERENCE TO RELATED APPLICATIONS
`
`The present application claims priority to and incorpo-
`rates the following applications by reference: DYNAMIC
`SYMBOLIC LINK RESOLUTION, Prov. No. 60/157,728,
`filed on Oct. 5, 1999; SNAPSHOT VIRTUAL
`TEMPLATING, Prov. 60/157,729 filed on Oct. 5, 1999;
`SNAPSHOT RESTORE OF APPLICATION CHAINS
`AND APPLICATIONS, Prov. No. 60/157,833, filed Oct. 5,
`1999; VIRTUAL RESOURCE-ID MAPPING, Prov. No.
`60/157,727, filed on Oct. 5, 1999; and VIRTUAL PORT
`MULTIPLEXING, Prov. No. 60/157,834 filed on Oct. 5,
`1999.
`
`FIELD OF THE INVENTION:
`
`invention relates broadly to client server
`The present
`computer architectures. Specifically, the present invention
`relates to creating virtual application templates for
`the
`purpose of propagating a single application snapshot into
`multiple, distinct images.
`
`BACKGROUND
`
`Global computer networks such as the Internet have
`allowed electronic commerce (“e-commerce”) to flourish to
`a point where a large number of customers purchase goods
`and services over websites operated by online merchants.
`Because the Internet provides an effective medium to reach
`this large customer base, online merchants who are new to
`the e-commerce marketplace are often flooded with high
`customer traffic from the moment their websites are rolled
`
`out. In order to effectively serve customers, online mer-
`chants are charged with the same responsibility as conven-
`tional merchants: they must provide quality service to cus-
`tomers in a timely manner. Often, insufficient computing
`resources are the cause of a processing bottleneck that
`results in customer frustration and loss of sales. This phe-
`nomena has resulted in the need for a new utility: leasable
`on-demand computing infrastructure. Previous attempts at
`providing computing resources have entailed leasing large
`blocks of storage and processing power. However, for a new
`online merchant having no baseline from which to judge
`customer traffic upon rollout, this approach is inefficient.
`Either too much computing resources are leased, depriving
`a start up merchant of financial resources that are needed
`elsewhere in the operation, or not enough resources are
`leased, and a bottleneck occurs.
`To make an on-demand computer infrastructure possible,
`computer applications must be ported across computer net-
`works to different processing locations. However,
`this
`approach is costly in terms of overhead for the applications
`to be moved across the network must be saved, shut down,
`stored, ported and then restored and re-initialized with the
`previously running data. The overhead is prohibitive and
`negates any performance improvements realized by trans-
`ferring the application to another computer. Thus,
`there
`remains a heartfelt need for a system and method for
`effecting a transfer of applications across computer networks
`without incurring costly processing overhead.
`
`SUMMARY
`
`invention solves the problems described
`The present
`above by creating virtual application templates for
`the
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`purpose of propagating a single application snapshot into
`multiple, distinct images. Snapshot virtual templates allow
`multiple application instances to use the same fixed resource
`identifier by making the resource identifier virtual, privatiz-
`ing it, and dynamically mapping it
`to a unique system
`resource identifier. When a snapshot is cloned from a virtual
`template, the common or shared data is used exactly as is,
`whereas the non-sharable data is either copied-on-write,
`multiplexed, virtualized, or customized-on-duplication. The
`present invention greatly reduces the required administrative
`setup per application instance. Snapshot virtual templating
`works by noting access to modified resources, fixed system
`IDs/keys and unique process-related identifies and automati-
`cally inserting a level of abstraction between these resources
`and the application. The resources contained in a snapshot
`virtual template can be dynamically redirected at restore
`time. Access to memory and storage is managed in a
`copy-on-write fashion. System resource handles are man-
`aged in a virtualize-on-allocate fashion or by a multiplex-
`on-access mechanism. Process-unique resources are man-
`aged in a redirect-on-duplicate fashion. Rules may be
`defined through an application configurator that allows some
`degree of control over the creation of non-sharable data.
`The snapshot virtual template is constructed by dividing
`the snapshot image into sharable and non-sharable data.
`Knowledge of which system resources can be shared is
`encoded in the application snapshot/restore framework.
`These and many other attendant advantages of the present
`invention will be understood upon reading the following
`detailed description in conjunction with the drawings.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a high level block diagram illustrating the
`various components of a computer network used in connec-
`tion with the present invention;
`FIG. 2 is a high level block diagram illustrating the
`various components of a computer used in connection with
`the present invention;
`FIG. 3 illustrates how application state is tracked using
`library and operating system kernel interposition;
`FIG. 4 illustrates the capture of an application’s run-time
`state;
`FIG. 5 is a flow chart illustrating the logical sequence of
`steps executed to create a snapshot image of an application
`instance;
`FIG. 6 is a flow chart illustrating the logical sequence of
`steps executed to restore an application instance from a
`snapshot image.;
`FIG. 7 is an illustration of the format of a snapshot virtual
`template;
`FIG. 8 is a flowchart illustrating the logical sequence of
`steps executed to create a snapshot virtual template;
`FIG. 9 is a flowchart illustrating the logical sequence of
`steps executed to clone a snapshot virtual template;
`FIG. 10 illustrates the registration of an application using
`virtual resource identifiers;
`FIG. 11 illustrates the allocation of a virtual resource;
`FIG. 12 illustrates the translation of a virtual resource to
`a system resource;
`FIG. 13 illustrates the translation of a system resource to
`a virtual resource;
`FIG. 14 is a flowchart illustrating the logical sequence of
`steps executed to create a virtual translation table; and
`FIG. 15 is a flowchart illustrating the logical sequence of
`steps executed to translate a virtual resource.
`
`

`

`US 6,917,963 B1
`
`3
`DETAILED DESCRIPTION
`
`A. Snapshot Restore
`FIG. 1 illustrates in high level block diagram form the
`overall structure of the present invention as used in connec-
`tion with a global computer network 100 such as the
`Internet.
`
`Remote users 102-1 and 102-2 can connect through the
`computer network 100 to a private network of computers
`106 protected by firewall 104. Computer network 106 is a
`network comprising computers 150-1, 150-2, through 150-11,
`where n is the total number of computers in network 106.
`Computers 150 are used to run various applications, as well
`as host web sites for access by remote users 102. The present
`invention is implemented on computer network 106 in the
`form of virtual environments 110-1 and 110-2. While only
`two virtual environments are illustrated, it is to be under-
`stood that any number of virtual environments may be
`utilized in connection with the present invention.
`FIG. 2 illustrates in high level block diagram form a
`computer that may be utilized in connection with the present
`invention. Computer 150 incorporates a processor 152 uti-
`lizing a central processing unit (CPU) and supporting inte-
`grated circuitry. Memory 154 may include RAM and
`NVRAM such as flash memory,
`to facilitate storage of
`software modules executed by processor 152, such as appli-
`cation snapshot/restore framework 200. Also included in
`computer 150 are keyboard 158, pointing device 160, and
`monitor 162, which allow a user to interact with computer
`150 during execution of software programs. Mass storage
`devices such as disk drive 164 and CD ROM 166 may also
`be in computer 150 to provide storage for computer pro-
`grams and associated files. Computer 150 may communicate
`with other computers via modem 168 and telephone line 170
`to allow the computer 150 to be operated remotely, or utilize
`files stored at different locations. Other media may also be
`used in place of modem 168 and telephone line 170, such as
`a direct connection or high speed data line. The components
`described above may be operatively connected by a com-
`munications bus 172.
`
`FIG. 3 shows how application states are tracked via
`library and kernel interposition. The application snapshot/
`restore framework 200 is a software module that processes
`transactions between the operating system 206 and the
`applications 208. Requests for system resources or changes
`to process state are routed internally and the application
`snapshot/restore framework 200 tracks these events in
`anticipation of a snapshot. The application snapshot/restore
`framework 200 is transparent to running (and snapshotted)
`applications. From an application’s perspective, the appli-
`cation is always running. An application snapshot may
`consist of multiple processes and multiple threads and
`includes shared resources in use by a process, such as shared
`memory or semaphores. A process may be snapshotted &
`restored more than once. The computer on which a process
`is restored on must be identically configured and have an
`identical environment (hardware, software, and files) that
`matches the environment of the computer where the process
`was snapshotted. All processes that are snapshotted together
`in the form of an application chain share the same applica-
`tion ID (“AID”). As used herein, an application chain is the
`logical grouping of a set of applications and processes that
`communicate with each other and share resources to provide
`a common function.
`
`The virtual environment 110 is a layer that surrounds
`application(s) 208 and resides between the application and
`the operating system 206. Resource handles are abstracted to
`present a consistent view to the application although the
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`actual system resource handles may change as an application
`is snapshot/restored more than once. The virtual environ-
`ment also allows multiple applications to compete for the
`same resources where exclusion would normally prohibit
`such behavior to allow multiple snapshots to coexist without
`reconfiguration. Preload library 214 is an application library
`that interposes upon an application for the express purpose
`of intercepting and handling calls and system calls. Once the
`library has been preloaded it
`is attached to the process’
`address space. Preload library 214 interposes between appli-
`cation 208 and operating system 206. It is distinguished
`from kernel interposition in that it operates in “user mode”
`(i.e., non-kernel and non-privileged mode). Application 208
`can make application programming interface (API) calls that
`modify the state of the application. These calls are made
`from the application 208 to the operating system API
`interfaces 210 via the application snapshot restore frame-
`work 200 or the preload library 214. The preload library can
`save the state of various resources by intercepting API
`interface calls and then saves the state at a pre-arranged
`memory location. When the process’memory is saved as part
`of the snapshot/restore mechanism, this state is saved since
`it resides in memory. The state as it is modified is saved to
`non-volatile storage (i.e. a file on disk). The preload library
`notify the snapshot/restore framework through one of its
`private interface.
`FIG. 4 illustrates the capture of an application’s run time
`state. The OS API interfaces 210 are standard programming
`interfaces defined by international standards organizations
`such as XOPEN. The open ( ) system call which allows an
`application to open a file for reading is an example of an API
`interface. The process management system 216 is a com-
`ponent of the operating system 206 that allows one process
`to examine or alter the state of another process. The inter-
`faces that are provided by this component are usually not
`standardized interfaces (not part of a recognized standard
`API) and are OS-implementation dependent. However, such
`interfaces usually allow access to more state than standard-
`ized API interfaces. The run-time information captured from
`a process is used by the snapshot driver 218.
`An application needs to be snapshotted if it is idle and is
`not currently in use or there are higher priority requests that
`require the application be scheduled out and preempted in
`favor of another application. A snapshot request is initiated
`by an application scheduler that determines when an appli-
`cation needs to be restored on-demand and when the appli-
`cation is no longer needed (can be snapshotted to free up
`resources). The application scheduler does this based on web
`traffic, server load, request response time, and a number of
`other factors. An application needs to be restored if there is
`an incoming request (i.e. a web browser request) and the
`application required to handle that request (ie a particular
`web site) is not currently running. Alternatively, an appli-
`cation needs to be restored if there is an incoming request
`(i.e. a web browser request) and the application required to
`handle that request (ie a particular web site) is currently
`overloaded, so another instance of that application is
`restored to handle that request.
`FIG. 5 illustrates the logical sequence of steps executed
`by Snapshot driver 218 to make a snapshot image of a
`process. Beginning at step 250, an snapshot image of a
`runnable application is requested. The AID is looked up
`(decision step 252) in a table in memory 154 containing a list
`of every AID present on-computer 150. If the AID is not
`found control returns at step 254. However, if the AID is
`found, control continues to decision step 256 where the
`snapshot/restore framework 200 searches for a process
`
`

`

`US 6,917,963 B1
`
`5
`belonging to the application having the matched ADD. If a
`process is found, control continues to step 258, where the
`process is suspended. For a process to be snapshotted, it
`must be completely suspended with no activity present and
`no ambiguous state (i.e.,
`in a transitory state). Since a
`process may be represented by asynchronous threads of
`activity in the operating system that are not formally part of
`the process state, any activity in the operating system that is
`executing on behalf of the process must be stopped (i.e. disk
`I/O activity). In other words, there may be moments where
`temporarily a process cannot be snapshotted. This is a finite
`and short period of time, but it can occur. If the state is
`consistent and the threads are quiesced (decision step 260),
`control
`loops to step 256 and the remaining processes
`belonging to the application are located and suspended.
`However,
`if a process is located that does not have a
`consistent state or a thread is not quiesced, suspended
`processes are resumed and the snapshot cannot be com-
`pleted.
`Once all related processes are suspended, for each state
`(step 266) of each suspended process (step 264), the state is
`checked to see if it is virtualized (step 268). A virtualized
`state is any process state that reflects a virtualized resource.
`If the state is virtualized, it is retrieved at step 270; otherwise
`the non-virtualized state is retrieved at step 272. State
`retrieval is performed as described above by the snapshot
`driver 218 querying the application snapshot/restore frame-
`work 200, operating system API interfaces 210, and process
`management subsystem 216. If the state has changed since
`the last snapshot (step 274), the new state is recorded (step
`276). Control then loops to step 264 (through decision steps
`278 and 280) and executes through the above sequence of
`steps until all states of all processes are checked. Once
`completed, control proceeds to step 282 (through decision
`steps 278 and 280),
`the registered global state, such as
`semaphores, is removed. Registered global state is state that
`is not specifically associated with any one process (ie private
`state). Global state is usually exported (accessible) to all
`processes and its state can be modified (shared) by all
`processes. Control proceeds to step 284, where the process
`is terminated. If there are remaining processes (step 286),
`these are also terminated. This sequence of steps is con-
`cluded to create a snapshot image which is stored as a file
`and made available for transmission to another computer
`within public computer network 100 or private computer
`network 106.
`
`FIG. 6 illustrates the sequence of steps executed by the
`restore driver 220 to restore a snapshot image. The snapshot
`image is accessed via a shared storage mechanism and a
`restore call is made at step 300. The restore driver 220 looks
`up the AID for the snapshot image and (decision step 302)
`if not found control returns and the snapshot image cannot
`be restored. However, if the AID is found, control continues
`to decision step 304 where, if the snapshotted image match-
`ing the AID is located,
`the global/shared state for each
`process associated with the snapshot are found. Control then
`continues to step 308, where remaining global or shared
`state for the processes are recreated. Since global and shared
`state is not associated with a single process and may be
`referenced by multiple processes, it is created first. Recre-
`ating the state entails creating a global resource that is
`functionally identical
`to the resource at
`the time of the
`snapshot. For example if during a snapshot, a semaphore
`with id 5193 is found with a value of 7, then to recreate the
`state at restore time a new semaphore must be created having
`the exact same ID as before (ie 5193) and it also must have
`the same state (ie value 7) as before. Then, for each image,
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`a process is created that inherits the global/shared state
`restored in step 308, and each created process is isolated to
`prevent inter-process state changes. When a process is being
`restored, process state is being registered with the kernel,
`inter-process mechanisms are being restored and recon-
`nected and I/O buffers in the kernel may be being restored.
`Some of these actions in one process may have the unin-
`tended side effect of disturbing another process that is also
`being restored. For example if an I/O buffer that is in the
`operating system as a result of a processx performing a write
`to a socket connection, then processy could unintentionally
`be delivered an asynchronous signal that notifies it of I/O
`being present (for reading) prior to the process being fully
`restored. At step 314, for each type of state within the
`processes,
`the process-private resources are recreated to
`their state at the time the snapshot image was taken. If the
`state is virtualized (decision step 316), the system state is
`bound to a virtual definition. As part it, of the restore an extra
`step must be done to create a virtual mapping. This is done
`by taking the system resource that was created in step 314
`and binding it to the virtual definition that was saved during
`the snapshot in step 266. This allows the application to see
`a consistent view of resources, since it cannot be guaranteed
`that at restore time the exact same system resource will be
`available. If the state is shared with another process, such as
`via a pipe (decision state 320), the shared state is recon-
`nected with the other process at step 322. If there are more
`states (decision step 324) steps 314 through 322 are
`repeated. Once steps 314 through 322 have been executed
`for all states, control continues to step 326, where the
`process is placed in synchronized wait. If there are remain-
`ing images in the snapshot image (decision step 328), steps
`310 through 326 are repeated. Once all images have been
`processed, control continues to step 330, where traces and
`states induced during restore of the process are removed, and
`a synchronized resume of all processes occurs at step 332.
`Once steps 300 through 332 have executed without error
`on whatever computer the restore driver 220 was executed,
`the restored application can continue to run without inter-
`ruption. Thus, the present invention avoids the overhead and
`delay of shutting down an application, storing data to a
`separate file, moving both the application and data file
`elsewhere, and restarting the program.
`B. Snapshot Virtual Templating
`In another aspect, the present invention provides a system,
`method, and computer program product for creating snap-
`shot virtual application templates for the purpose of propa-
`gating a single application snapshot into multiple distinct
`instances. Snapshot virtual templates allow multiple appli-
`cation instances to use the same fixed resource ID (“RID”)
`by making the resource ID virtual, privatizing the virtual
`RID, and dynamically mapping it
`to a unique system
`resource ID. A RID is the identifier assigned to represent a
`specific system resource and acts as a handle when refer-
`encing that system resource. Anonymous resources are
`resources that are read-only or functionally isolated from
`other applications. Anonymous resources are also shareable
`resources. An anonymous resource is a non-fixed resource
`allocated by the operating system and identified by a per-
`process handle. These are functionally