1111111111111111 IIIIII IIIII 11111 1111111111 111111111111111 111111111111111 1111111111 11111111
`US 20050160413Al
`
`(19) United States
`(12) Patent Application Publication
`Broussard et al.
`
`(10) Pub. No.: US 2005/0160413 Al
`Jul. 21, 2005
`(43) Pub. Date:
`
`(54) METHOD AND SYSTEM FOR A
`GRID-ENABLED VIRTUAL MACHINE WITH
`MOVABLE OBJECTS
`
`(75)
`
`Inventors: Scott J. Broussard, Cedar Park, TX
`(US); Ying Liu, Austin, TX (US);
`Eduardo N. Spring, Round Rock, TX
`(US)
`
`Correspondence Address:
`IBM CORP (JRB)
`C/O LAW OFFICE OF JOSEPH R BURWELL
`PO BOX 28022
`AUSTIN, TX 78755-8022 (US)
`
`(73) Assignee: INTERNATIONAL
`MACHINES
`ARMONK, NY
`
`BUSINESS
`CORPORATION,
`
`(21) Appl. No.:
`
`10/761,998
`
`(22) Filed:
`
`Jan.21,2004
`
`Publication Classification
`
`(51)
`
`Int. CI.7 ....................................................... G06F 9/45
`
`(52) U.S. Cl. .............................................................. 717/148
`
`(57)
`
`ABSTRACT
`
`A method, an apparatus, a computer program product, and a
`data processing system provide for operation of a virtual
`machine with embedded functionality for interoperating
`with other virtual machines in a computational grid. A
`plurality of virtual machines are run on one or more devices
`within a data processing system; each virtual machine in the
`plurality of virtual machines incorporates functionality for
`interoperating and associating with other virtual machines in
`a virtual machine cluster in a grid-like manner. Each virtual
`machine in the virtual machine cluster acts as a node within
`the virtual machine cluster. A virtual machine manages its
`objects in association with an object group, and each virtual
`machine may manage multiple object groups. The virtual
`machines share information such that the object groups can
`be moved between virtual machines in the virtual machine
`cluster, thereby allowing the virtual machine cluster to act as
`one logical virtual machine.
`
`MIGRATION OF OBJECT GROUP FROM GJVM
`
`BEGIN
`
`PREPARE TO MIGRATE OBJECT GROUP
`FROM LOCAL GJVM TO REMOTE GJVM
`1502
`
`SEND REQUEST TO PRIMARY GJVM TO
`CHANGE MIGRATION STATE OF OBJECT
`GROUP FROM STABLE TO MIGRATING
`1504
`
`RECEIVE MIGRATION STATE CHANGE
`ACKNOWLEDGMENT FROM PRIMARY GJVM
`~
`
`FLATTEN /SERIALIZE EACH OBJECT IN
`MIGRATING OBJECT GROUP
`1508
`
`PUSH FLATTENED/SERIALIZED OBJECTS
`FROM LOCAL GJVM TO REMOTE GJVM
`1510
`
`RECEIVE ACKNOWLEDGMENT
`AT LOCAL GJVM FROM REMOTE GJVM
`1512
`
`DESTROY OBJECT GROUP/HEAP
`AT LOCAL GJVM
`1514
`
`CLEAR OBJECT GROUP ENTRY IN LOCAL
`OBJECT GROUP TABLE OF LOCAL GJVM
`1filQ
`
`END
`
`Netflix, Inc. - Ex. 1016, Page 000001
`IPR2022-00322 (Netflix, Inc. v. CA, Inc.)
`
`

`

`Patent Application Publication Jul. 21, 2005 Sheet 1 of 15
`
`US 2005/0160413 Al
`
`100"\
`109
`
`103
`102
`
`105
`
`106
`
`CLIENT
`
`NETWORK
`11Q
`
`104
`
`114
`
`. _ _ _ . 116
`
`PERSONAL
`DIGITAL ASSISTANT
`
`~ 107
`STORAGE ~ ~
`L ~ rl_ 113
`A._ 112 7 :s:
`FIG. IA
`(PRIOR ART)
`
`111
`
`PE~A:
`DIGITAL ASSISTANT
`
`WIRELESS
`PHONE
`
`122
`
`120
`\
`
`CPU
`
`DISPLAY
`ADAPTER
`
`124
`
`RAM
`
`123
`
`144
`
`(cid:143).
`
`DISPLAY
`
`146
`
`130
`
`ROM
`
`1/0 ADAPTER
`
`132
`
`COMMUNICATION
`ADAPTER
`
`1 - - - - 1
`
`USER INTERFACE
`ADAPTER
`
`148
`
`142
`
`MOUSE
`
`140
`
`KEYBOARD
`
`136
`
`COMMUNICATION
`LINK
`
`FIG. JB
`(PRIOR ART}
`
`Netflix, Inc. - Ex. 1016, Page 000002
`
`

`

`Patent Application Publication Jul. 21, 2005 Sheet 2 of 15
`
`US 2005/0160413 Al
`
`PRIMARY
`GJVM NODE
`~
`
`GJVM
`NODE
`202
`
`GJVM
`NODE
`203
`
`GJVM
`NODE
`205
`
`GJVM
`NODE
`204
`
`VIRTUAL MACHINE
`CLUSTER/GRID
`
`FIG. 2
`
`GJVM NODE ~
`
`lr-P-2_P_C_O_M_M __ U-N-IT-L-02--,1 I UNIQUE GJVM ID ~ I I GJVM GROUPNAME ~ I
`APPLICATION ENVIRONMENT/CONTEXT PARAMETERS~ I PRIMARY GJVM IP:PORT .llQ I
`
`(GRID-ENABLED JAVA VIRTUAL MACHINE)
`
`GJVM REGISTRY ill
`I~ G-J-VM-l(cid:143) -L-14--,j I IP:PORT lli II~ R_U_N_S_T-AT_E_L-18~1
`
`Ir--O-BJ_E_C_T_G_R_O_U_P_ID_L_2_2--,I I GJVM ID~ I lr-M_I_G_RA_T-IO_N_ST_A_T_E_~--,1
`
`OBJECT GROUP REGISTRY ~
`
`Ir--O-BJ_E_C_T_G_R_O_U_P_ID_L_5_2--,I I POINTER TO THREAD LOCAL HEAP~ I ... I -TH_R_E_A_D-ID-~-~
`
`LOCAL OBJECT GROUP MANAGEMENT TABLE ~
`
`THREAD LOCAL HEAP~
`
`OBJECT GROUP 334
`
`OBJECT GROUP 336
`
`MASTER GLOBAL OBJECT GROUP 360
`
`I OBJECT~ I I OBJECT~ I
`I OBJECT~ I I OBJECT~ I
`
`THREAD
`328
`
`THREAD LOCAL HEAP 332
`
`OBJECT GROUP 338
`
`THREAD
`MANAGEMENT
`OBJECT
`~
`
`I OBJECT MQ I r-S_T_A-CK-FRA_M_E--,
`.__ ___ ~
`I OBJECT M2 I
`
`OBJECT
`M§
`
`FIG. 3A
`
`Netflix, Inc. - Ex. 1016, Page 000003
`
`

`

`Patent Application Publication Jul. 21, 2005 Sheet 3 of 15
`
`US 2005/0160413 Al
`
`GJVM NODE 300
`
`I OBJECT GROUP~ I
`I OBJECT GROUP 336 I
`I OBJECT GROUP m I
`
`GJVM NODE 300
`
`I OBJECT GROUP~ I
`I OBJECT GROUP~ I
`
`GJVM NODE 370
`
`I OBJECT GROUP ill I
`
`MOVEMENT OF
`OBJECT GROUPS WITHIN
`VIRTUAL MACHINE
`CLUSTER/GRID
`
`GJVM NODE 380
`
`I OBJECT GROUP 382 I
`
`FIG. 3B
`
`GJVM NODE 370
`
`I OBJECT GROUP 372
`
`FIG. 3C
`
`I \
`
`GJVM NODE 380
`
`I OBJECT GROUP 382 I
`I OBJECT GROUP 338 I
`
`GJVM NODE llQ
`I OBJECT GROUP 372
`
`I
`
`I OBJECT GROUP 334 ,_ I OBJECT GROUP~ I
`
`GJVM NODE 380
`
`I OBJECT GROUP 336
`
`I
`
`I OBJECT GROUP~ I
`
`FIG. 3D
`
`Netflix, Inc. - Ex. 1016, Page 000004
`
`

`

`Patent Application Publication Jul. 21, 2005 Sheet 4 of 15
`
`US 2005/0160413 Al
`
`INITIALIZATION OF PRIMARY GJVM NODE
`
`BEGIN
`, ,
`
`INITIALIZE PRIMARY GJVM
`402
`
`w
`
`REGISTER GJVM CLUSTER NAME
`404
`' ,
`GET GJVM APPLICATION
`ENVIRONMENT/CONTEXT PARAMETERS
`406
`
`, f
`
`INITIALIZE OBJECT GROUP REGISTRY
`408
`
`' ,
`INITIALIZE GJVM REGISTRY
`410
`
`' ,
`END
`
`FIG. 4
`
`Netflix, Inc. - Ex. 1016, Page 000005
`
`

`

`Patent Application Publication Jul. 21, 2005 Sheet 5 of 15
`
`US 2005/0160413 Al
`
`ALLOCATION OF UNIQUE ID TO
`NON-PRIMARY GJVM BY PRIMARY GJVM
`( BEGIN
`)
`i
`
`RECEIVE REQUEST AT PRIMARY GJVM
`FROM NON-PRIMARY GJVM
`FOR UNIQUE GJVM ID
`fill2
`i
`
`GENERATE UNIQUE GJVM ID
`504
`
`t
`
`ADD NEWLY GENERATED GJVM ID
`TO GJVM REGISTRY
`506
`
`t
`
`SET RUN STATE OF NON-PRIMARY GJVM
`IN GJVM REGISTRY TO STARTING
`508
`i
`
`RESPOND TO REQUESTING GJVM WITH
`UNIQUE GJVM ID
`510
`
`t
`
`SEND UPDATED GJVM REGISTRY TO ALL
`NON-PRIMARY GJVM NODES
`ill
`t
`
`END
`
`)
`(
`FIG. 5A
`
`REGISTRATION OF UNIQUE ID FOR
`NON-PRIMARY GJVM BY PRIMARY GJVM
`( BEGIN
`i
`
`RECEIVE REQUEST AT PRIMARY GJVM
`FROM NON-PRIMARY GJVM
`TO REGISTER NEW GJVM ID
`522
`t
`
`ADD NEW GJVM ID TO GJVM REGISTRY
`524
`t
`
`SET RUN STATE OF NON-PRIMARY GJVM
`IN GJVM REGISTRY TO STARTING
`222
`t
`
`RESPOND TO REQUESTING WITH STATUS
`528
`i
`
`SEND UPDATED GJVM REGISTRY TO ALL
`NON-PRIMARY GJVM NODES
`530
`i
`
`END
`
`)
`(
`FIG. 5B
`
`Netflix, Inc. - Ex. 1016, Page 000006
`
`

`

`Patent Application Publication Jul. 21, 2005 Sheet 6 of 15
`
`US 2005/0160413 Al
`
`PROVISION OF CLUSTER-SPECIFIC INFO TO
`NON-PRIMARY GJVM BY PRIMARY GJVM
`
`BEGIN
`
`RECEIVE REQUEST AT PRIMARY GJVM
`FOR CLUSTER-SPECIFIC INFORMATION
`702
`
`RESPOND TO REQUESTING GJVM WITH
`CLUSTER-SPECIFIC INFORMATION
`704
`
`END
`
`FIG. 7
`
`ALLOCATION OF OBJECT GROUP ID TO
`NON-PRIMARY GJVM BY PRIMARY GJVM
`( BEGIN
`)
`i
`
`RECEIVE REQUEST AT PRIMARY GJVM
`FOR UNIQUE OBJECT GROUP ID
`§Q2
`.!-
`GENERATE UNIQUE OBJECT GROUP ID
`filM
`i
`
`ADD NEWLY GENERATED OBJECT GROUP
`ID TO OBJECT GROUP REGISTRY
`§Q§
`
`i
`
`SET MIGRATION STATE OF OBJECT GROUP
`IN OBJECT GROUP REGISTRY
`.§Q§
`
`i
`
`RESPOND TO REQUESTING GJVM WITH
`UNIQUE OBJECT GROUP ID
`.filQ
`
`i
`
`SEND UPDATED OBJECT GROUP REGISTRY
`TO ALL NON-PRIMARY GJVM NODES
`612
`i
`
`END
`
`)
`(
`FIG. 6
`
`Netflix, Inc. - Ex. 1016, Page 000007
`
`

`

`Patent Application Publication Jul. 21, 2005 Sheet 7 of 15
`
`US 2005/0160413 Al
`
`CHANGE OF RUN STATE BY
`NON-PRIMARY GJVM AT PRIMARY GJVM
`( BEGIN
`)
`i
`
`RECEIVE REQUEST AT PRIMARY GJVM
`FOR CHANGE OF RUN STATE
`802
`i
`
`UPDATE GJVM REGISTRY AS REQUESTED
`804
`i
`
`RESPOND TO REQUESTING GJVM WITH
`WITH ACKNOWLEDGMENT MESSAGE
`806
`i
`
`SEND UPDATED GJVM REGISTRY TO ALL
`NON-PRIMARY GJVM NODES
`808
`i
`
`END
`
`(
`)
`FIG. 8
`
`CHANGE OF OBJECT GROUP MIGRATION
`STATE BY NON-PRIMARY GJVM
`AT PRIMARY GJVM
`( BEGIN
`i
`
`RECEIVE REQUEST FOR CHANGE OF
`MIGRATION STATE OF OBJECT GROUP
`fill2.
`i
`
`UPDATE OBJECT GROUP REGISTRY WITH
`MIGRATION STATE AS REQUESTED
`904
`i
`
`RESPOND TO REQUESTING GJVM WITH
`WITH ACKNOWLEDGMENT MESSAGE
`906
`i
`
`SEND UPDATED OBJECT GROUP REGISTRY
`TO ALL NON-PRIMARY GJVM NODES
`wa
`i
`
`END
`
`)
`(
`FIG.9
`
`Netflix, Inc. - Ex. 1016, Page 000008
`
`

`

`Patent Application Publication Jul. 21, 2005 Sheet 8 of 15
`
`US 2005/0160413 Al
`
`INITIALIZATION OF NON-PRIMARY GJVM NODE
`( BEGIN
`i
`
`INITIALIZE NON-PRIMARY GJVM
`1002
`i
`
`OBTAIN GJVM CLUSTER NAME
`1004
`
`i
`
`LOOKUP INFORMATION ABOUT
`GJVM CLUSTER
`1QOO
`
`i
`
`LOCATE PRIMARY GJVM
`1008
`i
`
`REQUEST AND RECEIVE UNIQUE GJVM ID
`FROM PRIMARY GJVM
`1Q1Q.
`
`i
`
`REQUEST AND RECEIVE CLUSTER-WIDE
`PARAMETERS FROM PRIMARY GJVM
`1012
`i
`
`REQUEST AND RECEIVE GJVM REGISTRY
`FROM PRIMARY GJVM
`1014
`i
`
`REQUEST AND RECEIVE OBJECT GROUP
`REGISTRY FROM PRIMARY GJVM
`1016
`I
`
`i
`
`SEND RUN STATE CHANGE REQUEST TO
`PRIMARY GJVM TO UPDATE FROM
`STARTING TO RUNNING
`1018
`l
`RECEIVE RUN STATE CHANGE
`ACKNOWLEDGMENT FROM PRIMARY GJVM
`1020
`i
`
`END
`
`(
`)
`FIG. 10
`
`Netflix, Inc. - Ex. 1016, Page 000009
`
`

`

`Patent Application Publication Jul. 21, 2005 Sheet 9 of 15
`
`US 2005/0160413 Al
`
`THREAD CONSTRUCTION IN GJVM NODE
`( BEGIN
`)
`i
`
`RECEIVE REQUEST AT GJVM TO
`CONSTRUCT THREAD
`1102
`~
`REQUEST AND RECEIVE UNIQUE OBJECT
`GROUP ID FROM PRIMARY GJVM
`llQi
`i
`
`CONSTRUCT LOCAL THREAD
`.1.1QQ
`
`i
`
`ASSIGN UNIQUE THREAD ID TO NEWLY
`CONSTRUCTED THREAD
`11.Qa
`
`i
`
`ALLOCATE THREAD LOCAL HEAP
`1110
`
`i
`
`CREATE OBJECT GROUP WITHIN
`THREAD LOCAL HEAP
`1112
`i
`
`ASSOCIATE OBJECT GROUP ID WITH
`NEWLY CONSTRUCTED THREAD
`.11.li
`i
`
`SEND REQUEST TO PRIMARY GJVM TO
`CHANGE MIGRATION STATE OF OBJECT
`GROUP FROM STARTING TO STABLE
`ill&
`i
`
`RECEIVE MIGRATION STATE CHANGE
`ACKNOWLEDGMENT FROM PRIMARY GJVM
`1118
`
`I
`
`)
`
`THREAD DESTRUCTION IN GJVM NODE
`( BEGIN
`~
`RECEIVE REQUEST TO DESTROY THREAD
`~
`~
`DISSASSOCIATE THREAD ID FROM EACH
`ASSOCIATED OBJECT GROUP
`1204
`i
`
`UPDATE LOCAL OBJECT GROUP
`MANAGEMENT TABLE
`1206
`i
`
`MOVE OBJECT GROUP FROM THREAD
`LOCAL HEAP TO GENERAL HEAP
`IF NECESSARY
`1208
`~
`DESTROY THREAD
`1210
`~
`RETURN THREAD DESTRUCTION
`RESPONSE TO REQUESTER
`1212
`i
`
`END
`
`(
`)
`FIG. 12
`
`i
`
`RETURN RESPONSE FOR THREAD
`CONSTRUCTION (THREAD ID, ... )
`1120
`
`i
`
`END
`
`(
`)
`FIG. 11
`
`Netflix, Inc. - Ex. 1016, Page 000010
`
`

`

`Patent Application Publication Jul. 21, 2005 Sheet 10 of 15
`
`US 2005/0160413 Al
`
`OBJECT ALLOCATION IN GJVM NODE
`( BEGIN
`)
`i
`
`RECEIVE REQUEST AT GJVM TO
`CONSTRUCT OBJECT
`1302
`J,
`DETERMINE THREAD ID OF REQUESTING
`THREAD
`1304
`J,
`RETRIEVE OBJECT GROUP INFORMATION
`FOR REQUESTING THREAD
`noo
`J,
`ALLOCATE OBJECT IN ONE OF
`REQUESTING THREAD'S OBJECT GROUPS
`~
`
`i
`
`ASSIGN OBJECT ID TO OBJECT
`1310
`i
`
`GENERATE OBJECT REFERENCE/HANDLE
`CONTAINING OBJECT GROUP ID
`AND OBJECT ID
`1312
`i
`
`RETURN OBJECT REFERENCE/HANDLE TO
`REQUESTING THREAD
`1314
`J,
`END
`
`)
`(
`FIG. 13
`
`OBJECT REFERENCING IN GJVM NODE
`
`BEGIN
`
`INTERCEPT ATTEMPT
`TO REFERENCE OBJECT
`VIA OBJECT HANDLE
`H..QZ
`
`GET OBJECT GROUP ID
`FROM OBJECT HANDLE
`1404
`
`YES
`
`GET POINTER TO
`THREAD LOCAL
`HEAP
`1iQ.a
`
`GET GJVM ID FOR
`OBJECT GROUP
`1412
`
`GET IP ADDRESS
`AND PORT ID
`FORGJVM ID
`1414
`
`ACCESS OBJECT
`IN THREAD
`LOCAL HEAP
`USING OBJECT ID
`1410
`
`SEND OBJECT
`ACCESS
`REQUEST TO
`IDENTIFIED GJVM
`NODE
`1416
`
`END
`
`FIG. 14
`
`Netflix, Inc. - Ex. 1016, Page 000011
`
`

`

`Patent Application Publication Jul. 21, 2005 Sheet 11 of 15
`
`US 2005/0160413 Al
`
`MIGRATION OF OBJECT GROUP FROM GJVM
`( BEGIN
`)
`i
`
`PREPARE TO MIGRATE OBJECT GROUP
`FROM LOCAL GJVM TO REMOTE GJVM
`1502
`i
`
`SEND REQUEST TO PRIMARY GJVM TO
`CHANGE MIGRATION STATE OF OBJECT
`GROUP FROM STABLE TO MIGRATING
`1504
`i
`
`RECEIVE MIGRATION STATE CHANGE
`ACKNOWLEDGMENT FROM PRIMARY GJVM
`1506
`i
`
`FLATTEN /SERIALIZE EACH OBJECT IN
`MIGRATING OBJECT GROUP
`1508
`i
`
`PUSH FLATTENED/SERIALIZED OBJECTS
`FROM LOCAL GJVM TO REMOTE GJVM
`.1filQ
`
`i
`
`RECEIVE ACKNOWLEDGMENT
`AT LOCAL GJVM FROM REMOTE GJVM
`ill2.
`i
`
`DESTROY OBJECT GROUP/HEAP
`AT LOCAL GJVM
`1514
`i
`
`CLEAR OBJECT GROUP ENTRY IN LOCAL
`OBJECT GROUP TABLE OF LOCAL GJVM
`1516
`i
`
`END
`
`(
`)
`FIG. 15
`
`MIGRATION OF OBJECT GROUP TO GJVM
`( BEGIN
`)
`i
`
`RECEIVE REQUEST FOR OBJECT GROUP
`MIGRATION FROM REMOTE GJVM
`AT LOCAL GJVM
`1602
`i
`
`ALLOCATE THREAD LOCAL HEAP
`1606
`i
`
`RECEIVE FLATTENED/SERIALIZED OBJECTS
`FROM REMOTE GJVM AT LOCAL GJVM
`1604
`!
`
`READ/INFLATE OBJECTS INTO HEAP
`1608
`!
`
`ASSOCIATE OBJECT GROUP WITH THREAD
`.1filQ
`.i
`
`CREATE OBJECT GROUP ENTRY IN LOCAL
`OBJECT GROUP TABLE OF LOCAL GJVM
`1filZ
`i
`
`SEND REQUEST TO PRIMARY GJVM TO
`CHANGE MIGRATION STATE OF OBJECT
`GROUP FROM MIGRATING TO STABLE
`1614
`i
`
`RECEIVE MIGRATION STATE CHANGE
`ACKNOWLEDGMENT FROM PRIMARY GJVM
`.1filQ
`
`i
`
`SEND ACKNOWLEDGMENT
`TO REMOTE GJVM FROM LOCAL GJVM
`FOR MIGRATION OF OBJECT GROUP
`1618
`i
`F 1G. 16 C END
`
`)
`
`Netflix, Inc. - Ex. 1016, Page 000012
`
`

`

`Patent Application Publication Jul. 21, 2005 Sheet 12 of 15
`
`US 2005/0160413 Al
`
`MIGRATION OF THREAD FROM GJVM
`( BEGIN
`)
`i
`
`PREPARE TO MIGRATE THREAD
`FROM LOCAL GJVM TO REMOTE GJVM
`1702
`i
`
`PUSH THREAD MANAGEMENT
`INFORMATION FROM LOCAL GJVM
`TO REMOTE GJVM
`1704
`i
`
`RECEIVE ACKNOWLEDGMENT
`AT LOCAL GJVM FROM REMOTE GJVM
`1706
`!
`
`UPDATE THREAD MANAGEMENT TABLES
`OF LOCAL GJVM
`1708
`i
`
`END
`
`(
`)
`FIG. 17
`
`MIGRATION OF THREAD TO GJVM
`( BEGIN
`)
`!
`
`RECEIVE REQUEST TO MIGRATE THREAD
`FROM REMOTE GJVM AT LOCAL GJVM
`1802
`!
`
`RECEIVE THREAD MANAGEMENT INFO
`FROM REMOTE GJVM AT LOCAL GJVM
`1804
`!
`
`UPDATE THREAD MANAGEMENT TABLES
`OF LOCAL GJVM
`.1M2
`!
`
`SEND ACKNOWLEDGMENT
`TO REMOTE GJVM FROM LOCAL GJVM
`FOR MIGRATION OF THREAD
`1808
`!
`
`END
`
`)
`(
`FIG. 18
`
`Netflix, Inc. - Ex. 1016, Page 000013
`
`

`

`Patent Application Publication Jul. 21, 2005 Sheet 13 of 15
`
`US 2005/0160413 Al
`
`GATHERING OF PERFORMANCE
`INFORMATION FROM GJVM NODES
`
`BEGIN
`
`GET NEXT REMOTE GJVM IN GJVM
`REGISTRY AT LOCAL GJVM
`2002
`
`SEND REQUEST FOR
`LOAD INFORMATION FROM
`LOCAL GJVM TO REMOTE GJVM
`~
`
`RECEIVE RESPONSE WITH LOAD
`INFORMATION FROM REMOTE GJVM
`2.0.QQ
`
`YES.
`
`END
`
`FIG.20
`
`SHUTDOWN OF GJVM NODE
`)
`( BEGIN
`i
`
`DETERMINE THAT LOCAL GJVM NEEDS TO
`PERFORM SHUTDOWN OPERATION
`~
`
`J
`
`SEND REQUEST TO PRIMARY GJVM TO
`CHANGE RUN STATE OF GJVM FROM
`RUNNING TO STOPPING
`1904
`i
`
`RECEIVE RUN STATE CHANGE
`ACKNOWLEDGMENT FROM PRIMARY GJVM
`1906
`J
`
`MIGRATE THREADS
`TO OTHER GJVM NODES
`~
`
`t
`
`MIGRATE OBJECT GROUPS
`TO OTHER GJVM NODES
`1910
`J,
`SEND REQUEST TO PRIMARY GJVM TO
`CHANGE RUN ST ATE OF GJVM FROM
`STOPPING TO STOPPED
`.1fil.2
`t
`
`RECEIVE RUN STATE CHANGE
`ACKNOWLEDGMENT FROM PRIMARY GJVM
`~
`
`i
`
`END
`
`(
`)
`FIG. 19
`
`Netflix, Inc. - Ex. 1016, Page 000014
`
`

`

`Patent Application Publication Jul. 21, 2005 Sheet 14 of 15
`
`US 2005/0160413 Al
`
`PERFORMANCE INFORMATION
`TRANSFER FROM GJVM NODE
`
`BEGIN
`
`RECEIVE REQUEST FOR
`LOAD INFORMATION AT REMOTE
`GJVM FROM LOCAL GJVM
`2102
`
`SEND RESPONSE WITH
`LOAD INFORMATION FROM
`REMOTE GJVM TO LOCAL GJVM
`2104
`
`END
`
`FIG. 21
`
`NO
`
`SELECT REMOTE GJVM WITH
`AVAILABLE MEMORY RESOURCE
`2216
`
`MIGRATE OBJECT GROUPS TO
`REMOTE GJVM FROM LOCAL GJVM
`2218
`
`END
`
`FIG. 22
`
`LOAD BALANCING BETWEEN
`GJVM NODES
`
`BEGIN
`
`START LOAD BALANCE OPERATION
`2202
`
`CHECK LOCAL LOAD VALUES
`2204
`
`GET LOADS AT REMOTE NODES
`2206
`
`SELECT REMOTE GJVM WITH
`AVAILABLE CPU RESOURCE
`2210
`
`MIGRATE THREADS TO REMOTE
`GJVM FROM LOCAL GJVM
`2212
`
`Netflix, Inc. - Ex. 1016, Page 000015
`
`

`

`'"""'
`>
`'"""' ~
`,i;;..
`0
`'"""' O'I
`0
`~
`0
`0
`N
`'JJ.
`d
`
`'"""' Ul
`0 ....,
`'"""' Ul
`~ ....
`'JJ. =(cid:173)~
`
`~
`
`Ul
`0
`0
`N
`!'"'
`N
`~
`
`.... 0 =
`~ ....
`O' -....
`~
`.... 0 =
`~ "Cl -....
`~ .... ~ = ....
`
`(')
`
`~ ....
`
`(')
`
`""C
`
`1
`LOGICAL GJVM "GRID_B" ~ :
`r--------------------
`
`PRIMARY GJVM NODE "A"
`
`1
`
`THREAD "C"
`MAIN THREAD SECONDARY
`APPLICATION
`
`2322
`"A"
`
`2324
`
`2344
`
`2342
`
`GROUP"E"
`
`GROUP "B"
`
`OBJECT
`
`OBJECT
`
`____________________ J
`I
`:
`:
`I
`:
`
`,..__ __________ __.,
`
`(JAVA-MAIN -GRID_B APPNAME)
`
`2312
`
`FIG. 23
`
`---------------------
`
`(JAVA-GRID_A APPNAME)
`
`2320
`
`--------------------
`-~-----------------
`
`2326
`
`2328
`
`THREAD "F"
`SECONDARY
`
`THREAD"E"
`SECONDARY
`
`I,..,--· ------------------
`
`2340
`
`2338
`
`GROUP "D"
`
`GROUP "A"
`
`JAVA APPLICATION 2334
`
`,-
`I L-------------------
`
`2330
`
`THREAD "B"
`SECONDARY
`
`I
`I
`THREAD"D"
`I
`I SECONDARY
`
`2332
`
`: ------·\--,----,-?
`
`2336
`
`2346
`
`GROUP"F"
`
`GROUP"C"
`
`i---
`
`NON-PRIMARY GJVM NODE "D"
`
`OBJECT
`
`OBJECT
`
`OBJECT
`
`OBJECT
`
`(JAVA-GRID_A APPNAME)
`
`(JAVA-GRID_A APPNAME)
`
`(JAVA-MAIN -GRID_A APPNAME)
`
`2318
`
`2316
`
`2314
`
`NON-PRIMARY GJVM NODE "C"
`
`NON-PRIMARY GJVM NODE "B"
`
`PRIMARY GJVM NODE "A"
`
`2310
`
`--------------------
`
`LOGICAL GJVM "GRID_A"
`
`r-----------------------
`
`2306
`
`MACHINE "C"
`
`2304
`
`MACHINE "B"
`
`2302
`
`MACHINE "A"
`
`Netflix, Inc. - Ex. 1016, Page 000016
`
`

`

`US 2005/0160413 Al
`
`Jul. 21, 2005
`
`1
`
`METHOD AND SYSTEM FOR A GRID-ENABLED
`VIRTUAL MACHINE WITH MOVABLE OBJECTS
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`[0001] The present invention is related to the following
`patent application, hereby
`incorporated by reference:
`"Method and System for Grid-enabled Virtual Machines
`with Distributed Management Of Applications", U.S. patent
`application Ser. No. ___ (attorney docket no.
`AUS920030872US1), filed (TED).
`
`BACKGROUND OF THE INVENTION
`
`[0002] 1. Field of the Invention
`[0003] The present invention relates to an improved data
`processing system and, in particular, to a method and
`apparatus for multicomputer data transferring and program
`execution.
`[0004] 2. Description of Related Art
`[0005] The widespread adoption of Internet-related and
`Web-related technology has enabled the growth of a global
`network of interconnected computers that are physically
`supported by many thousands of organizations and busi(cid:173)
`nesses and many millions of individuals. Recently, enter(cid:173)
`prises have exerted effort to organize the computational
`capacity of many computers into a grid, which is a logical
`organization of many computers for providing a collective
`sharing of computational capacity and datastore capacity
`while maintaining local autonomous control over individual
`computers.
`[0006] For example, many enterprises are cooperating
`within the Global Grid Forum™, which is supported by
`GGF, Inc., to develop community-driven standards for grid
`computing. The Globus Projects is one effort that has been
`supported by government agencies, corporations, and uni(cid:173)
`versities to develop grid-related technologies, which has
`resulted in the development of the Open Grid Services
`Architecture (OGSA), which is an initiative for incorporat(cid:173)
`ing grid concepts within a service-oriented framework based
`on Web services. The Globus Toolkit® is an implementation
`of the Open Grid Services Architecture that provides soft(cid:173)
`ware programming tools for developing grid-enabled appli(cid:173)
`cations, and the Java® Community Grid (CoG) Kit is the
`portion of the Globus Toolkit® that provides access to grid
`services through a Java® framework.
`[0007] Although the Java® CoG Kit supports a Java®
`application in accessing grid services, the Java® CoG Kit
`does not incorporate the computational characteristics of a
`grid into a Java® virtual machine (JVM). Therefore, it
`would be advantageous to have a method, a system, an
`apparatus, and/or a computer program product that incor(cid:173)
`porates computational characteristics of a grid within a
`virtual machine. It would be particularly advantageous to
`perform object management within a virtual machine such
`that the virtual machine may be implemented within a
`dynamically changing, grid computing environment.
`
`SUMMARY OF THE INVENTION
`
`[0008] A method, an apparatus, a computer program prod(cid:173)
`uct, and a data processing system provide for the operation
`
`of a virtual machine with embedded functionality for inter(cid:173)
`operating with other virtual machines in a computational
`grid. A plurality of virtual machines are run on one or more
`devices within a data processing system; each virtual
`machine in the plurality of virtual machines incorporates
`functionality for interoperating and associating with other
`virtual machines in a virtual machine cluster in a grid-like
`manner. Each virtual machine in the virtual machine cluster
`acts as a node within the virtual machine cluster. A virtual
`machine manages its objects in association with an object
`group, and each virtual machine may manage multiple
`object groups. The virtual machines share information such
`that the object groups can be moved between virtual
`machines in the virtual machine cluster, thereby allowing the
`virtual machine cluster to act as one logical virtual machine.
`A single application image conforming to a typical applica(cid:173)
`tion model can be executed within the grid-enabled virtual
`machine cluster in a way that enables the execution of the
`application to span multiple instances of a grid-enabled
`virtual machine, thereby allowing the execution of the
`application to span multiple devices.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`[0009] The novel features believed characteristic of the
`invention are set forth in the appended claims. The invention
`itself, further objectives, and advantages thereof, will be best
`understood by reference to the following detailed descrip(cid:173)
`tion when read in conjunction with the accompanying draw(cid:173)
`ings, wherein:
`[0010] FIG. lA depicts a typical network of data process(cid:173)
`ing systems, each of which may implement the present
`invention;
`[0011] FIG. lB depicts a typical computer architecture
`that may be used within a data processing system in which
`the present invention may be implemented;
`[0012] FIG. 2 depicts a block diagram that shows a virtual
`machine cluster in a distributed data processing system in
`accordance an implementation of the present invention;
`[0013] FIG. 3A depicts a block diagram that shows com(cid:173)
`ponents and/or information that may be used to enhance a
`virtual machine as a clusterable virtual machine in accor(cid:173)
`dance with an implementation of the present invention;
`[0014] FIGS. 3B-3D depict a set of block diagrams that
`shows the transference of object groups between clustered
`virtual machines;
`[0015] FIG. 4 depicts a flowchart that shows a process for
`initializing a primary cJVM node;
`[0016] FIG. SA depicts a flowchart that shows a process
`at a primary cJVM for allocating a unique cJVM identifier
`to a non-primary cJVM that is being initialized;
`[0017] FIG. SB depicts a flowchart that shows an alter(cid:173)
`native process at a primary cJVM for allocating a unique
`cJVM identifier to a non-primary cJVM that is being ini(cid:173)
`tialized;
`[0018] FIG. 6 depicts a flowchart that shows a process at
`a primary cJVM for allocating a unique object group iden(cid:173)
`tifier to a non-primary cJVM;
`[0019] FIG. 7 depicts a flowchart that shows a process at
`a primary cJVM for providing information about the cJVM
`cluster to a non-primary cJVM;
`
`Netflix, Inc. - Ex. 1016, Page 000017
`
`

`

`US 2005/0160413 Al
`
`Jul. 21, 2005
`
`2
`
`[0020] FIG. 8 depicts a flowchart that shows a process at
`a primary cJVM for changing a run state of a non-primary
`cJVM;
`[0021] FIG. 9 depicts a flowchart that shows a process at
`a primary cJVM for changing a migration state of an object
`group;
`[0022] FIG. 10 depicts a flowchart that shows a process
`for initializing a non-primary cJVM node;
`[0023] FIG.11 depicts a flowchart that shows a process on
`a clustered virtual machine during which a thread is con(cid:173)
`structed;
`[0024] FIG.12 depicts a flowchart that shows a process on
`a clustered virtual machine during which a thread is
`destroyed;
`[0025] FIG.13 depicts a flowchart that shows a process on
`a clustered virtual machine during which an object is con(cid:173)
`structed;
`[0026] FIG.14 depicts a flowchart that shows a process on
`a clustered virtual machine for handling accesses to objects;
`[0027] FIG. 15 depicts a flowchart that shows a process on
`a local clustered virtual machine for migrating an object
`group to a remote clustered virtual machine;
`[0028] FIG. 16 depicts a flowchart that shows a process on
`a local clustered virtual machine for accepting the migration
`of an object group from a remote clustered virtual machine;
`[0029] FIG. 17 depicts a flowchart that shows a process on
`a local clustered virtual machine for migrating a thread to a
`remote clustered virtual machine;
`[0030] FIG. 18 depicts a flowchart that shows a process on
`a local clustered virtual machine for accepting the migration
`of a thread from a remote clustered virtual machine;
`[0031] FIG. 19 depicts a flowchart that shows a process on
`a local clustered virtual machine for performing a shutdown
`operation on a clustered virtual machine;
`[0032] FIG. 20 depicts a flowchart that shows a process on
`a local clustered virtual machine for gathering load or
`performance information from a remote clustered virtual
`machine;
`[0033] FIG. 21 depicts a flowchart that shows a process on
`a local clustered virtual machine for responding to a request
`from a clustered virtual machine for the current load or
`performance information at the local clustered virtual
`machine;
`[0034] FIG. 22 depicts a flowchart that shows a process on
`a local clustered virtual machine for determining to migrate
`threads and/or object groups from the local clustered virtual
`machine to a remote clustered virtual machine in accordance
`with load or performance information; and
`[0035] FIG. 23 depicts a block diagram that shows the
`execution of a single application image that spans multiple
`grid-enabled virtual machines in a single logical virtual
`machine over multiple devices.
`
`DETAILED DESCRIPTION OF IBE
`INVENTION
`[0036]
`In general, the devices that may comprise or relate
`to the present invention include a wide variety of data
`
`processing technology. Therefore, as background, a typical
`organization of hardware and software components within a
`distributed data processing system is described prior to
`describing the present invention in more detail.
`[0037] With reference now to the figures, FIG. 1A depicts
`a typical network of data processing systems, each of which
`may implement a portion of the present invention. Distrib(cid:173)
`uted data processing system 100 contains network 101,
`which is a medium that may be used to provide communi(cid:173)
`cations links between various devices and computers con(cid:173)
`nected together within distributed data processing system
`100. Network 101 may include permanent connections, such
`as wire or fiber optic cables, or temporary connections made
`through telephone or wireless communications. In the
`depicted example, server 102 and server 103 are connected
`to network 101 along with storage unit 104. In addition,
`clients 105-107 also are connected to network 101. Clients
`105-107 and servers 102-103 may be represented by a
`variety of computing devices, such as mainframes, personal
`computers, personal digital assistants (PDAs ), etc. Distrib(cid:173)
`uted data processing system 100 may include additional
`servers, clients, routers, other devices, and peer-to-peer
`architectures that are not shown.
`
`[0038]
`In the depicted example, distributed data process(cid:173)
`ing system 100 may include the Internet with network 101
`representing a worldwide collection of networks and gate(cid:173)
`ways that use various protocols to communicate with one
`another, such as Lightweight Directory Access Protocol
`(LDAP), Transport Control Protocol/Internet Protocol
`(TCP/IP), Hypertext Transport Protocol (HTTP), Wireless
`Application Protocol (WAP), etc. Of course, distributed data
`processing system 100 may also include a number of dif(cid:173)
`ferent types of networks, such as, for example, an intranet,
`a local area network (LAN), or a wide area network (WAN).
`For example, server 102 directly supports client 109 and
`network 110, which incorporates wireless communication
`links. Network-enabled phone 111 connects to network 110
`through wireless link 112, and PDA 113 connects to network
`110 through wireless link 114. Phone 111 and PDA 113 can
`also directly transfer data between themselves across wire(cid:173)
`less link 115 using an appropriate technology, such as
`Bluetooth™ wireless technology, to create so-called per(cid:173)
`sonal area networks (PAN) or personal ad-hoc networks. In
`a similar manner, PDA 113 can transfer data to PDA 107 via
`wireless communication link 116. The present invention
`could be implemented on a variety of hardware platforms;
`FIG. lA is intended as an example of a heterogeneous
`computing environment and not as an architectural limita(cid:173)
`tion for the present invention.
`
`[0039] With reference now to FIG. 1B, a diagram depicts
`a typical computer architecture of a data processing system,
`such as those shown in FIG. lA, in which the present
`invention may be implemented. Data processing system 120
`contains one or more central processing units (CPUs) 122
`connected to internal system bus 123, which interconnects
`random access memory (RAM) 124, read-only memory 126,
`and input/output adapter 128, which supports various 1/0
`devices, such as printer 130, disk units 132, or other devices
`not shown, such as an audio output system, etc. System bus
`123 also connects communication adapter 134 that provides
`access to communication link 136. User interface adapter
`148 connects various user devices, such as keyboard 140 and
`mouse 142, or other devices not shown, such as a touch
`
`Netflix, Inc. - Ex. 1016, Page 000018
`
`

`

`US 2005/0160413 Al
`
`Jul. 21, 2005
`
`3
`
`screen, stylus, microphone, etc. Display adapter 144 con(cid:173)
`nects system bus 123 to display device 146.
`[0040] Those of ordinary skill in the art will appreciate
`that the hardware in FIG. 1B may vary depending on the
`system implementation. For example, the system may have
`one or more processors, such as an Intel® Pentium®-based
`processor and a digital signal processor (DSP), and one or
`more types of volatile and non-volatile memory. Other
`peripheral devices may be used in addition to or in place of
`the hardware depicted in FIG. 1B. The depicted examples
`are not meant to imply architectural limitations with respect
`to the present invention.
`[0041]
`In addition to being able to be implemented on a
`variety of hardware platforms, the present invention may be
`implemented in a variety of software environments. A typi(cid:173)
`cal operating system may be used to control program
`execution within each data processing system. For example,
`one device may run a Unix® operating system, while
`another device contains a simple Java® runtime environ(cid:173)
`ment. A representative computer platform may include a
`browser, which is a well known software application for
`accessing hypertext documents in a variety of formats and
`languages, such as graphic files, word processing files,
`Extensible Markup Language (XML), Hypertext Markup
`Language (HTML), Handheld Device Markup Language
`(HDML), Wireless Markup Language (WML), and various
`other formats and types of files.
`[0042] Virtual Machine Cluster/Grid
`[0043] The present invention may be implemented on a
`variety of hardware and software platforms, as described
`above with respect to FIG. 1A and FIG. 1B. More specifi(cid:173)
`cally, though, the present invention is directed to an
`improved virtual machine that includes computational char(cid:173)
`acteristics of a grid within the improved virtual machine
`such that the improved virtual machine may interact with
`similar improved virtual machines in a grid-like manner, as
`described with respect to the remaining figures. Although
`the following figures are described as incorporating Java®
`technology such that the present invention may be imple(cid:173)
`mented as improved Java® virtual machines (JVMs), the
`present invention may be implemented in conjunction with
`technology for other types of virtual machines.
`[0044] With reference now to FIG. 2, a block diagram
`illustrates a virtual machine cluster/grid in a distributed data
`processing system in accordance with an implementation of
`the present invention. Network 200 connects a set of virtual
`machines 201-205; network 200 is similar to network 101 in
`FIG. 1, and virtual machines 201-205 may be supported on
`a variety of devices, such as any of the devices that are
`shown in FIG. 1.
`[0045] By way of introducing terminology, a cluster is a
`set of two or more interacting entities. A cl