United States Patent (19)
`HOSOkawa et al.
`
`54 CLUSTER CONTROLLING SYSTEM
`OPERATING ON A PLURALITY OF
`COMPUTERS IN A CLUSTER SYSTEM
`
`75 Inventors: Takehiko Hosokawa; Kaoru Tsuru,
`both of Tokyo, Japan
`
`73 Assignee: Mitsubishi Denki Kabushiki Kaisha,
`Tokyo, Japan
`
`21 Appl. No.: 08/953,632
`22 Filed:
`Oct. 17, 1997
`30
`Foreign Application Priority Data
`Oct. 28, 1996
`JP
`Japan .................................... 8-285398
`Mar. 27, 1997
`JP
`Japan .................................... 9-075254
`51 Int. Cl."
`G06F 13/00
`
`US006088727A
`Patent Number:
`11
`(45) Date of Patent:
`
`6,088,727
`Jul. 11, 2000
`
`5,129,080 7/1992 Smith.
`5,598,529
`1/1997 Garay et al..
`5,781,736 7/1998 Schmidt ............................. 395/200.54
`5,787,249 7/1998 Badovinatz et al. ............... 395/200.31
`
`Primary Examiner Zarni Maung
`57
`ABSTRACT
`A cluster controlling System transferS packages which have
`been operating on one computer to another computer when
`a fault or failure has occurred by monitoring and controlling
`the packages in the entire System. When the respective
`packages are Started-up, cluster daemons on the respective
`computerS monitor and control resources on the operating
`computers. The monitored and controlled data are Stored in
`the respective computers as local data. A manager commu
`
`O
`
`-1 - O
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
`
`nicates with cluster daemons on the respective computers,
`
`E. For search - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - "...i. and Stores data in a global data memory to monitor and
`395/800.28, 200.31, 200.35, 200.43, 200.53,
`control the entire System. The manager is actually one of the
`200.73; 709/223, 201, 205, 213, 243, 225
`packages operating in the cluster System. If a fault or failure
`• -1s
`s a Yu us a Yu as a u-1s a
`-1s
`occurs in the manager or in the computer running the
`References Cited
`manager, the manager is re-started on another computer by
`a cluster daemon.
`
`56)
`
`U.S. PATENT DOCUMENTS
`4,356,546 10/1982 Whiteside et al..
`
`20 Claims, 47 Drawing Sheets
`
`re
`
`N
`
`103a2
`
`C PACKAGE B1
`
`O3a1
`
`
`
`
`
`O3b1
`
`06;
`GLOBALY
`DATA
`
`(EACKAGE2): 1 O 3 n 2
`
`
`
`
`
`
`Ex.1007 / Page 1 of 66Ex.1007 / Page 1 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 1 of 47
`
`6,088,727
`
`is a go
`
`was
`
`a
`
`a
`
`us
`
`N NOWEV/C]
`
`HELST?TO
`
`€ NOWEV/C]
`HE 1ST TO
`
`\/ NOWEV/C)]
`
`HELST?TO
`
`
`
`
`
`
`
`was
`
`a
`
`is
`
`as
`
`a
`
`a
`
`sm - - - -n a
`
`in
`c
`C
`O
`y
`
`Y - - - - - - - - - a
`
`1
`
`
`Ex.1007 / Page 2 of 66Ex.1007 / Page 2 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 2 of 47
`
`6,088,727
`
`
`
`a1VLSSOHNOSAY
`
`ADYNOSSY
`
`ALVLS
`
`SNITIOWULNOD
`
`AOYNOSAY
`
`ONISSSOOUd
`
`AdIAAG
`
`SNISILON
`
`QNVW34d
`
`ADNVHOALVLS
`
`aOYNOSIAY
`
`
`
`JDNVHDALVLS
`
`SNISSADOUd
`
`SSIAAG
`
`
`
`SDIAAGONILLAS
`
`ASILON
`AAILON
`
`ASILON
`
`2ted
`
`YADVNVA
`
`ONVW3G ASIAAG
`
`
`
`
`SLVLSAOWNOSAYAHL=QNyW3dSNTIOHLNOOJINSSYSSIOO"d«=ALVISSOWNOSSY4O
`
`NIvLa0OLGNVWAGFOYUNOSSYFDNVHOSHLASILLON
`
`
`NOWSVdY31SN10WOHSHaSNOL
`
`NOWavdY3LSN10OLYasnWOouSs
`s0¢
`TSTI
`
`—_—_—_—_—_—__——_—_—_—_——
`AOWNOSSAY
`
`NOILIGNOSee
`
`SOILON
`
`
`
`VivdTvg019
`
`viadgaonuwiasbeled"
`
`
`Ex.1007 / Page 3 of 66Ex.1007 / Page 3 of 66
`Ex.1007 / Page 3 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 3 of 47
`
`6,088,727
`
`80€
`
`LOE
`
`ONVWSG
`
`JOYNOSSYSHLNIVLEO
`Y3ASNAHLOLLINSAY
`dd3LVLSJowNosaY
`
`
`$S390udLINSNVYL
`ALVLSAHLWHOSNI
`
`ASILLONSLV0dN
`FiviSJOUNOSSY1aS=@0E
`
`
`JOILONSDNVHOdO3dAL
`ADYNOSAYAHLTOULNOSD
`
`
`Y3SNSHLWOYsGNVAWSCV8OsLIVM
`90€
`
`
`LINSAYSSADOWdLINSNVYL
`YALNdWODdVOLGNYWad
`AOYNOSSYSHLWOuS
`YALNdWODVNIV.LEO
`
`TOXLNOSLINSNVEL
`ADYNOSAYHLIM
`
`ADYNOSAYHLIM
`
`
`ALVLSJOYNOSAYNIVLEO
`AOWNOSAYWOYS
`YW3SnSHLOL
`
`gdSNILLAS
`
`
`
`ALVLSNIVLEO
`
`60€LINWSNVEL
`
`ALVLSSHL
`
`SOE
`
`gd3LlVvLs
`
`gd3A1V1s
`
`Ole
`
`LlLe
`
`
`
`WOUSALVLS
`
`Y3SNAHLOL
`
`ZOE
`
`
`
`Y¥aSNAHLOL
`
`€Sls
`
`
`Ex.1007 / Page 4 of 66Ex.1007 / Page 4 of 66
`Ex.1007 / Page 4 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 4 of 47
`
`6,088,727
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FOR ALL THE
`RESOURCES
`
`ALL RESOURCES
`ARE PROCESSED
`
`4O2
`
`OBTAIN THE
`RESOURCE STATE
`
`SAME AS RESOURCE
`STATE DB
`
`FALSE
`
`404
`
`UPDATE THE RESOURCE
`STATE DB
`
`REGISTERED IN
`NOTICE
`SETTING DB2
`
`TRUE
`
`4O6
`
`NOTIFY THE STATE
`CHANGE TO THE USER
`
`STOPFOR A
`PREDETERMINED TIME
`
`FIG. 4
`
`
`Ex.1007 / Page 5 of 66Ex.1007 / Page 5 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 5 of 47
`
`6,088,727
`
`
`
`
`
`
`
`\/ SSE OOHd
`
`
`
`
`
`
`
`
`
`
`Ex.1007 / Page 6 of 66Ex.1007 / Page 6 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 6 of 47
`
`6,088,727
`
`UZO !
`
`N NOWEV/C)]
`HE 1ST TO
`
`LI 109
`
`TWOOT
`
`N VIVO Y uÞ0||
`
`N HELLT, WOO
`
`U 101© ?? ? ?%101
`
`E HELT\d|WOO
`
`- - - - a
`
`or
`
`a
`
`s
`
`a
`
`is is a
`
`as
`
`on a
`
`„ - - - - - - - - -- - - )
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`\/NOWEV/C)]
`
`HE 1ST TO
`
`
`
`
`
`
`
`
`
`
`Ex.1007 / Page 7 of 66Ex.1007 / Page 7 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 7 of 47
`
`6,088,727
`
`DNISSSDOWd
`
`AOYWNOSSY
`
`
`
`AOnNosay
`
`L0Z
`
`W3SNOL————
`ee—_————
`—_—
`IN35vVWOH
`yWasnWows
`INSDVOL
`zizaYa435uY
`
`AONVHOSLVLS
`
`
`
`
`
`ONVWAdIOHLNOOLInsaySSa00udFONVHOALVIS
`3OuNOSSYFOUNOSSYFHLASILLON
`
`
`
`
`FSIAAC|po2qEleniONITIOWULNODDNISSSOOUdSNIAALLONSIAR
`39YNOSsaYONVWadFONVHOALVLS7SOYNOSAY
`
`
`
`=Z°Old
`siwis||SCONuRs
`
`
`VLVGIW80TD
`<a
`
`SIACONILLAS
`SOLLON|ASILLON
`ONVWadYasn
`YADVNVAN
`
`QNVW3d
`
`AALLON
`
`
`Ex.1007 / Page 8 of 66Ex.1007 / Page 8 of 66
`Ex.1007 / Page 8 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 8 of 47
`
`6,088,727
`
`NIVLGO0OLGNVWSd=JOKLNOOS0u8NOSsad
`
`
`
`NOWAVGY¥3.LSN19OL
`
`renee
`
`el
`
`AOWNOS3Y
`
`ALVLS
`
`SOYNOSSY
`
`
`
`
`
`SONVHOSLVLS
`
`FOYUNOSSY
`
`AOIAIG
`
`vOG|ONIYOLINOW
`ENITIOULNOO
`JOIN|po2q
`
`
`
`ALVLSSOYWNOS3AYSHLQNVWSd
`
`
`
`NOWSVdYSLSN19WOWSs
`
`ONISSAOOW¥daOnOssyY
`
`
`SOtAadaivdadnSLVLS
`
`PONISSADOY"dSNIASLLON
`
`
`ONVWaqaSDNVHOALVLS
`
`SolAgaADIAAG
`
`ad
`
`JOWLNOOADNOS3HY
`
`ONVW3d
`
`ADYNOSSYASILON
`HADVNVAWWoks
`SJOYNOSSY
`Enalee
`YSDVNVINOL
`
`ADNVHOSLVLS
`8Vis
`
`
`Ex.1007 / Page 9 of 66Ex.1007 / Page 9 of 66
`Ex.1007 / Page 9 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 9 of 47
`
`6,088,727
`
`ADIAIO
`
`|
`
`3ouNOssY
`
`AHLASILON
`
`NOWAVdYALSN19OL
`LN1——
`
`SAONVHOALVLS
`ONISSADOUd
`SOWNOSsY
`
`|NOLLVOISLLON ONVAWSa
`
`
`
`NOWS3VdH3LSN1DWOH4JOYHINOD3JOuNOSSY
`
`FOIARGDSNILLESSOIAAG|40zq30IASG
`
`
`
`
`SJOLLONONITIOWULNODONISSSOOHd
`
`YADVNVAWOdS
`
`—___—_e_—_—e—enee
`
`YADVNVAOL
`
`SOYNOSSYAHLASILON
`
`
`
`ADNVHOALVLS
`
`
`
`ADNVHOALVLS
`
`NOLIVWHOSWI
`
`ADYNOSAY
`
`SSIASG
`
`6Dis
`
`
`Ex.1007 / Page 10 of 66Ex.1007 / Page 10 of 66
`Ex.1007 / Page 10 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 10 0f 47
`
`6,088,727
`
`
`
`
`
`
`
`
`
`
`
`WAIT FOR RESOURCE
`STATE CHANGE NOTICE
`
`1001
`
`UPDATE THE RESOURCE
`STATE DB
`
`1002
`
`1003
`
`REGISTERED IN
`NOTICE
`SETTING DB2
`
`NOTIFY THE STATE
`CHANGE TO THE USER too
`
`FIG. 10
`
`
`Ex.1007 / Page 11 of 66Ex.1007 / Page 11 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 11 0f 47
`
`6,088,727
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FOR ALL RESOURCE
`ON THE COMPUTER
`
`OBTAIN THE
`RESOURCE STATE
`
`1102
`
`STATE IS SAMEAS
`THE STATE DB
`
`UPDATE THE RESOURCE
`STATE DB
`
`NOTIFY THE STATE
`CHANGE TO THE MANEGER 1105
`
`STOPFOR APREDETERMINEDIY
`TIME PERIOD
`1106
`
`FIG. 11
`
`
`Ex.1007 / Page 12 of 66Ex.1007 / Page 12 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 12 of 47
`
`6,088,727
`
`UZOl
`
`NNOWSVd
`
`YALSN19
`
`ULOL
`
`NYALNdWOO
`
`ee we i ee
`c
`oO
`oS-
`
`-C
`
`eLOL@HALNdNOOVHSLNdWOO
`901~~IVdO1D
`@NOWSVvd
`YALSNT9
`ZVJOWHOWdD|LeEOl IYdDvVMOVdD>}
`CSViVvdCS
`obOld
`
`Vivd
`
`ee ee ee oe ee
`
`cPEOL
`
`
`
`I
`
`1Ii
`
`LOL
`
`L
`
`WO01
`
`€vivd
`
`VNOWAVG
`
`YALSNT9D
`
`ep0l
`
`
`Ex.1007 / Page 13 of 66Ex.1007 / Page 13 of 66
`Ex.1007 / Page 13 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 13 of 47
`
`6,088,727
`
`1301
`
`1302
`
`USER
`PROCESSA
`
`
`
`USER
`PROCESS B
`
`ors
`
`COMPUTER 2 . . . . . . . . . . . COMPUTER n
`
`101 a
`
`101b.
`
`101 in
`
`FIG. 13A
`
`1301
`
`1302
`
`USER
`PROCESSA
`
`
`
`
`
`601 a
`
`
`
`601n
`
`COMPUTER 1
`
`COMPUTNER 2 . . . . . . . . . . .
`
`COMPUTER in
`
`1 O1a
`
`101n
`
`FIG. 13B
`
`
`Ex.1007 / Page 14 of 66Ex.1007 / Page 14 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 14 of 47
`
`6,088,727
`
`
`
`
`
`
`
`W
`
`V
`
`U 10 ||
`
`N HELLT, WOO
`
`N NOWEV/C]
`
`HELST?TO
`
`V NOWEVCI
`HE 1ST TO
`
`
`Ex.1007 / Page 15 of 66Ex.1007 / Page 15 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 15 of 47
`
`6,088,727
`
`15O1
`
`15O2
`
`USER
`PROCESSA
`
`USER
`PROCESS B
`
`105
`
`2 601 a
`
`6O1b
`
`601 n
`
`COMPUTER
`
`COMPUTER 2
`
`. . . . . . . . . . . COMPUTER
`
`1 Oa
`
`1Ob
`
`1 On
`
`
`
`
`
`COMPUTER 1
`
`101 a
`
`601n
`
`COMPUTER n
`
`1 On
`
`
`Ex.1007 / Page 16 of 66Ex.1007 / Page 16 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`
`NOW3VdY3LSN10WOHsensaasna—_—_—_—_—_—_—_—_—_—_—_—_—
`errand
`
`
`
`
`
`ONVWadONITIOWINOSSOuNOSSYLinsgusss00udFONVHOALVLS
`
`
`
`
`
`——K—K—_—_—_—_—ey
`
`
`
`NOW3VdHALSNT9OLWasnWOW
`
`
`
`SLVLS3OYUNOSAY
`
`a0unNossHY
`
`FDNVHO
`
`ALVLS
`
`
`SNITIOU.LNOODSNISSSOO"ddivls
`aowNossYaNvWwadaOUNOSSe
`
`
`
`
`
`aoAadSONVHO
`
`SNIASLLON
`
`SdIAAG
`
`ONITIOULNOD
`
`DILVWOLNY
`
`Jul. 11, 2000
`
`Sheet 16 of 47
`
`6,088,727
`
`
`
`
`
`SNISSSO0UdSNILLASSOLLON
`
`aAOYUNosSAY
`
`gqSLVLs
`
`WadOTSD
`
`viva
`
`9}‘Ola
`
`YHSaDVNVA
`
`
`Ex.1007 / Page 17 of 66Ex.1007 / Page 17 of 66
`Ex.1007 / Page 17 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 17 Of 47
`
`6,088,727
`
`
`
`RESOURCE NAME (EVENTNAME)
`PROCESS
`
`if (pkg2.state == down)
`pkg2->startup
`
`10:
`11: pkg 1 (startup) {
`12:
`if (pkg2.state == up 8& pkg2.node == pkg1.node)
`13:
`pkg2->halt
`14:
`15: }
`16:
`17: pkg2 (startup)
`18:
`if (pkg1.state == up &&. pkg2.node == pkgi.node){
`19:
`pkg2->halt
`
`}
`
`21: }
`
`1: if PACKAGENAME: STATE
`2: pkg 1: Ipkg2
`3: pkg2 ... pkg 1
`4. pkg3 : pkg1 pkg2
`
`FIG. 18
`
`
`Ex.1007 / Page 18 of 66Ex.1007 / Page 18 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`
`
`NOWSVCH3LSN10OL=
`
`
`
`—___Y3SNOL
`
`¥W3aSNWOYS
`
`
`
`NOWAVdH3.LSM10WOHsTOXLNOD30uNOSSsYFONVHOSHLASILON
`
`U.S. Patent
`
`
`
`
`
`
`
`Jul. 11, 2000
`
`SNISSADOWd
`
`QNVW3d
`
`SSIS
`
`
`
`SDNVHOALVLS
`
`SDYNOS3AY
`
`SNIASILON
`
` SOIAAG
`
`
`divlsJOWNOSsaYQNVWSd0Y3Sn
`
`Sheet 18 of 47
`
`6,088,727
`
`
`
`ADNVHSALVLS
`
`SNISSSOOkd
`
`aDYNOSaY
`
`SDIAIG
`
`aSNOSSY
`
`ALVLS
`
`8a
`
`
`
`VivdTwdOTd
`
`YADVNVA
`
`6+Sls
`
`
`Ex.1007 / Page 19 of 66Ex.1007 / Page 19 of 66
`Ex.1007 / Page 19 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`
`
`
`
`
`
`ONVWaAGLINSSYSSA00udFLVLSJOUNOSAYJO
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 19 of 47
`
`6,088,727
`
`
`
`OPERATING
`
`OPERATION SYSTEM STOP
`
`START-UP (OPERATING)
`
`START-UP (WAITING)
`
`FIG. 20
`
`
`Ex.1007 / Page 20 of 66Ex.1007 / Page 20 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 20 0f 47
`
`6,088,727
`
`2012
`
`10 12
`
`
`
`
`
`y 6Md
`
`
`Ex.1007 / Page 21 of 66Ex.1007 / Page 21 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 21 of 47
`
`6,088,727
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`UPDATE
`
`
`
`5DNISSE OOH,
`
`EOLAECI
`
`EOLAEC]
`?N?TTOH LNO O
`
`
`
`©NISSE OOHc]
`
`EOLAECI | LOZ
`
`
`
`EOLAECI SON|| || ES
`
`EO||LON
`
`
`Ex.1007 / Page 22 of 66Ex.1007 / Page 22 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 22 of 47
`
`6,088,727
`
`NOTICE
`
`DEMAND
`
`NOTICE
`PROCESSING
`DEVICE
`
`MODE
`CONTROLLING|2304
`DEVICE
`
`NOTICE
`
`105
`
`MANAGING
`TABLE
`
`MAGING
`UPDATE/REFER DEVICE
`
`MANAGER
`
`FIG. 23
`
`
`
`PACKAGENAME
`
`OPERATING
`
`WAITING
`
`FIG. 24
`
`
`Ex.1007 / Page 23 of 66Ex.1007 / Page 23 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 23 of 47
`
`6,088,727
`
`INITIALIZE THE TABLE
`
`25O1
`
`WAIT FOR THE NOTICE
`
`25O2
`2503
`START-UP THE COMPUTER
`
`
`
`TYPE OF
`THE NOTICE
`
`STOP THE COMPUTER
`
`
`
`CHANGE THE MODE
`OF THE TABLE TO "STOP"
`
`START-UP THE
`PACKAGE
`
`IS THERE A
`PACKAGE
`WITH MODE
`"WAITING" ?
`
`IS THERE A
`PACKAGE
`WITH MODE
`"OPERATING" 2
`
`
`
`2507
`
`2508
`
`2506
`
`
`
`SWITCH THE MODE OF
`THE PACKAGE TO "OPERATING"
`
`CHANGE THE MODE
`OF THE PACKAGE
`TO "OPERATING"
`
`2509
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`UPDATE THE TABLE
`
`251O
`
`FIG. 25
`
`
`Ex.1007 / Page 24 of 66Ex.1007 / Page 24 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 24 of 47
`
`6,088,727
`
`Geol
`
`PE0l
`
`qe0l
`
`U9‘Sls
`
`V9C
`‘Old
`
`(@HALNdWOD)
`
`ONILVHSdO
`
`LOL
`
`PLOL
`
`qeol
`
`PEOL
`
`@byd
`
`(@HALNdNOD)
`(WHALNdWOD)vByd
`
`ONILIVM
`
`ONILVY3dO
`
`¢SGON
`
`tJGON
`
`LOL)ONILVYadO
`
`ONILIVM
`
`BLOL
`
`qLOL
`
` JGON
`
`¢JOON
`
`(@HALNdWOD)g6Byd
`(VWHALNdWODd)
`
`Y¥Byd
`
`(4HALNdNOD)g5yd
`
`ONILIVM
`
`Q9eDls
`
`d9e
`‘Old
`
`
`Ex.1007 / Page 25 of 66Ex.1007 / Page 25 of 66
`Ex.1007 / Page 25 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 25 of 47
`
`6,088,727
`
`NOTICE
`
`DEMAND
`
`NOTICE
`PROCESSING
`DEVICE
`
`OUTPUT
`SUPPRESSING
`DEVICE
`
`DEMAND
`
`NOTICE
`
`105
`
`MANAGNG
`TABLE
`
`RENAENG
`UPDATE/REFER DEVICE
`
`MANAGER
`
`FIG. 27
`
`
`
`
`
`PACKAGENAME
`
`pkg A
`
`RELEASE
`
`
`
`FIG. 28
`
`
`Ex.1007 / Page 26 of 66Ex.1007 / Page 26 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 26 of 47
`
`6,088,727
`
`NITALIZE THE TABLE
`
`2901
`
`WAIT FOR THE NOTICE
`
`2902
`2503
`START-UP THE COMPUTER
`
`TYPE OF
`THE NOTICE
`
`STOP THE COMPUTER
`
`CHANGE THE OUTPUT
`SUPPRESSION TO "SUPPRESS"
`
`START-UP THE
`PACKAGE
`
`IS THERE A
`PACKAGE
`WHOSE OUTPUT IS
`SUPPRESSED?
`
`
`
`IS THERE A
`PACKAGE
`WITH RELEASED
`OUTPUT
`UPPRESSIO
`
`2907
`
`2908
`
`RELEASE THE OUTPUT
`SUPPRESSION OF THE
`PACKAGE
`
`RELEASE THE OUTPUT
`SUPPRESSION OF THE
`PACKAGE
`
`2909
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`UPDATE THE TABLE
`
`291O
`
`FIG. 29
`
`
`Ex.1007 / Page 27 of 66Ex.1007 / Page 27 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 27 of 47
`
`6,088,727
`
`asva1sy
`
`
`
`INdLnoINdiLno
`
`
`
`NoIssaudns‘+-4--NoissavdansKSSSitis
`
`
`
`
`
`
`
`GeolqeoleE0l
`
`NOISS3udNs
`
`1NdLno
`
`
`
`QqLOLe101
`
`VOeSis
`
`S$Sd00udqByd
`
`ssd00udd
`
`vyByd
`
`¢SGON!JOON
`
`
`
`qLOLeLOLqLoL
`
`
`
`Q0de‘SlsdoeSis
`
`
`Ex.1007 / Page 28 of 66Ex.1007 / Page 28 of 66
`Ex.1007 / Page 28 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 28 of 47
`
`6,088,727
`
`
`
`HE-JEH / E LVC]dn
`
`
`
`€)NISSE OOHc]
`
`EOLAEC]
`
`
`Ex.1007 / Page 29 of 66Ex.1007 / Page 29 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 29 of 47
`
`6,088,727
`
`
`
`E
`W
`
`z dnou6 | dnou6 | dnou6
`
`
`
`VN ETIOH!3) | HB L?ldWOO ?NLLÍTOBOXE
`
`ECJOWN
`
`ZEGION
`
`
`
`5DNI LIV/WW
`
`
`
`5DNI LIV/WW
`
`
`
`
`
`
`
`| ?NDVEJO
`€28 ’5)|–|
`
`NOLLICINOO
`HELLflc|WOO
`
`| ECJON
`
`ZEICION
`
`€EGION
`
`†7EGION
`
`
`Ex.1007 / Page 30 of 66Ex.1007 / Page 30 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 30 0f 47
`
`6,088,727
`
`INITIALIZE THE TABLE
`
`WAIT FOR THE NOTICE
`
`32O2
`3203 START-UP THE
`COMPUTER
`
`
`
`
`
`TYPE OF
`THE NOTICE
`STOP THE
`COMPUTER
`CHANGE THE MODE OF
`THE TABLE TO "STOP"
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`IS THERE A
`PAKCAGE WITH MODE
`"WAITING" IN THE
`
`SWITCH THE OPERATION OF THE
`PACKAGETO "OPERATING"
`
`CHANGE THE MODE OF
`THE TABLE TO "OPERATING"
`
`
`
`3208
`
`IS THERE
`A COMPUTER WHICH
`CAN OPERATE THE
`STOPPED
`PACKAGE2
`
`YES
`START-UP THE PACKAGE
`
`NO
`
`
`
`
`
`
`
`
`
`IS THERE A
`PACKAGE WITH
`MODE "OPERATING"
`THE GROUP2
`
`YES
`
`
`
`NO
`SWITCH THE OPERATION OF THE
`PACKAGETO "OPERATING
`
`321
`
`FIG. 33
`
`UPDATE STATE
`
`3212
`
`
`Ex.1007 / Page 31 of 66Ex.1007 / Page 31 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 31 of 47
`
`6,088,727
`
`
`
`ULOLthot
`
`(@HALNdWOD)
`
`ONILVHSdO
`
`
`
`EGONU3GON
`
`
`
`
`EOLr~--.___9E0L“ens
`ooo1onuivm‘\ONILVEHSdO‘\(¥HSLNdWoO)}(gWELAAWOO)}\vodJagby/\,/
`
`ee30ON\.u3CON
`
`PEOL
`
`(@HALNANOD)
`(¥YaLNdWwod)
`
`ONILIVAA
`ONILWHSdO
`
`gSyd
`
`vyBud
`
`VveSls
`
`
`
`GLO}BLOLa
`
`(@HSLNdNOD)qSyd
`
`ONILIVM
`
`
`
`
`
`UbOFqLoLULOL
`
`dveéSls
`
`dveSls
`
`
`Ex.1007 / Page 32 of 66Ex.1007 / Page 32 of 66
`Ex.1007 / Page 32 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 32 of 47
`
`6,088,727
`
`
`
` dgseSis ONILVYAdOu30ONONILWYAdO2JGON
`
`Ii1!
`
`dOLsLlJGON
`IIII
`
`
`ALVLSYALNdWOO
`
`
`
`HALNdNWOODNILNOZOXSJWNVNSDvVyOVd
`
`|3GONvvByd
`
`2SGONqByd
`
`
`
`(t-UJQ0ON(}-u)Byd
`
`VSeDis
`
`
`Ex.1007 / Page 33 of 66Ex.1007 / Page 33 of 66
`Ex.1007 / Page 33 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 33 of 47
`
`6,088,727
`
`NITIALIZE THE TABLE
`
`WAIT FOR THE NOTICE
`
`3502
`3503
`START-UP THE COMPUTER
`
`
`
`TYPE OF
`THE NOTICE
`
`STOP THE
`COMPUTER
`RETRIEVE A VACANT
`COMPUTER
`FROM THE TABLE
`
`
`
`IS THERE A
`VACANT
`COMPUTER2
`
`
`
`UPDATE THE TABLE
`
`
`
`3508
`
`IS THERE
`A COMPUTER
`HICH CAN EXCECUT
`THE STOPPED
`PAKCAGE2
`
`3507
`
`NO
`
`RE-START THE
`STOPPED PACKAGE
`ON THE VACANT
`COMPUTER
`
`EXCECUTE
`THE PACKAGE 3509
`
`UPDATE THE TABLE
`
`350
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 36
`
`
`Ex.1007 / Page 34 of 66Ex.1007 / Page 34 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 34 of 47
`
`6,088,727
`
`U
`
`
`
`
`
`
`
`
`
`
`Ex.1007 / Page 35 of 66Ex.1007 / Page 35 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 35 of 47
`
`6,088,727
`
`
`[aresF3GONvox
`
`
`
`SWVNdNOYD|YALNdWOOONILNDADXS|ANVNSADVHOVd
`
`
`LAGON|AUWLSHaLNdWOO
`ONILWYAdOcAIGON
`
`ONILVYadO
`age“DI4
`V8eSid
`
`
`Ex.1007 / Page 36 of 66Ex.1007 / Page 36 of 66
`Ex.1007 / Page 36 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 36 of 47
`
`6,088,727
`
`U 10 ||
`
`UECION
`
`(290||
`
`SSHOOHd g 6}{d
`
`
`
`SSE OOHdSSE OOHCH g 6×dy
`6Md
`
`
`
`
`
`
`
`
`
`
`
`
`Ex.1007 / Page 37 of 66Ex.1007 / Page 37 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 37 0f 47
`
`6,088,727
`
`
`
`
`
`EGEO A LIHOIH.) | HELTldWOO ?NLLT OBOXE | EWVN EÐ\/>''OVd
`
`
`
`
`
`E_L\/LSHELLTldWOO
`
`
`Ex.1007 / Page 38 of 66Ex.1007 / Page 38 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 38 of 47
`
`6,088,727
`
`INITIALIZE THE TABLE
`
`4001
`
`WAIT FOR THE NOTICE
`
`4002
`
`4003
`START-UP
`THE COMPUTER
`
`
`
`TYPE OF
`THE NOTICE
`
`STOP THE
`COMPUTER
`
`RETRIEVE A COMPUTER
`WHICH CANNOT
`EXCECUTE A PACKAGE
`
`RETRIEVE A STOPPED
`PACKAGE FROM
`THE TABLE
`
`IS THERE
`A STOPPED
`PACKAGE 2
`
`4007
`
`IS THERE A
`COMPUTER
`WHICH CAN
`EXCECUTE A
`PACKAGE 2
`
`4009
`
`S THERE A
`PACKAGE
`WHICH CAN BE
`STOPPED 2
`
`START-UP THE
`PACKAGE HAVING
`HIGHEST PRIOR
`
`STOP THE PACKAGE
`WHICH CAN BE
`STOPPED
`
`EXCECUTE THE
`PACKAGE
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`UPDATE
`THE TABLE
`
`FIG. 41
`
`
`Ex.1007 / Page 39 of 66Ex.1007 / Page 39 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 39 of 47
`
`6,088,727
`
`ALIMWOIddgByd
`
`ALIWOl¥d
`
`ALIMOldd
`
`daddO1s
`
`SigByd
`
`GaddOls
`
`SINByd
`
`UEO
`
`L
`
`ALI¢OIddNByd
`
`
`
`GLoLELOL
`
`
`
`Z300Nacon
`
`sees
`
`ULOL
`
`L30ON
`
`ULO
`
`
`
`QLOLBLOL
`
`.bYaGHO
`
`
`
`ZAQONLSGON
`
`“_
`
`Acv‘Sls
`
`
`
`Uov‘DIA
`
`
`Ex.1007 / Page 40 of 66Ex.1007 / Page 40 of 66
`Ex.1007 / Page 40 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 40 of 47
`
`devSis|og|eNuvuado|YSGON_|ONILVeAdO
`cI0ON
`
`ONILVYAdOL30ON
`
`ALVLSYALNdWOD
`ONILNOAOXA|ADVWHOVd
`LL3GONvbxd
`VevDis
`[—zacon|35]|eacon|Od
`
`eeYaLNdWOOAWVN
`ZI0ONgbyd
`
`[ao
`
`6,088,727
`
`
`Ex.1007 / Page 41 of 66Ex.1007 / Page 41 of 66
`Ex.1007 / Page 41 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 41 of 47
`
`6,088,727
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`INTIALIZE THE TABLE
`
`4301
`
`WAIT FOR THE NOTICE
`
`LOAD OF THE COMPUTER
`EXCEEDS 100%
`
`4303
`
`
`
`SELECTA PACKAGE
`ON THE COMPUTER 4304
`
`START-UP/STOP
`THE COMPUTER
`
`UPDATE
`THE TABLE
`
`4309
`
`SELECT A STOPPED
`PACKAGE WITH
`THE HIGEST PRIORITY 4310
`
`OBTAIN A COMPUTER
`WHICH CANNOT
`EXCECUTE A PACKAGE
`FROM THE TABLE
`
`STOP THE
`PACKAGE 4305
`
`
`
`UPDATE
`THE TABLE
`
`HAVE
`ALL THE
`PAKCAGES ON THE
`COMPUTER BEEN
`PROCESSED?
`
`DOES LOAD
`OF THE COMPUTER
`EXCEEDS
`100%2
`
`
`
`
`
`
`
`
`
`
`
`NO
`
`FIG. 44
`
`IS THERE A
`COMPUTER WHICH
`CAN EXCECUTE A
`PACKAGE
`
`
`
`
`
`START UP
`THE PACKAGE
`
`
`
`UPDATE
`THE TABLE
`
`S THERE A
`PACKAGE WHICH
`CAN BE
`STOPPED2
`
`
`
`
`
`STOP THE
`PACKAGE WHICH
`CAN BE STOPPED
`
`
`
`435
`
`HAVE ALL
`THE STOPPED
`PACKAGES BEEN
`PROCESSED2
`
`
`
`YES
`
`UPDATE
`THE TABLE 4318
`
`
`Ex.1007 / Page 42 of 66Ex.1007 / Page 42 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 42 of 47
`
`6,088,727
`
`PEOL
`
`ALIWOIddaByd
`
`%0Z¢HS0HO
`
`2EO1GeOt
`
`WOERWC
`
`9H#AdYWOS$HaAGHO'
`
`401E01~aer"
`ALIWONWdALMOWd!+Byd3Byd\Cgbyd,
`
`
`
`ALIYO!d>Byd
`
`€H3qYO
`
`%Ov
`
`ALIYOINdqByd
`
`%0P@WaqHO
`
`ALIWONdd
`
`%0PL¥aGHO
`
`VvByd
`
`%09-0V01
`
`%08-dVO1
`
`%08-:0V01
`
`UAGON
`
`cAqON
`
`tS0G0N
`
`LoL
`
`
`
`OSr‘Dis
`
`
`
`sE04SE0LPEOL
`
`ALMdOldd4Byd
`
`%0G9HACHO
`
`ALIMOIdd3Byd
`
`%0S§H3q0HO
`
`
`
`9E0!GEOLBE0l
`
`ALIWNOIUdqyd
`
`%O0ZyHa0YO
`
`ALIOIdd©Syd
`
`%0b&HS0HO
`
`ALIPOldd
`
`%OP@YaqdYO
`
`qByd
`
`ALIMOIdd
`
`VvByd
`
`%OPLHaGHO
`
`%09:0VO71
`
`%09:0VO1
`
`U30ON
`
`cAqON
`
`%09:0V01
`
`bS0ON
`
`VSvSls
`
`qtol
`
`BLOL
`
`JE01
`
`SEOPEOL
`
`ALIMOIdd
`
`%0Z9H3q0YuO
`
`sjByd
`
`ALIWOIdd3Byd
`
`S$HAqYO
`
`%02
`
`9EOLqeolPE0l
`
`ALMOlYdqSyd
`
`%0ebHAqUO
`
`ALMOldd5Byd
`
`€YACHO
`
`%0P
`
`ALWOlddgByd
`
`%0V¢HACdHO
`
`ALIMOIkd
`
`|dAaGHO
`
`vyByd
`
`%09-0VOT1]......
`%03:0V071
`
`U3dON
`
`c300N
`
`GtOL
`
`dsv‘Sis
`
`
`Ex.1007 / Page 43 of 66Ex.1007 / Page 43 of 66
`Ex.1007 / Page 43 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 43 of 47
`
`6,088,727
`
`GaddOlssi4Bd
`
`AlldOldd
`
` HAGHO
`
`%O0V
`
`ALHONd2Byd
`
`%0b&HAGHO
`
`ALHOIdd2Byd
`
`€H3qYO
`
`%OF
`
`ALMOldd
`
`%OV&YWAqCHO
`
`gqByd
`
`ALIWOIdd
`
`%0e’HAGHO
`
`ALIWOId¥Bod
`
`%0bLHAGHO
`
`
`
`#08-VO1]......%09:0VO1
`
`
`
`UAGONcAGON
`
`%039:0V01
`
`%09:0VOT
`%09:0V01
`
`UAGON
`
`cAGON
`
`LAdON
`
`
`
`ULOLQqLOLBLOL
`
`LOL
`
`BLOL
`
`
`
`O9r‘DIA
`
`VorSls
`
`#E0}
`
`E01
`
`ALIdOlddvByd
`
`%0P}HaqHO
`
`ALIdOldd5Byd
`
`%0P©Y3GYO
`
`
`
`%08:0VO1}......%08-0VO1
`
`
`
`USGONcSQON
`
`ULOLGLoLBLOL
`
`
`
`Q9r‘Sls
`
`ALIWONdd3Byd
`
`%0eSHA0HO
`
`ALWOIYd5Byd
`
`€HaGYuO
`
`%Ov
`
`%09:0V07
`
`USGON
`
`qLOL
`
`BLOL
`
`%09-0VOT
`cAGON
`ntavo
`
`
`
`dor‘Ola
`
`
`Ex.1007 / Page 44 of 66Ex.1007 / Page 44 of 66
`Ex.1007 / Page 44 of 66
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 44 of 47
`
`6,088,727
`
`NOTICE
`
`DEMAND
`
`NOTICE
`PROCESSING
`DEVICE
`
`RESOURCE
`ASSIGNMENT 4603
`CONTROLLING
`DEVICE
`
`DEMAND
`
`NOTICE
`
`MANAGING
`TABLE
`
`RESOURE
`ASSIGNMENT
`UPDATE/REFERMANAGING
`DEVICE
`
`MANAGER
`
`FIG. 47
`
`PACKAGE | EXCECUTING | PRIORITY
`NAME
`COMPUTER
`ORDER
`
`
`
`
`
`FIG. 48
`
`
`Ex.1007 / Page 45 of 66Ex.1007 / Page 45 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 45 0f 47
`
`6,088,727
`
`INITIALIZE THE TABLE
`
`480
`
`WAIT FOR THE NOTICE
`4802
`
`UPDATE
`THE TABLE 4803
`
`SELECT A
`COMPUTER FROM
`THE CLUSTER
`
`4804
`
`SELECT APACKAGE
`WITH THE HIGHEST
`PRIORITY
`
`4805
`
`CALCULATE
`THE RESOURCE
`ASSIGNED TO THE PACKAGE 4806
`
`ASSIGN THE
`RESOURCE Y
`4807
`
`4808
`
`HAVE ALL
`THE PACKAGES ON
`THE COMPUTER BEEN
`PROCESSED?
`
`
`
`4809
`HAVE ALL
`THE COMPUTERS
`BEEN
`PROCESSED2
`
`YES
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 49
`
`
`Ex.1007 / Page 46 of 66Ex.1007 / Page 46 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 46 of 47
`
`6,088,727
`
`U
`
`- - - a
`
`- a
`
`a
`
`see
`
`
`
`
`
`
`
`
`
`- - - - - - - a
`
`/
`
`me a
`
`m
`v
`c
`cr)
`
`O t
`
`N NOWEV/C]
`
`HELST?TO
`
`\/ NOWEVC]
`
`HELST?TO
`
`
`Ex.1007 / Page 47 of 66Ex.1007 / Page 47 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 47 0f 47
`
`6,088,727
`
`U 10,72
`
`N HELLTldWOO
`
`N NOWEV/C]
`HE 1ST TO
`
`*N
`\/ EOHTOSEH
`
`10:72
`\/ HELLT, WOZ
`(})
`
`
`
`
`
`
`Ex.1007 / Page 48 of 66Ex.1007 / Page 48 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`6,088,727
`
`45
`
`50
`
`55
`
`60
`
`65
`
`25
`
`1
`CLUSTER CONTROLLING SYSTEM
`OPERATING ON A PLURALITY OF
`COMPUTERS IN A CLUSTER SYSTEM
`BACKGROUND OF THE INVENTION
`1. Field of the Invention
`The present invention relates to a cluster controlling
`System for monitoring and controlling a cluster System, and
`for transferring package programs which have been operat
`ing on a computer Suffering from a failure to another
`computer which is within the cluster to execute the pro
`grams.
`2. Description of the Prior Art
`A “cluster' in the conventional art includes both a close
`coupled cluster, whose main memory is shared by a CPU,
`15
`and a loose-coupled cluster, whose data is shared by com
`puters using a LAN or a common disk. The present invention
`is applied only to the latter type of cluster.
`FIG. 50 shows an example of a conventional cluster
`system. In FIG. 50, a plurality of computers A-N
`(101a–101n) comprise a cluster. The respective computers
`are executing cluster daemon A-N (102a-102n), and the
`respective cluster daemons Start-up corresponding package
`programs A1-N2 (103a1 103n2). The term “package pro
`gram' is a general term referring to application and Service
`programs.
`The respective cluster daemons monitor and control
`resources (various Services or network addresses provided
`by a CPU, LAN, disk and package programs) on the
`executing computers, and Store the data on the respective
`computers as local data A-N (104a–104n).
`The operation of the cluster System is explained using
`FIG. 51. When a resource A (2401a) required by computer
`A (101a) is lost, the cluster daemon A (102a) stops the
`35
`computer A (101a). The cluster daemon N (102n) on another
`computer N (101n) detects the stop of the computer A
`(101a), and the computer N (101n) executes the package
`program A1 (103a), which had been executed by the com
`puter A (101a).
`40
`Thus, the Specific package program is executed on any
`one of the computers within the cluster. When a user utilizes
`Services provided by the package program by assigning a
`network address for every package program, it is not nec
`essary for the user to know exactly which computer in the
`cluster is executing the package program.
`Exemplary Systems for concentratedly monitoring and
`controlling distributed resources are disclosed in references
`such as Japanese Laid-open Patent publication No. 5-75628,
`"Network resource monitoring System', Japanese Laid open
`Patent publication No. 5-134902, “Information Managing
`System In A Distributed Computing System”, and Japanese
`Laid-open Patent publication No. 6-223020, “Network Man
`agement System And Managing Method Of Objects”.
`These Systems are achieved by using managing
`computers, or by incorporating a managing process
`(manager). However, none of the above references Suggest
`a Solution for when a failure occurs in the managing com
`puters or the managing process.
`Since conventional cluster Systems are constructed in the
`manner explained above, when making a program for moni
`toring and controlling the entire System, it has been neces
`Sary to communicate with all computers in the cluster Since
`the data is distributed to all computers. Therefore, there have
`been Some difficulties in making Such programs.
`In addition, in Systems which concentratedly monitor and
`control distributed resources, the monitoring function com
`
`2
`pletely stops when a fault or failure occurs in either the
`computer itself or in the proceSS for monitoring and con
`trolling the entire System. Moreover, because the
`interrelationship, or priority order among various package
`programs could not be defined, it has been difficult to
`transfer the data from other multiplexed Systems.
`In addition, it takes a long time to restart and recover the
`package. Moreover, Since Switching and the processing of
`packages takes a long time after the System has been
`recovered, a package to be processed by a parallel operation
`cannot be processed well. Therefore, the performance of the
`System deteriorates after recovery.
`SUMMARY OF THE INVENTION
`It is an object of the present invention to solve the
`problems described above. The present invention provides a
`cluster controlling System which makes it easy to create
`programs for monitoring and controlling the entire System,
`which in turn makes the transfer from another System
`possible, at a high Speed.
`According to one aspect of the invention, a cluster con
`trolling System “is provided which operates on a plurality of
`computers in a cluster System for moving package programs
`to other computers when a fault or failure occurs on one
`computer in the cluster. The cluster controlling System
`compriseS cluster daemons for communicating with the
`computers and for monitoring and controlling resources,
`local data memories for Storing the monitoring result as local
`data; and wherein one of the computers in the cluster System
`further comprises: global data memory for Storing data
`collected from local data memories of the respective
`computers, and for being referred to from any of the other
`computers in the cluster; and a manager for communicating
`with the global data memory and the cluster daemons of the
`respective computers, and for monitoring and controlling the
`overall cluster System.
`Preferably the manager in the cluster controlling System
`comprises a demand processing device for receiving
`demands from the user, and a resource controlling device
`which receives a demand from the demand processing
`device, refers to the resource State data base in the global
`data memory and then transmit the obtained resource State
`corresponding to the demand to the cluster daemon.
`According to another aspect of the invention, the cluster
`controlling System further comprises agents for communi
`cating with the manager and the cluster daemon on the
`computer.
`Preferably, the agent in the cluster controlling System
`comprises a demand processing device for receiving
`resource control demands from the manager, and a resource
`controlling device which receives a demand from the
`demand processing device, refers to a resource State data
`base, and transmits the obtained resource State correspond
`ing to the demand to the cluster daemon.
`According to a further aspect of the invention, a cluster
`controlling System is provided which operates on a plurality
`of computers in a cluster System for moving packages to
`other computers when a fault or failure occurs on one
`computer in the cluster. The cluster controlling System
`compriseS cluster daemons for communicating with pack
`ages and the computers and for monitoring and controlling
`resources, agents for communicating with a corresponding
`cluster daemon on the respective computers, with a global
`data memory and with the other agents, local data memories
`for Storing the monitoring result as local data, and global
`data memory for Storing data collectcd from the local data
`
`
`Ex.1007 / Page 49 of 66Ex.1007 / Page 49 of 66
`
`TESLA, INC.TESLA, INC.
`
`

`

`6,088,727
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`3
`memories of the respective computers and for being referred
`to by any of the computers via the agents.
`According to a still further aspect of the invention, the
`manager includes a mode managing device for managing
`operation control for a package, according to a respective
`operation mode representing “operating”, “waiting” and
`“testing”.
`According to a still further aspect of the invention, the
`manager includes a log managing device for collecting logs
`regarding resource State changes which occur in the cluster
`System.
`According to a still further aspect of the invention, the
`manager includes a notice processing means for receiving a
`monitored result Such as a resource State change notice from
`the cluster daemons, and a mode controlling device for
`receiving a mode controlling demand from the notice pro
`cessing means, and for transmitting it to the cluster daemons
`for operating a Second package on a Second computer by
`controlling the cluster daemons when a fault or failure has
`occurred in a first computer, and which forces a first package
`in the Started-up State on a first computer to wait when the
`first computer has recovered from the fault or failure.
`According to a still further aspect of the invention, the
`manager includes an output managing device for receiving
`a resource State change notice from the cluster daemons, and
`an output Suppressing device which receives an output
`Suppression/release demand from the output managing
`device, and which transmits it to the cluster daemon,
`wherein the manager manages the output of the Second
`package instead of the first package by controlling the
`cluster daemons when a failure occurs in the first computer
`while the first package is outputting an output, and wherein
`the manager maintains an output from a Second package
`until a fail