United States Patent
`Graf
`
`[19]
`
`lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll
`US005619656A
`5,619,656
`Patent Number:
`Date of Patent:
`Apr. 8, 1997
`
`[45]
`
`[11]
`
`[54] SYSTEM FOR UNINTERRUPTIVELY
`DISPLAYING ONLY RELEVANT AND
`NON-REDUNDANT ALERT MESSAGE OF
`THE HIGHEST SEVERITY FOR SPECIFIC
`CONDITION ASSOCIATED WITH GROUP OF
`COMPUTERS BEING MANAGED
`
`[75]
`
`Inventor: Lars 0. Graf, Rensselaer, N.Y.
`
`[73] Assignee: OPENService, Inc., Albany, N.Y.
`
`[21] Appl. No.: 238,476
`
`[22] Filed:
`
`May 5, 1994
`
`[51]
`
`Int. Cl.6
`
`..••..•.••.••••••••••••.••.• G06F 11/00; G06F 11/30;
`G06F 13/10
`[52] U.S. Cl •................................ 395/200.11; 395/185.01;
`3951185.1; 395/839; 364/141; 364/152;
`364/265; 3641267; 364/280; 364/284.3
`[58] Field of Search ..................................... 395/650, 575,
`395/700, 200.11, 185.Dl, 185.1, 835, 839;
`364/141, 144, 146, 152, 264.5, 284.3, 280,
`286.3, 974.5
`
`[56]
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,589,068
`4,815,030
`4,866,712
`4,888,690
`
`5/1986 Heinen, Jr ............................... 364/300
`3/1989 Cross et al .............................. 364/900
`9/1989 Chao ........................................ 37115.1
`12/1989 Huber ...................................... 364/200
`
`911991 Hill et al. ................................ 364/900
`5,047,977
`5,109,486
`411992 Seymour ................................. 395/200
`5,111,384
`511992 Aslanian et al ......................... 395/575
`5,146,587
`911992 Francisco ................................ 395/575
`5,155,842
`10/1992 Rubin ...................................... 395/575
`5,163,151
`11/1992 Bronikowski et al. ................. 395n25
`5,226,150
`7/1993 Callander et al ....................... 395/575
`5,230,048
`7/1993 Moy ........................................ 395/600
`5,418,966
`511995 Madduri .................................. 395/725
`5,432,932
`711995 Chen et al .............................. 395/650
`5,528,759
`611996 Moore ..................................... 395/200
`Primary Examiner-Thomas C. Lee
`Assistant Examiner-Ki S. Kim
`[57]
`ABSTRACT
`
`The system and method of this invention automatically
`manages a group of computers by automatically gathering
`data, storing the data, analyzing the stored data to identify
`specified conditions, and initiating automated actions to
`respond to the detected conditions. The invention, hereafter
`"SYSTEMWatch AI-L", comprises a SYSTEMWatch AI-L
`client which turns a computer into a managed computer, a
`SYSTEMWatch AI-L console, which turns a computer into
`a monitoring computer, a SYSTEMWatch AI-L send facility,
`which allows a system administrator to send commands to
`various SYSTEMWatch AI-L clients through the SYSTEM(cid:173)
`Watch AI-L console, and a SYSTEMWatch AI-L report
`facility which allows a system administrator to query infor(cid:173)
`mation collected and processed by the SYSTEMWatchAI-L
`clients and SYSTEMWatch AI-L consoles.
`
`22 Claims, 12 Drawing Sheets
`
`READ MAILBOX FOR
`MESSAGES & EXECUTE
`COMMANDS PLACED IN
`MAILBOX. READ DATA
`IN MAILBOX
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 1
`
`

`
`="'
`="' Ol
`-.
`\C
`="' ~
`-.
`Ol
`
`N
`1-"
`s,
`00. =-i
`
`1-"
`
`-....)
`\C
`\C
`1-"
`~
`:"l
`"'CC
`>
`
`~
`•
`rJJ
`~ •
`
`~ = """"
`
`fig. 1
`
`COMPUTER
`MONITORING
`
`COMPUTER
`MONITORING
`
`COMPUTER
`MANAGED
`
`COMPUTER
`MANAGED
`
`COMPUTER
`MANAGED
`
`COMPUTER
`MANAGED
`
`GROUP OF MANAGED COMPUTERS
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 2
`
`

`
`~
`Ol
`~
`\C
`~
`~
`Ol
`
`~
`
`~
`
`N
`iooo-0
`g,
`N
`r:l:l =-$!.
`
`\C
`iooo-0
`
`::s
`
`~
`
`~r:x;
`~
`
`= ~
`~ ;-
`~
`rJ1 •
`•
`Cj
`
`SYSTEMWotch Al-L
`
`CLIENT
`
`13
`
`OPERATING SYSTEM
`
`COMPONENTS
`
`OPERA liNG SYSTEM
`
`OPERATING SYSTEM
`
`OPERATING SYSTEM
`
`11
`
`USER PROGRAMS
`
`I
`
`I
`
`_l
`
`I
`
`I
`
`.... ...
`... -
`
`-
`
`MEMORY
`
`10
`
`14
`
`fig. 2
`
`9
`
`CPU
`
`15
`
`PERIPHERIALS
`
`OTHER
`
`16
`
`COMMUNICATIONS
`
`INTERFACE
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 3
`
`

`
`~
`(It
`~
`....
`\C
`~
`~
`....
`(It
`
`~
`
`"""' N
`s,
`rJJ. =-m.
`
`"'-l
`\C
`"""'
`\C
`S'J
`:'1
`~
`
`~ = """"
`Pi
`~
`•
`rJ'l
`0 •
`
`SYSTEMWatch Al-L
`
`CONSOLE
`
`21
`
`OPERATING SYSTEM
`
`COMPONENTS
`
`OPERATING SYSTEM
`
`OPERATING SYSTEM
`
`OPERATING SYSTEM
`
`19
`
`USER PROGRAMS
`
`I
`
`I
`
`I
`
`I
`
`I
`
`... ....
`
`-...
`
`....
`
`MEMORY
`
`18
`
`22
`
`fig. 3
`
`23
`
`PERIPHERIALS
`
`OTHER
`
`CPU
`
`24
`
`COMMUNICATIONS
`
`INTERFACE
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 4
`
`

`
`="
`=" 01
`-.
`\C
`=" ~
`-.
`01
`
`r ol::o
`
`N
`~
`g,
`
`.....:J
`'-=
`'-=
`S'J
`:"i
`~
`
`~
`
`~ = ~
`
`~
`~
`•
`00
`d •
`
`l}3
`SYSTEMWotch Al-L I 31
`
`SYSTEMWatch Al-L
`
`CLIENT
`
`29
`
`fig. 4
`
`. I-)
`
`CONSOLE
`
`OPERATING SYSTEM
`
`COMPONENTS
`
`OPERATING SYSTEM
`
`27
`
`OPERATING SYSTEM
`
`USER PROGRAMS
`
`I
`
`I
`
`I
`
`I
`
`1
`
`... -
`... ...
`...
`
`MEMORY
`
`26
`
`30
`
`PERIPHERIALS
`
`OTHER
`
`CPU
`
`31
`
`COMMUNICATIONS
`
`INTERFACE
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 5
`
`

`
`U.S. Patent
`
`Apr. 8, 1997
`
`Sheet 5 of 12
`
`5,619,656
`
`SYSTEMWatch Al-L CLIENT _5 32
`
`CORE LAYER
`
`APPLICATION LAYER
`
`3 3
`
`3 4
`
`SYSTEMWatch Al-L CONSOLE _5 35
`
`CORE LAYER
`
`APPLICATION LAYER
`
`3 3
`
`3 6
`
`fig. 5
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 6
`
`

`
`U.S. Patent
`
`Apr. 8, 1997
`
`Sheet 6 of 12
`
`5,619,656
`
`38
`
`42
`
`39
`
`Al-L LANGUAGE
`INTERPRETER
`
`COMMUNICATIONS
`FACILITY
`
`M----t
`
`ROUTINES
`ROUTINES
`ROUTINES
`
`41
`
`40
`
`DATABASE
`ENTITIES
`
`Vl w
`1-a::
`w a.
`0 a:: a.
`
`DATA
`
`DATA
`
`DATA
`
`DATA
`
`EXPERT SYSTEM
`
`RULES
`
`RULES
`
`RULES
`
`fig. 6
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 7
`
`

`
`I!
`
`....:J
`~
`"'""' ~
`
`~ss
`
`I PROPERTY I _..,DATA I--""
`I
`
`I
`
`i
`
`fig. 7
`
`•----•----•
`
`I
`
`I
`
`---
`
`I
`
`I
`
`I
`
`PROPERTY
`
`PROPERTY ~ DATA I
`
`56
`
`--48
`
`PROPERTY t---f DATA I DATA I
`
`I
`
`471
`
`54--..
`
`46
`
`45
`
`I
`
`44
`
`C\
`til
`C\
`-..
`\C
`~
`C\
`-..
`til
`
`"'""' N
`0 .....,
`....:J
`~ ......
`~
`::r
`00
`
`I
`
`I
`
`I
`
`I
`
`57
`
`58
`
`I r"\ATA I r"\ATA I
`
`I
`
`'-J PROPERTY I
`"
`
`I
`
`I
`
`~ f"'t(cid:173) n> = f"'t-
`
`00. •
`d •
`
`c.--53
`
`ENTITY
`
`43
`
`ENTITY
`
`41
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 8
`
`

`
`U.S. Patent
`
`Apr. 8, 1997
`
`Sheet 8 of 12
`
`5,619,656
`
`START
`
`60
`
`SORT RULE IN DECENDING
`SPECIFICITY ORDER; IN
`CASE OF TIE, SORT
`TIES IN ASCENDING
`DECLARATION ORDER
`
`INITIALIZE RULES BY:
`1. SET EACH RULE'S
`LASTCHECK TIME TO ZERO.
`~~ 2. SET EACH RULE'S ONCE
`STATE TO TRUE
`3. EXECUTE EACH RULE'S
`INITIALIZATION SECTION
`
`y
`
`SET EXPERT SYSTEM
`RULE STATE TO
`EMPTY STRING
`
`SET CURRENT RULE
`POINTER TO RULE #1
`
`63
`
`64
`
`N
`
`71
`
`69
`
`y
`
`70
`
`EXECUTE THE STATEMENTS
`IN THE CURRENT RULE'S
`ELSE CLAUSE, IF IT EXISTS
`
`EXECUTE THE STATEMENTS
`IN THE CURRENT RULE'S
`THEN CLAUSE
`
`CD
`
`fig. BA
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 9
`
`

`
`U.S. Patent
`
`Apr. 8, 1997
`
`Sheet 9 of 12
`
`5,619,656
`
`65
`
`~
`
`MAKE CURRENT RULE
`BE THE RULE POINTED
`TO BY THE CURRENT
`RULE POINTER
`
`MAKE THE CURRENT
`RULE POINTER POINT
`TO THE NEXT RULE IN
`THE EXPERT SYSTEM
`
`73
`
`y
`
`fig. BB
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 10
`
`

`
`U.S. Patent
`
`Apr. 8, 1997
`
`Sheet 10 of 12
`
`5,619,656
`
`75
`
`READ MAILBOX FOR
`MESSAGES & EXECUTE
`COMMANDS PLACED IN
`MAILBOX. READ DATA
`IN MAILBOX
`
`76
`
`N
`
`77
`
`78
`
`79
`
`CALL EXECRULES
`TO RUN THROUGH
`THE EXPERT SYSTEM
`
`CALL CHECKALERT
`TO ESCALATE/CLEAR
`ALERTS AS NEEDED
`
`DISPLAY ALERTS
`
`SLEEP 5 SECONDS
`
`80
`
`fig. 9
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 11
`
`

`
`U.S. Patent
`
`Apr. 8, 1997
`
`Sheet 11 of 12
`
`5,619,656
`
`81
`
`READ MAILBOX FOR
`MESSAGE & EXECUTE
`COMMANDS IN
`MAILBOX. READ DATA
`IN MAILBOX
`
`82
`
`83
`
`DISPLAY ALERTS
`
`SLEEP 5 SECONDS
`
`84
`fig. 10
`
`FORMAT MESSAGES
`
`87
`
`88
`
`fig. 11
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 12
`
`

`
`U.S. Patent
`
`Apr. 8, 1997
`
`Sheet 12 of 12
`
`5,619,656
`
`READ REQUEST & LIB PROGRAMS
`
`89
`
`SETUP REPORT WINDOWS _5
`
`90
`
`91
`
`READ SPECIFIED REPORT FILE
`
`QUERY DATABASE &
`AWAIT RESULTS
`
`OISPLA Y INFORMATION
`
`93
`
`94
`
`99
`
`fig. 12
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 13
`
`

`
`5,619,656
`
`1
`SYSTEM FOR UNINTERRUPTIVELY
`DISPLAYING ONLY RELEVANT AND
`NON-REDUNDANT ALERT MESSAGE OF
`THE HIGHEST SEVERITY FOR SPECIFIC
`CONDITION ASSOCIATED WITH GROUP OF
`COMPUTERS BEING MANAGED
`
`FIELD OF THE INVENTION
`
`This invention relates in general to system administration
`and in particular to automated management of a group of
`computers and its associated hardware and software.
`
`BACKGROUND ART
`
`5
`
`15
`
`2
`In a typical system administration environment, the work
`cycle consists of the following. A problem occurs on the
`computer which prevents the end user from carrying out
`some task. The end user detects that problem some time after
`it has occurred, and calls the complaint desk. The complaint
`desk dispatches a system administrator to diagnose and
`remedy the problem. This has three important consequences:
`First, problems are detected after they have blocked a user's
`work. This can be of substantial impact in organizations
`10 which use their computers to run their businesses. Second,
`problems which do not necessarily block a user's work, but
`which may nonetheless have important consequences, are
`difficult to detect. For example, one vendor supplies an
`electronic mail package which is dependent upon a func-
`tional mail daemon process. This mail daemon process has
`a tendency to die on an irregular, but frequent basis. In such
`situations, the end user typically does not realize that he is
`not capable of receiving electronic mail until after they've
`missed a meeting scheduled by electronic mail. Third,
`20 because problems are not detected until after they block a
`user's work, a problem which at an earlier state might have
`been easier to fix cannot be fixed until it has escalated into
`something more serious, and more difficult to correct.
`Currently, system administrators manage a group of com-
`25 puters by performing most actions manually. Typically, the
`system administrator periodically issues a variety of com(cid:173)
`mands to gather information regarding the state of the
`various computers in the group. Based upon the information
`gathered, and based upon a variety of non-computer infor-
`30 mation, the system administrator detects problems and for(cid:173)
`mulates action plans to deal with the detected problems.
`Automation of a system administration's task is difficult for
`several reasons:
`1. Data regarding the state of the computer is difficult to
`obtain. Typically, the system administrator must issue a
`variety of commands and consider several pieces of
`information from each command in order to diagnose a
`problem. If the system administrator is responsible for
`several machines, these commands must be repeated on
`each machine.
`2. When the system administrator detects a problem, the
`appropriate action plan may vary depending on a variety
`of external factors. For example, suppose a particular
`computer becomes slow and unresponsive when the sys(cid:173)
`tem load on that computer crosses a certain threshold. If
`this problem occurs during normal business hours under
`ordinary circumstances, it will probably be a problem
`which must be resolved in a timely manner. On the other
`hand, suppose this problem occurs in the middle of the
`night. While this situation might still be a problem, the
`resolution need not be as timely since the organization's
`work will not be impacted, unless the problem still exists
`by the start of the business day. Now suppose the account(cid:173)
`ing department, at the end of each month, runs a processor
`intensive task to do the end-of-month accounting, which
`normally forces the load average above that threshold. If
`the system load crosses that same average during the time
`when the accounting department runs their end of month
`program, that's not a problem. In order to build a tool to
`handle situations like these using current tools would
`require writing a large series of inter-related complex
`boolean expressions. Unfortunately, writing and testing
`such a series of complex boolean expressions are difficult.
`3. Current system administration tools view the universe of
`computer problems as a static universe. Computer prob(cid:173)
`lems, however, evolve over time as hardware and soft(cid:173)
`ware are added, removed, and replaced in a computer.
`
`The following documents are hereby incorporated by
`reference in its entirety:
`1. Object Oriented Programming, Coad P., and Nicola J.,
`YourDon Press Computing Series, 1993., ISBN 0-13-
`032616-X.
`2. The C Programming Language, Kernighan B., and
`Ritchie D., 1st Edition, Prentice-Hall Inc., ISBN 0-13-
`110163-3
`3. The Unix Programming Environment, Kernighan and
`Pike, Prentice-Hall Inc., ISBN 013-937699-2
`4. Unix Network Programming, Stevens, Prentice Hall Soft(cid:173)
`ware Series, 1990, ISBN 0-13-949876-1.
`5. Intemetworking with TCP/IP, Volume I, Principles, Pro(cid:173)
`tocols, and Architecture, 2d Ed, Prentice Hall, 1991,
`ISBN 0-13-468505-9
`6. Solaris 1.1, SMCC VersionA, AnswerBook for SunOS
`4.1.3 and Open Windows Version 3, Sun Microsystems
`Computer Corporation, Part Number 704-3183-10, Revi- 35
`sian A.
`7. Artificial Intelligence, Rich E., McGraw-Hill, 1983, ISBN
`0-07-052261-8.
`8. Artificial Intelligence, Winston P., 2d Edition, 1984, ISBN
`0-201-08259-4.
`9. Documentation for the SunOS 4.1.3 operating system
`from Sun Microsystems, Inc.
`10. SunOS 4.1.3 manual pages ("man pages") from Sun
`Microsystems, Inc.
`As used within this document and its accompaning draw- 45
`ings and figures, the following terms are to be construed in
`this manner:
`1. "CPU" shall refer to the central processing unit of a
`computer if that computer has a single processing unit. If
`the computer has multiple processors, the term CPU shall 50
`refer to all the processing units of such a system.
`2. "Managing a computer" shall refer to the steps necessary
`to manage a computer, for example, gathering and storing
`information, analyzing information to detect conditions,
`and acting upon detected conditions.
`The problem of system administration for a computer
`with a complex operating system such as the UNIX oper(cid:173)
`ating system is a complex one. For example, in the UNIX
`workstation market, it is common for an organization to hire
`one system administrator for every 20-50 workstations 60
`installed, with each such administrator costing a company
`(including salary and overhead) between $60,000 and $100,
`000. Indeed, some corporations have discovered that despite
`freezing or cutting back hardware and software purchases,
`the rising cost of retaining system administrators has nev- 65
`ertheless continued to escalate the cost of maintaining an
`Information Services organization at a substantial rate.
`
`40
`
`55
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 14
`
`

`
`5,619,656
`
`4
`FIG. 11 illustrates one embodiment of the
`Watch AI-L request facility; and
`FIG. 12 illustrates one embodiment of the
`Watch AI-L report facility.
`
`SYSTEM-
`
`SYSTEM-
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENT
`
`3
`4. Furthermore, an automated tool should also flexibly alter
`its behavior based on the nature of the commands a
`system administrator issues to it in guiding it in to resolve
`problems. Thus, if the system administrator routinely
`ignores a particular problem, the automated tool should 5
`warn the system administrator less frequently if the rou(cid:173)
`tinely ignored problem reoccurs.
`What is needed is a tool which will automatically gather
`the necessary computer information to manage a group of
`computers, detect problems based upon the gathered infor- 10
`mation, inform the system administrator of detected prob(cid:173)
`lems, and automatically perform corrective actions to
`resolve detected problems.
`
`SUMMARY OF THE INVENTION
`
`The shortcomings of the prior art are overcome and
`additional advantages are provided in accordance with the
`principles of the present invention through the provision of
`SYSTEMWatch AI-L, which automatically manages at least
`one computer by automatically gathering computer infor(cid:173)
`mation, storing the gathered information, analyzing the
`stored information to identify specific computer conditions,
`and performing automatic actions based on the identified
`computer conditions.
`
`BRIEF DESCRIPTION OF DRAWINGS
`
`20
`
`25
`
`45
`
`One preferred embodiment of the technique of the present
`invention of managing a group of computers is targeted at
`groups of workstations running the UNIX operating system.
`Alternative embodiments of the present invention can con(cid:173)
`sist of groups of computers running other operating systems,
`such as, Microsoft's Windows NT and IBM's OS/2. As
`viewed in FIG. 1, the invention comprises, for instance, 2
`15 groups of computers:
`a. A group of managed computers, 1, which includes
`computers, 2-5, comprising, for example, (see FIG. 2)
`a CPU, 9, memory, 10, disks, 14, communications
`interface, 16, other peripherals, 15, and a SYSTEM(cid:173)
`Watch AI-L client, 13. The size of the managed group
`of computers can range from 1 to several thousand.
`Data which is gathered from a managed computer is
`stored on the managed computer. From time to time, a
`managed computer may send data to a monitoring
`computer (see below).
`b. A group of monitoring computers, 6, which includes
`computers comprising, for example, (see FIG. 3) a
`CPU, 17, memory,18, disks, 22, communications inter(cid:173)
`face, 24, other peripherals, 23, and a SYSTEMWatch
`AI-L console, 21. The size of the monitoring group of
`computers can range from 0 to several hundred.
`Although data gathered from a managed computer is
`stored on the managed computer, from time to time a
`managed computer may send data to a monitoring
`computer. A monitoring computer can also explicitly
`request data from a managed computer. Data which is
`received by the monitoring computer from a managed
`computer is stored on the monitoring computer. Fur-
`thermore, since a monitoring computer can receive data
`from several managed computers, a monitoring com(cid:173)
`puter may perform post-processing on data received
`from several managed computer, and/or perform addi(cid:173)
`tional data gathering itself, in which case that data is
`stored on the monitoring computer.
`In another embodiment the two groups of computers may
`be the same group (all managed computers are also moni(cid:173)
`toring computers), two distinct groups (no managed com(cid:173)
`puters are monitoring computers), or overlap (some man-
`50 aged computers are monitoring computers). The computers
`which form the groups of computers may be heterogeneous
`or homogeneous. The only requirement is that each managed
`computer have the capability to communicate with at least
`one monitoring computer. One preferred embodiment of this
`invention is to have all the computers on a computer
`network, but any other means of communication, e.g., over
`a modem using a telecommunications network, is adequate.
`The differentiation between managed and monitoring com(cid:173)
`puters are the SYSTEMWatch AI-L client and the SYS-
`60 TEMWatch AI-L console, which are described below:
`a. As show in FIG. 2, a computer is a managed computer
`if the computer is running the SYSTEMWatch AI-L
`client, which provides a means for the computer to
`automatically detect and respond to problems. Addi(cid:173)
`tionally, the SYSTEMWatch AI-L client also accepts
`and responds to commands issued by a SYSTEMWatch
`AI-L console described below.
`
`The subject matter which is regarded as the invention is
`particularly pointed out and distinctly claimed in the claims
`at the conclusion of the specification. The foregoing and 30
`other objects, features, and advantages of the invention will
`be apparent from the following detailed description taken in
`conjunction with the accompaning drawings in which:
`FIG. 1 illustrates an embodiment of the present invention
`which comprises two groups of computers, a group of 35
`managed computers and a group of monitoring computers.
`FIG. 2 illustrates one example of the structure of a
`managed computer, comprising a processing unit, memory,
`disk, network interface, peripherals, and a SYSTEMWatch
`AI-L client;
`· 40
`FIG. 3 illustrates one embodiment of the structure of a
`monitoring & command computer, comprising a processing
`unit, disk, network interface, peripherals, and a SYSTEM(cid:173)
`Watch AI-L console;
`FIG. 4 illustrates one embodiment of the structure of a
`computer which is both a managed computer and a moni(cid:173)
`toring computer, comprising a processing unit, disk, network
`interface, peripherals, a SYSTEMWatch AI-L console, and
`a SYSTEMWatch AI-L client;
`FIG. 5 illustrates one embodiment of the SYSTEMWatch
`AI-L client and the SYSTEMWatch AI-L console, compris(cid:173)
`ing of a core layer plus an application layer.
`FIG. 6 illustrates one embodiment of the logical structure
`of the core layer in accordance with the principles of the
`present invention;
`FIG. 7 illustrates one example of an embodiment of data
`within the database of the core layer accordance with the
`principles of the present invention;
`FIGS. Sa-Sb illustrates one embodiment of the operation
`of the expert system found in the core layer of SYSTEM(cid:173)
`Watch AI-L;
`FIG. 9 illustrates one embodiment of the SYSTEMWatch
`AI-L client's "client loop";
`FIG. 10 illustrates one embodiment of the SYSTEM(cid:173)
`Watch AI-L console's "console loop";
`
`55
`
`65
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 15
`
`

`
`5,619,656
`
`5
`b. As shown in FIG. 3, a computer is a monitoring
`computer if the computer is running the SYSTEM(cid:173)
`Watch AI-L console, which provides a means for the
`computer to receive and display notifications of
`detected problems, and to display the corrective actions
`taken. Additionally, the SYSTEMWatch Al-L console
`is also able to issue commands to any group of man(cid:173)
`aged computers.
`
`5
`
`6
`7, in the context of the database, SYSTEMWatch AI-L uses
`two concepts: ENTITYs, 43, 53, and PROPERTYs, 44, 47,
`49, 54, 56. These two features are now described in greater
`detail:
`1. PROPERTY
`Conceptually, PROPERTY s are similar to field descrip(cid:173)
`tions. In one embodiment, a PROPERTY has the following
`features:
`
`FEATURE
`
`DESCRIPTION
`
`TABLE 1
`
`NAME
`TYPE
`
`FORMAT
`
`HEADER
`
`DISPLAYUNIT
`
`DISPLAYTYPE
`
`SHORTDESC
`
`LONGDESC
`
`A property must have a name.
`A property must have a type, which corresponds to the type of the data to
`be stored in the field.
`A property may optionally have a string which describes how the data in
`the field should be formatted. The format string is similar to the C
`language's printf( )' s formatting control.
`A property may optionally contain a string which will be displayed as the
`column header when a report featuring records containing the property is
`displayed.
`A string used by the reporting facility which is appended to the data in the
`field during a report. Thus, if the PROPERTY is a description of memory
`utilization in kilobytes, an appropriate DISPLAYUNIT might be "kb"
`Some display formats are commonly used through SYSTEMWatch AI-L.
`DISPLAYTYPES are keywords which corresponds to a particular
`FORMAT. Examples of DISPLAYTYPEs include STRING20, for a string
`limited to 20 characters in width, DATESMALL, for displaying date in
`mm/dd format, PERCENT, for automatically display numbers between
`0.0 and 1.0 as percentages (e.g.: 0.52 is displayed as 52%)
`A PROPERTY may optionally contain an abbreviated description of the
`PROPERTY.
`A PROPERTY may optionally contain a long description of the
`PROPERTY.
`
`c. As shown in FIG. 4, a computer is both a managed
`computer and a monitoring computer if it contains both 35
`SYSTEMWatch AI-L client, 13, and SYSTEMWatch
`AI-L console, 21.
`An Overview of the SYSTEM Watch AI-L Client
`The task of the SYSTEMWatch AI-L client is to manage
`a computer and to provide notification of management 40
`actions to the SYSTEMWatch AI-L console. Before explain(cid:173)
`ing how the SYSTEMWatch AI-L client operates, however,
`it is necessary to consider how the SYSTEMWatch AI-L
`client is organized. As previously mentioned, the SYSTEM(cid:173)
`Watch AI-L client is bifurcated into a core layer, 33, which 45
`provides the SYSTEMWatchAI-Lclient with the underlying
`mechanism for detecting and responding to problems, and
`an application layer, 34, which configures the SYSTEM(cid:173)
`Watch AI-L client to operate in a useful manner. The
`SYSTEMWatch AI-L client was designed this way because 50
`the nature of a particular computer~s problem is not static.
`For example, problems may evolve as changes are made to
`the hardware and software of the computer, and if the
`computer is a multi-user computer, as users are added and
`removed from the system. As computer problems change, 55
`only the SYSTEMWatch AI-L client's application layer
`need be modified. As shown in FIG. 6, the core layer is
`composed of four elements: a database, 41, an expert
`system, 40, a language interpreter, 39, and a communica(cid:173)
`tions mechanism, 42. One example of a preferred embodi- 60
`ment of the application layer, 34, is a series of programs
`written in a language which can be interpreted by the
`language interpreter of the core layer.
`Care Layer Description-Database
`The first element of the core layer is SYSTEMWatchAI-L 65
`database, 41. The database is used for storing gathered data,
`intermediate results, and other information. Refering to FIG.
`
`2. ENTITY
`Conceptually, ENTITY s are similar to database tables. In
`SYSTEMWatch AI-L, ENTITYs are used to group related
`PROPERTYs.
`FIG. 7 illustrates the concept that each piece of data in the
`database is associated with a given PROPERTY and a given
`ENTITY. In this document, it will be necessary to refer to
`certain combinations of ENTITY s and PROPERTY s. The
`(e.g.:
`construction <entity name>_ <property name>
`IGNORE_IGNORETIME) will refer to a database entry
`with an entity equal to <entity name> and a property equal
`to <property name>.
`In addition to ENTITY s and PROPERTY s, the database,
`41, in SYSTEMWatch AI-L also has these additional fea(cid:173)
`tures:
`1. Host Information
`Each piece of data in database, 41, automatically has host
`information associated with it. Thus, as data is stored in the
`database, the database automatically associates the host
`from which the data originated from. This is because in
`SYSTEMWatch AI-L, data is "owned" by the host from
`where the data originated. Other hosts may request a copy of
`the data since SYSTEMWatch AI-L has communications
`capabilities. Some data may be stored in a central location
`(e.g.: a SYSTEMWatch AI-L console) if it is relevant to
`multiple computers. Because each piece of data has host
`information associated with it, a SYSTEMWatch AI-L con(cid:173)
`sole can conslidate data from multiple hosts.
`2. Time Information
`Each piece of data in database, 41, has a time field
`associated with it. The time field by default has the last time
`the data was updated, but SYSTEMWatch AI-L provides a
`mechanism of changing the time field so its possible to store
`some other time in the field.
`
`Petitioners Old Republic Gen'l Insur. Group, et al., Ex. 1005, p. 16
`
`

`
`10
`
`20
`
`35
`
`3. Name
`Each piece of data in database, 41, has a key field which
`is called the name field. A name field must be unique for a
`given ENTITY, PROPERTY, and host (the name of a
`computer). Thus, within an ENTITY and PROPERTY used 5
`for tracking computer processes, the name field might be the
`process id since process ids are unique on each computer, so
`by specifying the ENTITY name, PROPERTY name, and
`host name, the name field forms a unique key to locate the
`data.
`4. Value
`Of course, a database stores data. In SYSTEMWatch
`AI-L, the term value refers to the data stored in the database.
`In one example, database, 41, is currently implemented as
`a relational database: One table is used for describing
`ENTITYs. This table is used to associate ENTITYs with 15
`PROPERTYs. Another table is used for describing PROP(cid:173)
`ERTY s. Finally, another table holds the information, which
`can be located by providing an ENTITY name, PROPERTY
`name, and the name field of the data. This table also contains
`the associated host and time information.
`In another embodiment, database, 41, can also be imple(cid:173)
`mented with a database which is object oriented, i.e, a
`database which supports the ability to inherit data and
`methods from super and sub classes.
`Additional requirements of database, 41, used in the core 25
`is that the database must support certain query operations
`and certain set operations. Specifically, the query operations
`supported by the database include:
`1. regular expression matching in queries.
`2. creation time or update time query, i.e., searching for a 30
`data item based upon the time the data was stored in the
`database or based on the time the data was last updated in
`the database.
`3. host of origin in queries, i.e., searching for a data item
`based on the host which created the data.
`4. time comparison query, i.e., searching for data based upon
`a time comparison. Note: SYSTEMWatch AI-L stores its
`time in a manner similar to the UNIX operating system.
`That is to say, all time is converted to seconds elapsed
`since the beginning of UNIX time. The advantages of 40
`using this method is that time comparisons are easily
`made, and a time plus an interval can be added to obtain
`a future time.
`The set operations which database, 41, supports include:
`1. set intersections (ANDs)-given 2 or more sets of data, 45
`return the elements present in all sets.
`2. set union (ORs)-given 2 or more sets of data, return the
`elements in all sets.
`3. set exclusion (NOTs)-given a first set and a second set,
`return elements in the first set which are not elements of
`the second set.
`Care Layer Description-The Expert System
`The second element of the core layer is an expert system,
`40, which is used for problem detection and action initiation.
`The expert system, 40, is a forward chaining rule based
`expert system using a rule specificity algorithm. When
`SYSTEMWatch AI-L client, 13, is started, the expert system
`contains no rules. Rules are declared and incorporated into
`the core layer. Rules support both the IF-THEN rules as well
`as IF-THEN-ELSE rules. The rules used in SYSTEMWatch
`AI-L permit assignments and function calls within the
`condition of the rule. Additionally, SYSTEMWatch AI-L
`expert system, 40, also has the following features:
`a. Rules can declare variables. All variables declared
`within a rule are static variables.
`b. Rules can have an initialization section. The initializa(cid:173)
`tion section contains actions which must be performed
`
`50
`
`55
`
`60
`
`65
`
`7
`
`5,619,656
`
`8
`only once, and before the rule is ever tested. It can, for
`example, contain a state declaration and an interval
`declaration (states and intervals are described below).lt
`may contain variable declarations for variables used by
`the rules, and it may contain code to do a variety of
`actions.
`c. Rules can have, for instance, an INTERVAL and a
`LASTCHECK time. In accordance with the principles
`of the present invention, in order for a rule to be eligible
`for testing by the expert system, at the time of testing
`the clock time must be equal to or greater than the
`LASTCHECK time plus the INTERVAL time. The
`LASTCHECK time for each rule is set to the clock time
`whenever a rule is actually tested. This way, the
`INTERVAL specifies the minimum amount of time
`which must elapse since the last time a rule was
`checked before the rule becomes eligible for testing
`again.
`d. The expert system and its rules have a state property.
`One example of the possible states is described below.
`Under expert system, 40, in order for a rule to be
`eligible for testing, the rule's state must equal the
`expert system's state. All rules except one must declare
`a state for the rule in its initialization section. The one
`rule without such a declaration is a rule used by expert
`system, 40, to switch it into the DATA state. Other rules
`are responsible for managing the transition from DATA
`to DATA2, and from DATA2 to EXCEPT. These states
`are described below:
`Data:
`The data state is assigned to rules which gather raw data
`from the computer system. Examples of such rules would be