Case 6:20-cv-01152-ADA Document 1-1 Filed 12/16/20 Page 1 of 16
`Case 6:20-cv-01152-ADA Document 1-1 Filed 12/16/20 Page 1 of 16
`
`
`
`
`
`
`
`EXHIBIT 1
`
`
`EXHIBIT 1
`
`

`

`Case 6:20-cv-01152-ADA Document 1-1 Filed 12/16/20 Page 2 of 16
`I IIIII IIIIIIII Ill lllll lllll lllll lllll lllll lllll lllll lllll 111111111111111111
`US007177886B2
`
`c12) United States Patent
`Pruet, III
`
`(IO) Patent No.:
`(45) Date of Patent:
`
`US 7,177,886 B2
`Feb.13,2007
`
`(54) APPARATUS AND METHOD FOR
`COORDINATING LOGICAL DATA
`REPLICATION WITH HIGHLY AVAILABLE
`DATA REPLICATION
`
`(75)
`
`Inventor: Clarence Madison Pruet, III, Flower
`Mound, TX (US)
`
`(73) Assignee: International Business Machines
`Corporation, Armonk, NY (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 534 days.
`
`(21) Appl. No.: 10/360,403
`
`(22) Filed:
`
`Feb. 7, 2003
`
`(65)
`
`Prior Publication Data
`
`US 2004/0158588 Al
`
`Aug. 12, 2004
`
`(51)
`
`Int. Cl.
`G06F 17/30
`(2006.01)
`(52) U.S. Cl. ...................... 707/204; 707/202; 707/203;
`707/1; 709/203; 370/389
`(58) Field of Classification Search ........ 707/202-204,
`707 /1; 370/389; 709/203
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,594,900 A
`5,640,561 A
`5,842,222 A
`6,148,412 A
`6,199,069 Bl
`6,199,074 Bl
`
`1/1997 Cohn et al.
`6/1997 Satoh et al.
`11/1998 Lin et al.
`11/2000 Cannon et al.
`3/2001 Dettinger et al.
`3/2001 Kern et al.
`
`6/2001 Bogantz et al.
`6,243,715 Bl
`7/2001 Smith
`6,269,432 Bl
`9/2001 Parker
`6,289,357 Bl
`9/2001 Wallach et al.
`6,292,905 Bl
`6,304,882 Bl * 10/2001 Strellis et al.
`.............. 707/202
`6,324,693 Bl
`11/2001 Brodersen et al.
`6,405,220 Bl
`6/2002 Brodersen et al.
`6/2002 Hart
`6,408,310 Bl
`6,421,688 Bl
`7/2002 Song
`
`(Continued)
`
`OTHER PUBLICATIONS
`
`"Efficient Logging of Transactions on Persistent Information in
`General and Databases in Particular", IBM Technical Disclosure
`Bulletin, vol. 40, No. 11, Nov. 1997, pp. 117-120.
`
`(Continued)
`
`Primary Examiner-Sathyanarayan Pannala
`(74)Attorney, Agent, or Firm-Fay, Sharpe, Fagan, Minnich
`& McKee, LLP
`
`(57)
`
`ABSTRACT
`
`In a database apparatus (10), a critical database server (12)
`includes a primary server (20) supporting a primary database
`instance and a secondary server (22) supporting a secondary
`database instance that mirrors the primary database instance.
`The secondary server (22) generates an acknowledgment
`signal (60) indicating that a selected critical database trans(cid:173)
`action ( 42) is mirrored at the secondary database instance. A
`plurality of other servers (14, 16, 18) each support a data(cid:173)
`base. A data replicator (30) communicates with the critical
`database server (12) and the other servers (14, 16, 18) to
`replicate the selected critical database transaction ( 42) on at
`least one of said plurality of other servers (14, 16, 18)
`responsive to the acknowledgment signal (60).
`
`10 Claims, 6 Drawing Sheets
`
`12 r -
`~
`1
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`~ Ccnt_Serv _Sec
`~ n
`R-1
`
`I
`
`I
`1
`
`;
`34
`................... ~ ....... ,
`
`~--------1 -------i---------------- I
`C::::
`=::::>
`
`Dat_Replic
`
`.1Q
`
`/
`10
`
`Rmt_l
`14
`
`Rmt_2
`16_
`
`Rmt_3
`.lli
`
`

`

`Case 6:20-cv-01152-ADA Document 1-1 Filed 12/16/20 Page 3 of 16
`
`US 7,177,886 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`2005/0114285 Al*
`
`5/2005 Cincotta ........................ 707/1
`
`9/2002
`6,446,075 Bl
`6,654,891 Bl* 11/2003
`6,671,757 Bl* 12/2003
`7,065,541 B2
`6/2006
`2002/0116457 Al*
`8/2002
`2002/0163910 Al * 11/2002
`
`Velasco
`Borsato et al. ................ 726/6
`Multer et al. ............... 710/100
`Gupta et al.
`.......... 709/203
`Eshleman et al.
`Wisner et al.
`.............. 370/389
`
`OTHER PUBLICATIONS
`
`Abstracted Patent No. JP200259473 of Shibata Hirohito, Entitled:
`Database Management System, Publication Date Sep. 22, 2000.
`IBM Technical Disclosure Bulletin, vol. 30, No. 4, Sep. 1987, pp.
`1738-1740 - Database Recovery Using A Transaction Log File.
`
`* cited by examiner
`
`

`

`Case 6:20-cv-01152-ADA Document 1-1 Filed 12/16/20 Page 4 of 16
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`,
`
`I Cent_Serv _Sec I :
`I Cent_Serv _Prim I .,_.
`. . . • . .
`............. .?.~.~ ...... .l L ~
`~-------~~-----~------------------(cid:173)
`. . .
`
`12 r -
`~
`1
`I
`
`R-1
`32
`
`20
`
`:
`
`22
`
`R-1
`
`. I
`
`I
`I
`
`/
`10
`
`R-1
`32'
`
`Dat_Replic 30
`
`Rmt 1
`14
`
`Rmt 2
`16
`
`Rmt_3
`18
`
`FIG 1
`
`e •
`
`00
`•
`~
`~
`~
`
`~ = ~
`
`"f'j
`
`~ ....
`
`~
`"'
`N
`0
`0
`-....J
`
`rJJ =(cid:173)
`('D a ....
`0 ....
`
`O'I
`
`d
`rJl.
`"'-.....l
`"""'
`-.....l
`-.....l
`00
`00
`
`O', = N
`
`

`

`Case 6:20-cv-01152-ADA Document 1-1 Filed 12/16/20 Page 5 of 16
`
`e •
`
`00
`•
`~
`~
`~
`
`~ = ~
`
`.
`34
`:
`. . .
`•..••..•••.•••.•••• ~ •••.... !
`~--------~---------~-~-~----------(cid:173)
`. . . . . •
`
`R-1
`
`"f'j
`
`~ ....
`
`~
`"'
`N
`0
`0
`-....J
`
`Dat_Replic 30
`
`rJJ =(cid:173)
`('D a
`0 ....
`
`N
`
`O'I
`
`/
`10
`
`Rmt_l
`14
`
`Rmt_2
`16
`
`Rmt_3
`18
`
`R-1
`32'
`
`R-1
`32'
`
`R-1
`32'
`
`FIG2
`
`d r.,;_
`"'-.....l
`"""'
`-.....l
`-.....l
`00
`00
`
`O', = N
`
`

`

`Case 6:20-cv-01152-ADA Document 1-1 Filed 12/16/20 Page 6 of 16
`
`········································································-···················~·-·······································
`
`~ _ J Transaction on primary ~ - - - - - -
`' l
`I
`42
`Logs ship to HDR
`secondary 52
`
`I
`
`Dat_Rep Snooping
`44
`
`Reconstruct transaction
`46
`
`Send Queue 48
`
`40
`
`Transactions applied and
`logged on secondary 54
`
`Secondary transmits
`acknowledgment 56
`
`....
`
`Clear send queue 66
`
`•
`
`-
`
`•
`
`-
`
`;;>
`
`.,1 Remote servers
`14, 1§, _IB
`
`I
`
`64
`I
`~
`. FIG 3
`
`60
`t__J
`
`..
`
`e •
`
`00
`•
`~
`~
`~
`
`~ = ~
`
`"f'j
`('D
`
`?' ....
`
`~
`"'
`N
`0
`0
`-....J
`
`('D
`('D
`
`~
`
`rJJ =(cid:173)
`.....
`0 ....
`
`O'I
`
`d r.,;_
`"'-.....l
`"""'
`-.....l
`-.....l
`00
`00
`
`O', = N
`
`

`

`Case 6:20-cv-01152-ADA Document 1-1 Filed 12/16/20 Page 7 of 16
`
`Log_Buf
`72
`
`P_HDR_Buf
`74
`
`10 -
`
`11 -12 -Ll
`
`6 -7 -8 -2
`
`P _log file
`70
`
`6 -7 -8 -9 -
`
`HDR_Cont_Struct
`86
`
`I Last Ack=9 I
`i
`I Send Queue :Hi I
`
`Cent_Serv _Prim
`20
`
`App_Comp
`82
`
`S_HDR_Buf
`
`80 -6 -1
`8 -9 -
`
`6 -7 -8 -
`s;nt_Serv_Sec FIG 4
`
`2
`
`e •
`
`00
`•
`~
`~
`~
`
`~ = ~
`
`"f'j
`('D
`?'
`....
`
`~
`"'
`N
`0
`0
`-....J
`
`('D
`('D
`
`.i;...
`
`rJJ =(cid:173)
`.....
`0 ....
`
`O'I
`
`d r.,;_
`"'-.....l
`"""'
`-.....l
`-.....l
`00
`00
`
`O', = N
`
`

`

`Case 6:20-cv-01152-ADA Document 1-1 Filed 12/16/20 Page 8 of 16
`
`Replication transaction is applied and logged at the primary server 90
`
`+
`
`Post Dat_Rep acknowledgment indexed to current primary log position 92
`-------~-----~--------------------------------------------------------------------------------------------
`Retrieve most recently
`backed up log pos lQ2
`
`r" •
`
`100
`V"'\:
`
`Create dummy
`transaction 110
`
`1
`I
`I
`I
`
`I
`I
`I
`I
`
`e •
`
`00
`•
`~
`~
`~
`
`~ = ~
`
`"f'j
`('D
`
`?' ....
`
`~
`"'
`N
`0
`0
`-....J
`
`: _....._
`
`I
`
`•
`
`,....
`
`i . .._.( Posted Dat_Rep
`ack list 94
`
`Send 106
`
`No
`
`Flush log buffers 112
`
`('D
`('D
`
`rJJ =(cid:173)
`......
`Ul
`0 ....
`
`O'I
`
`64-F'f
`·
`i
`Clear send queue 66 ------------------------------------
`
`I Post an error alert .116 ~--~~--
`
`---
`
`1 Q 5
`Fl
`
`d r.,;_
`"'-.....l
`"""'
`-.....l
`-.....l
`00
`00
`
`O', = N
`
`

`

`Case 6:20-cv-01152-ADA Document 1-1 Filed 12/16/20 Page 9 of 16
`
`/120
`Ji.
`
`152
`
`/
`
`-170
`
`'> ~
`
`170
`
`156./ 110-f vo
`
`150
`
`_c..
`
`177
`
`lnl
`
`\
`
`'\ ("-.,, 170
`
`IEJ
`
`...-
`El I
`
`1 -1 1-,
`
`1.-Wnl
`
`I
`
`«
`
`)\,,,,,tr:::;--
`
`130
`
`132
`
`FIG 6 134
`
`136
`
`e •
`
`00
`•
`~
`~
`~
`
`~ = ~
`
`"f'j
`('D
`
`?' ....
`
`"'~
`N
`0
`0
`-....J
`
`162
`
`('D
`
`rJJ =-('D
`.....
`O'I
`0 ....
`
`O'I
`
`d r.,;_
`
`"'-.....l
`"""'
`-.....l
`-.....l
`00
`00
`
`O', = N
`
`

`

`Case 6:20-cv-01152-ADA Document 1-1 Filed 12/16/20 Page 10 of 16
`
`US 7,177,886 B2
`
`1
`APPARATUS AND METHOD FOR
`COORDINATING LOGICAL DATA
`REPLICATION WITH HIGHLY AVAILABLE
`DATA REPLICATION
`
`BACKGROUND OF THE INVENTION
`
`The present invention relates to the information arts. In
`finds particular application in relational database systems
`that distribute data across a plurality of computers, servers, 10
`or other platforms, and will be described with particular
`reference thereto. However, the invention also finds appli(cid:173)
`cation in many other systems including distributed informa(cid:173)
`tion systems, in information backup systems, and the like.
`Relational database systems are widely used in business, 15
`govermnent, and other organizations to record, store, pro(cid:173)
`cess, share, and otherwise manipulate information. Because
`such organizations are commonly regional, national, or
`global in scope, the relational database is preferably acces(cid:173)
`sible from regionally, nationally, or globally distributed 20
`computers, terminals, or other devices across local area
`networks, Internet links, wireless links, and other commu(cid:173)
`nication pathways. For example, worldwide offices of a
`corporation preferably access a single corporate database or
`selected portions thereof.
`A problem arises in that accessing a single database by a
`large number of remote computer systems creates substan(cid:173)
`tial communication and data processing bottlenecks that
`limits database speed. To overcome such bottlenecks, a
`distributed database system is used, in which database 30
`information is shared or distributed among a plurality of
`database servers that are distributed across the communica(cid:173)
`tion network.
`A distributed database system typically includes a central
`database and various remote databases that are synchronized
`with the central database using various techniques. The
`remote databases can contain substantially the entire central
`database contents, or selected portions thereof. Moreover,
`transactions can be generated at the central database server
`or at one of the remote servers. In a commercial enterprise,
`for example, remote database servers at sales offices receive
`and generate purchase order transactions that propagate by
`data distribution to the central database server and in some
`cases to other database servers. Similarly, remote servers at
`billing centers generate sales invoice transactions that propa(cid:173)
`gate through the distributed database system, and so forth.
`The central database server provides a repository for all
`database contents, and its contents are preferably highly
`robust against server failures.
`To provide for recovery in the event that the central
`database fails, the central database can include primary and
`secondary database instances. The secondary database
`instance mirrors the primary database instance and acts as a
`hot backup providing failover recovery in the event of a
`primary database failure. Mirroring is maintained by ship- 55
`ping logical log files of the primary database instance to the
`secondary instance as they are being copied to disk or other
`non-volatile storage on the primary instance. The secondary
`instance remains in recovery mode as it is receiving and
`processing the shipped logical log files. Since all log records 60
`are processed at the secondary instance, the secondary
`instance provides a mirror image backup of the primary
`database instance, except for recent transactions that may
`not have been copied to the secondary instance yet. The
`primary and secondary database instances are in some cases 65
`configured such that a transaction commit is not completed
`at the primary until the log of that transaction is shipped to
`
`25
`
`2
`the secondary instance. Such a central database is robust
`against primary database failure and provides a fail-safe
`solution for high availability. However, it is limited in
`functionality, supporting only a single or limited number of
`5 synchronized secondary instances, which must be substan(cid:173)
`tially compatible. For example, the primary log records
`should be interpretable by the secondary server without
`introducing substantial translation processing overhead.
`Remote databases which store some or all information
`contained in the central database are typically maintained by
`synchronous or asynchronous data replication. In synchro(cid:173)
`nous replication, a transaction updates data on each target
`remote database before completing the transaction. Synchro(cid:173)
`nous replication provides a high degree of reliability and
`substantially reduced latency. However, synchronous repli(cid:173)
`cation introduces substantial delays into data processing,
`because the replication occurs as part of the user transaction.
`This increases the cost of the transaction, and can make the
`transaction too expensive. Moreover, a problem at a single
`database can result in an overall system failure. Hence,
`synchronous replication is usually not preferred except for
`certain financial transactions and other types of transactions
`which require a very high degree of robustness against
`database failure.
`Asynchronous replication is preferred for most data dis(cid:173)
`tribution applications. In asynchronous replication, transac(cid:173)
`tion logs of the various database servers are monitored for
`new transactions. When a new transaction is identified, a
`replicator rebuilds the transaction from the log record and
`distributes it to other database instances, each of which
`apply and commit the transaction at that instance. Such
`replicators have a high degree of functionality, and readily
`support multiple targets, bi-directional transmission of rep(cid:173)
`licated data, replication to dissimilar machine types, and the
`35 like. However, asynchronous replicators have a substantial
`latency between database updates, sometimes up to a few
`hours for full update propagation across the distributed
`database system, which can lead to database inconsistencies
`in the event of a failure of the central database server. Hence,
`40 asynchronous replicators are generally not considered to be
`fail-safe solutions for high data availability.
`Therefore, there remains a need in the art for a method and
`apparatus for fail-safe data replication in a distributed data(cid:173)
`base system, which provides for reliable fail-safe recovery
`45 and retains the high degree of functionality of asynchronous
`replication. Such a method and/or apparatus should be
`robust against a failure at a critical node within the replica(cid:173)
`tion domain, and should ensure the integrity of transaction
`replications to other servers within the replication domain in
`50 the face of such a critical node failure.
`The present invention contemplates an improved method
`and apparatus which overcomes these limitations and others.
`
`SUMMARY OF THE INVENTION
`
`In accordance with one aspect, a database apparatus
`includes a critical database server having a primary server
`supporting a primary database instance and a secondary
`server supporting a secondary database instance that mirrors
`the primary database instance. The secondary server gener(cid:173)
`ates an acknowledgment signal indicating that a selected
`critical database transaction is mirrored at the secondary
`database instance. A plurality of other servers each support
`a database. A data replicator communicates with the critical
`database server and the other servers to replicate the selected
`critical database transaction on at least one of said plurality
`of other servers responsive to the ackuowledgment signal.
`
`

`

`Case 6:20-cv-01152-ADA Document 1-1 Filed 12/16/20 Page 11 of 16
`
`US 7,177,886 B2
`
`4
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`3
`In accordance with another aspect, a method is provided
`for integrating a high availability replication system that
`produces at least one mirror of a critical database node, with
`The invention may take form in various components and
`a data distribution replication system that selectively repli(cid:173)
`arrangements of components, and in various process opera(cid:173)
`cates data at least from the critical database node to one or 5
`tions and arrangements of process operations. The drawings
`more remote databases. In the data distribution replication
`are only for the purposes of illustrating preferred embodi(cid:173)
`system, an object at the critical database node targeted for
`ments and are not to be construed as limiting the invention.
`replication is identified. In the high availability replication
`FIG. 1 is a block diagram showing a distributed relational
`system, objects including the identified object are replicated
`database system including a central database server with a
`at the mirror and a mirror acknowledgment indicative of
`primary database server and a hot-backnp secondary data(cid:173)
`completion of replication of the identified object at the
`base server, a highly available data replication component
`mirror is generated. In the data distribution replication
`for maintaining the hot-backnp secondary database, and a
`system, the identified object is replicated responsive to the
`logical data replication component for selectively distribut-
`mirror acknowledgment.
`15 ing data amongst remote servers and the central database.
`In accordance with another aspect, a method is provided
`FIG. 2 is a block diagram showing the distributed rela(cid:173)
`for coordinating data replication to distributed database
`tional database system of FIG. 1 after the primary server of
`servers with a hot-backup instance of a database. Database
`the central database server has failed and failover recovery
`transactions are backed up at the hot-backnp instance. A
`control has passed to the secondary database server.
`backup indicator is maintained that identifies database trans(cid:173)
`FIG. 3 is a flowchart showing a preferred method for
`actions backed up at the hot-backup source. Data replication 20
`synchronizing logical data replication with highly available
`of a database transaction is delayed until the backup indi(cid:173)
`data replication.
`cator identifies the database transaction as having been
`backed up at the hot-backup source.
`FIG. 4 is a block diagram showing a preferred embodi(cid:173)
`In accordance with yet another aspect, an article of
`ment of the highly available data replication component that
`manufacture includes a program storage medium readable
`25 includes communication of a synchronizing acknowledg(cid:173)
`by a computer and embodying one or more instructions
`ment signal to a send queue of the logical data replication
`executable by the computer to perform process operations
`component.
`for executing a command to perform a database operation on
`FIG. 5 is a flowchart showing a modification of the
`a relational database connected to the computer. A transac(cid:173)
`process of FIG. 3 for providing robust synchronization of
`tion performed in the relational database is identified. The 30
`logical data replication with highly available data replication
`identified transaction is replicated responsive to an indica(cid:173)
`in a case where the logical data replicator sends a replication
`tion that the identified transaction has been backed up at the
`transaction to the primary server.
`relational database.
`FIG. 6 is a block diagram showing another distributed
`In accordance with still yet another aspect, an apparatus
`relational database system, which has a tree topology with
`for supporting a distributed relational database includes
`three critical nodes, each critical node having a highly
`primary and secondary servers. The primary server supports
`available data replication pair including a primary database
`a primary database instance that includes a primary database
`server and a hot-backup secondary database server, and a
`instance log file. The secondary server supports a secondary
`logical data replication component for selectively distribut-
`database instance that includes a secondary instance log file.
`A plurality of other servers each support a database instance. 40 ing data amongst servers of the tree topology.
`A highly available data replication component communi(cid:173)
`DETAILED DESCRIPTION OF THE
`cates with the primary and secondary servers to transfer
`PREFERRED EMBODIMENTS
`primary database instance log file entries from the primary
`server to the secondary server. The secondary server pro(cid:173)
`duces an acknowledgment indicating that the transferred log 45
`file entries have been received. A logical data replication
`component communicates with the primary server and the
`other servers to identify a log record in the primary database
`instance log file, construct a replication transaction corre(cid:173)
`sponding to the identified log record, and, responsive to the 50
`highly available data replication component indicating that
`the identified log record has been received at the secondary
`server, cause one or more of the other servers to perform the
`replication transaction.
`One advantage resides in avoiding data inconsistencies 55
`among remote servers in the event of a failure of the central
`database primary server.
`Another advantage resides providing asynchronous rep(cid:173)
`lication functionality that is robust with respect to primary
`database failure.
`Yet another advantage resides in providing for fail-safe
`recovery via a high availability replication system, while
`retaining the broad functionality of data distribution by
`asynchronous replication.
`Still further advantages and benefits will become apparent
`to those of ordinary skill in the art upon reading and
`understanding the following detailed description.
`
`10
`
`35
`
`With reference to FIG. 1, a distributed relational database
`system 10 of a spokes-and-hub topology includes a central
`database server 12 and a plurality of remote database servers
`14, 16, 18. The central database server 12 includes a primary
`server 20 and a secondary server 22 that mirrors the primary
`server 20. The mirroring is provided by a highly available
`data replication (HDR) component 26 that transfers log
`records of the central database primary server 20 to the
`secondary server 22. The log records are applied and logged
`at the secondary server 22. In this manner, the secondary
`server 22 is maintained as a mirror image of the primary
`server 20, except for a set of most recent primary server
`transactions which may not yet have been transferred by the
`highly available data replication component 26.
`Although the primary and secondary server components
`60 20, 22 of the central database 12 are shown together in FIG.
`1, the combination is a logical combination, and is not in
`general a physical combination. That is, the primary and
`secondary server components 20, 22 can be spatially remote
`from one another and in operative communication via a
`65 communication network, which may also include the remote
`servers 14, 16, 18. The servers 20, 22 are preferably logi(cid:173)
`cally compatible. For example, the log files of the primary
`
`

`

`Case 6:20-cv-01152-ADA Document 1-1 Filed 12/16/20 Page 12 of 16
`
`US 7,177,886 B2
`
`5
`
`5
`server 20 are preferably readily interpretable by the second(cid:173)
`ary server 22 without computationally intensive translation
`processing.
`The distributed database 10 is of the spokes-and-hub
`topology, in which there is one critical node, namely the
`central database server 12, which serves as the hub. The
`plurality of remote database servers 14, 16, 18 are spokes
`that connect at the hub. The central database server 12 is a
`critical node because a failure of that server results in service
`interruption for a number of other servers, such as the
`remote database servers 14, 16, 18. Rather than a spokes(cid:173)
`and-hub topology, other topologies can be employed, such
`as a tree topology, in which there is more than one critical
`node. In topologies which include more than one critical
`node, each critical node is preferably supplied with its own
`highly available data replication (HDR) hot backup.
`Data distribution by asynchronous replication amongst
`the primary server 12 and the remote servers 14, 16, 18 of
`the database system 10 is performed by an asynchronous
`logical data replication component 30. The data replication
`component 30 produces computation threads that monitor
`transaction logs of the primary server 20 of the central
`database 12 and of the remote servers 14, 16, 18 to identify
`recent transactions. Advantageously, such log monitoring
`does not significantly slow operation of the servers 12, 14,
`16, 18. When a recently logged transaction is identified, the
`data replication component 30 constructs one or more rep(cid:173)
`lication transactions that effect replication of the logged
`transaction.
`Because replication transactions are generated by the data
`replication component 30, the replication transaction can be
`different in form but equivalent in function to the original
`transaction. This allows the central database server 12 and
`the various remote database servers 14, 16, 18 to be dis(cid:173)
`similar, for example with respect to operating system, com(cid:173)
`puter type, and the like. Replication to multiple targets,
`bi-directional transmission of replicated data, replication to
`dissimilar machine types, and the like are readily supported
`by the data replication component 30. Data replication can
`also be selective. That is, only certain data on the central 40
`database 12 or the remote servers 14, 16, 18 can be repli(cid:173)
`cated to selected remote servers 14, 16, 18. For example, if
`remote servers 14, 16, 18 are Eastern, Midwestern, and
`Western regional servers, then data is suitably regionally
`filtered and selectively distributed to
`the appropriate 45
`regional remote server 14, 16, 18 by the data replication
`component 30.
`In FIG. 1, an exemplary row insertion "R-1" transaction
`32 is performed at the primary server 20 of the central
`database 12. Although an exemplary row insertion transac- 50
`tion is described herein for purposes of illustrating a pre(cid:173)
`ferred embodiment, substantially any type of relational
`database transaction can be similarly processed. The row
`insertion transaction 32 is logged at the primary database,
`identified by the data replication component 30, and a 55
`replication transaction 32' is generated. However, the repli(cid:173)
`cation transaction 32' is not immediately sent to the remote
`servers 14, 16, 18. Rather, the data replication component 30
`initially waits for an indication that the transaction 32 has
`been backed up at the secondary server 22 of the central 60
`database 12 before sending it to the remote servers 14, 16,
`18.
`Specifically, in the embodiment of FIG. 1 the highly
`available data replication component 26 transfers recent log
`records of the primary server 20, including a log record of
`the row insertion transaction 32, to the secondary server 22.
`At the secondary server 22, the transferred log records are
`
`6
`applied and logged, including a row insertion transaction
`32" that mirrors the row insertion transaction 32 which was
`performed at the primary server 20. The secondary server 22
`generates an acknowledgment indicating that the row inser-
`tion transaction 32" is applied and logged.
`In response to this acknowledgment, the highly available
`data replication component 26 produces a mirror acknowl(cid:173)
`edgment 34 indicating that the transaction 32 of the primary
`server 20 is mirrored at the secondary server 22. Responsive
`10 to the mirror acknowledgment 34, the data replication com(cid:173)
`ponent 30 begins sending the replication transaction 32' to
`the remote servers 14, 16, 18.
`With continuing reference to FIG. 1 and with further
`reference to FIG. 2, a significant advantage of delaying
`15 transmission of the replication transaction 32' to the remote
`servers 14, 16, 18 until receipt of the mirror acknowledg(cid:173)
`ment 34 is described. In FIG. 2, the primary server 20 of the
`central database 12 is shown by its absence in FIG. 2 as
`having failed after the transaction 32' has been transmitted to
`20 the remote server 14, but before the transaction 32' has been
`transmitted to the remote servers 16, 18. Because the data
`replication component 30 delayed sending the transaction
`32' until after receipt of the mirror acknowledgment 34, it is
`assured that the transaction 32 is mirrored at the secondary
`25 server 22 by the mirror transaction 32" before the replication
`transaction is distributed. Moreover, the replication transac(cid:173)
`tion 32' remains queued for sending at the data replication
`component 30, which continues to forward the replication
`transaction 32' to the remaining remote servers 16, 18 so that
`30 all remote servers 14, 16, 18 scheduled for receipt of the
`replication transaction 32' actually receive the transaction.
`As a result, there are no data inconsistencies between the
`central database server 12 and the remote servers 14, 16, 18.
`In contrast, in a conventional arrangement in which there
`35 are no delays, replication transactions are transmitted as
`soon as they are reconstructed. As a result, none, some, or all
`of the remote servers may or may not receive the replication
`transaction in the event of a failure of the central database
`primary server. Furthermore, the transaction being repli(cid:173)
`cated may or may not have been copied to the secondary
`server prior to failover. Thus, data inconsistencies may result
`between the remote servers, and between remote servers and
`the central database server, in the event of a failure of the
`central database primary server.
`In addition to the highly available data replication com(cid:173)
`ponent 26 providing the synchronizing mirror acknowledg(cid:173)
`ment 34, to ensure data consistency in the event of a fail over
`recovery, the data replicator 30 preferably generates trans(cid:173)
`action replication threads that communicate only with the
`primary server 20, and not with the secondary server 22. In
`its preferred form, this is accomplished during replication
`thread generation by checking whether a server of the
`replication thread is acting as a secondary server of a highly
`available data replication component. If it is, then the thread
`is canceled or a suitable error indicator generated. Prefer(cid:173)
`ably, the distributed database 10 is configured so that the
`central server 12 appears as a single logical entity to the data
`replicator 30.
`With continuing reference to FIG. 1 and with further
`reference to FIG. 3, the preferred data replication method 40
`executed by the relational database system 10 is described.
`A transaction 42 occurs on the primary server 20 of the
`central database 12. The data replicator 30 monitors, or
`snoops 44, the log files of the primary server 20 and
`65 identifies a log record corresponding to the transaction 42.
`The data replicator 30 reconstructs 46 the transaction 42
`based on the identified transaction log record to generate a
`
`

`

`Case 6:20-cv-01152-ADA Document 1-1 Filed 12/16/20 Page 13 of 16
`
`US 7,177,886 B2
`
`7
`replication transaction that is placed in a send queue 48.
`However, the replication transaction is not immediately sent.
`The highly available data replication component 26 also
`processes the transaction 42, by shipping 52 log files includ(cid:173)
`ing a log of the transaction 42 to the secondary server 22.
`The transaction logs are applied and logged 54 at the
`secondary server 22, and the secondary sever 22 transmits
`56 an acknowledgment 60 to the primary server 20.
`Responsive to the acknowledgment 60, a transmit gate 62
`transmits the corresponding replication transaction in the
`send queue 48 to the remote servers 14, 16, 18. Each remote
`server receives, applies, and logs the replication transaction,
`and generates a replication acknowledgment 64. Responsive
`to the replication acknowledgment 64, the data replicator 30
`clears 66 the corresponding replication transaction from the
`send queue 48.
`With reference to FIG. 4, the preferred configuration of
`the highly available data replication component 26 is
`described. The component 26 generates a gating signal for
`synchronizing the data replicator 30 with the highly avail(cid:173)
`able data replication component 26. The primary server 20
`maintains a primary server log file 70. Recent transactions
`are stored in a primary server log buffer 72. The contents of
`the log buffer 72 are from time to time flushed and written
`to the primary server log file 70 which is stored on a
`magnetic disk or other non-volat