(12) United States Patent
`Mithyantha
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,972,602 B2
`Mar. 3, 2015
`
`US008972602B2
`
`SYSTEMIS AND METHODS FORUSING
`ECMP ROUTES FOR TRAFFIC
`DISTRIBUTION
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`7/2009 Aysan et al. .................. TO9,238
`7,568,047 B1 *
`2013/0286846 A1* 10/2013 Atlas et al. .........
`370,236
`2013/0336329 A1* 12/2013 Gopinath et al. ............. 370/401
`* cited by examiner
`Primary Examiner — Frantz Jean
`(74) Attorney, Agent, or Firm — Foley & Lardner LLP:
`Christopher J. McKenna
`(57)
`ABSTRACT
`The present application is directed towards systems and
`methods for using equal cost multi-path routing for traffic
`distribution in a cluster environment. Each intermediary
`device of a cluster may advertise, via a routing protocol to a
`router, a corresponding internet protocol (IP) address of a
`virtual server and one or more connection metrics having
`predetermined values. Upon determining that another inter
`mediary device of the cluster is unavailable, each active
`device may re-advertise the IP address of the virtual server
`executing on the intermediary device and the one or more
`connection metrics with the previously advertised value
`reduced by a predetermined amount. In some embodiments,
`each active device may wait a predetermined time period,
`Such as a time period for expiration of routing protocol tables,
`and then re-advertise the IP address of the virtual server
`executing on the intermediary device and the one or more
`connection metrics with the predetermined values.
`18 Claims, 20 Drawing Sheets
`
`(54)
`
`(75)
`
`(73)
`
`Inventor: Sharvari Mithyantha, Karnataka (IN)
`
`Assignee: Citrix Systems, Inc., Fort Lauderdale,
`FL (US)
`
`(*)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 312 days.
`
`(21)
`
`Appl. No.: 13/524,799
`
`(22)
`
`Filed:
`
`Jun. 15, 2012
`
`(65)
`
`(51)
`
`(52)
`
`(58)
`
`Prior Publication Data
`US 2013/0339544A1
`Dec. 19, 2013
`
`Int. C.
`G06F 5/73
`U.S. C.
`USPC ........... 709/238; 709/249; 370/235; 370/352;
`370/392
`
`(2006.01)
`
`Field of Classification Search
`CPC ...... H04L 45/24; H04L 45/138: H04L 47/125
`USPC ................... 709/238, 249; 370/235,352,392
`See application file for complete search history.
`
`
`
`
`
`Client 102a
`
`Client 102
`
`
`
`Client 102n
`
`Server
`
`106n
`
`Metacluster LT, UAB
`EX1009
`Page 1 of 52
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar.3, 2015
`
`Sheet 1 of 20
`
`US 8,972,602 B2
`US 8,972,602 B2
`
`seAleg=n———ge
`
`89
`
`egOL
`
`
`
`
`
`YIOMJON
`
`‘VOL
`
`aouelddy
`
`yIomMjaN
`
`vOL
`
`()
`
`
`
`U90,s9AIES
`
`VlSls
`
`
`
` 000q901Janes
`
`00
`
`UZOL=ywUaIID
`
` BZOLywuaDail
`GzZ0b=ywaid
`
`Page 2 of 52
`
`Page 2 of 52
`
`
`
`
`
`

`

`U.S. Patent
`
`Mar. 3, 2015
`
`Sheet 2 of 20
`
`US 8,972,602 B2
`
`
`
`
`
`
`
`
`
`eZ0), quello
`
`Page 3 of 52
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar. 3, 2015
`Mar.3, 2015
`
`Sheet 3 of 20
`Sheet 3 of 20
`
`US 8,972,602 B2
`US 8,972,602 B2
`
`
`
`
`
`
`
`
`
`B90;saalag
`
`jualDSS
`eZOl
`
`
`
`q90)JaAlag
`
`{eolAap(aolAep0uoneziwydQ
`
`uoneziwundo
`
`
`
`aouelddyaoueljddy
`
`
`
`NVMNVMazObward
`
`
`
`ugOL—_JaAiegJb‘SlsuzOLwena
`
`Page 4 of 52
`
`Page 4 of 52
`
`
`
`
`

`

`U.S. Patent
`
`Mar. 3, 2015
`
`Sheet 4 of 20
`
`US 8,972,602 B2
`
`
`
`V790), J?AuÐS
`
`
`
`
`
`
`
`
`
`00Z
`
`H = H
`
`§
`
`Page 5 of 52
`
`

`

`U.S. Patent
`
`Mar. 3, 2015
`
`Sheet 5 of 20
`
`US 8,972,602 B2
`
`100
`
`Y.
`
`
`
`128
`
`101
`
`120
`
`Memory
`
`150
`
`123
`
`I/O
`CTRL
`
`Display
`device(s)
`
`126
`
`127 N-124a-n
`
`Keyboard
`
`Pointing
`
`Installation
`Device
`116
`
`NetWOrk
`Interface
`118
`
`Fig. IE
`
`Page 6 of 52
`
`

`

`U.S. Patent
`
`Mar. 3, 2015
`
`Sheet 6 of 20
`
`US 8,972,602 B2
`
`101
`
`140
`
`
`
`
`
`Main
`Processor
`
`I/O I/O Memory
`POrtPOrt POrt
`
`1
`
`Fig. IF
`
`Page 7 of 52
`
`

`

`U.S. Patent
`
`Mar. 3, 2015
`
`Sheet 7 of 20
`
`US 8,972,602 B2
`
`
`
`101
`
`101
`
`101'
`
`CPU
`
`GPU
`
`Fig. IH
`
`Page 8 of 52
`
`

`

`
`
`
`
`OrzsulBuzgjeyoegpozei69}u|
`
`VeSlsnar 992SHOd09S97JOSSO90/d
`obsweiboldgvie
`
`
`
`YIOMIONJOSS9I01q||10SS9901guondAusuy902
`
`uolssaidwo4yayoe9
`
`
`BuloyuoWyyeoHSODIAIBS|]9US
`
`
`2-@4eKe7peeds-yBiH0Ezv0e
`8&7Jsabeuew
`
`~lawiayaoedsg
`Zzjaulay
`
`U.S. Patent
`U.S. Patent
`
`Mar. 3, 2015
`Mar.3, 2015
`
`Sheet 8 of 20
`Sheet 8 of 20
`
`US 8,972,602 B2
`US 8,972,602 B2
`
`
`
`
`
`Jodo}o1d-njnNW
`
`
`
`
`
`812saolnsagUOWaeGWa}sks
`
`YIOMJON
`
`19749e1S
`
`uondAssug
`
`veceulbug
`
`bonnenegeeeiauempsey
`
`Page 9 of 52
`
`Page 9 of 52
`
`

`

`U.S. Patent
`
`Mar. 3, 2015
`
`Sheet 9 of 20
`
`US 8,972,602 B2
`
`e90 || 13 Au3S
`
`
`
`987 SNC]
`
`
`
`
`
`
`
`
`
`
`
`
`
`06Z NAH ddwf
`
`Page 10 of 52
`
`

`

`U.S. Patent
`
`Mar. 3, 2015
`
`Sheet 10 of 20
`
`US 8,972,602 B2
`
`Client 102
`
`user mode 303
`
`App 1
`
`App 2
`
`1st Program
`322
`App N
`
`Network
`Stack
`310
`
`r
`
`APII data
`structure 325
`
`monitoring
`agent/script 197
`
`Streaming Client
`306
`
`Collection Agent
`3O4.
`
`Acceleration
`Program 302
`
`interceptor
`350
`
`Client Agent 120
`
`
`
`100
`
`Page 11 of 52
`
`

`

`U.S. Patent
`
`Mar. 3, 2015
`
`Sheet 11 of 20
`
`US 8,972,602 B2
`
`
`
`virtualized environment 400
`VIRTUALIZATION LAYER
`
`device 100
`
`Virtual Machine 406a
`
`Virtual Machine 406b
`
`Virtual Machine 406c
`
`Control
`Operating
`System
`405
`
`Tools
`Stack 404
`
`Guest
`Operating
`System
`
`410a
`
`Guest
`Operating
`System
`
`41 Ob
`
`HYPERVISORLAYER
`
`Hypervisor 401
`
`HARDWARE LAYER
`
`Physical Disk(s) 428
`
`Physical CPU(s) 421
`
`Fig. 4A
`
`Page 12 of 52
`
`

`

`U.S. Patent
`
`Mar. 3, 2015
`
`Sheet 12 of 20
`
`US 8,972,602 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Computing Device 100a
`Virtual
`Virtual
`Machine
`Machine
`406a
`406b
`
`
`
`
`
`Control OS
`405a
`Management
`component
`404a
`
`Guest
`Operating
`System
`410a
`
`Computing Device 100b
`Virtual
`Virtual
`Machinc
`Machine
`406c
`406d
`
`
`
`Control
`OS 405b
`Mgmt
`component
`404a
`
`Guest
`Operating
`System
`41 Ob
`
`Hypervisor
`401 a
`
`
`
`
`
`
`
`Virtual
`Resources
`432a, 442a :
`
`Computing Device 100c
`Virtual Machine 450e
`Guest Operating System 410c
`
`
`
`
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
`
`Virtual Machine
`406f
`
`Control OS
`405C
`Management
`component
`404a
`
`Page 13 of 52
`
`

`

`U.S. Patent
`
`Mar. 3, 2015
`
`Sheet 13 of 20
`
`US 8,972,602 B2
`
`virtualized application delivery Controller 450
`
`
`
`VServer A 275a
`
`VServer A 275a
`
`VServer A 275n
`
`VServer A 275n
`
`SSL VPN 280
`
`Intranet P 282
`
`Switching 284
`
`SSL VPN 280
`
`Intranet P 282
`
`Switching 284
`
`DNS 286
`
`DNS 286
`
`ACCeleration 288
`
`ACCeleration 288
`
`App FW 290
`
`App FW 290
`
`Virtualized environment 400
`
`Computing device 100
`
`Fig. 4C
`
`Page 14 of 52
`
`

`

`U.S. Patent
`
`Mar. 3, 2015
`
`Sheet 14 of 20
`
`US 8,972,602 B2
`
`"Si6A
`
`- - - - - - - - - - -
`
`510C;
`
`. -----------
`
`Functional
`
`Parallelism 500
`41
`
`I/O
`
`SSL ;
`
`;
`
`;...
`
`:
`
`:
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`542C
`
`............:
`542A
`i........... VIPs
`542B:
`; VIP2 :
`
`515
`
`
`
`VIP1
`
`-
`
`- - - -
`
`-
`
`- - -
`
`- -
`
`542D:
`
`v-----------
`...........:
`52D;
`NIC2
`
`i.
`
`NIC1
`
`Data
`
`Parallelism 540
`41
`
`...
`
`Flow-Based Data
`Paul- 520
`
`. i"536B"Side..."g:536F";
`
`536D
`
`. .
`
`.
`
`.
`
`536A
`
`.
`
`.
`
`.
`
`.
`
`
`
`---,
`
`536Ni
`
`Page 15 of 52
`
`

`

`U.S. Patent
`
`Mar. 3, 2015
`
`Sheet 15 of 20
`
`US 8,972,602 B2
`
`545
`
`548A
`Packet
`Engine A
`
`548B
`Packet
`Engine B
`
`548N
`Packet
`Engine N
`
`
`
`Memory Bus
`
`556
`
`Core 1
`
`Core 2 I Core 3
`
`Core 4
`
`Core 5
`
`Core 6
`
`Core 7
`
`505A
`
`505B
`
`505C
`
`505D
`
`505E
`
`505 F
`
`5OSG
`
`Core N
`
`505N
`
`Flow Distributor
`
`550
`
`Fig. 5B
`
`Page 16 of 52
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar.3, 2015
`
`Sheet 16 of 20
`
`US 8,972,602 B2
`US 8,972,602 B2
`
`Na109
`
`NSOS
`
`
`
`08Ssyoedjeqo|D
`
`GLS
`
`9S0S
`
`
`
`
`
`ga109ge10pa10pE9109Zal04)
`
`
`
`
`
`
`
`ASO0SASO¢Sdg0¢6OG0SgS0¢S
`
`US‘Sls
`
`Page 17 of 52
`
`18101)
`
`
`
`
`
`
`
`I
`
`lBUI!jo1]U0D
`
`Page 17 of 52
`
`
`
`

`

`U.S. Patent
`
`Mar. 3, 2015
`
`Sheet 17 of 20
`
`US 8,972,602 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`O ; O ;
`
`| EHEF
`
`009
`
`• • • • • • • • • • • • • • •
`
`e o o a a de
`
`e o 0
`
`8
`
`a
`
`Page 18 of 52
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar.3, 2015
`
`Sheet 18 of 20
`
`US 8,972,602 B2
`US 8,972,602 B2
`
`
`
`aue|qyorg
`
`909
`
`
`
`2002S31A9q
`
`——__/
`
`02/81
`
`40Z09SuedeqjUa!/D
`
`709eue|dejegJaAIEg
`
`
`
`ePOZJOWUOWBZ0L
`
`
`
`SNJE}SJOAIOSA
`
`
`
`QPOZ4oyUOLUqzoz
`
`
`
`snjelsJBAIOSA
`
`
`
`qo0zeo1,0q
`
`
`
`9002891A9q
`
`
`
`SNJE}SJOAIOSA
`
`U=dU}8lWBunnoy
`
`e902
`
`q907U=OUJeWBunnoy
`
`U=OujeWBulnoy
`
`
`
`
`
`dPOLJoWUOWIZOL
`
`
`
`POL4oyUOLWUWPZOL
`
`
`
`snjelsJBAIOSA
`
`P00ZS9IANq
`
`
`
`snjejsgJOAIOSA
`
`
`
`
`
`UpO/JOWUOWuzOL
`
`U =
`
`U=DU]8lWBunnoy
`
`ugos
`
`p90LU=SUEBulnoy
`
`Jaynoy
`
`002
`
`V2Z‘9Slh
`
`Page 19 of 52
`
`Page 19 of 52
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar.3, 2015
`
`Sheet 19 of 20
`
`US 8,972,602 B2
`US 8,972,602 B2
`
`
`
`
`
`”v7_Pooesomed
`
`
`
`QOLJOWUOWQé02
`
`
`
`snjeysJOAISSA
`
`
`
`q00z82!A9q
`
`
`
`3002801Aeq
`
`
`
`snjeysJOAISSA
`
`
`
`
`
`2PO0/JOWUOW9ZOL
`
`aue|gyoeg
`
`909
`
`JO209aue|qejeqyUaI|/D
`
`709sue|qBeQJ8AIBg
`
`EPOLJOWUOLWecoL
`
`
`
`L-U=OUISLUBulInoY
`
`
`
`snjeisJOAIOSA
`
`
`
`F00299/A9q
`
`—.*
`
`esol
`
`
`
`
`
`le----y
`ontoon
`
`I>cJBAISSA"he.
`
`\
`
`
`
`
`
`UpOZJOWUOWuzOL
`
`
`
`u00z89!Aaq
`
`
`
`snjeysJOAISSA
`
`L-U=SujauBunnoy
`
`U8OL
`
`qZ‘Sls
`
`Page 20 of 52
`
`p-U=OujelWBunoy
`
`g804
`
`L-U=oujewWBunnoy
`
`u=oujewWBunnoy
`
`PSoz
`
`J9jnoy
`
`00Z
`
`Page 20 of 52
`
`
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar.3, 2015
`
`Sheet 20 of 20
`
`US 8,972,602 B2
`US 8,972,602 B2
`
`
`
`SSolppeOSILOAPY
`
`
`
`PULJOAIOSAJO
`
`OpZUOlWewW
`
`
`
`posedowl}
`
`9rZ
`
`Jayjouesey
`
`
`
`SW098qBIIABP
`
`
`
`ssalppeasiLisApy
`
`
`
`DUEJSAIOSAJO
`
`PPLL-UOUjOW
`
`
`zalaeleneunPSUILUS}apeldeA,
`
`
`
`JZSls
`
`Page 21 of 52
`
`Page 21 of 52
`
`
`

`

`US 8,972,602 B2
`
`1.
`SYSTEMS AND METHODS FOR USING
`ECMP ROUTES FORTRAFFIC
`DISTRIBUTION
`
`FIELD OF THE INVENTION
`
`The present application generally relates to data commu
`nication networks. In particular, the present application
`relates to systems and methods for using equal cost multi
`path routing for traffic distribution in a cluster environment.
`
`10
`
`BACKGROUND OF THE INVENTION
`
`Equal-cost multi-path (ECMP) routing, and similar routing
`schemes, are used to allow routers to load balance traffic to a
`destination among a plurality of network paths, potentially
`offering substantial increases in bandwidth over that of a
`single path. In some embodiments of cluster environments in
`which a plurality of nodes of the cluster communicate with an
`upstream router, it may be desirable to use ECMP routing or
`a similar strategy to load balance traffic to the plurality of
`nodes from the upstream router. In some embodiments, the
`plurality of nodes of the cluster may appear to the upstream
`router be a single node, accessible via multiple paths, and
`accordingly, the router may distribute traffic amongst the
`multiple paths. However, in many such embodiments, if a
`node of the cluster goes down or becomes unavailable, the
`upstream router may not be able to detect this condition,
`because it may still be properly communicating with other
`nodes at the same virtual or mapped IP address. Accordingly,
`the router may continue sending packets to the unavailable
`node until a timeout or expiration interval of the routing
`protocol expires, and the packets may be lost.
`
`15
`
`25
`
`30
`
`BRIEF SUMMARY OF THE INVENTION
`
`35
`
`2
`the determination, the corresponding IP address of the virtual
`server with the at least one routing metric value reduced by a
`predetermined amount. The method further includes re-ad
`Vertising, by each intermediary device that is active and
`responsive to waiting a predetermined time period, the cor
`responding IP address of the virtual server and the at least one
`routing metric with the predetermined value.
`In Some embodiments, the method includes advertising, by
`each intermediary device of the cluster via the routing proto
`col comprising a dynamic routing protocol, the correspond
`ing IP address of the virtual server to the router to install
`equal-cost multi-path (ECMP) routes to the cluster on the
`router. In other embodiments, the method includes determin
`ing that the first intermediary device is one of down or is no
`longer able to communicate with the router. In yet other
`embodiments, the method includes identifying, the first inter
`mediary device as one of down or not active responsive to
`detecting that a health metric of the first intermediary device
`is below a predetermined threshold.
`In one embodiment, the method includes determining, by
`each intermediary device of the cluster, that the first interme
`diary device is not part of an active node set for the cluster. In
`another embodiment, the method includes comparing, by
`each intermediary device that is active, a current active node
`set with a previous active node set and, responsive to the
`comparison, determining that the current active node set is a
`Subset of the previous active node set. In some embodiments,
`the method includes re-advertising, by each intermediary
`device, the corresponding IP address of the virtual server with
`the at least one routing metric value reduced by a predeter
`mined amount of one. In other embodiments, the method
`includes installing, by the router, a new set of routes respon
`sive to the re-advertising with the at least one routing metric
`value reduced by a predetermined amount.
`In one embodiment, the method includes waiting, by each
`intermediary device that is active, a predetermined time
`period after re-advertising with the at least one routing metric
`with the predetermined value reduced by a predetermined
`amount before re-advertising with the at least one routing
`metric with the predetermined value. In another embodiment,
`the method includes waiting, by each intermediary device
`that is active, the predetermined time period comprising a
`time period that is configurable.
`In another aspect, the present disclosure is directed to a
`system for using routes for traffic distribution across a cluster
`of intermediary devices. The system includes a cluster com
`prising a plurality of intermediary devices in communication
`with a router. Each intermediary device of the cluster is con
`figured for executing a virtual server having an internet pro
`tocol (IP) address, wherein each intermediary device adver
`tises, via a routing protocol to the router, a corresponding
`internet protocol (IP) address of the virtual server and at least
`one routing metric with a predetermined value. Each inter
`mediary device of the system that is active determines that a
`first intermediary device of the cluster is one of down or not
`active, and re-advertises, responsive to the determination, the
`corresponding IP address of the virtual server with the at least
`one routing metric value reduced by a predetermined amount.
`Each intermediary device that is active further re-advertises,
`responsive to waiting a predetermined time period, the cor
`responding IP address of the virtual server and the at least one
`routing metric with the predetermined value.
`In one embodiment of the system, each intermediary
`device of the cluster advertises, via the routing protocol com
`prising a dynamic routing protocol, the corresponding IP
`address of the virtual server to the router to install equal-cost
`multi-path (ECMP) routes to the cluster on the router. In
`
`45
`
`The present application is directed towards systems and
`methods for using equal cost multi-path routing for traffic
`distribution in a cluster environment. In some embodiments,
`each intermediary device of a cluster comprising a plurality of
`40
`intermediary devices may advertise, via a routing protocol to
`a router, a corresponding internet protocol (IP) address of a
`virtual server executing on each intermediary device and one
`or more connection metrics having predetermined values.
`Upon determining that another intermediary device of the
`cluster is down or unavailable, each active device may re
`advertise the IP address of the virtual server executing on the
`intermediary device and the one or more connection metrics
`with the previously advertised value reduced by a predeter
`mined amount. In some embodiments, each active device
`may wait a predetermined time period. Such as a time period
`for expiration of routing protocol tables, and then re-advertise
`the IP address of the virtual server executing on the interme
`diary device and the one or more connection metrics with the
`predetermined values.
`In one aspect, the present disclosure is directed to a method
`of using routes for traffic distribution across a cluster of
`intermediary devices. The method includes advertising, by
`each intermediary device of a cluster comprising a plurality of
`intermediary devices, via a routing protocol to a router, a
`corresponding internet protocol (IP) address of a virtual
`server executing on each intermediary device and at least one
`routing metric with a predetermined value. The method fur
`ther includes determining, by each intermediary device that is
`active, that a first intermediary device of the cluster is one of
`down or not active. The method also includes re-advertising,
`by each intermediary device that is active and responsive to
`
`50
`
`55
`
`60
`
`65
`
`Page 22 of 52
`
`

`

`US 8,972,602 B2
`
`3
`another embodiment of the system, each intermediary device
`that is active determines that the first intermediary device is
`one of down or is no longer able to communicate with the
`router. In yet another embodiment of the system, each inter
`mediary device that is active identifies the first intermediary
`device as one of down or not active responsive to detecting
`that a health metric of the first intermediary device is below a
`predetermined threshold.
`In some embodiments of the system, each intermediary
`device that is active determines that the first intermediary
`device is not part of an active node set for the cluster. In other
`embodiments of the system, each intermediary device that is
`active compares a current active node set with a previous
`active node set and, responsive to the comparison, determines
`that the current active node set is a subset of the previous
`active node set. In still other embodiments of the system, each
`intermediary device re-advertises the corresponding IP
`address of the virtual server with the at least one routing
`metric value reduced by a predetermined amount of one.
`In one embodiment of the system, the router installs a new
`set of routes responsive to the re-advertising with the at least
`one routing metric value reduced by a predetermined amount.
`In another embodiment of the system, each intermediary
`device that is active waits a predetermined time period after
`re-advertising with the at least one routing metric with the
`predetermined value reduced by a predetermined amount
`before re-advertising with the at least one routing metric with
`the predetermined value. In still another embodiment of the
`system, the predetermined time period comprises a time
`period that is configurable.
`The details of various embodiments of the invention are set
`forth in the accompanying drawings and the description
`below.
`
`BRIEF DESCRIPTION OF THE FIGURES
`
`The foregoing and other objects, aspects, features, and
`advantages of the invention will become more apparent and
`better understood by referring to the following description
`taken in conjunction with the accompanying drawings, in
`which:
`FIG. 1A is a block diagram of an embodiment of a network
`environment for a client to access a server via an appliance;
`FIG. 1B is a block diagram of an embodiment of an envi
`ronment for delivering a computing environment from a
`server to a client via an appliance;
`FIG. 1C is a block diagram of another embodiment of an
`environment for delivering a computing environment from a
`server to a client via an appliance;
`FIG. 1D is a block diagram of another embodiment of an
`environment for delivering a computing environment from a
`server to a client via an appliance;
`FIGS. 1 E-1H are block diagrams of embodiments of a
`computing device;
`FIG. 2A is a block diagram of an embodiment of an appli
`ance for processing communications between a client and a
`server;
`FIG. 2B is a block diagram of another embodiment of an
`appliance for optimizing, accelerating, load-balancing and
`routing communications between a client and a server,
`FIG. 3 is a block diagram of an embodiment of a client for
`communicating with a server via the appliance;
`FIG. 4A is a block diagram of an embodiment of a virtu
`alization environment;
`FIG. 4B is a block diagram of another embodiment of a
`virtualization environment;
`
`4
`FIG. 4C is a block diagram of an embodiment of a virtu
`alized appliance;
`FIG.5A are block diagrams of embodiments of approaches
`to implementing parallelism in a multi-core system;
`FIG. 5B is a block diagram of an embodiment of a system
`utilizing a multi-core system;
`FIG. 5C is a block diagram of another embodiment of an
`aspect of a multi-core system;
`FIG. 6 is a block diagram of an embodiment of a cluster
`system;
`FIG. 7A is a block diagram of an embodiment of a system
`for using multi-path routing for traffic distribution in a cluster
`environment;
`FIG. 7B is a block diagram of an embodiment of the system
`for using multipath routing for traffic distribution in a cluster
`environment illustrating the loss of a cluster node; and
`FIG.7C is a flow chart of an embodiment of a method for
`using multi-path routing for traffic distribution in a cluster
`environment.
`The features and advantages of the present invention will
`become more apparent from the detailed description set forth
`below when taken in conjunction with the drawings, in which
`like reference characters identify corresponding elements
`throughout. In the drawings, like reference numbers gener
`ally indicate identical, functionally similar, and/or structur
`ally similar elements.
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`For purposes of reading the description of the various
`embodiments below, the following descriptions of the sec
`tions of the specification and their respective contents may be
`helpful:
`Section A describes a network environment and computing
`environment which may be useful for practicing
`embodiments described herein;
`Section B describes embodiments of systems and methods
`for delivering a computing environment to a remote
`user,
`Section C describes embodiments of systems and methods
`for accelerating communications between a client and a
`server;
`Section D describes embodiments of systems and methods
`for virtualizing an application delivery controller;
`Section E describes embodiments of systems and methods
`for providing a multi-core architecture and environment;
`Section F describes embodiments of systems and methods
`for providing a clustered appliance architecture environ
`ment; and
`Section G describes embodiments of systems and methods
`for using multi-path routing in a cluster environment.
`A. Network and Computing Environment
`Prior to discussing the specifics of embodiments of the
`systems and methods of an appliance and/or client, it may be
`helpful to discuss the network and computing environments
`in which such embodiments may be deployed. Referring now
`to FIG. 1A, an embodiment of a network environment is
`depicted. In brief overview, the network environment com
`prises one or more clients 102a-102n (also generally referred
`to as local machine(s) 102, or client(s) 102) in communica
`tion with one or more servers 106a-106n (also generally
`referred to as server(s) 106, or remote machine(s) 106) via
`one or more networks 104, 104 (generally referred to as
`network 104). In some embodiments, a client 102 communi
`cates with a server 106 via an appliance 200.
`Although FIG. 1A shows a network 104 and a network 104
`between the clients 102 and the servers 106, the clients 102
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Page 23 of 52
`
`

`

`5
`and the servers 106 may be on the same network 104. The
`networks 104 and 104' can be the same type of network or
`different types of networks. The network 104 and/or the net
`work 104" can be a local-area network (LAN), such as a
`company Intranet, a metropolitan area network (MAN), or a
`wide area network (WAN), such as the Internet or the World
`WideWeb. In one embodiment, network 104" may be a private
`network and network 104 may be a public network. In some
`embodiments, network 104 may be a private network and
`network 104" a public network. In another embodiment, net
`works 104 and 104" may both be private networks. In some
`embodiments, clients 102 may be located at a branch office of
`a corporate enterprise communicating via a WAN connection
`over the network 104 to the servers 106 located at a corporate
`data center.
`The network 104 and/or 104" be any type and/or form of
`network and may include any of the following: a point to point
`network, a broadcast network, a wide area network, a local
`area network, a telecommunications network, a data commu
`nication network, a computer network, an ATM (Asynchro
`nous Transfer Mode) network, a SONET (Synchronous Opti
`cal Network) network, a SDH (Synchronous Digital
`Hierarchy) network, a wireless network and a wireline net
`work. In some embodiments, the network 104 may comprise
`a wireless link, such as an infrared channel or satellite band.
`The topology of the network 104 and/or 104 may be a bus,
`star, or ring network topology. The network 104 and/or 104
`and network topology may be of any such network or network
`topology as known to those ordinarily skilled in the art
`capable of Supporting the operations described herein.
`As shown in FIG. 1A, the appliance 200, which also may be
`referred to as an interface unit 200 or gateway 200, is shown
`between the networks 104 and 104". In some embodiments,
`the appliance 200 may be located on network 104. For
`example, a branch office of a corporate enterprise may deploy
`an appliance 200 at the branch office. In other embodiments,
`the appliance 200 may be located on network 104". For
`example, an appliance 200 may be located at a corporate data
`center. In yet another embodiment, a plurality of appliances
`200 may be deployed on network 104. In some embodiments,
`e.g., as shown in FIGS. 1A-1D, a plurality of appliances 200
`may be deployed on network 104". In one embodiment, a first
`appliance 200 communicates with a second appliance 200'. In
`other embodiments, the appliance 200 could be a part of any
`client 102 or server 106 on the same or different network
`104,104" as the client 102. One or more appliances 200 may
`be located at any point in the network or network communi
`cations path between a client 102 and a server 106.
`In Some embodiments, the appliance 200 comprises any of
`the network devices manufactured by Citrix Systems, Inc. of
`Ft. Lauderdale Fla., referred to as Citrix NetScaler devices. In
`other embodiments, the appliance 200 includes any of the
`product embodiments referred to as WebAccelerator and
`Big IP manufactured by F5Networks, Inc. of Seattle, Wash. In
`another embodiment, the appliance 205 includes any of the
`DX acceleration device platforms and/or the SSL VPN series
`of devices, such as SA 700, SA 2000, SA 4000, and SA 6000
`devices manufactured by Juniper Networks, Inc. of Sunny
`vale, Calif. In yet another embodiment, the appliance 200
`includes any application acceleration and/or security related
`appliances and/or Software manufactured by Cisco Systems,
`Inc. of San Jose, Calif., such as the Cisco ACE Application
`Control Engine Module service software and network mod
`ules, and Cisco AVS Series Application Velocity System.
`In one embodiment, the system may include multiple, logi
`cally-grouped servers 106. In these embodiments, the logical
`group of servers may be referred to as a server farm 38. In
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 8,972,602 B2
`
`5
`
`10
`
`15
`
`6
`some of these embodiments, the serves 106 may be geo
`graphically dispersed. In some cases, a farm 38 may be
`administered as a single entity. In other embodiments, the
`server farm 38 comprises a plurality of server farms 38. In one
`embodiment, the server farm executes one or more applica
`tions on behalf of one or more clients 102.
`The servers 106 within each farm 38 can be heterogeneous.
`One or more of the servers 106 can operate according to one
`type of operating system platform (e.g., WINDOWS NT,
`manufactured by Microsoft Corp. of Redmond, Wash.), while
`one or more of the other servers 106 can operate on according
`to another type of operating system platform (e.g., Unix or
`Linux). The servers 106 of each farm 38 do not need to be
`physically proximate to another server 106 in the same farm
`38. Thus, the group of servers 106 logically grouped as a farm
`38 may be interconnected using a wide-area network (WAN)
`connection or medium-area network (MAN) connection. For
`example, a farm 38 may include servers 106 physically
`located in different continents or different regions of a conti
`nent, country, state, city, campus, or room. Data transmission
`speeds between servers 106 in the farm 38 can be increased if
`the servers 106 are connected using a local-area network
`(LAN) connection or some form of direct connection.
`Servers 106 may be referred to as a file server, application
`server, web server, proxy server, or gateway server. In some
`embodiments, a server 106 may have the capacity to function
`as either an application server or as a master application
`server. In one embodiment, a server 106 may include an
`Active Directory. The clients 102 may also be referred to as
`client nodes or endpoints. In some embodiments, a client 102
`has the capacity to function as both a client node seeking
`access to applications on a server and as an application server
`providing access to hosted applications for other clients
`102a-102n.
`In some embodiments, a client 102 communicates with a
`server 106. In one embodiment, the client 102 communicates
`directly with one of the servers 106 in a farm 38. In another
`embodiment, the client 102 executes a program neighbor
`hood application to communicate with a server 106 in a farm
`38. In still another embodiment, the server 106 provides the
`functionality of a master node. In some embodiments, the
`client 102 communicates with the server 106 in the farm 38
`through a network 104. Over the network 104, the client 102
`can, for example, request execution of various applications
`hosted by the servers 106a-106n in the farm 38 and receive
`output of the results of the application execution for display.
`In some embodiments, only the master node provides the
`functionality required to identify and provide address infor
`mation associated with a server 106" hosting a requested
`application.
`In one embodiment, the server 106 provides functionality
`of a web server. In another embodiment, the server 106a
`receives requests from the client 102, forwards the requests to
`a second server 106b and responds to the request by the client
`102 with a response to the request from the server 106b. In
`still another embodiment, the server 106 acquires an enu
`meration of applications available to the client 102 and
`address information associated with a server 106 hosting an
`application identified by the enumeration of applications. In
`yet another embodiment, the server 106 presents the response
`to the request to the client 102 using a web interface. In one
`embodiment, the client 102 communicates directly with the
`server 106 to access the identified application. In another
`embodiment, the client 102 receives application output data,
`Such as display data, generated by an execution of the iden
`tified application on the server 106.
`
`Page 24 of 52
`
`

`

`15
`
`7
`Referring now to FIG. 1B, an embodiment of a network
`environment deploying multiple appliances 200 is depicted.
`A first appliance 200 may be deployed on a first network 104
`and a second appliance 200' on a second network 104". For
`example a corporate enterprise may deploy a first appliance
`200 at a branch office and a second appliance 200' at a data
`center. In another embodiment, the first appliance 200 and
`second appliance 200' are deployed on the same network 104
`or network 104. For example, a first appliance 200 may be
`deployed for a first server farm 38, and a second appliance
`200 may be deployed for a second server farm 38'. In another
`example, a first appliance 200 may be deployed at a first
`branch office while the second appliance 200' is deployed at a
`second branch office'. In some embodiments, the first appli
`ance 200 and second appliance 200' work in cooperation or in
`conjunction with each other to accelerate network traffic or
`the delivery of application and data between a client and a
`SeVer
`Referring now to FIG. 1C, another embodiment of a net
`work environment deploying the appliance 200 with one or
`more other types of appliances, such as between one or more
`WAN optim