ARTO I AAA Y
`
`US 20030035430A1
`
`as United States
`a2) Patent Application Publication co) Pub. No.: US 2003/0035430 Al
`
` Islam et al. (43) Pub. Date: Feb. 20, 2003
`
`
`(54) PROGRAMMABLE NETWORKDEVICE
`
`Publication Classification
`
`(76)
`
`Inventors: Junaid Islam, San Jose, CA (US);
`Homayoun Valizadeh, Danville, CA
`(US); Jeffrey S. Payne, Seattle, WA
`(US)
`
`Correspondence Address:
`Shailesh Mehra
`Suite 200
`4695 Chabot Drive
`Pleasanton, CA 94588 (US)
`
`(21) Appl. No.:
`
`09/918,363
`
`(22)
`
`Filed:
`
`Jul. 30, 2001
`
`Related U.S. Application Data
`
`(63) Continuation-in-part of application No. 09/679,321,
`filed on Oct. 3, 2000.
`
`(SL) MB. CDacc cccccecsseessssssesessneeessseees HO4L 12/28
`(52) U.S. Ch.ects 370/401; 709/223
`
`(57)
`
`ABSTRACT
`
`A programmable network device is described. The program-
`mable network device executes software modules resident
`on its hardware to support assorted applications and network
`managementservices. These modules may be dynamically
`loaded, unloaded, or modified without interrupting network
`traffic routed through the device. The loading and unloading
`of modules can be administered remotely, via a network
`backbone, service provider network, LAN,or other inter-
`network coupled to the device. Alternatively, administrators
`may alter the operating parameters of individual manage-
`ment modules via the networkto effect performancegains or
`modify existing operating parameters.
`
`104
`
`106
`
`108
`
`108
`
`Google Exhibit 1090
`Google v. VirtaMove
`
`Google Exhibit 1090
`Google v. VirtaMove
`
`

`

`Patent Application Publication
`
`US 2003/0035430 Al
`
`106
`
`es
`o—
`
`Feb. 20,2003 Sheet 1 of 8
`
`Figure1
`
`

`

`Patent Application Publication
`
`Feb. 20, 2003 Sheet 2 of 8
`
`US 2003/0035430 Al
`
`
`
`v02
`
`
`
`00¢
`
`eevee
`
`

`

`US 2003/0035430 Al
`
`Feb. 20, 2003 Sheet 3 of 8
`
`Patent Application Publication
`
`

`

`Patent Application Publication
`
`Feb. 20, 2003 Sheet 4 of 8
`
`US 2003/0035430 Al
`
`
`
`
`
`
`
`
`
`
`

`

`Patent Application Publication
`
`Feb. 20,2003 Sheet 5 of 8
`
`US 2003/0035430 Al
`
`

`

`Patent Application Publication
`
`Feb. 20,2003 Sheet 6 of 8
`
`US 2003/0035430 Al
`
`610
`
`~---+--+ ponon--4qe------
`
`G24
`6220!
`'
`614
`~~~ q------ -b ------4-------
`608
`i
`612
`
`—-~—~——te
`
`606
`
`~oee pecw ee ee nee ne ee
`j
`i
`j
`!
`~o-eb tee
`
`602
`
`a gece cece ee ee ene eee
`
`604
`———— teeLe
`
`GE620005;0
`
`a eea
`
`ee ee eee eee ee
`
`ee ei rmeea ee ee
`
`600
`
`

`

`Patent Application Publication
`
`Feb. 20,2003 Sheet 7 of 8
`
`US 2003/0035430 Al
`
`700
` TCPAP Stack
`
`Figure 7
`
`

`

`Patent Application Publication
`
`Feb. 20,2003 Sheet 8 of 8
`
`US 2003/0035430 Al
`
`800
`
`802 a
`
`
`
`Figure 8
`
`

`

`US 2003/0035430 Al
`
`Feb. 20, 2003
`
`PROGRAMMABLE NETWORK DEVICE
`
`[0001] This application claimspriority to U.S. application
`Ser. No. 09/679,321, entitled “Programmable Network
`Application Server,” filed Oct. 3, 2000, inventors Junaid
`Islam, Jeffery S. Payne, Homayoun Valizadeh, which is
`hereby incorporated by reference in its entirety
`
`FIELD OF THE INVENTION
`
`[0002] The invention is a networking device. More spe-
`cifically, the invention comprises a programmable network-
`ing device used to perform a variety of networking appli-
`cations while maintaining a specified throughput.
`
`DESCRIPTION OF RELATED ART
`
`[0003] The Inadequacies of Pre-Programmed Network
`Devices
`
`[0004] Existing network environments are characterized
`by a disjunction between programmable components, which
`are generally CPUs in workstations connected to the net-
`work, and pre-programmedunits in the infrastructure of the
`network, such as routers and switches. By design, these
`pre-programmed network devices are closed from the per-
`spective of network users and service providers.
`
`[0005] The rigidity of pre-programmed network devices
`results in inefficiencies in the maintenance of networks and
`inflexibility in the deployment of new services or enhance-
`ment of existing services. For instance, the provisioning of
`new applications at a node in a network typically entails the
`overhead of one or more of the following: 1) developing
`hardware to support the new applications 2) writing new
`software for existing network platforms to support
`the
`desired applications 3) deploying workforce to the network
`node to install hardware and/or software developed to sup-
`port the desired applications 4) interrupting or re-routing
`traffic that would otherwise pass through the device while
`the device is upgraded with the new hardware and/orsoft-
`ware.
`
`[0006] The priorart does include some network devices in
`which parameters may be changed via a network, without
`requiring the network device to be restarted or interrupting
`traffic through the device. One such example is IOS from
`Cisco®. Such systems, however, only allow parameters to
`be adjusted without restarting the device. They do not allow
`for the addition or deletion of software modules without
`
`interruption to network services.
`
`[0007] As a result of this inflexibility, network service
`providers are constrained in the geographical breadth of
`their services by physical resources. As personnel must be
`dispatched to install and administer existing network
`devices, service providers are constrained to offer services
`only where they have sufficient manpower and physical
`resources. Consequently,
`there are currently no network
`service providers with global reach.
`
`[0008] The Coupling of Hardware and Software in Exist-
`ing Network Devices
`
`[0009] The pre-programmed nature of existing network
`devices also results in a tight coupling between hardware
`and software used on the network devices.
`In the vast
`majority of network devices, new application modules may
`not be added dynamically, as such devices typically utilize
`
`a single, monolithic program which executes a finite set of
`services. Though routers have been developed for platforms
`such as Windows NT®, such technologies are too slow for
`widespread use in service provider networks and do not
`allow for the dynamic loading and unloadingof applications
`without interrupting packet forwarding. As such, to provide
`newservices, service providers are often forced to replace
`existing network devices with new devices that
`include
`software for the respective service, a process that may take
`years. The replacement of boxes to support new functions
`has grown particularly problematic, as the amortization
`period of network devices continues to shrink. As such, the
`coupling of hardware and software places an onerousfinan-
`cial constraint on service providers.
`
`[0010] Moreover, the coupling of hardware and software
`on network devices precludes third parties from developing
`applications for the devices. Given existing network tech-
`nology, third parties wishing to develop new applications for
`the devices would have to co-operate with the device
`manufacturers to have their software included in the device
`prior to deployment. Existing network devices make no
`provisions for the inclusion of new modules after deploy-
`ment. As the development of new services accelerates,
`network devices become obsolete before generating an
`adequate return on investment.
`
`[0011]
`Devices
`
`Inability to Place Agents on Existing Network
`
`[0012] The inability to load modules, or agents, on exist-
`ing network devices presents difficulties in the analysis of
`network parameters. Existing network devices do not allow
`agents to be uploaded in order to analyze or act upon
`networktraffic. An example of this inefficiency is evident in
`existing support of Service Level Agreements (SLAs).
`Existing SLA techniques typically utilize SNMP or an
`another architecture which polls network devices periodi-
`cally to read counters. Such data is collected and then
`transported over the network for post-facto analysis, 1e., to
`determine packet discard rate and other relevant parameters.
`This architecture demands substantial overheadto scale to a
`large numberof devices and doesnotoffer traffic analysis in
`true real-time.
`
`[0013] The inadequacies of current network devices
`evince a need for reprogrammable devices that support
`multiple network management functions. Code supporting
`network managementfunctions should be dynamically load-
`able on network devices, thereby alleviating the need install
`new devices at network nodes. Devices should also be
`
`remotely configurable in order to eliminate the costs of
`deploying manpowerto service the devices. Such devices
`should also be scalable to accommodate network expansion,
`and should facilitate load balancing and redundancy.
`
`SUMMARYOF THE INVENTION
`
`[0014] The present invention includes systems and meth-
`ods for supporting a programmable network device. The
`programmable network device is capable of executing soft-
`ware modules resident on its hardware to support assorted
`applications and network managementservices. These mod-
`ules may be dynamically loaded, unloaded, or modificd
`without
`interrupting network traffic routed through the
`device; without interrupting or otherwise affecting other
`modules executing at the time; and without requiring the
`
`

`

`US 2003/0035430 Al
`
`Feb. 20, 2003
`
`device to be restarted or rebooted. Modules may be loaded,
`unloaded, or modified either locally, or remotely via any
`type of network in communication with the device. Alter-
`natively, administrators may alter the operating parameters
`of individual management modules via the networkto effect
`performance gains or modify existing operating parameters.
`
`Insome embodiments,the device mayreside at any
`[0015]
`point within a network or between two or more networks. In
`embodiments of the invention, the programmable network
`device may reside at the edge of a Wide Area Network
`(WAN) and fan out to one or more Local Area Networks
`(LANs). The WAN may be an Autonomous System, a
`Service Provider Network, or other type of internetwork. In
`some such embodiments, the WAN may be administered by
`a Service Provider, while the one or more LANs aresituated
`at customer premises. In other embodiments, the program-
`mable network device maybe located at a customersite and
`connect to a service provider network via the customer’s
`Local Area Network. In some such embodiments, the pro-
`grammable network device may tunnel to the service pro-
`vider network via a Virtual Private Network, or VPN.
`
`[0016] The invention enables administrators to load,
`unload, or alter modules on the programmable network
`devices remotely, via one or more networks in communica-
`tion with the device. These modules may emulate legacy
`systems, provide VPN services such as tunneling protocols,
`support network management functions, or provide new
`types of applications developed by network service provid-
`ers or third party developers. By enabling the remote upload-
`ing of new modules, the invention helps to eliminate the lag
`lime in the provision of new networkservices. Likewise, by
`enabling remote administration of the programmable net-
`work device, the invention pre-empts the necessity of allo-
`cating, personnel to maintain the devices.
`
`[0017] By decoupling hardware and software on program-
`mable network devices, the invention allows hardware and
`software components to be retailed to subscribers separately.
`This feature of the invention also allows third party devel-
`opment of networking applications.
`
`[0018] Embodiments of the invention employ a multi-
`tiered software architecture comprising a forwarding engine,
`an application tier, and a network management
`tier.
`In
`embodiments, the forwardingtier is responsible for forward-
`ing packets between networks coupled to the programmable
`network device. In embodiments,the forwarding engine also
`includes encryption and authentication mechanisms for
`accessing modules in the programmable network device.
`The forwarding engine is also a conduit between modules
`resident on the programmable network device and data
`packets traversing the programmable network device.
`[0019] The application tier contains modules for network-
`ing applications. Such applications may correspond to VPN
`functions, including but not limited to applications such as
`Multiprotocol Label Switching, or MPLS, Layer Two Tun-
`neling Protocol, or L2TP, and IP Sec. This allows the
`programmable network device to emulate any type of VPN.
`The modules mayalso be unrelated to VPNs, and support
`applications such as Traffic Shaping or Multicasting. Mod-
`ules in the application tier may also be encoded to support
`entirely new types of applications.
`[0020] Another tier in the software architecture comprises
`a network management
`layer. Modules in this ticr may
`
`support remote network monitoring and managementpro-
`tocols, such as the Simple Network Management Protocol
`(SNMP) and the Common Management Information Proto-
`col (CMIP). Modules may include support for CORBA
`Object Request Broker or an XML based messaging proto-
`col handler. The network managementtier may also include
`modules facilitating the monitoring and enforcement of
`service level maintenance functions in support of Service
`Level Agreements (SLAs).
`
`the program-
`In cmbodiments of the invention,
`[0021]
`mable network device is implemented by use of a hardware
`configuration which may include one or more of the fol-
`lowing: one or more processors dedicated to the forwarding
`engine, one or more processors dedicated to the applications
`and network management tiers, an data ports which, by way
`of non-limiting example, may be any one or more of an
`Ethernetport, an Asynchronous Transfer Mode (ATM)port,
`a SONET/SDH port. Modules on the programmable net-
`work device are executed on the general execution proces-
`sors. In some embodimentsof the invention, the forwarding
`engine may be encoded in microcode. The separation
`between the processors supporting the forwarding engine
`and the application processors allow packets to be streamed
`through the forwarding engine continuously, irrespective of
`loading, unloading, modification, or failure of one or more
`modules running on the general execulion processors.
`
`the program-
`In embodiments of the invention,
`[0022]
`mable network device may be configured to operate in
`parallel with similar devices. For instance, a cluster of
`programmable network devices may be stacked, in order to
`facilitate distributed processing and redundancy. In embodi-
`ments of the invention, stacked servers may be coupled by
`a local network or via a WAN,such as a service provider
`network or the Internet. In embodiments of the invention,
`the devices may be stacked, or coupled, by daisy chaining;
`in other embodiments, the devices may be coupled via a hub
`configuration. In embodiments of the invention, the modules
`are executed as threads distributed over multiple program-
`mable network devices. These and other aspects and
`embodiments of the invention shall be elaborated herein.
`
`DESCRIPTION OF FIGURES
`
`[0023] FIG. 1 illustrates a location of a programmable
`network device between a Local Area Network and a Wide
`Area Network according to embodiments of the invention.
`
`FIG.2 illustrates a multi-tiered software architec-
`[0024]
`ture of the programmable network device.
`
`FIG.3 illustrates line cards used in embodiments
`[0025]
`of the programmable network device.
`
`FIG.4 illustrates a stacked configuration of mul-
`[0026]
`tiple programmable network devices.
`
`FIG.5 illustrates a model of software organization
`[0027]
`within processors in the programmable network device.
`
`FIG.6 illustrates a packet format for a Multi CPU
`[0028]
`Communication Protocol used internally by embodiments of
`the programmable network device.
`
`FIG.7 illustrates components of the programmable
`[0029]
`network device used to add and delete flows in embodiments
`of the invention.
`
`

`

`US 2003/0035430 Al
`
`Feb. 20, 2003
`
`[0030] FIG. 8 illustrates a method of adding a flow to the
`programmable device according to embodiments of the
`invention.
`
`DETAILED DESCRIPTION
`
`[0031] A. Overview of
`Device
`
`the Programmable Network
`
`[0032] Somc embodiments of the invention include a
`Programmable Network Device, which may be located at
`any point within a network or between networks. In some
`embodiments,
`the device may be located at customer, or
`enterprise premises; in other embodiments, the device may
`be located at an edge of a service provider network. In some
`embodiments, the Programmable Network Device may be
`owned and/or operated by a Service Provider (SP)orcarrier
`connecting the customer, or enterprise,
`to a Wide Area
`Network (WAN). The WAN maybe an AutonomousSystem,
`service provider backbone, or other type of internetwork.
`Alternatively, the device may be owned/and or operated by
`the enterpriseitself.
`
`In embodiments of the invention illustrated sche-
`[0033]
`matically in FIG.1, the Programmable Network Device 102
`may be a self-contained unit which resides behind an access
`router 104 and supports IP services to the enterprise 100. In
`alternative
`embodiments,
`the Programmable Network
`Device maybe instantiated as an access router.
`
`the Program-
`In embodiments of the invention,
`[0034]
`mable Network Device may include two or more physical
`interfaces 106108 for carrying data; in embodiments, these
`interfaces may operate at rates of 1 Gbps or higher. In some
`such embodiments,
`the physical
`interfaces 106108 may
`comprise Gigabit Ethernet interfaces; in other embodiments,
`one or more of the physical interfaces may comprise 10/100
`Ethernet interfaces. One of these interfaces 106 may connect
`to the access router 104, and the other 108 to the enterprise
`network 100. In embodiments of the invention, the device
`102 may include additional
`interfaces for management,
`which may include, but are not limited to a console or
`modem to a serial port, or a 10/100 Ethernet port.
`
`[0035] B. Multi-Tiered Logical Architecture
`
`[0036] FIG. 2 illustrates a logical architecture of the
`Programmable Network Device. Multiple logical
`layers
`200202204210 are depicted. At the lowest level is a hard-
`ware instantiated data-forwarding layer 204. This layer
`provides hardware acceleration for forwarding data speci-
`fied line rates. In embodiments of the invention, the hard-
`ware data forwarding layer 204 supports line rates of a
`gigabit or higher. The hardware layer 204 continues to
`forward data in case of software failures. That is, if one or
`more software modules operating on the programmable
`network device fail, the hardware layer 204 may continue
`forwarding data in order to preserve connectivity between
`networks coupled to the Programmable Network Device.
`
`[0037] Embodiments depicted in FIG. 2 also include a
`core application laycr 202. This layer mayinclude numcrous
`types of applications such as, by way of non-limiting
`example, Virtual Private Network (VPN)applications, Net-
`work Address Translation (NAT), IPSEC applications, firc-
`wall applications, etc. Software modules may be loaded onto
`the programmable network device 102 either prior
`to
`deployment or via the service provider network 100 at any
`
`time in its operation. Software modules may be loaded or
`unloaded from the programmable network device 100 dur-
`ing its operation, without disrupting packet forwarding
`through the programmable network device. It is desirable for
`such applications to be very stable, to recover from failure
`without customerintervention, and to perform in accordance
`with any Service Level Agreements (SLAs) in effect. In
`some embodiments of the invention, core applications may
`be assigned higher priority than other applications in order
`to ensure the applications adequate time and resources to
`achieve defined performance objectives.
`
`layer 200
`[0038] FIG. 2 also includes a management
`comprised of managementapplications. In embodiments of
`the invention, these managementapplications employ Appli-
`cation Programming Interfaces (APIs) exposed by core
`applications 202 and the system infrastructure. By way of
`non-limiting
`example, management
`applications may
`sample the system statistics periodically in order to ensure
`that any SLAs in effect are satisfied. In some embodiments
`of the invention, these managementapplications are granted
`a specified number of CPU cycles. In embodiments, the
`managementapplications employ the open APIs provided by
`the system and the core applications.
`
`[0039] An infrastructure layer 210 includes tools which
`may be used by all applications in the programmable net-
`work device, which mayinclude, but are not limited to, any
`one or more of the following: an operating system for the
`application; APIs to the forwarding engine, hardware offsets
`for security, hardware offsets for compression, hardware for
`packet reassembly;
`
`[0040] C. Hardware Architectures of the Programmable
`Network Device
`
`[0041] A hardware architecture used by embodiments of
`the invention to implementthe logical view of the architec-
`ture is illustrated in FIG. 3. In embodiments of the inven-
`tion, the programmable network device unit includes one or
`more Application Processor Cards,
`(APC’s)
`farm card
`302304, each APC including multiple CPUs 306-320. In
`embodiments, these CPUs 306-320 may be general purpose
`CPUs, such as processors from the Intel Pentium® family,
`the Power PC® series, or those offered by Transmeta® Inc;
`alternative CPUs will be apparent to those skilled in theart.
`Core and management applications are executed on the
`CPUs 306-320 resident on the Application Processor Cards
`302304.
`
`In embodiments of the invention, the Application
`[0042]
`Processor Card may include one or more encryption pro-
`cessors 322324 to perform cneryption services for the CPUs
`306-320. These encryption services mayinclude, but are not
`limited to Diffie-Hellman operations, RSA signatures, RSA
`verifications, etc. In embodiments, each CPU 306-320in the
`Application Processor Cards 302304 has its own encryption
`processor 322324. Examples of commercial encryption pro-
`cessors that maybe utilized include the HiFn 6500 and the
`Broadcom BCM 5820. Alternative security processors will
`be apparent to those skilled in the art.
`
`In embodiments, each of the Application Processor
`[0043]
`Cards 302304 also includes a switch 326328342 allowing
`the processors 306-320 to communicate with a backplane
`330332 of the device. In embodiments, the backplane may
`include two or more unidircctional buses,
`including an
`
`

`

`US 2003/0035430 Al
`
`Feb. 20, 2003
`
`uplink 332 and a downlink 330. The uplink and downlink
`each transmit data at rates of 10 Gbps or higher. In embodi-
`ments,
`the uplink and downlink operate by use of Low
`Voltage Differential Signaling, or LVDS. In embodiments of
`the invention, the switches 326328342 may comprise cus-
`tomized ASICs; in other embodiments, the switches may be
`implemented on FPGAs. Examples of FPGAs that may be
`used for the switch include those produced by Xilinx®,Inc.
`Alternative FPGAs will be apparent to those skilled in the
`art.
`
`In embodiments of the invention, the forwarding
`[0044]
`engine 204 is implemented in a Network Processor Card
`(NPC) 300, also depicted in FIG. 3. The Network Processor
`Card 300 may include one or more network processors to
`perform functions on inbound and outbound packet flows. In
`embodimentsasillustrated in FIG.3, the Network Processor
`Card may have two sets of network processors 334336
`which handle outbound 338 and inbound 340traffic respec-
`tively. In particular, an inbound PHYinterface 340 and an
`outbound PHY interface 338 may both interact with Gigabit
`Ethermet ports. Examples of suitable Network Processors
`334336 include the Intel® IXP Chip, the Agere family of
`Network Processors, and Motorola Inc.’s C-Port network
`processor; other suitable network processors will be appar-
`ent to those skilled in the art. Alternatively, a special purpose
`ASIC may be used to support functions on traffic flows.
`
`[0045] The Network Processor Card 300 mayalso contain
`one or more controller CPUsreferred to as controller CPUs
`326 for controlling and managing the network processors
`334336. The controller CPUs may also be general purpose
`CPUs.
`
`[0046] FIG. 4 illustrates a configuration by which mul-
`tiple programmable network deviccs 406408410 may be
`stacked via the high speed bus 330332. In embodiments, a
`first programmable network device 406 includes a Network
`Processor Card 300 and an Application Processor Card 302
`in a first chassis. In embodiments, the chassis is designed for
`inclusion in a standard carrier rack which is NEPS compli-
`ant. The first programmable network device 406 may be
`coupled via the bus to one or more programmable network
`devices 408410. In embodiments, each of the programmable
`network devices 408410 includes two or more Application
`Processor Cards 304400402. In other embodiments, for
`redundancy purposes, one of the programmable network
`devices may contain a standby Network Processor Card,
`which maybeactivated if the main Network Processor Card
`300 fails.
`
`[0047] FIG. 3 also depicts an internal communications
`bus comprised by internal buses 348344346 in the Processor
`Cards 302304306, the stacking logic between the Processor
`Cards 300302304 and the bus 330332. In embodiments of
`the invention, the local buses 344346348 within the Proces-
`sor Cards 302304306 may be PCI buses; alternative imple-
`mentations of the local buses will be apparent
`to those
`skilled in theart.
`
`processors 334336 may include, by way of non-limiting
`example, calls that set filters, add and removetree elements,
`etc. In embodiments of the invention, such software resides
`on the Controller CPU 326. In such embodiments, the API
`is extendedto applications on other CPUs 306-322 by use of
`a Multi-CPU Communication Protocol, described elsewhere
`in this specification. In embodiments, the API may also be
`used to readstatistics from the Network Processors 334336.
`
`In embodiments of the invention, each of the
`[0050]
`network processors 334336 compriscs a sct of micro-coded
`engines. In embodiments, the micro-code for these proces-
`sors is stored in a localfile system, and is downloaded from
`a remote server. In embodiments,
`the remote server is
`coupled to the programmable network device via an inter-
`network. In some embodiments, the micro-code determines
`which applications are executed on the programmable net-
`work device, as well the sequence in which they are run. The
`micro-code may also provide hooks whereby new applica-
`tions can filter out packets and re-insert them into the data
`stream.
`
`In embodiments of the invention, encryption/de-
`[0051]
`cryption/key generation engines 322324 are attached one or
`more of the application CPU s 306-322. A driver for these
`engines makes these functions available in user and kernel
`space.
`
`a compression/decompression
`In embodiments,
`[0052]
`engine is attached to one or more of the application CPUs
`306-322. In some such embodiments, the driver for these
`engines makes these functions available in user and kernel
`space
`
`[0053] Embodiments of the programmable network device
`include a file system contained in a micro-drive 348 in the
`Network Processor Card 300. In embodiments of the inven-
`
`tion,the file system may based on a Unix/Linuxfile; in other
`embodiments, the file system may be based on a DOS/
`WindowsFile Allocation Table. Alternative file systems will
`be apparent
`to those skilled in the art. In embodiments
`supporting Linux,the file system may include configuration
`files, application and OSbinaries, shared libraries, etc.
`
`In embodimentsof the invention, the file system is
`[0054]
`directly attached to the Controller CPU 326 In embodiments
`of the invention, the Controller CPU 326 exports the file
`system to the application CPUs 306-322, which may mount
`the file system as part of diskless operation.
`
`[0055] D. Software Services Supported within the Pro-
`grammable Network Device
`
`In embodiments of the invention, once the control-
`[0056]
`ler CPU 326 and other CPUs 306-322 are loaded with
`operating systems, a number of manager/server applications
`are slarted. They maybestarted on any CPU 306-322 in the
`system. Non-limiting examples of the standard services may
`include file servers, telnet servers, console I/O, etc. Other
`services may include one or more of the following:
`
`[0048] Hardware Acceleration in the Forwarding Engine
`
`[0057] Name Registry
`
`the programmable network
`In embodiments,
`[0049]
`device may include one or more scts of dedicated processors
`334336 for packet forwarding; these sets may include, by
`way of non-limiting example general purpose CPUs, cus-
`tomizcd ASICs, or nctwork processors. API calls to these
`
`In embodiments of the invention, every application
`[0058]
`program in the programmable network server offcring a
`service registers with the Name Server. The Name Registry
`maintains information which may include the application’s
`name, version, and a local address where it can be reached
`
`

`

`US 2003/0035430 Al
`
`Feb. 20, 2003
`
`by other applications. The Name Registry itself is available
`at a well-known address, and runs on the Controller CPU
`after it boots up.
`
`[0059] Programmable Network Device Manager and
`CPU Manager.
`
`[0060] Embodiments of the invention include a Program-
`mable Network Device Manager (PND Manager) which is
`usedto start all applications other than those that are part of
`the infrastructure. The PND Manager, which mayrun on the
`Controller CPU 326, reads the configuration information,
`and starts applications on various CPUs. In embodiments,
`the PND performsthis function in conjunction with a CPU
`Manager, which has instances running on the other CPUs
`306-322. In some embodiments of the invention, the CPU
`Manager
`runs in every application CPU 306-322.
`In
`embodiments of the invention, the PND Manager balances
`load based on the loading of CPUs as measured by the CPU
`Manager;alternatively, the PND Manager mayselect a fixed
`CPU for an application based onits configuration. When an
`application is started up, the CPU Manager allocates CPU
`resources for a given application, such as, by way of
`non-limiting example, the application’s priority or real-time
`quota. In embodiments of the invention, the CPU manager
`starts up in a CPU as soon as it boots up, and has a
`well-known address.
`
`[0061] Statistics Manager.
`
`In embodimentsof the invention, applications peri-
`[0062]
`odically make their statistics available to a statistics man-
`ager. The statistics manager may run on any CPU inthe
`Programmable Network Device. The Statistics Manager can
`be queried by management applications through an API. In
`embodiments of the invention, the Statistics Manager reg-
`isters with the Name Registry, so applications will be able to
`locate it by querying the Name Registry.
`
`[0063] E. Software Organization within CPUs
`
`In embodiments of the invention, all of the CPUs
`[0064]
`306-322 include identical operating system kernels. The
`software architecture of individual CPUs is illustrated in
`FIG. 5. The CPUs 300-322 in the CPU cards 330-334 run
`
`core 504 and network management 508 applications. Non-
`limiting examples of core applications may includeI'irewall,
`Network Address Translation (NAT), IPSEC/VPN,Layer 2
`Tunneling Protocol (1.2TP), Routing, Quality of Service
`(QoS), Multi Protocol Label Switching (MPLS), IP Multi-
`cast; other examplesof core applications will be apparent to
`those skilled in the art. In embodiments of the invention,
`core applications 504 are allocated sizeable ratios of CPU
`resources for meeting performance goals, while manage-
`ment applications 508 are allocated a smaller, pre-defined
`percentage of a CPU. In some such embodiments,
`this
`pre-defined percentage may be on or about 5% of CPU
`resources. All of the management applications 408 will
`share this allocation. If core applications 504 do not use the
`CPUresources allocated to them, these CPU resources will
`be available for managementapplications 508.
`
`In embodiments of the invention,all of the appli-
`[0065]
`cations are loaded dynamically, and into their own memory
`protected segments. While core applications 504 may have
`driver components loaded into the kernel 500, in embodi-
`ments of the invention, management applications 508 do not
`have driver componcnts
`
`the Controller
`In embodiments of the invention,
`[0066]
`CPU 326 controls the startup ofall of the sub-systemsin the
`programmable network device. In some embodimentsof the
`invention, this CPU 326 includes a flash memoryunit anda
`hard disk micro-drive which store the operating system and
`application binaries for all of the CPUs 300-322, along with
`any configuration information. In embodiments of the inven-
`tion, the Controller CPU 326 also includes a serial port for
`attachment of a console, modem, and/or an Ethernet port—
`such as a a 10/100 Mbit/s Ethernet port—for management.
`The Controller CPU 326 may also support telnet/console
`sessions. In embodiments of the invention, the application
`CPUs 300-322 mounttheir file systems from the Controller
`CPU 326, and will see the same files as any application
`running on the Controller CPU 326.
`
`[0067] Dynamic Loading and Unloading of Drivers and
`Applications
`
`In the environment of the program