| HAI LALA AT MATA ANI MAMA
`
`
`
`
`
`US009830182B2
`
`
`
`I IIIII IIIIIIII 1111111111111!IsIJIII811,11!!11 11111 111111 IIII
`
`( 12 ) United States Patent
`(12) United States Patent
`Ackley
`Ackley
`
`( 10 ) Patent No . :
`(10) Patent No.:
`( 45 ) Date of Patent :
`(45) Date of Patent:
`
`US 9 , 830 , 182 B2
`US 9,830,182 B2
`* Nov . 28 , 2017
`*Nov. 28, 2017
`
`( * ) Notice :
`(* ) Notice:
`
`( 54 ) TRANSLATING MEDIA ACCESS CONTROL
`(54) TRANSLATING MEDIA ACCESS CONTROL
`( MAC ) ADDRESSES IN A NETWORK
`(MAC) ADDRESSES IN A NETWORK
`HIERARCHY
`HIERARCHY
`( 71 ) Applicant : Rackspace US , Inc . , San Antonio , TX
`(71) Applicant: Rackspace US, Inc., San Antonio, TX
`( US )
`(US)
`( 72 ) Inventor : Jason Ackley , Cibolo , TX ( US )
`(72)
`Inventor: Jason Ackley, Cibolo, TX (US)
`( 73 ) Assignee : Rackspace US , Inc . , San Antonio , TX
`(73) Assignee: Rackspace US, Inc., San Antonio, TX
`( US )
`(US)
`Subject to any disclaimer , the term of this
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`patent is extended or adjusted under 35
`U . S . C . 154 ( b ) by 320 days .
`U.S.C. 154(b) by 320 days.
`This patent is subject to a terminal dis
`This patent is subject to a terminal dis-
`claimer .
`claimer.
`( 21 ) Appl . No . : 14 / 606 , 184
`(21) Appl. No.: 14/606,184
`( 22 ) Filed :
`Jan . 27 , 2015
`(22) Filed:
`Jan. 27, 2015
`( 65 )
`Prior Publication Data
`(65)
`Prior Publication Data
`US 2015 / 0143371 A1 May 21 , 2015
`US 2015/0143371 Al May 21, 2015
`Related U . S . Application Data
`Related U.S. Application Data
`( 63 ) Continuation of application No . 13 / 474 , 958 , filed on
`(63) Continuation of application No. 13/474,958, filed on
`May 18 , 2012 , now Pat . No . 8 , 964 , 735 .
`May 18, 2012, now Pat. No. 8,964,735.
`( 51 ) Int . Cl .
`(51) Int. Cl.
`H04L 12 / 28
`H04L 12/28
`G06F 9 / 455
`G06F 9/455
`H04L 12 / 743
`H04L 12/743
`H04L 29 / 12
`H04L 29/12
`H04L 29 / 06
`H04L 29/06
`H04L 12 / 24
`H04L 12/24
`U . S . CI .
`( 52 )
`(52) U.S. Cl.
`CPC . . . . GO6F 9 / 45558 ( 2013 . 01 ) ; H04L 29 / 06095
`CPC .... G06F 9/45558 (2013.01); H04L 29/06095
`( 2013 . 01 ) ; H04L 41 / 50 ( 2013 . 01 ) ; H04L
`(2013.01); H04L 41/50 (2013.01); H04L
`45 / 7457 ( 2013 . 01 ) ; H04L 61 / 103 ( 2013 . 01 ) ;
`45/7457 (2013.01); H04L 61/103 (2013.01);
`
`( 2006 . 01 )
`(2006.01)
`( 2006 . 01 )
`(2006.01)
`( 2013 . 01 )
`(2013.01)
`( 2006 . 01 )
`(2006.01)
`( 2006 . 01 )
`(2006.01)
`( 2006 . 01 )
`(2006.01)
`
`( 56 )
`(56)
`
`H04L 61 / 2596 ( 2013 . 01 ) ; H04L 61 / 6022
`H04L 61/2596 (2013.01); H04L 61/6022
`( 2013 . 01 ) ; G06F 2009 / 45562 ( 2013 . 01 ) ; G06F
`(2013.01); G06F 2009/45562 (2013.01); G06F
`2009 / 45595 ( 2013 . 01 )
`2009/45595 (2013.01)
`Field of Classification Search
`( 58 )
`(58) Field of Classification Search
`None
`None
`See application file for complete search history .
`See application file for complete search history.
`References Cited
`References Cited
`U . S . PATENT DOCUMENTS
`U.S. PATENT DOCUMENTS
`2003 / 0123387 A1 *
`7 / 2003 Jackson . . . . . . . . . . H04L 12 / 4625
`2003/0123387 Al *
`7/2003 Jackson
` H04L 12/4625
`370 / 230
`370/230
`2007 / 0201490 A1 *
`8 / 2007 Mahamuni . . . . . . . . . HO4L 12 / 4625
`2007/0201490 Al * 8/2007 Mahamuni
` H04L 12/4625
`370/395.54
`370 / 395 . 54
`( Continued )
`(Continued)
`OTHER PUBLICATIONS
`OTHER PUBLICATIONS
`U . S . Appl . No . 13 / 352 , 852 entitled “ Optimizing Allocation of
`U.S. Appl. No. 13/352,852 entitled "Optimizing Allocation of
`On - Demand Resources Using Performance Zones , ” filed Jan . 18 ,
`On-Demand Resources Using Performance Zones," filed Jan. 18,
`2012 , by Erik V . Carlin , et al .
`2012, by Erik V. Carlin, et al.
`( Continued )
`(Continued)
`Primary Examiner — Kwang B Yao
`Primary Examiner — Kwang B Yao
`Assistant Examiner — Hardikkumar Patel
`Assistant Examiner — Hardikkumar Patel
`( 74 ) Attorney , Agent , or Firm — Dykema Gossett
`(74) Attorney, Agent, or Firm — Dykema Gossett
`P . L . L . C .
`P.L.L.C.
`( 57 )
`ABSTRACT
`ABSTRACT
`(57)
`In one embodiment a method includes receiving a packet
`In one embodiment a method includes receiving a packet
`including a destination media access control ( MAC ) address
`including a destination media access control (MAC) address
`field having a MAC address of a hypervisor and a destina
`field having a MAC address of a hypervisor and a destina-
`tion Internet protocol ( IP ) address field having an IP address
`tion Internet protocol (IP) address field having an IP address
`of a virtual machine ( VM ) coupled to the hypervisor . The
`of a virtual machine (VM) coupled to the hypervisor. The
`method further determines a MAC address of the VM using
`method further determines a MAC address of the VM using
`the IP address of the VM and applies the VM MAC address
`the IP address of the VM and applies the VM MAC address
`to the destination MAC address field of the packet to
`to the destination MAC address field of the packet to
`forward the packet to the VM .
`forward the packet to the VM.
`17 Claims , 5 Drawing Sheets
`17 Claims, 5 Drawing Sheets
`
`Core
`Core
`Z11
`
`Agg
`A99
`60
`
`10
`10
`
`Switched
`Switched
`Network
`Ne work
`
`20x
`20x
`
`CAM Table
`CAM Table
`45
`1' 5
`
`Top Of Rack Switch
`Top Of Rack Switch
`40
`41
`
`250
`250
`Server
`Server
`
`HV
`394
`
`25
`25,
`Server
`Server
`
`HV
`HV
`30 .
`2.Qn
`
`VMO
`VM0
`35 .
`154
`
`VMX
`VMX
`35,
`
`VM0
`VMO
`35 .
`2.5.2
`
`VMX
`35.
`35
`
`-
`
`-
`-
`-
`-
`-
`-
`-
`-
`Mapping Table - -
`j
`Mapping Table
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`200
`200
`
`-
`-
`-
`-
`-
`-
`-
`-
`Mapping Table
`Mapping Table
`32
`324
`-
`
`- -
`
`I -
`
`WIZ, Inc. EXHIBIT - 1054
`WIZ, Inc. v. Orca Security LTD.
`
`

`

`US 9 , 830 , 182 B2
`US 9,830,182 B2
`Page 2
`Page 2
`
`( 56 )
`(56)
`
`References Cited
`References Cited
`U . S . PATENT DOCUMENTS
`U.S. PATENT DOCUMENTS
`2010 / 0014526 A1 *
`1 / 2010 Chavan . . . . . . . . . . . . . . . . . H04L 49 / 35
`2010/0014526 Al* 1/2010 Chavan
`H04L 49/35
`370 / 395 . 53
`370/395.53
`1 / 2010 Vahdat et al .
`2010 / 0020806 AL
`1/2010 Vandat et al.
`2010/0020806 Al
`2010 / 0031258 A1 *
`2 / 2010 Takano
`G06F 9 / 4856
`. . . . . . . . . . . . . . . . .
`G06F 9/4856
`2010/0031258 Al* 2/2010 Takano
`718 / 1
`718/1
`2010 / 0257269 A1 * 10 / 2010 Clark . . . . . . . . . . . .
`G06F 9 / 4856
`G06F 9/4856
`2010/0257269 Al* 10/2010 Clark
`709 / 226
`709/226
`2011 / 0238975 A1 *
`9 / 2011 Amemiya . . . . . . . . . . . . . . . HO4W 4 / 20
`H04W 4/20
`2011/0238975 Al *
`9/2011 Amemiya
`713 / 150
`713/150
`2011 / 0286380 A1 * 11 / 2011 Zhu . . . . . . . . . . . . . . . . . . H04W 88 / 02
`H04W 88/02
`2011/0286380 Al * 11/2011 Zhu
`370 / 315
`370/315
`2011 / 0299537 A1 * 12 / 2011 Saraiya
`H04L 61 / 2596
`H04L 61/2596
`2011/0299537 Al * 12/2011 Saraiya
`370 / 392
`370/392
`2012 / 0236761 A1 *
`9 / 2012 Yang . . . . . . . . . . . . . . . . . H04L 61 / 103
`H04L 61/103
`2012/0236761 Al* 9/2012 Yang
`370 / 259
`370/259
`2013 / 0124750 A1 *
`5 / 2013 Anumala . . . . . . . . . . . . HO4L 12 / 4625
`H04L 12/4625
`2013/0124750 Al *
`5/2013 Anumala
`709 / 232
`709/232
`
`OTHER PUBLICATIONS
`OTHER PUBLICATIONS
`U . S . Appl . No . 13 / 036 , 219 entitled “ Automated Hybrid Connec
`U.S. Appl. No. 13/036,219 entitled "Automated Hybrid Connec-
`tions Between Multiple Environments in a Data Center , ” filed Feb .
`tions Between Multiple Environments in a Data Center," filed Feb.
`28 , 2011 , by Christopher Kuehl .
`28, 2011, by Christopher Kuehl.
`* cited by examiner
`* cited by examiner
`
`

`

`U . S . Paten
`
`lualud °S11
`atent
`
`Nov . 28 , 2017
`LJOZ `8Z *A0N1
`
`Sheet 1 of 5
`S Jo 1 WIN
`
`US 9 , 830 , 182 B2
`Zll Z8111£8% Sfl
`
`Mapping Table
`
`-
`
`-
`
`-
`
`-
`
`-
`
`
`Mapping Table
`32n
`
`32n
`
`-
`
`-
`
`-
`
`FIG. 1
`
`FIG . 1
`
`200
`
` \
`
`VMX
`35x
`
`35
`VMX
`
`VMO 350
`
`350
`VM0
`
`-
`
`i
`i
`
`r
`
`
`
`300
`HV
`
`30n
`HV
`
`Server
`
`Server
`
`251
`
`25.
`
`350 3
`VMX 35 %
`VMO
`
`35
`VMX
`
`350
`VM0
`
`Server
`
`Server
`/
`250
`
`250
`
`030
`HV
`
`HV 30 .
`
`Mapping Table
`
`320
`
`! Mapping Table
`
`320
`
`Top Of Rack Switch
`
`
`
`Top Of Rack Switch 40
`
`40
`
`CAM Table
`
`
`
`CAM Table 45
`
`45
`
`/
`20x
`
`20x
`
`10
`
`Switched Network 50
`
`Network
`Switched
`
`50
`
`CAggD
`
`Agg 60
`
`60
`
`Core 70
`
`70
`Core)
`
`

`

`U . S . Patent
`lualud °S11
`
`N ov . 28 , 2017
`LJOZ `8Z *A0N1
`
`Sheet 2 of 5
`S Jo Z WIN
`
`US 9 , 830 , 182 B2
`Zll Z8111£8% Sfl
`
`|
`
`FIG. 2
`
`FIG . 2
`
`Provisioning Manifests 120
`
`Manifests
`
`Provisioning
`
`120
`
`Provisioning Manager 110
`
`Manager
`
`Provisioning
`
`110
`
`VMs
`
`To/From
`
`140
`Logic
`
`140
`
`Packet Mapping
`
`To / From VMs
`
`Packet Mapping Logic
`To / From TOR Switch
`
`MAC Cache
`
`
`
`MAC Cache 150
`
`150
`
`A
`Switch
`TOR
`
`To/From _
`
`100
`
`100
`
`

`

`U . S . Patent
`U.S. Patent
`
`Nov . 28 , 2017
`Nov. 28, 2017
`
`Sheet 3 of 5
`Sheet 3 of 5
`
`US 9 , 830 , 182 B2
`US 9,830,182 B2
`
`Receive Request To
`Receive Request To
`Create Virtual Machine
`Create Virtual Machine
`
`Create State For Virtual Machine
`Create State For Virtual Machine
`And Instantiate Virtual Machine
`And Instantiate Virtual Machine
`On Physical Server
`On Physical Server
`
`210
`210
`
`N
`220
`220
`
`Create Provisioning Manifest For VM
`Create Provisioning Manifest For VM
`
`
`
`k 2
`
`230
`30
`
`200
`200
`
`Mapping
`Mapping
`Table
`Table
`Enabled ?
`Enabled?
`
`240
`240
`
`End
`(Endp
`
`Store Association Of An Identifier For
`Store Association Of An Identifier For
`The VM With A Virtual MAC Address
`The VM With A Virtual MAC Address
`In The MAC Translation Table
`In The MAC Translation Table
`
`250
`250
`
`FIG . 3
`FIG. 3
`
`

`

`U . S . Patent
`U.S. Patent
`
`Nov . 28 , 2017
`Nov. 28, 2017
`
`Sheet 4 of 5
`Sheet 4 of 5
`
`US 9 , 830 , 182 B2
`US 9,830,182 B2
`
`300
`300
`
`Forward Packet
`Forward Packet
`TO VM
`To VM
`
`340
`340
`
`Receive Packet From
`Receive Packet From
`Virtual Machine In Hypervisor
`Virtual Machine In Hypervisor
`
`305
` 305
`
`Destination
`Destination
`Local VM ?
`Local VM?
`
`NA MAC
`MAC
`Translation
`Translation
`Enabled ?
`Enabled?
`
`310
`310
`
`315
`315
`
`Replace VM MAC Address With
`Replace VM MAC Address With
`Hypervisor MAC Address
`Hypervisor MAC Address
`
`•
`Forward Packet To Switch Device
`Forward Packet To Switch Device
`
`320
` 320
`
`330
`330
`
`FIG . 4
`FIG. 4
`
`

`

`U . S . Patent
`U.S. Patent
`
`Nov . 28 , 2017
`Nov. 28, 2017
`
`Sheet 5 of 5
`Sheet 5 of 5
`
`US 9 , 830 , 182 B2
`US 9,830,182 B2
`
`350
`
`350 *
`
`Drop
`Drop
`Packet
`Packet
`
`395
`395
`
`nominere
`
`Receive Packet In Hypervisor
`Receive Packet In Hypervisor
`From Switch Device
`From Switch Device
`
`Broadcast
`Broadcast
`Packet Not Directed
`Packet Not Directed
`To HV or VMs of
`To HV or VMs of
`HV ?
`HV?
`
`355
`355
`
`360
`360
`
`MAC
`MAC
`Translation
`I
`Translation
`Enabled ?
`Enabled?
`
`IN
`
`365
`
`365
`
`Access Metadata To Determine
`Access Metadata To Determine
`Destination Virtual Machine
`Destination Virtual Machine
`
`Replace Hypervisor MAC Address
`Replace Hypervisor MAC Address
`With VM MAC Address
`With VM MAC Address
`
`Forward Packet To
`Forward Packet To
`Destination Virtual Machine
`Destination Virtual Machine
`
`370
`k70
`
`380
`k380
`
`390
`390
`
`FIG . 5
`FIG. 5
`
`

`

`US 9 , 830 , 182 B2
`US 9,830,182 B2
`
`15
`
`1
`2
`FIG . 2 is a block diagram of components of a hypervisor
`TRANSLATING MEDIA ACCESS CONTROL
`TRANSLATING MEDIA ACCESS CONTROL
`FIG. 2 is a block diagram of components of a hypervisor
`( MAC ) ADDRESSES IN A NETWORK
`in accordance with an embodiment of the present invention .
`(MAC) ADDRESSES IN A NETWORK
`in accordance with an embodiment of the present invention.
`FIG . 3 is a flow diagram of a method for instantiating a
`HIERARCHY
`HIERARCHY
`FIG. 3 is a flow diagram of a method for instantiating a
`virtual machine in accordance with an embodiment of the
`virtual machine in accordance with an embodiment of the
`This application is a continuation of U . S . patent applica - 5 present invention .
`5 present invention.
`This application is a continuation of U.S. patent applica-
`FIG . 4 is a method for handling virtual machine egress
`tion Ser . No . 13 / 474 , 958 , filed May 18 , 2012 , the content of
`FIG. 4 is a method for handling virtual machine egress
`tion Ser. No. 13/474,958, filed May 18, 2012, the content of
`traffic in accordance with one embodiment of the present
`which is hereby incorporated by reference .
`traffic in accordance with one embodiment of the present
`which is hereby incorporated by reference.
`invention .
`invention.
`FIG . 5 is a flow diagram of a method for handling virtual
`BACKGROUND
`FIG. 5 is a flow diagram of a method for handling virtual
`BACKGROUND
`10 machine ingress traffic in accordance with an embodiment of
`10 machine ingress traffic in accordance with an embodiment of
`In typical data center configurations , each virtual machine
`the present invention .
`the present invention.
`In typical data center configurations, each virtual machine
`( VM ) includes a media access control ( MAC ) address . In
`(VM) includes a media access control (MAC) address. In
`DETAILED DESCRIPTION
`DETAILED DESCRIPTION
`turn , these multiple virtual machines couple to and are
`turn, these multiple virtual machines couple to and are
`controlled by a hypervisor . Thus these VMs below a hyper 15
`controlled by a hypervisor. Thus these VMs below a hyper-
`In various embodiments , a MAC address translation tech
`In various embodiments, a MAC address translation tech-
`visor all require a unique MAC address . Within a given
`visor all require a unique MAC address. Within a given
`nique can be implemented at a level below a TOR switch to
`nique can be implemented at a level below a TOR switch to
`cabinet , there can be many hypervisor instances that couple
`cabinet, there can be many hypervisor instances that couple
`reduce the number of unique MACs communicated to the
`reduce the number of unique MACs communicated to the
`into a top of rack ( TOR ) switch . This switch includes content
`into a top of rack (TOR) switch. This switch includes content
`TOR switch , thus enabling a greater number of resources ,
`TOR switch, thus enabling a greater number of resources,
`both physical and virtual , that can be coupled below the
`addressable memory ( CAM ) tables that store an association
`addressable memory (CAM) tables that store an association
`both physical and virtual, that can be coupled below the
`of MAC addresses to physical locations , both for the VMs 20 TOR switch . Note that in different embodiments , the mask
`of MAC addresses to physical locations, both for the VMs
`20 TOR switch. Note that in different embodiments, the mask-
`below it , as well as the MACs for all other VMs within a
`ing of MAC addresses can occur at different locations within
`below it, as well as the MACs for all other VMs within a
`ing of MAC addresses can occur at different locations within
`switching domain / huddle , which is a collection of cabinets
`a network hierarchy depending on a desired implementation .
`switching domain/huddle, which is a collection of cabinets
`a network hierarchy depending on a desired implementation.
`or other aggregation of a set of servers or other computing
`In one embodiment , a single MAC address associated
`or other aggregation of a set of servers or other computing
`In one embodiment, a single MAC address associated
`with a hypervisor can be communicated from the hypervisor
`resources of the data center .
`resources of the data center.
`with a hypervisor can be communicated from the hypervisor
`Due to the limited size of this CAM table , a constraint is 25 up to a TOR switch . This MAC address can be used as an
`Due to the limited size of this CAM table, a constraint is
`25 up to a TOR switch. This MAC address can be used as an
`identifier for all the VMs coupled below the hypervisor . In
`placed on the number of VMs that can be supported by a
`identifier for all the VMs coupled below the hypervisor. In
`placed on the number of VMs that can be supported by a
`switch in a network topology .
`turn , each hypervisor may include a mechanism such as a
`turn, each hypervisor may include a mechanism such as a
`switch in a network topology.
`translation table to provide a mapping between this single
`translation table to provide a mapping between this single
`SUMMARY OF THE INVENTION
`MAC address exposed to the upper layers of the network
`MAC address exposed to the upper layers of the network
`SUMMARY OF THE INVENTION
`30 architecture and all of the VMs associated with the hyper
`30 architecture and all of the VMs associated with the hyper-
`In one aspect , the present invention includes a method for
`visor . As will be described further below , the hypervisor can
`visor. As will be described further below, the hypervisor can
`In one aspect, the present invention includes a method for
`leverage information already available to it in order to
`receiving , in a hypervisor , a packet including a destination
`leverage information already available to it in order to
`receiving, in a hypervisor, a packet including a destination
`media access control ( MAC ) address field having a MAC perform the MAC address translation ( actually , replacement
`perform the MAC address translation (actually, replacement
`media access control (MAC) address field having a MAC
`address of the hypervisor and a destination Internet protocol
`as described further below ) and to direct packets to the
`as described further below) and to direct packets to the
`address of the hypervisor and a destination Internet protocol
`( IP ) address field having an IP address of a virtual machine 35 correct destination , both in upstream and downstream direc
`35 correct destination, both in upstream and downstream direc-
`(IP) address field having an IP address of a virtual machine
`( VM ) coupled to the hypervisor . In turn , the hypervisor can
`tions . In this way , many fewer entries of the TOR switch are
`tions. In this way, many fewer entries of the TOR switch are
`(VM) coupled to the hypervisor. In turn, the hypervisor can
`determine a MAC address of the VM using the IP address of
`consumed , enabling a greater amount of hypervisors to be
`consumed, enabling a greater amount of hypervisors to be
`determine a MAC address of the VM using the IP address of
`the VM and apply the VM MAC address to the destination
`allocated to a single TOR switch . This has the benefit of
`allocated to a single TOR switch. This has the benefit of
`the VM and apply the VM MAC address to the destination
`MAC address field of the packet and forward the packet to
`decoupling the scaling constraint of the network topology
`decoupling the scaling constraint of the network topology
`devy
`MAC address field of the packet and forward the packet to
`the VM from the hypervisor . As an example , application of 40 from the VM count , and ties it to the unit of scale deploy
`40 from the VM count, and ties it to the unit of scale deploy-
`the VM from the hypervisor. As an example, application of
`the VM MAC address may include replacing the hypervisor
`ment , the hypervisor .
`ment, the hypervisor.
`the VM MAC address may include replacing the hypervisor
`Another embodiment can instead provide this exposure of
`MAC address with the VM MAC address .
`Another embodiment can instead provide this exposure of
`MAC address with the VM MAC address.
`The method may additionally provide for receiving a
`a single MAC address at a higher level . For example the
`a single MAC address at a higher level. For example the
`The method may additionally provide for receiving a
`second packet in the hypervisor that includes a source MAC TOR switch itself can expose a single TOR MAC address to
`TOR switch itself can expose a single TOR MAC address to
`second packet in the hypervisor that includes a source MAC
`address field having the MAC address of the VM , replacing 45 higher layers and in turn , the TOR switch provides a
`45 higher layers and in turn, the TOR switch provides a
`address field having the MAC address of the VM, replacing
`the MAC address of the VM in the source MAC address
`mapping or translation table to map this single MAC address
`mapping or translation table to map this single MAC address
`the MAC address of the VM in the source MAC address
`field with the MAC address of the hypervisor , and forward
`to the devices coupled below it ( namely hypervisors and
`to the devices coupled below it (namely hypervisors and
`field with the MAC address of the hypervisor, and forward-
`ing the second packet to a switch coupled to the hypervisor .
`connected VMs ) .
`connected VMs).
`ing the second packet to a switch coupled to the hypervisor.
`Another aspect of the present invention is directed to a
`Embodiments thus provide an insulation layer for the
`Embodiments thus provide an insulation layer for the
`Another aspect of the present invention is directed to a
`system with a server resource having a hypervisor and a 50 network from performance improvements that may take
`so network from performance improvements that may take
`system with a server resource having a hypervisor and a
`multiple VMs managed by the hypervisor . Each of these
`place within the generic server / hypervisor equipment ( ala
`place within the generic server/hypervisor equipment (ala
`multiple VMs managed by the hypervisor. Each of these
`VMs may have a provisioning manifest including an IP
`Moore ' s Law ) . As an example , if a given amount of hyper
`Moore's Law). As an example, if a given amount of hyper-
`VMs may have a provisioning manifest including an IP
`address for the VM and a MAC address for the VM . In turn ,
`visor switch ports are upgraded from one generation of
`visor switch ports are upgraded from one generation of
`address for the VM and a MAC address for the VM. In turn,
`a switch may be coupled to this and other server resources .
`hypervisor to another , there may 2 - 4x the amount of VMs
`hypervisor to another, there may 2-4x the amount of VMs
`a switch may be coupled to this and other server resources.
`The switch may include a content addressable memory 55 may be present due to increased CPU and memory capaci
`55 may be present due to increased CPU and memory capaci-
`The switch may include a content addressable memory
`( CAM ) having entries each to associate a MAC address of
`ties of the new hypervisor platform . By performing MAC
`ties of the new hypervisor platform. By performing MAC
`(CAM) having entries each to associate a MAC address of
`a hypervisor with a physical location . The MAC addresses
`translations as described herein , there is no impact to the
`translations as described herein, there is no impact to the
`a hypervisor with a physical location. The MAC addresses
`of the VMs can be transparent to the switch . In some
`switching layer as this layer does not store the end - VM
`switching layer as this layer does not store the end-VM
`of the VMs can be transparent to the switch. In some
`embodiments , a mapping table may store entries each map -
`MAC addresses , only the MAC addresses associated with
`MAC addresses, only the MAC addresses associated with
`embodiments, a mapping table may store entries each map-
`ping an IP address for one of the VMs with a MAC address 60 the hypervisors that perform the MAC translation . The
`60 the hypervisors that perform the MAC translation. The
`ping an IP address for one of the VMs with a MAC address
`hypervisor count , as seen by the switching layer ( or any
`for the VM , to enable efficiency of translations .
`hypervisor count, as seen by the switching layer (or any
`for the VM, to enable efficiency of translations.
`other layer higher than the hypervisor that performs the
`other layer higher than the hypervisor that performs the
`BRIEF DESCRIPTION OF THE DRAWINGS
`translations ) may thus remain static during upgrades of the
`translations) may thus remain static during upgrades of the
`BRIEF DESCRIPTION OF THE DRAWINGS
`hypervisor hardware .
`hypervisor hardware.
`FIG . 1 is a high - level block diagram of a network archi - 65
`FIG . 1 is a high - level block diagram of a network archi
`FIG. 1 is a high-level block diagram of a network archi-
`FIG. 1 is a high-level block diagram of a network archi-
`65
`tecture in accordance with an embodiment of the present
`tecture in accordance with an embodiment of the present
`tecture in accordance with an embodiment of the present
`tecture in accordance with an embodiment of the present
`invention . As shown in FIG . 1 , network 10 may be a portion
`invention .
`invention. As shown in FIG. 1, network 10 may be a portion
`invention.
`
`

`

`US 9 , 830 , 182 B2
`US 9,830,182 B2
`
`4
`3
`ized . Specifically by exposing only hypervisor MAC
`of a data center or other networked computer environment .
`ized. Specifically by exposing only hypervisor MAC
`of a data center or other networked computer environment.
`For purposes of discussion herein , assume that the data
`addresses to the TOR switch , the needed storage within
`addresses to the TOR switch, the needed storage within
`For purposes of discussion herein, assume that the data
`center is a multi - tenant data center that provides both
`CAM table 45 can be reduced , rather than also exposing
`CAM table 45 can be reduced, rather than also exposing
`center is a multi-tenant data center that provides both
`virtual MAC addresses for each VM below each of the
`dedicated and cloud - based resources for a number of dif -
`virtual MAC addresses for each VM below each of the
`dedicated and cloud-based resources for a number of dif-
`ferent customers . However , understand that the scope of the 5 hypervisors .
`5 hypervisors.
`ferent customers. However, understand that the scope of the
`present invention is not limited in this regard and embodi -
`For example , assume that each hypervisor can allocate 40
`present invention is not limited in this regard and embodi-
`For example, assume that each hypervisor can allocate 40
`ments may be applicable to single - tenant data centers , along
`VMs below it , and assume 20 hypervisors within a cabinet .
`ments may be applicable to single-tenant data centers, along
`VMs below it, and assume 20 hypervisors within a cabinet.
`In this instance , 800 of the 8K entries in the CAM table may
`with various other network architectures .
`with various other network architectures.
`In this instance, 800 of the 8K entries in the CAM table may
`In the embodiment illustrated , individual computing
`be completely consumed by these hypervisors of one cabi
`In the embodiment illustrated, individual computing
`be completely consumed by these hypervisors of one cabi-
`resources may be provided within one or more cabinets 10 net . Instead , using transparent MAC addresses for the hyper
`resources may be provided within one or more cabinets
`10 net. Instead, using transparent MAC addresses for the hyper-
`20 . - x ( generally cabinet 20 ) . In one example configuration ,
`visors in accordance with an embodiment of the present
`200_x (generally cabinet 20). In one example configuration,
`visors in accordance with an embodiment of the present
`the data center can be arranged into a plurality of so - called
`invention , only 20 CAM entries may be consumed within
`the data center can be arranged into a plurality of so-called
`invention, only 20 CAM entries may be consumed within
`huddles or aggregation of cabinets in which a variety of
`the CAM table of the TOR switch for this cabinet .
`huddles or aggregation of cabinets in which a variety of
`the CAM table of the TOR switch for this cabinet.
`different types of computing resources may be provided . For
`Still referring to FIG . 1 , cabinet 20 may communicate
`different types of computing resources may be provided. For
`Still referring to FIG. 1, cabinet 20 may communicate
`purposes of discussion , assume that cabinet 20 includes a 15 with higher layers of a network hierarchy , including a
`purposes of discussion, assume that cabinet 20 includes a
`15 with higher layers of a network hierarchy, including a
`plurality of rack - mounted servers 25 . - 25 , . For example , a
`switched network 50 that may be coupled in addition to a
`plurality of rack-mounted servers 250-25„. For example, a
`switched network 50 that may be coupled in addition to a
`given cabinet may include between approximately 20 and 80
`number of other cabinets of similar configuration . In gen
`given cabinet may include between approximately 20 and 80
`number of other cabinets of similar configuration. In gen-
`physical servers . As is known , each server can include
`eral , network hierarchy may be configured of various layers ,
`physical servers. As is known, each server can include
`eral, network hierarchy may be configured of various layers,
`various components including one or more processors ,
`including a layer 3 ( L3 ) which may correspond to an IP
`various components including one or more processors,
`including a layer 3 (L3) which may correspond to an IP
`memory , a network interface , mass storage , and so forth . In
`20 layer , a layer 2 ( L2 ) which may correspond to a MAC layer ,
`memory, a network interface, mass storage, and so forth. In
`20 layer, a layer 2 (L2) which may correspond to a MAC layer,
`turn , each server may include one or more hypervisors 30 ,
`and a layer 1 ( L1 ) which corresponds to actual physical
`turn, each server may include one or more hypervisors 30,
`and a layer 1 (L1) which corresponds to actual physical
`each of which may be an orchestration layer to enable
`locations , e . g . , Ethernet locations .
`each of which may be an orchestration layer to enable
`locations, e.g., Ethernet locations.
`instantiation and control of one or more virtual machines
`Routing of packets within the network hierarchy can
`instantiation and control of one or more virtual machines
`Routing of packets within the network hierarchy can
`that operate on the physical server . Thus as shown in FIG .
`occur via L3 information , e . g . , IP addresses , at higher levels
`that operate on the physical server. Thus as shown in FIG.
`occur via L3 information, e.g., IP addresses, at higher levels
`1 , each hypervisor 30 . - 30 , may include a plurality of VMs , 25 of the network hierarchy . At a router level , e . g . , correspond
`1, each hypervisor 300-30, may include a plurality of VMs,
`25 of the network hierarchy. At a router level, e.g., correspond-
`e . g . , VMs 350 - 35x , coupled below them . As examples ,
`ing to switch network 50 , a translation from L3 to L2 may
`e.g., VMs 350-35,, coupled below them. As examples,
`ing to switch network 50, a translation from L3 to L2 may
`between 40 and 160 VMs may be allocated by a single
`occur based on information in an address resolution protocol
`between 40 and 160 VMs may be allocated by a single
`occur based on information in an address resolution protocol
`hypervisor . Note that each VM may be allocated a portion of
`( ARP ) table , which may provide for mappings between L3
`hypervisor. Note that each VM may be allocated a portion of
`(ARP) table, which may provide for mappings between L3
`CPU and memory resources of the given server .
`and L2 information , namely IP addresses to MAC addresses .
`CPU and memory resources of the given server.
`and L2 information, namely IP addresses to MAC addresses.
`Still referring to FIG . 1 , each cabinet 20 can include at 30 In this hierarchy of the network , the TOR switches may
`Still referring to FIG. 1, each cabinet 20 can include at
`30 In this hierarchy of the network, the TOR switches may
`least one top of rack ( TOR ) switch 40 . In some embodi
`operate to provide a translation between L2 information and
`least one top of rack (TOR) switch 40. In some embodi-
`operate to provide a translation between L2 information and
`ments , each cabinet may include up to 3 such switches that
`Li information , namely providing a physical location iden
`ments, each cabinet may include up to 3 such switches that
`L1 information, namely providing a physical location iden-
`are used to enable communication of packets between
`tifier , e . g . , a port number corresponding to a given MAC
`are used to enable communication of packets between
`tifier, e.g., a port number corresponding to a given MAC
`resources within the cabinet and other network components
`address .
`resources within the cabinet and other network components
`address.
`external to the cabinet . As seen in FIG . 1 , a content address - 35
`As further seen in FIG . 1 , in turn , switch 50 may com
`external to the cabinet. As seen in FIG. 1, a content address-
`35 As further seen in FIG. 1, in turn, switch 50 may com-
`able memory ( CAM ) table 45 may be present within TOR
`municate with an aggregation layer 60 , which may be in
`able memory (CAM) table 45 may be present within TOR
`municate with an aggregation layer 60, which