(12) United States Patent
`Wang et al.
`
`111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US006292705Bl
`US 6,292, 705 Bl
`Sep.18,2001
`
`(10) Patent No.:
`(45) Date of Patent:
`
`(54) METHOD AND APPARATUS FOR ADDRESS
`TRANSFERS, SYSTEM SERIALIZATION,
`AND CENTRALIZED CACHE AND
`TRANSACTION CONTROL, IN A SYMETRIC
`MULTIPROCESSOR SYSTEM
`
`(75)
`
`Inventors: Yuanlong Wang, Sunnyvale; Zong Yu,
`Cupertino; Xiaofan Wei, Sunnyvale;
`Earl T. Cohen, Fremont; Brian R.
`Baird, Pleasanton; Daniel Fu,
`Sunnyvale, all of CA (US)
`
`(73) Assignee: Conexant Systems, Inc., Newport
`Beach, CA (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.: 09/163,294
`
`(22) Filed:
`
`Sep. 29, 1998
`
`Int. Cl? ..................................................... GOSB 19/18
`(51)
`(52) U.S. Cl. ................................. 700/5; 700/90; 710/100;
`711!146
`(58) Field of Search .......................... 700/5, 90; 709/233;
`710/100, 29, 146, 141, 122; 711!110, 200,
`202, 143, 119
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,315,308
`4,438,494
`
`2/1982 Jackson . ... ... .... ... ... ... ... ... .... ... . 710/33
`3/1984 Budde eta!. ............................ 714/2
`
`(List continued on next page.)
`
`01HER PUBLICATIONS
`
`Technical White Paper, Sun™Enterprise™lOOOO Server,
`Sun Microsystems, Sep. 1998.
`Alan Charlesworth, Starfire: Extending the SMP Envelope,
`IEEE Micro, Jan./Feb. 1998, pp. 39-49.
`
`Joseph Heinrich, Origin™ and Onyx2™ Theory of Opera(cid:173)
`tions Manual, Document No. 007-3439-002, Silicon
`Graphics, Inc., 1997.
`
`White Paper, Sequent's NUMA--Q SMP Architecture,
`Sequent, 1997.
`
`(List continued on next page.)
`
`Primary Examiner-William Grant
`Assistant Examiner--Kidest Bahta
`(74) Attorney, Agent, or Firm-Keith Kind; Kelly H. Hale
`
`(57)
`
`ABSTRACT
`
`A preferred embodiment of a symmetric multiprocessor
`system includes a switched fabric (switch matrix) for data
`transfers that provides multiple concurrent buses that enable
`greatly increased bandwidth between processors and shared
`memory. A Transaction Controller, Transaction Bus, and
`Transaction Status Bus are used for serialization, centralized
`cache control, and highly pipelined address transfers. The
`shared Transaction Controller serializes transaction requests
`from Initiator devices that can include CPU/Cache modules
`and Peripheral Bus modules. The Transaction Bus of an
`illustrative embodiment is implemented using segmented
`buses, distributed muxes, point-to-point wiring, and sup(cid:173)
`ports transaction processing at a rate of one transaction per
`clock cycle. The Transaction Controller monitors the Trans(cid:173)
`action Bus, maintains a set of duplicate cache-tags for all
`CPU/Cache modules, maps addresses to Target devices,
`performs centralized cache control for all CPU /Cache
`modules, filters unnecessary Cache transactions, and routes
`necessary transactions to Target devices over the Transac(cid:173)
`tion Status Bus. The Transaction Status Bus includes both
`busbased and point-to-point control of the target devices. A
`modified rotating priority scheme is used to provide
`Starvation-free support for Locked buses and memory
`resources via backoff operations. Speculative memory
`operations are supported to further enhance performance.
`
`20 Claims, 33 Drawing Sheets
`
`FLOW CONTROL UNIT
`(FCU)
`
`152
`
`200/
`
`151
`
`NETAPP, INC. EXHIBIT 1016
`Page 1 of 46
`
`

`
`US 6,292, 705 Bl
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`4,480,307
`5,313,609
`5,335,335
`5,440,698
`5,511,226
`5,513,335
`5,524,234
`5,535,363
`5,537,569
`5,537,575
`5,553,310
`5,561,779
`5,568,620
`5,574,868
`5,577,204
`5,581,729
`5,588,131
`5,594,886
`5,606,686
`5,634,043
`5,634,068
`5,644,754
`5,655,100
`5,657,472
`5,682,516
`5,684,977
`5,696,910
`5,796,605
`5,829,034
`5,895,495
`5,897,656
`5,940,856
`5,946,709
`6,065,077 *
`
`10/1984
`5/1994
`8/1994
`8/1995
`4/1996
`4/1996
`6/1996
`7/1996
`7/1996
`7/1996
`9/1996
`* 10/1996
`10/1996
`11/1996
`11/1996
`12/1996
`12/1996
`1!1997
`2/1997
`5/1997
`5/1997
`7/1997
`8/1997
`8/1997
`10/1997
`11/1997
`12/1997
`8/1998
`10/1998
`4/1999
`4/1999
`8/1999
`8/1999
`5!2000
`
`........................ 710/100
`Budde et a!.
`Baylor et a!. ........................ 711!121
`Jackson et a!. ...................... 711!121
`Sindhu eta!. ......................... 710/33
`Zilka ...................................... 711!21
`McClure .............................. 711!130
`Martinez, Jr. et a!. .............. 711!121
`Prince .................................. 711!121
`Masubuchi ........................... 709/225
`Foley et a!. .......................... 711!146
`Taylor et a!.
`........................ 711!130
`Jackson et a!. ...................... 711!122
`Sarangdhar et a!. ................. 711!141
`Marisetty ............................... 710/40
`Brewer et a!. ....................... 711!122
`Nishtala et a!.
`..................... 710/105
`Borrill .................................. 711!146
`Smith et a!.
`......................... 711!131
`Tarui et a!.
`.......................... 709/216
`Self eta!. ............................ 711!141
`Nishtala et a!.
`..................... 713/503
`Weber et a!.
`........................ 345/501
`Ebrahim eta!. ..................... 711!144
`Van Loo et a!. ..................... 711!158
`Sarangdhar et a!. ................. 711!146
`Van Loo et a!. ..................... 711!170
`Pawlowski ........................... 430/170
`Hagersten ................................ 700/5
`Hagersten et a!. ................... 711!141
`Arimilli et a!. ...................... 711!156
`Vogt eta!. ........................... 711!141
`Arimilli et a!. ...................... 711!119
`Arimilli et a!. ...................... 711!119
`Fu ........................................ 710/100
`
`01HER PUBLICATIONS
`
`White Paper, Eight-way Multiprocessing, Hewlett-Packard,
`Nov. 1997.
`
`George White & Pete Vogt, Profusion, a Buffered, Cache(cid:173)
`Coherent Crossbar Switch, presented at Hot Interconnects
`Symposium V, Aug. 1997.
`Alan Charlesworth, et al., Gigaplane-----}(B: Extending the
`Ultra Enterprise Family, presented at Hot Interconnects
`Symposium V, Aug. 1997.
`James Loudon & Daniel Lenoski, The SGI Origin: A
`ccNUMA Highly Scalable Server, Silcon Graphics, Inc.,
`presented at the Proc. of the 24th Int'l Symp. Computer
`Architecture, Jun. 1997.
`Mike Galles, Spider: A High-Speed Network Interconnect,
`IEEE Micro, Jan./Feb. 1997, pp. 34-39.
`T.D. Lovett, R.M. Clapp and R.J. Safranek, NUMA---Q: An
`SCI-based Enterprise Server, Sequent, 1996.
`Daniel E. Lenoski & Wolf-Dietrich Weber, Scalable
`Shared-Memory Multiprocessing, Morgan Kaufmann Pub(cid:173)
`lishers, 1995, pp. 143-159.
`David B. Gustavson, The Scalable Coherent Interface and
`Related Standards Projects, (as reprinted in Advanced Mul(cid:173)
`timicroprocessor Bus Architectures, Janusz Zalewski, IEEE
`Computer Society Press, 1995, pp. 195-207.).
`Kevin Normoyle, et al., UltraSPARC™ Port Architecture,
`Sun Microsystems, Inc., presented at Hot Interconnects III,
`Aug. 1995.
`Kai Hwang,Advanced Computer Architecture: Parallelism,
`Scalability, Programmability, McGraw-Hill, 1993, pp.
`355-357.
`Jim Handy, The Cache Memory Book, Academic Press,
`1993, pp. 161-169.
`AngelL. DeCegama, Parallel Processing Architectures and
`VLSI Hardware, vol. 1, Prentice-Hall, 1989, pp. 341-344.
`
`* cited by examiner
`
`NETAPP, INC. EXHIBIT 1016
`Page 2 of 46
`
`

`
`U.S. Patent
`U.S. Patent
`
`Sep.18,2001
`Sep. 18, 2001
`
`Sheet 1 of 33
`Sheet 1 0133
`
`US 6,292, 705 Bl
`US 6,292,705 B1
`
`L
`
`~
`
`0 T"""
`T"""L_
`
`§I
`
`3om?
`
`~I
`
`
`
`9.m:m_.=n_oz_<s_
`
`UJ
`
`:I: u
`<C:z
`2*
`Q'*l=
`::> a_
`u
`
`•
`•
`•
`
`
`
`M10520mIo<o5%
`
`UJ
`
`:I: u
`~~
`3
`::>
`a_
`u
`
`UJ
`oc.9
`1-Cl
`U) 0:::
`::> (()
`C?w
`::> ::>
`a_(() UQ
`
`
`
`
`
`33moemmm=m_-o\_E055.
`
`~I
`
`~I
`
`_..63
`
`bm<moat
`
`»\o9
`
`0
`0 T"""
`
`NETAPP, INC. EXHIBIT 1016
`
`Page 3 of 46
`
`NETAPP, INC. EXHIBIT 1016
`Page 3 of 46
`
`
`

`
`1--"
`~
`(It
`Q
`~
`N
`\0
`'N
`0'1
`rJ'l
`
`e
`
`~
`~
`0 ......,
`N
`~ .....
`'JJ. =(cid:173)~
`
`'"""'
`N c c
`'"""'
`~CIO
`~ '?
`'JJ.
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`153
`
`-....
`
`c1:
`
`~
`
`AGP
`
`1303
`MEMORY
`
`1302
`MEMORY
`
`1301
`MEMORY
`
`1300
`MEMORY
`
`....
`
`.....
`
`~
`
`......
`
`~
`
`)
`115
`
`230
`
`MCU
`
`230
`
`MCU
`
`....
`
`.....
`
`)
`114
`
`FIG. 2
`
`240
`
`BBU
`
`1
`
`151
`
`.....
`..
`
`200_/
`
`..--
`......_
`
`240
`
`BBU
`
`113\.._!
`
`152J
`
`~
`
`AGP
`
`210
`DCIU
`
`-
`
`t
`
`220
`
`FLOW CONTROL UNIT
`
`(FCU)
`
`• • •
`
`~1 l
`
`210
`DCIU
`
`• ~PU6 . ~CPU7
`
`120
`
`-
`
`120
`
`-
`
`•
`
`•
`
`~ ~PU1
`
`120
`
`-
`
`NETAPP, INC. EXHIBIT 1016
`Page 4 of 46
`
`

`
`1--"
`~
`(It
`Q
`~
`N
`\0
`'N
`0'1
`\Jl
`
`e
`
`~
`~
`0 ......,
`~
`
`~ .....
`'JJ. =(cid:173)~
`
`'"""'
`N c c
`'"""'
`~CIO
`'?
`~
`'JJ.
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`1303
`' I
`.-m-< -f-. SDRAM
`oc.;:::o
`::0::;!:! 0
`-I ::0 S:
`g~s: -+-I
`zmm
`~ ~ ~ : I SDRAM I
`CHANNEL ~ : I ~ (") z s: ~I SDRAM I
`
`--=--__J
`
`-
`
`:
`
`I
`I
`,
`
`I
`
`1301
`
`-
`
`-+--:---
`
`:
`
`m
`
`: I
`' ,
`
`I
`
`1300
`
`I
`'
`
`~::0!1:
`iTI m
`0
`
`.-----:-:---,
`
`I
`'(
`
`' :
`
`I
`
`I
`
`I
`II
`
`:
`
`115
`
`'
`
`305
`
`--------------f\_ 230
`
`~----------------1
`I
`
`CONTROL3
`
`;
`
`I
`
`t
`
`I
`
`--------
`
`t t s I
`
`I I
`
`: ·---________ L,
`230
`
`:
`:
`:
`:
`~220
`·r-----:-'~~--305
`---------------------------1
`
`3108 305 : 114
`
`"i'-31 02
`:
`
`• • •
`~ ____ ---------J c----------~~
`
`I
`
`I
`I
`
`:
`
`CONTROL N
`INTERFACE/
`
`DEVICE
`CACHING
`
`CIB
`
`: • • •
`l
`-:
`
`f200
`
`FIG. 3
`
`305
`
`INTERFACE
`CPU/CACHE
`
`DUAL
`
`• • •
`
`~210
`I
`I
`I
`
`120
`
`CPU?
`
`1 120
`
`I CPU6
`
`• • •
`
`I
`
`~-------------------------------------------------------------------------------------
`
`7
`
`I
`
`i
`
`i I
`
`t
`
`t
`
`~ I
`I
`
`2
`
`-
`
`I I
`
`t
`
`.. I
`
`t 1' ~I CHANNEL
`MEMORY
`; CONTROL 2
`t I' ~ I CHANNEL
`MEMORY
`:~J::::b:d~d~;;~~~t~~~~J:==tJ~ CHANNEL
`CONTROL 1
`MEMORY
`' CONTROL 0
`MEMORY
`h~IF/GART
`:
`:
`IIF:
`
`1/0
`CIB
`
`:
`
`J
`
`I
`
`-: :-
`
`L-.--~-L: :
`CONTROL 0 I
`
`-
`
`FCU
`
`f2 ~ 400
`0 f>
`~ ffi
`0 ~
`(")-I
`
`r---1
`
`'
`
`: IIF INTERFACE
`
`I
`
`1
`J :--
`
`1
`
`: :
`
`1/0
`CIB
`
`CONTROL N-1
`INTERFACE/
`
`DEVICE
`CACHING
`
`CIB
`
`:
`:
`: IIF
`:
`
`I
`
`I
`
`1
`
`~ ___ _t __ L~--.
`:
`: CONTROL 0
`:
`: : INTERFACE IIF:
`:
`-: 113 ;---
`
`305~~------~~~~~~~~~~~~--~~~~~~~~~~~~~~~~~:f~:~~---1:::_
`
`INTERFACE
`CPU/CACHE
`
`DUAL
`
`120 1
`
`CPU1
`
`120
`
`CPUO
`
`240' I
`: :
`
`I
`
`I
`
`I
`I
`I
`I
`I
`I
`I
`
`I
`I
`I
`I
`I
`I
`I
`
`-
`
`CIB _j : [-CIB
`
`I
`
`240:
`
`CONTROL
`INTERFACE
`
`BUS
`
`CONTROL
`INTERFACE
`
`BUS
`
`r---~----~-----r----
`
`PCI
`
`AGP
`
`154
`
`151
`
`t-'
`PCI
`
`152 -1
`AGP
`
`NETAPP, INC. EXHIBIT 1016
`Page 5 of 46
`
`

`
`U.S. Patent
`
`Sep.18,2001
`
`Sheet 4 of 33
`
`US 6,292, 705 Bl
`
`0
`.___.,....-~
`
`u:tt: ~I
`
`-< (
`::::?!<!)
`
`u_
`
`~I
`
`'<:f"
`......
`
`(
`
`("')
`
`......
`
`~
`
`C\1
`......
`
`!
`
`co co
`0 ......
`
`("')
`
`u_
`
`~I
`
`co
`0
`......
`("')
`
`- D r<-J
`-
`
`I
`
`Vff
`
`u_
`<!)
`u_
`(.)
`
`~I
`
`(.)
`I-
`
`~I
`
`co
`I-
`
`~
`
`_r-
`
`•
`
`0
`C\1
`
`7
`
`0
`C\1
`
`~
`
`co
`0:: 01
`<((j)
`~'<:f"
`
`0
`0
`......
`(
`
`("')
`
`~
`
`NETAPP, INC. EXHIBIT 1016
`Page 6 of 46
`
`

`
`1--"
`~
`(It
`Q
`~
`N
`\0
`'N
`0'1
`rJ'l
`
`e
`
`~
`~
`0 ......,
`Ul
`~ .....
`'JJ. =(cid:173)~
`
`'"""'
`N c c
`'"""'
`~CIO
`~ '?
`'JJ.
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`-""--4357
`'-4347
`
`---4337
`-""--4327
`-""--4317
`
`-
`
`14307
`l4300
`
`WQ7 (BANK1)
`
`TAG? (BANK1)
`
`WQ? (BANKO)
`
`TAG? (BANKO)
`
`TUB?
`•
`•
`•
`
`DIAGNOSIS INTERFACE
`
`480
`(DI)
`
`31~4
`
`TB
`
`INCOMING QUEUE
`
`460
`(IQ)
`
`4\
`
`--'--4350
`'-4340
`
`_
`
`'--4330
`--1'---4320
`--'--4310
`
`/
`
`WQO (BANK1)
`
`TAGO (BANK1)
`
`WQO (BANKO)
`
`TAGO (BANKO)
`
`TUBO
`
`440
`
`-
`
`COMMAND DECODER
`
`ADDRESS AND
`
`~441
`
`FROM CFGIF
`MEMORY MAP:
`
`4~7
`
`43\7
`
`•
`•
`•
`4~50
`
`4~~0
`4~0
`
`4~5
`
`420
`
`R
`E
`L
`u
`D
`E
`H
`c
`s
`
`3~6
`
`TSB
`
`400_/
`
`FIG. 48
`
`NETAPP, INC. EXHIBIT 1016
`Page 7 of 46
`
`

`
`U.S. Patent
`
`Sep.18,2001
`
`Sheet 6 of 33
`
`US 6,292, 705 Bl
`
`FIG. 4C
`
`CFGIF
`410
`-
`
`r-.-441
`
`DECODE
`440
`-
`
`4~
`
`3104 ~~
`
`~r
`
`IQ
`460
`
`LJ
`
`43\0
`
`4390
`
`\
`4253
`
`I TAG 1~1
`
`f----4310
`
`4302
`
`+
`\_.-f
`
`4350l 4303\..
`
`[4330
`
`1
`
`\
`4250
`
`4301
`-
`
`)
`
`4252
`
`I TAG 1~1 I TAG 1~1
`
`s
`c
`H
`E
`D
`u
`L
`E
`R
`
`420
`
`4304--.rl
`
`4305~
`
`43401 43061
`
`1 432o
`
`I TAG 1~1 I TAG 1~1
`
`1-'--3106
`
`\
`
`4251
`
`NETAPP, INC. EXHIBIT 1016
`Page 8 of 46
`
`

`
`1-"
`~
`(It
`Q
`~
`N
`\0
`'N
`0'1
`rJ'l
`
`e
`
`~
`~
`0 ......,
`-..J
`~ .....
`'JJ. =(cid:173)~
`
`""""
`N c c
`""""
`~CIO
`~ '?
`'JJ.
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`2M: 2 LINES/SECTOR
`
`4M: 2 LINES/SECTOR
`
`4M: 2 LINES/SECTOR
`
`-----·---
`
`512K, 1M: NO
`
`512K, 1M, 2M: NO
`
`512K, 1M, 2M: NO
`
`2-WAY SET ASSOCIATIVE
`
`2-WAY SET ASSOCIATIVE
`
`2-WAY SET ASSOCIATIVE
`
`SECTORING
`
`ASSOCIATIVITY
`TAG
`
`2
`
`2
`
`2
`
`#OF BANKS
`
`2-WAY SET ASSOCIATIVE
`DIRECT MAPPED &
`32 BYTES
`
`2-WAY SET ASSOCIATIVE
`DIRECT MAPPED &
`64 BYTES
`
`2-WAY SET ASSOCIATIVE
`DIRECT MAPPED &
`
`ASSOCIATIVITY
`L2
`
`64BYTES
`
`CACHE LINE SIZE
`
`512K, 1M, 2M
`
`512K, 1M, 2M, 4M
`
`512K, 1M, 2M, 4M
`
`L2 SIZE
`
`MODEC
`
`MODES
`
`MODE A
`
`ORGANIZATION
`
`TAG
`
`CONFIGURATION
`
`L2
`
`FIG. 5
`
`NETAPP, INC. EXHIBIT 1016
`Page 9 of 46
`
`

`
`1--"
`~
`(It
`Q
`~
`N
`\0
`'N
`0'1
`rJ'l
`
`e
`
`~
`~
`0 ......,
`CIO
`.....
`=-~
`
`'JJ.
`
`~
`
`'"""'
`c
`N c
`'"""'
`~CIO
`~ '?
`'JJ.
`
`=
`
`......
`~
`......
`~
`~
`•
`\Jl
`d •
`
`01
`
`CACHE LINE
`
`8 I 7161514
`
`LN
`
`IND
`
`BK
`
`01
`
`CACHE LINE
`
`514
`
`IND
`
`BK
`716
`
`8 I
`
`I
`
`\JT'\\,/IIL.. LlllL..
`
`I
`
`11'11..1
`
`1
`
`ur\.
`
`I
`
`o I
`0 I
`
`o I
`
`I
`
`0
`
`I
`
`0
`
`0
`
`CACHE LINE
`
`CACHE LINE
`
`CACHE LINE
`
`CACHE LINE
`
`CACHE LINE
`
`CACHE LINE
`
`5
`
`8 I
`
`8 I 7 I 6 I 5
`;K II~D I
`
`LN
`
`BK
`
`5
`
`5
`
`8
`
`8
`
`5
`
`8 I
`
`8 I 7 I 6 I 5
`;K II~D I
`
`LN
`
`BK
`
`CACHE LINE
`
`5
`
`8
`
`CACHE LINE
`
`8 I 7 I 6 I 5
`
`IND
`
`BK
`
`IND
`
`IND
`
`IND
`
`IND
`
`IND
`
`IND
`
`IND
`
`IND
`
`IND
`
`20 119
`
`19 118
`
`18 117
`
`TAG
`
`TAG
`
`TAG
`
`IND
`
`21 120
`
`20 119
`
`18
`
`19
`
`17
`
`18
`
`IND
`
`21 120
`
`20 119
`
`18
`
`19
`
`18 117
`
`TAG
`
`TAG
`
`TAG
`
`TAG
`
`TAG
`
`TAG
`
`TAG
`
`TAG
`
`I 33
`
`34
`
`I 37
`
`38
`
`I
`
`2M
`
`I ~~~ llLJJ/J 33
`
`1M
`
`FIG. 6C ~
`
`1 33
`
`34
`
`1 37
`
`512K I 38
`
`35 I 34
`
`I 38
`
`4M
`
`2M ~34
`
`1M
`
`35 I 34
`
`512K I 38
`
`I 33
`
`34
`
`I 37
`
`38
`
`I
`
`4M
`
`I 38 ~33
`
`2M
`
`I 33
`
`34
`
`I 37
`
`38
`
`I
`
`1M
`
`I 33
`
`34
`
`I 37
`
`38
`
`512K I
`
`FIG. 68 ~
`
`FIG. 6A ~
`
`NETAPP, INC. EXHIBIT 1016
`Page 10 of 46
`
`

`
`1--"
`~
`(It
`Q
`~
`N
`\0
`'N
`0'1
`rJ'l
`
`e
`
`~
`~
`0 ......,
`'0
`~
`
`'JJ. =(cid:173)~
`
`'"""'
`N c c
`'"""' 00
`~ '?
`'JJ.
`
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`RAM
`
`8K X 40 INTERNAL TAG
`
`OR
`
`8K X 20 INTERNAL TAG
`
`WAT1
`RAM
`
`WAYO
`RAM
`
`8K X 20 INTERNAL TAG
`
`4
`
`2 11 ~
`
`3 ~· 1 MES1~ I
`MES10 I
`
`TAG 15 BITS
`
`TAG 16 BITS
`
`TAG 17 BITS
`
`18
`
`18
`
`I PA~~TY 1
`
`2M
`
`1M
`
`512K I 19 118
`
`~ARITY
`
`FIG. 78
`
`RAM
`
`8K X 40 INTERNAL TAG
`
`OR
`
`8K X 20 INTERNAL TAG
`
`WAT1
`RAM
`
`WAYO
`RAM
`
`8K X 20 INTERNAL TAG
`
`0 I
`0 I
`
`MES10
`
`i-11
`
`-
`
`-
`
`-
`
`MES10
`
`3 ~ 1
`
`TAG 14 BITS
`
`TAG 16 BITS
`
`TAG 17 BITS
`
`18
`
`·-
`
`·-
`
`4M
`
`2M
`
`PARITY
`I 19 118
`PARITY
`512K I 19-118
`
`1M
`
`FIG. 7A
`
`NETAPP, INC. EXHIBIT 1016
`Page 11 of 46
`
`

`
`1--"
`~
`(It
`Q
`~
`N
`\0
`'N
`0'1
`rJ'l
`
`e
`
`~
`~
`0 ......,
`'""" c
`~ ......
`'JJ. =(cid:173)~
`
`'"""
`N c c
`~CIO
`'"""
`~ '?
`'JJ.
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`ACTIONS DESCRIBED ABOVE.
`S*_LO BEING INVALIDATED TO REPLACE. FOLLOWTHE
`BE INVALIDATED. THE CHOOSE THE WAY WITH
`NO TUB ALLOCATION. WAIT FOR SO_LO OR S1_LO TO
`
`UPDATE S1 ONCE S1_L 11NVALIDATED.
`INVALIDATION TO S1_L 1. TUB ALLOCATION.
`
`S1_L1.
`UPDATE S1. NO TUB ALLOCATION.INVALIDATE
`UPDATE S1. NO TUB ALLOCATION.
`
`UPDATE SO ONCE SO_L 1 INVALIDATED.
`INVALIDATION TO SO_L 1. TUB ALLOCATION.
`
`SO_L 1.
`UPDATE SO. NO TUB ALLOCATION. INVALIDATE
`UPDATE SO. NO TUB ALLOCATION.
`UPDATE SO ONCE SO_L 1 INVALIDATED.
`INVALIDATION TO SO_L 1. TUB ALLOCATION.
`
`S1_L 1.
`UPDATE S1. NO TUB ALLOCATION. INVALIDATE
`SO_L 1.
`UPDATE SO. NO TUB ALLOCATION. INVALIDATE
`UPDATE S1. NO TUB ALLOCATION.
`UPDATE SO. NO TUB ALLOCATION.
`
`ACTION
`
`FIG. 8
`
`MOESI
`
`MO
`
`ES
`I
`
`MOESI
`
`MOESI
`MOESI
`
`MO
`
`ES
`
`MOES
`
`I
`
`MOESI
`S1_L 1
`
`MOESI
`
`MOESI
`
`MOESI
`MOESI
`
`MO
`
`ES
`I
`
`MO
`
`MO
`
`ES
`
`MOES
`
`I
`
`SO_L 1
`
`MOES
`
`I
`
`I
`I
`
`MOESI
`
`MOESI
`MOESI
`
`I
`
`I
`
`I
`I
`I
`
`MOES
`
`MOESI
`
`MOESI
`MOESI
`
`I
`
`I
`I
`
`I
`
`I
`
`I
`I
`I
`
`S1_LO
`
`SO_LO
`
`NETAPP, INC. EXHIBIT 1016
`Page 12 of 46
`
`

`
`U.S. Patent
`
`Sep.18,2001
`
`Sheet 11 of 33
`
`US 6,292, 705 Bl
`
`FIG. 9A
`
`Cycle
`
`Transaction Activity
`
`-10
`-9
`-8
`-7
`-6
`-5
`-4
`-3
`-2*
`-1*
`0*
`1*
`2*
`
`3*
`
`4*
`
`5*
`
`6
`
`7
`8
`9
`
`Processor Bus
`*
`*
`*
`Processor Interface
`ccu
`CCU/CIB transmit
`channel wires
`(CCU to IIF)
`IIF I CIB receive
`IIF processing
`TB request; IIF has command
`TB grant; drive address/control onto TB
`address/control echoed;
`all interfaces latch address/control
`all interfaces observe address/control; compute backoff
`TC predecode; TC tag lookup
`MIF decodes for speculative read
`drive backoff
`TC combinatorial scheduler
`TC generates and drives "mem early release"
`initiator/MIF/TC samples backoff
`MIF observes "mem early release"
`backoff and "mem early release" used late in cycle
`by MIF to control speculative request
`TC drives Transaction Status Bus
`MIF/CIB transmit speculative request
`initiator/MIF read Transaction Status Bus
`channel wires
`(MIF to MCU)
`MCU/CIB receive
`MCU core has request
`
`(possible extra cycle for 100MHz alignment
`counted)
`
`-- not
`
`NETAPP, INC. EXHIBIT 1016
`Page 13 of 46
`
`

`
`U.S. Patent
`
`Sep.18,2001
`
`Sheet 12 of 33
`
`US 6,292,705 Bl
`
`FIG. 98
`
`Cycle Transaction Activity
`
`10
`11
`12
`13
`14
`15
`16
`17
`18
`19
`20
`
`21
`22
`23*
`24*
`25*
`26*
`
`27*
`28
`29
`30
`31
`
`32
`
`MCU drives DIMM pins
`
`(data arrives SOns later)
`
`II
`
`external driver for DIMM pins
`
`II
`
`DRAM has address
`
`(on MCU pins)
`data back from DRAM
`MCU has data; ECC compute
`(another cycle needed if bypass cannot happen -- assume
`bypass)
`(another cycle needed if ECC indicates correction needed)
`MCU/CIB transmit
`channel wires
`(MCU to MIF)
`MIF/CIB receive
`MIF receive
`on Data Switch
`IIF receive
`(IIF processing
`IIF/CIB transmit
`channel wires
`(IIF to CCU)
`CCU/CIB receive
`CCU processing
`Processor interface
`(0 - 3 cycles to align to Processor bus; assume 0 here)
`Processor transmit
`
`(not needed for data cycles))
`
`NETAPP, INC. EXHIBIT 1016
`Page 14 of 46
`
`

`
`U.S. Patent
`
`Sep.18,2001
`
`Sheet 13 of 33
`
`US 6,292, 705 Bl
`
`VALID TB COMMANDS
`RDU
`
`RDE
`
`RDS
`
`RDM, BE = OxOO
`
`RDM, BE = Oxff
`
`RDC
`
`WRU
`WRB, BE = OxOO
`
`WRB, BE = Oxff
`
`WRC, BE <> Oxff
`
`WRC, BE = Oxff
`
`E2M
`
`S2M
`
`lACK
`
`SHUT
`HALT
`INVD
`
`FIG. 10A
`
`DESCRIPTION
`NON-CACHEABLE READ. THE INITIATOR DOES NOT CACHE THE DATA. THIS TRANSACTION IS
`NOT CACHE COHERENT, WHICH MEANS NOT TAG LOOK-UP IS PERFORMED AND NO CACHE
`STATE TRANSITION IS PERFORMED.
`READ EXCLUSIVE. THE TAG LOOK-UP IS PERFORMED. THE INITIATOR WILL CACHE THE
`LINE IN ESTATE. THERE MAY BE LINE REPLACEMENT ASSOCIATED WITH THIS TRANSACTION.
`
`READ SHARED. THE TAG LOOK-UP IS PERFORMED. THE INITIATOR WILL CACHE THE LINE
`IN S STATE. THE CACHE LINE STATE TRANSITIONS ARE DEFINED BY TPT. THERE MAY BE
`LINE REPLACEMENT ASSOCIATED WITH THIS TRANSACTION.
`READ FOR MODIFY. TAG LOOK-UP IS PERFORMED. WHEN BE= OxOO, THE TRANSACTION
`DOES NOT DO DATA TRANSFER. THE FINAL CACHE LINE STATE IN THE L2 WILL BE M. THE
`CONTENT OF THE CACHE LINE IS UNPREDICTABLE. THERE MAY BE LINE REPLACEMENT
`ASSOCIATED WITH THIS TRANSACTION.
`READ FOR MODIFY. WHEN BE= Oxff, THE INITIATOR PERFORMS A CACHE LINE READ AND
`THE FINAL STATE OF THE CACHE WILL BE IN M STATE. THERE IS DATA TRANSFER ASSOCIATED
`WITH THIS COMMAND. THERE MAY BE LINE REPLACEMENT ASSOCIATED WITH THIS
`TRANSACTION.
`READ COHERENT. THE TAG LOOK-UP IS PERFORMED. THE INITIATOR WILL NOT CACHE THE
`DATA BEING READ
`NON-CACHEABLE WRITE. NO TAG LOOK-UP AND NO CACHE STATE TRANSITION.
`WRITE BACK WITH NO DATA TRANSFER. THIS TRANSACTION WILL CHANGE THE CACHE LINE IN
`THE TAG TO I STATE.
`WRITE BACK WITH DATA TRANSFER. THIS TRANSACTION WILL CHANGE THE CACHE LINE IN THE
`TAG TO I STATE AND WRITE BACK A DIRTY LINE TO THE MEMORY.
`WRITE COHERENT. SUB-BLOCK WRITE. IF THIS TRANSACTION HITS A DIRTY LINE IN ANOTHER
`L2, THE DIRTY LINE WILL BE WRITIEN BACK TO THE MEMORY BEFORE THIS TRANSACTION
`CAN GOON.
`WRITE COHERENT. BLOCK WRITE. IF THIS TRANSACTION HITS A DIRTY LINE IN ANOTHER L2,
`THE DIRTY LINE WILL BE INVALIDATE WITHOUT BEING WRITIEN BACK TO THE MEMORY AND
`THE MEMORY WILL BE UPDATED WITH THE NEW DATA PROVIDED WITH THIS TRANSACTION.
`CACHE STATE TRANSITION E TO M. THE CACHE LINE STATE IN THE TAG WILL BE CHANGED
`TO M. TAG LOOKUP IS PERFORMED AND ALL OTHER L2 COPIES OF THE LINE WILL BE
`INVALIDATED AND ALSO WILL BE WRITIEN BACK IF DIRTY.
`CACHE STATE TRANSITION S TOM. THE CACHE LINE STATE IN THE TAG WILL BE CHANGED
`TO M. TAG LOOKUP IS PERFORMED ALL OTHER L2 COPIES OF THE LINE WILL BE
`INVALIDATED AND ALSO WILL BE WRITIEN BACK IF DIRTY.
`INTERRUPT ACKNOWLEDGE, THIS TRANSACTION WILL BE FORWARDED TO THE COMPATIBILITY
`PCI BUS.
`SHUT DOWN. TREATED BY THE TC AS A NOP, BUT THE INITIATOR WILL BE ACKNOWLEDGED.
`HALT. TREATED THE SAME AS SHUTDOWN.
`EVICT. THE CACHE LIN IN THE TAG WILL BE INVALIDATED. NO DATA TRANSFER IS
`PERFORMED EVEN IF THE CACHE IS CURRENTLY IN THE M OR 0 STATE.
`
`NETAPP, INC. EXHIBIT 1016
`Page 15 of 46
`
`

`
`U.S. Patent
`
`Sep.18,2001
`
`Sheet 14 of 33
`
`US 6,292, 705 Bl
`
`VALID TB COMMANDS
`INVD
`
`WBINVD
`FLUSHA
`
`STPGNT
`
`SMIACK
`
`LOCKON
`
`LOCKOFF
`
`IDENTC
`
`IDENTB
`
`IDENTM
`
`MB
`MC
`DRAIN
`DRAINW
`WBOUND
`
`FIG. 108
`
`DESCRIPTION
`EVICt THE CACHE LINE IN THE TAG WILL BE INVALIDATED. NO DATA TRANSFER IS
`PERFORMED EVEN IF THE CACHE LINE IS CURRENTLY IN THE M OR 0 STATE.
`WRITE-BACK AND INVALIDATE. TREATED THE SAME AS SHUTDOWN.
`FLUSH ACKNOWLEDGE. THIS TRANSACTION WILL BE FORWARDED TO THE COMPATIBILITY
`PCI BUS. THE TARGET SHOULD SEND A REPLY TO THE INITIATOR.
`
`STOP GRANT. THIS TRANSACTION WILL BE FORWARDED TO THE COMPATIBILITY PCI BUS.
`THE TARGET SHOULD SEND A REPLY TO THE INITIATOR.
`
`SMI ACKNOWLEDGE .. THIS TRANSACTION WILL BE FORWARDED TO THE COMPATIBILITY PCI
`BUS. THE TARGET SHOULD SEND A REPLY TO THE INITIATOR.
`LOCK ON. BUS LOCK SET. IF THIS TRANSACTION IS NOT BACKED OFF, THE TC STARTS TO
`BACK OFF ALL TRANSACTIONS (EXCEPT LOCKOFF, ADDRBOFFRESUME) FROM OTHER
`CCUs.
`
`LOCK OFF. BUS LOCK RELEASE. THE TC WILL CLEAR THE BUS LOCK AFTER SEEN THIS
`COMMAND.
`
`CCU IDENTIFICATION. THIS TRANSACTION ACTS LIKE AN 10 WRITE COMMAND AND WILL
`BE FORWARDED TO A TBD IF.
`
`BBU IDENTIFICATION. THIS TRANSACTION ACTS LIKE AN 10 WRITE COMMAND AND WILL
`BE FORWARDED TO A TBD IF.
`
`MIF IDENTIFICATION. THIS TRANSACTION ACTS LIKE AN 10 WRITE COMMAND AND WILL BE
`FORWARDED TO A TBD IF.
`MEMORY BARRIER. NOT CLEAR WHAT TO DO.
`MEMORY BARRIER. NOT CLEAR WHAT TO DO.
`COMPLETE THINGS. TREATED AS SHUTDOWN.
`COMPLETE ALL WRITES THEN REPLY. TREATED AS SHUTDOWN.
`WRITES AFTER THIS TRANSACTION CANNOT PASS PRECEDING READS. TREATED AS
`SHUTDOWN.
`
`AVAIL
`ADDRBOFFRESUME
`
`RESOURCE AVAILABILITY. TREATED AS SHUTDOWN.
`ADDRESS BACK-OFF RESUME.USED TO RESUME TRANSACTIONS BACKED OFF BY A
`SPECIFIC ADDRESS CACHE (CACHE LINE BOUNDARY)
`
`NETAPP, INC. EXHIBIT 1016
`Page 16 of 46
`
`

`
`U.S. Patent
`
`Sep.18,2001
`
`Sheet 15 of 33
`
`US 6,292, 705 Bl
`
`FIG. 11A
`
`LEN CID SCID QID ADDR
`BE
`- CMD D TY
`6 6665 5 55 55555544 44 4444
`44 3333 3333
`3 2109 8 7 54321098
`76 5432
`10 9876 5432
`
`ADDR[34:3]
`33222222222211111111110000000000
`10987654321098765432109876543210
`
`FIG. 118
`
`bits Field Description
`63
`Reserved
`62:59 CMD
`Command
`0000 RDU
`0001 RDE
`0010 RDS
`0011 RDM
`
`Read Uncacheable
`Read Exclusive (reply may be RDE or RDS)
`Read Shared
`Read for Modify
`LEN field encoaes cache line length (32 or 64)
`BE field encodings:
`OxOO No data transfer
`Oxff Data transfer
`Read Coherent
`Similar to RDS but no cache state updates
`occur. Used for I/0.
`Reserved
`Reserved
`Reserved
`Write Uncacheable
`Reserved
`Reserved
`WriteBack
`LEN field encodes cache line length (32 or 64)
`BE field encodings:
`OxOO No data transfer
`Oxff Data transfer
`Write Coherent
`Reserved
`
`0100 RDC
`
`-
`0101
`-
`0110
`-
`0111
`1000 WRU
`1001
`-
`1010
`-
`1011 WRB
`
`1100 WRC
`1101
`-
`
`NETAPP, INC. EXHIBIT 1016
`Page 17 of 46
`
`

`
`U.S. Patent
`
`Sep.18,2001
`
`Sheet 16 of 33
`
`US 6,292,705 Bl
`
`FIG. 11C
`
`bits Field Description
`62:59 CMD
`Command
`1110 SOP
`
`(continued)
`Special Operation
`LEN field always 00
`BE field encodes which operation:
`Ox01
`SHUT
`Shutdown
`Ox03 HALT
`Halt
`Ox05
`INVD
`Cache Invalid
`Ox07 WBINVD
`WB Cache Invalid
`Ox09
`FLUSHA
`Flush Ack
`OxOb STPGNT
`Stop Grant
`OxOd SMIACK
`SMI Acknowledge
`OxOe DRAIN
`Complete things
`Ox10 LOCKON
`System Lock (reply needed)
`Ox12 LOCKOFF
`System Unlock (reply needed)
`Ox14 MB
`Memory Barrier (reply needed)
`Ox17 MC
`Machine Check
`Ox18 DRAINW
`Complete all writes then reply
`Ox1b WBOUND
`Writes after this cannot pass
`preceding reads
`CCU Ident (reply needed)
`BBU Ident (reply needed)
`MCU Ident (reply needed)
`ADDR encoded as:
`38:35 reserved
`34
`compatibility PCI bus
`33
`Mode B processor
`32
`Mode A processor
`31
`Mode C processor
`3 0 : 2 7 reserved
`cmd buffers counts
`26:21 COP
`20:15 non COP
`data buffer counts
`14:9 COP
`8:3 non COP
`if a COP value is zero, the
`non COP value covers both types
`
`IDENTC
`IDENTB
`IDENTM
`
`Ox82
`Ox84
`Ox86
`
`request gives chip values
`reply gives FCU values
`Ox90 AVAIL Resouce Availability
`(reply needed)
`ADDON encoded as:
`18:27 reserved
`cmd buffers counts
`26:21 COP
`20:15 non COP
`data buffer counts
`14:9 COP
`8:3 non COP
`if a COP calue is zero, the
`non COP value covers both types
`Only those SOPs with the low order BE bit
`off (ie: those that are even) get a reply.
`
`NETAPP, INC. EXHIBIT 1016
`Page 18 of 46
`
`

`
`U.S. Patent
`
`Sep.18,2001
`
`Sheet 17 of 33
`
`US 6,292,705 Bl
`
`FIG. 11D
`
`bits Field Description
`
`62:59 CMD
`
`001
`
`010
`
`011
`
`Command
`(continued)
`1110 COP Cache Operation
`LEN field always 00
`BE field encodes which operation:
`DCCxxNNN
`xx = Reserved
`D - Direction
`0
`from processor to FCU
`from FCU to processor
`1
`DCC - Direction and command
`No operation
`000
`NNN nust be zero
`E2M - processor E => M request
`NNN must be zero
`S2m - processor S -> M request
`NNN must be zero
`INVD - processor * -> I notice
`NNN must be zero
`No O~eration
`NNN 1s new cache line state
`CWR - send cache line as write request
`NNN is new cache line state
`CRD - send cache line as read reply
`NNN is new cache line state
`CEV - send cache line as write request
`(used to evict lines in tag-sectoring)
`NNN must be zero
`NNN - Next tag state
`000 N - no change
`001 E - exclusive
`S - shared
`010
`0 - owner
`011
`I - invalid
`100
`reserved
`101
`reserved
`110
`reserved
`111
`The reply for any of the read commands may have
`changed the CMD field to encode what the cache state
`of the line should be (ie, a RDE request may generate
`a RDE reply if the line is in exclusive state or a RDS
`reply if the line is in shared state)
`
`100
`
`101
`
`110
`
`111
`
`NETAPP, INC. EXHIBIT 1016
`Page 19 of 46
`
`

`
`U.S. Patent
`
`Sep.18,2001
`
`Sheet 18 of 33
`
`US 6,292, 705 Bl
`
`FIG. 11E
`
`58
`
`bits Field Description
`Data Cycles
`No data cycles follow this command header
`0
`At least one data cycle follows command header
`1
`(Must look at LEN and BE fields to determine how many)
`
`D
`
`57:56 TY
`
`55:48 TY
`
`47:46 LEN
`
`Request
`Hi-Priority request
`Reply
`Error Reply
`
`Type
`00
`RQ
`01
`HQ
`RP
`10
`EP
`11
`Byte Enable
`For reads or write encodes which of eight subblocks
`of the data transmitted is actually valid. Each
`bit of the BE filed will be a one 1f the corresponding
`eighth of the LEN field is valid.
`For SpecialOperation or CacheOperation commands
`encodes a sub-command.
`Length
`Length of request for read or write commands.
`00
`8 bytes (one data cycle)
`32 bytes (four data cycles)
`01
`64 bytes (ei9ht data cycles)
`10
`512 bytes (s1xty-four data cycles)
`11
`Always 00 for other types of commands
`
`45:42 CID Channel Identifier
`Which unit made the command request. Replies or Errors
`are returned to this unit. The following encodings of
`the CID field are representative, but not mandatory.
`An implementation might allow replacement of a CCU
`with additional BBUs, for example.
`0000 ccu 0
`0001 ccu 1
`0010 ccu 2
`0011 ccu 3
`0100 ccu 4
`0101 ccu 5
`0110 ccu 6
`0111 ccu 7
`1000 BBU 0
`1001 BBU 1
`1010 Reserved
`1011 Reserved
`1100 MCU 0
`1101 MCU 1
`1110 MCU 2
`1111 MCU 3
`
`NETAPP, INC. EXHIBIT 1016
`Page 20 of 46
`
`

`
`U.S. Patent
`
`Sep.18,2001
`
`Sheet 19 of 33
`
`US 6,292, 705 Bl
`
`FIG. 11F
`
`bits Field Description
`
`41:40 SCID Sub-Channel Identifier
`Each channel identifier may have a variety of devices
`associated with it. Any replies or errors must
`preserve the SCID value that was part of the request.
`The value is implementation specific within each unit.
`For BBUS 1 one SCID would be PCI bus 0 and another
`would be AGP bus 0 or PCI bus 1.
`For CCUs 1 the SCID might be used to extend the QID
`field by two more bits.
`
`39:36 QID
`
`Queue Identifier
`Each component that makes requests assigns a QID to
`the request. Any replies or errors must preserve the
`QID value that was part of the request.
`
`35:32 ADDR[38:35)
`
`31:0 ADDR[34:3)
`Address Bits
`ADDR[38) == 0 means I/O address space
`ADDR[38) == 1 means memory address space
`For Mode A Processors which have PA[34:3) 1 the following
`mapping occurs.
`PA[34] --> ADDR[38]
`0000 --> ADDR[37:34)
`PA [ 3 3 : 3] -- > ADDR [ 3 3 : 3)
`
`For Mode B Processors which have PA[34:3] 1 the following
`mapping occurs.
`(xored)
`PA[43) --> ADDR[38)
`PA[37: 3] --> ADDR [37: 34)
`PA[42:38] must be zero (???)
`For memory addresses going to am MCU 1 the FCU will
`remove the address bits that select which MCU port
`and which MCU 1 and shift down all higher order bits
`except for ADDR[38). ADDR[37) will be set to the
`port number for the MCU )which may be in single
`channel mode).
`
`NETAPP, INC. EXHIBIT 1016
`Page 21 of 46
`
`

`
`U.S. Patent
`
`Sep.18,2001
`
`Sheet 20 of 33
`
`US 6,292, 705 Bl
`
`FIG. 11G
`
`Command Mappings:
`
`PCI Command
`
`Interrupt Ack
`0000
`0001 Special Cycle
`I/0 Read
`0010
`I/O Write
`0011
`reserved
`0100
`reserved
`0101
`011 0 Memory Read
`0111 Memory Write
`reserved
`1000
`reserved
`1001
`1010 Configuration Read
`1011 Configuration Write
`1100 Memory Read Multiple
`1101 Dual Address Cycle
`1110 Memory Read Line
`1111 Memory Write/Invalidate
`
`Channel Command
`
`RDU in IACK space
`SOP
`RDU, LEN=B, BE= .. .
`WRU, LEN=8, BE= .. .
`
`RDC
`WRC
`
`RDU
`WRU
`RDC
`(internal to BBU)
`RDC, LEN=<Cl>
`WRC, LEN=<Cl>, BE=Oxff
`
`FIG. 11H
`
`Command Mappings:
`
`AGP Command
`
`0000
`0001
`0010
`0011
`0100
`0101
`0110
`0111
`1000
`1001
`1010
`1011
`1100
`1101
`1110
`1111
`
`Read
`Read (hi-priority)
`reserved
`reserved
`Write
`Write (hi-priority)
`reserved
`reserved
`Long Read
`Long Read (hi priority)
`Flush
`reserved
`Fence
`Dual Address Cycle
`reserved
`reserved
`
`Channel Command
`-----------
`RDU, LEN=64, BE= ...
`RDU, LEN=64, TY=HQ, BE= ...
`
`WRU, LEN=64, BE= ...
`WRU, LEN=64, TY=HQ, BE= ...
`
`RDU, LEN=64, BE= ...
`RDU, LEN=64, TY=HQ, BE= ...
`SOP, BE=DRAINW
`
`SOP, BE=WBOUND
`(internal to BBU)
`
`NETAPP, INC. EXHIBIT 1016
`Page 22 of 46
`
`

`
`U.S. Patent
`
`Sep.18,2001
`
`Sheet 21 of 33
`
`US 6,292, 705 Bl
`
`FIG. 12A
`
`SIGNAL
`NAME
`taddr[38:3]
`
`tcmd[3:0]
`
`DRIVER LISTENER
`
`DESCRIPTION
`
`IIF
`
`IIF
`
`TC
`ALLIFs
`TC
`ALLIFs
`
`taddr£38:3], TB address, taddr[38:3]follows the PP-Channel
`address format. See the PP-Channel definitions for details
`tcmd[3:0], TB address, tcmd[3:0] can be divided into two
`categories, PP-Channel commands and miscellaneous commands.
`tcmd[3] is used to identify these two groups. tcmd[3] = 0 indicates
`that tcmd[2:0] is defined according to PP-Channel command format
`while tcmd[3] = 1 indicate that tcmd[2:0] has miscellaneous
`commands. See PP-Channel definitions for details.
`
`PP-Channel Commands:
`Commands
`tcmd[3:0]
`4'b0000
`ROE
`4'b0001
`ROS
`ROM
`4'b0010
`4'b0011
`SOP
`WRI
`4'b0100
`WRB
`4'b0101
`4'b0110
`4'b0111
`
`COP
`
`Read Exclusive
`Read Shared
`Read for Modify
`Special Operation
`Write
`Write Back
`Reserved
`Cache Operation
`
`TB Miscellaneous Commands:
`Commands
`tcmd[3:0]
`LOCKOFF
`4'b1000
`
`AddrBof/Resume 4'b1001
`
`To restart the transactions backed-
`off by tlock
`To restart the transactions backed-
`off by taddrboff at the address
`indicated by taddr£38:3]
`tbe[l:O], TB byte enable or sub-channel commands. See PP-Channel
`definitions for details.
`tlen[1 :0], TB transaction length. See PP-Channel
`definitions for details.
`tiid£3:0], TB initiator ID.
`
`tqid[3:0], TB queue IO.It is a pointer pointing to the entry of the
`initiato~s internal data structure that holds the information of the TB
`transaction beino driven.
`tscid£1:0], TB sub-channeiiD.
`
`tc, TB cacheable. 0 indicates the transaction is not cacheable while 1
`ind