a2) United States Patent
`US 6,581,137 Bl
`(10) Patent No.:
`Jun. 17, 2003
`(45) Date of Patent:
`Sandorfi
`
`US006581137B1
`
`(54) DATA STORAGE SYSTEM
`
`(75)
`
`Inventor: Miklos Sandorfi, Foxboro, MA (US)
`
`(73) Assignee: EMC Corporation, Hopkinton, MA
`(US)
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.: 09/408,058
`
`(22)
`
`Filed:
`
`Sep. 29, 1999
`
`(SL) Unt, C1. ec eee cee ce cceeeeeeceeeeeseeeeneene GO06F12/00
`(52) US. Che eee ceereeenees 711/114; 710/52
`(58) Field of Search «0.00.00... 710/52, 129; 711/112,
`711/113, 114, 118, 131
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`4,545,077 A
`5,206,939 A
`5,550,804 A
`5,721,860 A
`5,734,848 A
`5,737,745 A *
`
`10/1985 Drapala et al.
`4/1993 Yanai etal.
`8/1996 Haussler et al.
`2/1998 Stolt etal.
`3/1998 Gates etal.
`4/1998 Matsumoto et al.
`
`........ TAL/114
`
`(List continued on next page.)
`FOREIGN PATENT DOCUMENTS
`
`JP
`JP
`WO
`WO
`
`05046525 A
`08171458 A
`WO 00/39690
`WO 00/39691
`
`2/1993
`7/1996
`7/2000
`7/2000
`
`OTHER PUBLICATIONS
`
`Co-Pending Patent Application Serial No. 09/408,429 filed
`Sep. 29, 1999 and Assigned to Art Unit 2851.
`Co—Pending Patent Application Serial No. 09/408,430 filed
`Sep. 29, 1999 and Assigned to Art Unit 2751.
`Co—Pending Patent Application Serial No. 09/408,807 filed
`Sep. 29, 1999 and Assigned to Art Unit 2751.
`
`Co-Pending Patent Application Serial No. 09/408,811 filed
`Sep. 29, 1999 and Assigned to Art Unit 2751.
`Co-Pending Patent Application Serial No. 09/408,234 filed
`Sep. 29, 1999 and Assigned to Art Unit 2751.
`
`Primary Examiner—Do Hyun Yoo
`Assistant Examiner—Christian P. Chace
`(74) Attorney, Agent, or Firm—Daly, Crowley & Mofford,
`LLP
`
`67)
`
`ABSTRACT
`
`A data storage system wherein a host computer is in com-
`munication with a bank of disk drives through an interface.
`The interface includes: a memory;a plurality of directors for
`controlling data transfer between the host computer and the
`bankof disk drives as such data passes through the memory;
`and a plurality of busses in communication with the direc-
`tors and the memory. Each one of the directors includes a
`central processing unit. The central processing unit includes:
`(A) a microprocessor; (B) a main memory; and (C) a
`microprocessor interface.
`‘The microprocessor interface
`includes: (i) a data rebuffering section disposed in the chip
`and adapted to couple data from a one ofa plurality of data
`ports to a data port of the microprocessor selectively in
`accordance with a control signal; and (ii) a main memory
`interface adapted for coupling to a main memory for the
`microprocessor, such main memoryinterface being adapted
`for coupling to the microprocessor and being coupled to the
`data rebuffering section for providing control signals to the
`main memorysection for enabling data transfer between the
`main memory and the microprocessor through the data
`rebuffering section. A controller is coupled to the data
`rebuffering section for producing the control signal. The
`central processing unit main memoryis a selected one of a
`plurality of memory types each type having a different data
`transfer protocol and the main memory interface is config-
`ured in accordance with the selected one of the plurality of
`memory types to provide a proper memory protocol to data
`being transferred between the microprocessor and the main
`memory through the main memory interface. One main
`memory is an SDRAM and another a RDRAM.
`
`19 Claims, 14 Drawing Sheets
`
`14
`DISK
`
`BHa on
`DRIVES
`TL
`
`DIRECTOR e-—|
`(FIG. 2)
`el}
`20;
`:
`>
`i
`
`”|DIRECTOR
`HOST
`
`
`COMPUTER
`
`
`
`
`
`
`
`
`
`
`
`
`
`LOWADDR
`MEM I~ 186
`
`Google Exhibit 1022
`Google Exhibit 1022
`Google v. Valtrus
`Google v. Valtrus
`
`

`

`US 6,581,137 B1
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`Tzelnic et al. oo... 711/162
`
`StolowitZ oo... 710/61
`Bui et al. wee 711/114
`
`6,148,380 A
`6,256,705
`6,269,424 B1 *
`6,286,083
`6,360,305
`
`* cited by examiner
`
`Dodd etal. .....
`
`Katsuragiet al.
`....
`
`Novaket al. oo... FAL/157
`
`

`

`U.S. Patent
`
`Jun. 17, 2003
`
`Sheet 1 of 14
`
`US 6,581,137 BI
`
`|peworoaual*||3:~S02|!!
`3!
`!———_iSIO|"024)HeuwHI1: ySiIg|NS
`L‘Old
`
`
`|HOL9FHIG
`
`
`
`T_[YOLOI4IC «FHLblogrf tyonazuia
`
`802
`
`él
`
`LSOH
`
`YILNdW0d
`
`

`

`U.S. Patent
`
`
`
`US 6,581,137 BI
`
`eeeceeneeeeseseeeeestisesustntenuneeeeeenaneeeetenesngs,AOUO3SNOX!!YOLITUICeoJOVIUSINI|!SING;|bb!YOSSIIOUOHOIN|!wsiq|||eaeeececaeeeeceeeeeeeeeeeeeceeed|et:MOOeeepee
`
`
`—¢OlASIMuC
`P952g3HOVO|_NIVTOUINOD|AYOWIWNIV|ogy|WOHOL|wteC(©‘D14)
`ySIT|wobaorNOLLIISNddA!SFOUNOSTY!GIYVHS
`
`
` _Su|TOHLNODwvus||!t'$Yl''iiOetNTowedV1HOd>||(+|4300930Es|140d|inda|JOHINOD_—_+¥__+TOXLNOD}cb|;105g@140dg30q7aW3|!1TOUINOD|!||pTOHINOD|||6|!'1[yoss3a0ud)MVE||:|!1“O8DIN!gs|a|18%!!ya|AHOWaW|tCygongyLONFOUN
`
`
`
`
`
`Jun. 17, 2003
`
`Sheet 2 of 14
`
`WOus/0L
`
`FHIVO
`
`1AYOWIN
`
`
`

`

`U.S. Patent
`
`US 6,581,137 BI
`
`Jun. 17, 2003
`
`Sheet 3 of 14
`
`7-iiBNIWSHANETY:|HOSSIOOUdOUOINvlVG|i|JOUINODProvivd:|wous/OL
`€Ol' a9_-]9i8Soul<4140d'|140d18G09Sp===2=222225=--513HITIOHINODwus1OUGiGgjaNi|}|i05woqvuomiasMivdTd-YIG0950—7OWINOD_!TES}}Add
`sDUINGD|FOVsHLNABO|Nog|ttNIV|eI::AYOWSIN|!NIVLtHITIOULNOD!Wou/01|ANOWINN267Ai
`Wous/0L*og7NGD)|[YNOXI
`
`Nido1+WOH09||JOULNOD
`YouuIHD1YAIX|+3140d1q
`
`!‘woqviviva|Pe!!UAIX'|TOHINOD
`
`
`xuowan15!|JOWIUEINI|beeeneeeeeeeeeee(p“Dld)ZOWAUFININIVYOSSIIOUdOHIIW|
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 17, 2003
`
`Sheet 4 of 14
`
`US 6,581,137 BI
`
`<0'5¢>
`
`doneg940d
`
`ccHA
`
`
`<0''SE>
`
`
`
`donedVikOd<o"LZ>donegdNdg
`
`<}-—dugooned9Luod|480003{|duzooned
`¥1YOd{wou
`———anornagP—_|9%Bossz00udOuOIMOL
`dupnomedviuod\,#300930
`
`Na'dyadWoud
`dasinndaWows
`dupyOMedG§lHOd05
`
`
`djagouAgyegg§4Od
`
`djagoucgieqYLuOd
`dupyoTMedNdd
`dagoudsiegNd
`9805
`alas|/<0"'12>eyequiau'OZova7OL
`0¢§¥300030Ndd
`
`

`

`U.S. Patent
`
`Jun. 17, 2003
`
`Sheet 5 of 14
`
`US 6,581,137 BI
`
`eeey
`
`djagoulsieqNdd
`
`dupyonurednda
`
`dupyoTuedNdd
`
`
`
`b-aé‘Old
`
`
`
`dupouegdVLYOd
`
`
`
`dupyouredgluOd
`
`
`
`
`
`djagoudgyeg¥LYOd4300930
`<Q’2>djasguogiegdNddNous
`
`
`
`dyasoudsiegalyOd05
`
`"ACi£6(=tg
`£6=9g
`
`£6&g
`
`YSOld
`
`OlL|(—8g
`£64g
`
`OVOFWOly:
`
`
`<0'1L>BeQuiayy:
`029vGIWOHI
`
`YOSSFIONJOYIINOL
`
`
`
`<Q"11>doyegNdd
`
`0S
`
`dujooyegdd
`4900030Ndd
`
`Wout
`
`olds
`
`£6(=cg
`£6lg
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 17, 2003
`
`Sheet 6 of 14
`
`
`
` dopedaud!'<|al—!'_Sal:0000_M.$€+}\:|HHIU.GE3!66“o'bg”|reT!<OLEP9'L9>BedaunJOSSAIOIMOIONIYWOl4inare\<ZE°E9'89"L/>WYYS
`JPUJaJUYWOJYHPy:!uigy,geLHa!_!<0"9E>BEGaMal4OdWoly—!
`core>Ji.rD3<O"1E'P9'L9>WYYSJeusayu]Wold
`
`
`<o'Ge>BedaMVleOdWold!<ZE"E9'99"L>RIEAMMJOSSIIOICOSOIYWOl4
`
`N69
`
`FOOUNIX!
`
`US 6,581,137 BI
`
`2-98“Id
`
`!101
`
`S)
`
`SIOIGIOISICISTIS
`
`
`

`

`U.S. Patent
`
`Jun. 17, 2003
`
`Sheet 7 of 14
`
`US 6,581,137 BI
`
`
`<0"GE>
`
`Gipyyuod
`
`beeee3409HNOX|!N¢9
`
`QUIOOLPYV10dWOUdqug00}bYgulddvaq0odddC|
`
`
`
`
`
`
`
`dipyguod
`<0''GE>
`
`Zit
`
`61
`
`/gnoxndg.Wh!
`5oldsjt
`
`bg
`
`(i<0"s6>bpyddSYOSSII0UdOUSINWOU
`
`eweeeeeeeeeeeeeeeeeeeeeeeerweeeeeeeeeeeeee
`
`Je“Sl4
`
`dus7WOud
`
`auyovYHq¥39G0030Ndd
`dyagouAsupyNd46
`
`
`
`

`

`U.S. Patent
`
`US 6,581,137 BI
`
`Jun. 17, 2003
`
`Sheet 8 of 14
`
`00000420ititBxosalBrie_pr9ilt1Sh50000
`
`
` CS}2ble169|oo9g1420iiESO|_Prle9\444sy0ToprUph|=<.0‘fol!!Or‘Wy¥as010]|
`ae“Bld0Ta]|010]
`
`
`(£2-1€SUGONUVONNYLHALSSquaayUYLINSHDIY(upe210z80)
`'\60¢>Lyey;0+10}||iTz]:jane!0'/o}0‘lol:!90!0cle0‘lal0(OhiRRmj{EtPRrH01{QO}i>0|)&gHH1g!30110}|15=0|9)xRi}yi18!2jfifitSsLL
`0420fme90¢
`
`
`
`
`
`
`
`(peg!40h(pee!§=—(W000.ai40)9dZ0)(upeztgogo)«4000d4Z0)
`SsauaayGiddy07GIISYNySYW=SSSNaY
`
`p|ole“|G:lerg7"602
`
`
`TWNWHOSININDIS=NIWZINTJS=Mbynd)
`
`0s)0S9ININDIS
`
`
`

`

`Jun. 17, 2003
`
`Sheet 9 of 14
`
`U.S. Patent
`
`US 6,581,137 BI
`
`WwoHs\awiopuAss:|ssPeduggWOu-/OL
`
`mx|ai=TTdent\usaooaa
`Ndif&PeOLLtnd9
`
`
`ejeq/SSaJppyi,duggag/Olig|~PLPpous]ajoquogpue=fitCd<0">OUDidisvajsibay
`
`wegByuog:|=+44<0aeus!\a[>DEU0S
`gySyBaoT“ag
`
`qapi=>anned;4<a1Z>a0nedNdd|0soRLE
`! 3S|NidOL:yasoy
`
`HlosONUO)adepajuya/Ouy/L
`J)ema[=$—it
`
`
`gocheyUa~RSsoBpiaquyLoweywey|=
`-t-.nao
`i)@TTy901)
`
`
`

`

`U.S. Patent
`
`Jun. 17, 2003
`
`Sheet 10 of 14
`
`US 6,581,137 B1
`
`NQ</1..0>
`
`s
`
`S
`Ss
`SSu[KkBas|Se
`==
`=
`sé
`
`FIG. 4B
`
`

`

`U.S. Patent
`
`Jun. 17, 2003
`
`Sheet 11 of 14
`
`US 6,581,137 BI
`
`0S4300930NddOL
`
`
`
`YOuALladSUGNOFHO
`
`NIVIVO-|wvayod
`YOIVYINIDra.:LI9NOFHO‘
`
`
`vivdYOs>Wha!!
`
`youuOl901¢D4
`dIHOAlla
`N3980“igYou\99HOXAYOxNiviva
`zl”211-Lg
`yOuviva¥Os
`
`
`0.YvdNIDyoaHo
`
`JWOUGNAS
`
`youuy
`
`YdNID
`
`YdMHI
`
`6
`
`LIGMOFHD
`
`YOIVUINIO
`
`xe)<oan”
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 17, 2003
`
`Sheet 12 of 14
`
`US 6,581,137 B1
`
`Interrupt
`Controller
`
`327
`
`325
`
`IRQnp<4>
`(RQnp<3> C
`32
`IRQnp<2> Ohanasoe
`IROnp<1> otiastoe
`Lb
`IRQnp<0> qe 32 ;
`esi
`peat
`f=
`yee
`f=
`et}—
`|nu}—|
`
`
`
`FIG. 6A
`
`329
`
`Source Mask
`
`523
`
`312
`
`304
`
`Current
`Interrupts
`REGS
`32
`
`REG Section
`
`32
`Interrupt
`Invert REG
`Section
`
`AOOO ©
`
`

`

`U.S. Patent
`
`Jun. 17, 2003
`
`Sheet 13 of 14
`
`US 6,581,137 B1
`
`°
`
`
`Fatal Maskar
`“|
`Asic Fatal
`Fatal Intlop<3..0>
`PULSE
`
`From
`Xcore
`Sel,83 SELECT
`
`Watchdog
`30;
`
`DATA
`
`Watchdog IRQ
`
`From
`Xcore} nara
`Sel, 83
`
`
`
`Match
`RO
`
`ADDR
`
`SELECT
`
`
`393
`
`FIG. 6B
`
`or
`el
`SSsx
`S35 Vv
`Sf
`S
`g
`
`cL
`
`8SS
`
`=
`
`
`
`

`

`U.S. Patent
`
`Jun. 17, 2003
`
`Sheet 14 of 14
`
`US 6,581,137 B1
`
`IRQ
`
`| INTERRUPT_TYPE REG
`
`1 INTERRUPT_TYPE
`
`FIG. 9
`
`

`

`US 6,581,137 B1
`
`1
`DATA STORAGE SYSTEM
`
`BACKGROUND OF THE INVENTION
`
`2
`bank of disk drives as such data passes through the memory;
`and a plurality of busses in communication with the direc-
`tors and the memory. Each one of the directors includes a
`central processing unit. The central processing unit includes:
`(A) a microprocessor; (B) a main memory; and (C) a
`microprocessor interface. The microprocessor interface
`includes a semiconductor integrated circuit having formed
`therein: (i) a data rebuffering section disposed in the chip
`and adapted to couple data from a one ofa plurality of data
`ports to a data port of the microprocessor selectively in
`accordance with a control signal; and (ii) a main memory
`interface adapted for coupling to a main memory for the
`microprocessor, such main memory interface being adapted
`for coupling to the microprocessor and being coupled to the
`data rebuffering section for providing control signals to the
`main memorysection for enabling data transfer between the
`main memory and the microprocessor through the data
`rebuffering section. A controller is coupled to the data
`rebuffering section for producing the control signal.
`In one embodimentof the invention, the central process-
`ing unit main memory is a selected one of a plurality of
`memory types each type having a different data transfer
`protocol and the main memory interface is configured in
`accordance with the selected one of the plurality of memory
`types to provide a proper memory protocol to data being
`transferred between the microprocessor and the main
`memory through the main memory interface.
`In accordance with one embodimentof the invention, one
`main memory type is an SDRAM andin another type is a
`RDRAM.
`
`In accordance with another feature of the invention, the
`central processing unit data rebuffering section includes: a
`selector responsive to the control signal for coupled data
`between a selected one of the data ports and the micropro-
`cessor.
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`60
`
`65
`
`This invention relates generally to data storage systems,
`and more particularly to data storage systems having redun-
`dancy arrangementsto protect against total system failure in
`the event of a failure in a component or subassembly of the
`storage system.
`As is knownin the art, large mainframe computer systems
`require large capacity data storage systems. These large
`main frame computer systems generally include data pro-
`cessors which perform many operations on data introduced
`to the computer system through peripherals including the
`data storage system. The results of these operations are
`output to peripherals, including the storage system.
`Onctype of data storage system is a magnetic disk storage
`system. Here a bank of disk drives and the main frame
`computer system are coupled together through an interface.
`The interface includes CPU,or “front end”, controllers (or
`directors) and “back end”disk controllers (or directors). The
`interface operates the controllers (or directors) in such a way
`that they are transparent to the computer. That is, data is
`stored in, and retrieved from, the bank of disk drives in such
`a waythat the mainframe computer system merely thinksit
`is operating with one mainframe memory. One such system
`is described in U.S. Pat. No. 5,206,939, entitled “System and
`Method for Disk Mapping and Data Retrieval’, inventors
`Moshe Yanai, Natan Vishlitzky, Bruno Alterescu and Daniel
`Castel,
`issued Apr. 27, 1993, and assigned to the same
`assignee as the present invention.
`As described in such U.S. Patent, the interface may also
`include,in addition to the CPU controllers (or directors) and
`disk controllers (or directors), addressable cache memories.
`The cache memory is a semiconductor memory and is
`providedto rapidly store data from the main frame computer
`In one embodiment, the data rebuffering section includes
`system before storage in the disk drives, and, on the other
`a data distribution unit having a plurality of ports each one
`hand, store data from the disk drives prior to being sent to
`of the ports being coupled to a corresponding oneof: (i) the
`the main frame computer. The cache memory being a
`selector; (ii) a random access memory; (iii) an interrupt
`semiconductor memory, as distinguished from a magnetic
`request controller; (iv) the microprocessor data port; and (v)
`memory as in the case of the disk drives, is muchfaster than
`the main memory interface.
`the disk drives in reading and writing data.
`In accordance with another feature Of the invention, a
`The CPU controllers, disk controllers and cache memory
`data storage system is provided wherein a host computeris
`are interconnected through a backplane printed circuit
`in communication with a bank of disk drives through an
`board. More particularly, disk controllers are mounted on
`interface. ‘The interface includes: a memory; a plurality of
`disk controller printed circuit boards. CPU controllers are
`directors for controlling data transfer between the host
`mounted on CPU controller printed circuit boards. And,
`computer and the bank of disk drives as such data passes
`cache memories are mounted on cache memory printed
`through the memory; and a plurality of busses in commu-
`circuit boards. The disk controller, CPU controller and cache
`nication with the directors and the memory. Each oneof the
`memory printed circuit boards plug into the backplane
`directors includes a central processing unit. The central
`printed circuit board. In order to provide data integrity in
`processing unit includes: (A) a microprocessor; (B) a main
`case ofa failure in a controller, the backplane printed circuit
`memory having a plurality of data storage sections, one
`board has a pair of buses. One set the disk controllers is
`section havingafirst set of addresses and a second section
`55
`connected to one bus and another set of the disk controllers
`having addresses a second set of addresses; (C) a micropro-
`is connected to the other bus. Likewise, one set the CPU
`cessor interface. The interface includes: (i) a memory con-
`controllers is connected to one bus and another set of the
`troller for producing addresses for the main memory, such
`CPU controllers is connected to the other bus. The cache
`memory controller having a decoder responsive to the
`memories are connected to both buses. Each oneof the buses
`produced addressed to determine whether the produced
`address is within the first set or the second set of addresses;
`and (ii) a main memory interface adapted for coupling to a
`main memory for the microprocessor, such main memory
`interface being adapted for coupling to the microprocessor
`and being coupled to the data rebuffering section for pro-
`viding control signals to the main memory section for
`enabling data transfer between the main memory and the
`microprocessor through the data rebuffering section. A con-
`
`provides data, address and control information.
`SUMMARY OF THE INVENTION
`
`In accordance with the present invention, a data storage
`system is provided wherein a host computer is in commu-
`nication with a bank of disk drives through an interface. The
`interface includes: a memory; a plurality of directors for
`controlling data transfer between the host computer and the
`
`

`

`US 6,581,137 B1
`
`3
`troller is responsive to the decoder, for enabling the second
`section in the memory when the decoder determines the
`produced addressis in the second set of addresses. The first
`section is enabled for addressing by the produced address
`when the decoder determines the produced addressis in the
`first set of addresses.
`
`In one embodiment, the system includes a maskto trans-
`form the address to an address in the second section of the
`memory.
`
`BRIEF DESCRIPTION OF THE DRAWING
`
`‘These and other features of the invention will become
`
`more readily apparent from the following detailed descrip-
`tion when read together with the accompanying drawings,in
`which:
`
`FIG. 1 is a block diagram of a data storage system
`according to the invention;
`FIG. 2 is a block diagram of an exemplary one of a
`plurality of directors used in the system of FIG. 1, such
`director having a central processing unit in accordance with
`the invention;
`FIG. 3 is a block diagram of a microprocessor interface
`used in the central processing unit of the director of FIG. 2;
`FIGS. 3A through 3C are block diagrams of an XCVR
`core and a CPU XCVRusedin the microprocessorinterface
`of FIG. 3, FIG. 3A showing the data rebuffering mechanism
`which supports microprocessor read operations, FIG. 3B
`showing the data rebuffering mechanism that supports
`microprocessor read operations, and FIG. 3C showing data
`rebuffering mechanism that supports the microprocessor
`address path;
`FIG. 3D is a diagram useful in understanding an address-
`ing feature provided by the microprocessorinterface of FIG.
`2 in accordance with the invention;
`FIG. 4 is a block diagram of a main memory interface
`according to the invention used in the microprocessor inter-
`face of FIG. 3;
`FIG. 5 is a block diagram of an error detector and
`corrector according to the invention used in the main
`memory interface of FIG. 4;
`FIG. 5A is a block diagram of an XOR arrangement used
`in the error detector and corrector of ['IG. 5;
`FIG. 6 is a block diagram of an interrupt request controller
`used in the microprocessor interface of FIG, 3;
`FIG. 7 is a block diagram of an interrupt inverter register
`used in the interrupt controller of FIG. 6;
`FIG. 8 is a block diagram ofan interrupt type register used
`in the interrupt request controller of FIG. 6; and
`FIG. 9 is a diagram of a fault detector adapted to detect
`hard faults on a bi-directional data line according to the
`invention.
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENTS
`
`DATA STORAGE SYSTEM
`
`Referring now to FIG. 1, a data storage system 10 is
`shown wherein a host computer 12 is coupled to a bank 14
`of disk drives through a system interface 16. The system
`interface 16 includes a cache memory 18. A plurality of
`directors 20,-20,,; is provided for controlling data transfer
`between the host computer 12 and the bank 14 ofdisk drives
`as such data passes through the cache memory 18. A pair of
`high address busses TH, BHis electrically connected to the
`
`10
`
`15
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`high address memory section 18H of cache memory 18 as
`described in U.S. patent application. Ser. No. 09/223,115
`entitled “Data Storage System”, inventors D. Castel et al,
`filed Dec. 30, 1998, assigned to the same assignee as the
`present invention,
`the entire subject matter thereof being
`incorporated into this application by reference. A pair of low
`address busses TL, BL is electrically connected to the low
`address memory section 18L of cache memory 18. The
`cache memory 18 has a plurality of storage location
`addresses. Here,
`the storage locations having the higher
`addresses are in the high address memory sections 18H and
`the storage locations having the lower addresses are in the
`low address memory sections 18L. It should be noted that
`each oneofthe directors 20,—20, , is electrically connected
`to one of the pair of high address busses TH, BH and one of
`the pair of low address busses TL, BL. Thus, each oneof the
`directors 20,-20,; is able to address all locations in the
`entire cache memory 18 (ie.,
`to both the high address
`memory sections 18H and the low address memory sections
`18L) and is therefore able to store data in and retrieve dala
`from any storage location in the entire cache memory 18.
`Moreparticularly, a rear-end portion of the directors, here
`directors 20.-20,,is electrically connected to the bank 14 of
`disk drives and a front-end portion of the directors, here
`directors 20,-20,5,
`is electrically connected to the host
`computer 12.
`In operation, when the hast computer 12 wishes to store
`data, the host computer 12 issues a write request to one of
`the front-end directors 20,-20,, to perform a write com-
`mand. One of the front-end directors 20,—20,, replies to the
`request and asks the host computer 12 for the data. After the
`request has passed to the requesting one of the front-end
`directors 20,-20,;, the director determines the size of the
`data and reserves space in the cache memory18to store the
`request. The front-end director then produces control signals
`on either a high address memory bus (TH or BH) or a low
`address memory bus (TL, BL) connected to such front-end
`director depending on the location in the cache memory 18
`allocated to store the data and enables the transfer to the
`cache memory 18. The host computer 12 then transfers the
`data to the front-end director. The front-end director then
`
`advises the host computer 12 that the transfer is complete.
`The front-end director looks up in a Table, not shown,stored
`in the cache memory 18 to determine which one of the
`rear-end directors 20,—20, is to handle this request. ‘The
`Table maps the host computer 12 address into an address in
`the bank 14 of disk drives. The front-end director then puts
`a notification in a “mail box” (not shown and stored in the
`cache memory 18) for the rear-end director which is to
`handle the request, the amount of the data and the disk
`address for the data. Other rear-end directors poll the cache
`memory 18 when they are idle to check their “mail boxes”.
`If the polled “mail box” indicates a transfer is to be made,
`the rear-end director processes the request, addresses the
`disk drive in the bank, reads the data from the cache memory
`and writes it into the addresses ofa disk drive in the bank 14.
`When data is to be read from the disk drive to the host
`
`computer 12 the system operates in a reciprocal manner.
`DIRECTOR
`
`Referring now to FIG. 2, an exemplary one of the
`directors, here director 20,, is shown. The director 20, has
`an X CPU section 22 and a Y CPU section 24 which share
`shared resources 40 such as flash memories, etc. The flash
`memory stores the BIOS, or boot-up routine, for the CPU
`sections 22, 24. The X and Y CPU sections 22, 24 are
`identical in construction, the X CPU section 22 being shown
`
`

`

`US 6,581,137 B1
`
`5
`in more detail in FIG. 2. Here, the X CPU section 22 shown
`is a rear end director and thusis coupled to the disk drives
`14 (FIG. 1), it being understood that had such section been
`in a front end director such X CPU section 22 would have
`been connected to the host computer 12. The X CPUsection
`22 is also coupled to the cache memory 18 (FIG. 1), as
`indicated.
`
`Referring in more detail to the X CPU section 22, it is
`noted that such section 22 includes a Direct Memory Access
`(DMA)section 42 which is an interface between the cache
`memory 18 (FIG. 1), the bank of disk drives 14 and the
`central processing unit 44 of the X CPU section 22. The
`central processing unit 44 includes a microprocessor 46,
`here a Power PC microprocessor, a main memory 48, a CPU
`decoder 50 (e.g., a programmable logic device), and a
`microprocessor interface 52, here an Application Specific
`Integrated Circuit (ASIC). The microprocessor interface 52
`will be described in more detail in connection with FIG. 3.
`
`10
`
`15
`
`the microprocessor
`that
`to say here, however,
`Suffice it
`interface 52 is a comprehensive Power PC microprocessor
`support integrated circuit chip having several discrete func-
`tional sections, including a main memoryinterface 54 hav-
`ing a set of registers 53, a data rebuffering section 56, an
`interrupt request (IRQ) controller 58, and an embedded
`Static Random Access Memory (SRAM)60. Herc, the main
`memory 48 is an SDPAM,however, as will be described,
`other types of memories may be used such as a PIBUS
`DRAM (RDRAM).
`Here, the main memory interface 54 is adapted to manage
`one or two banks of main memory 48 SDRAMs,providing
`up to 128 MB of memoryI/O space using 64-megabit RAM
`densities. The data is Error Correction Code (ECC)—
`protected and single-bit errors can be corrected as they are
`detected. The main memory SDPAM interface 54 fully
`supports byte, half-word and word reads and writes through
`built in Read-Modify-Write cycles.
`Theinterrupt request (IRQ) controller 58 provides flexible
`interrupt management. Here, the interrupt request controller
`58 supports up to 28 external interrupts and 4 fatal interrupts
`as well as internal
`interrupts. These interrupts can be
`assigned to any interrupt level by programming level and
`mask registers which will be described in more detail in
`connection with FIG. 6. In addition, the interrupt request
`controller provides facilities to assist in Motorola 68060-
`style interrupt generation as will also be described in more
`detail in connection with FIG. 6.
`
`The data rebuffering section 56 of the microprocessor
`interface 52 provides dual-bus address and data rebuffering.
`Fully flexible and controlled by the external CPU decoder
`50,
`the address and data paths are rebuffered to enable
`connection to local and shared resources 58. The options of
`registering data, assembly/disassembly function, and parity
`generation are controlled by the logic in the external CPJ
`decoder 50.
`
`The microprocessorinterface 52 also providesfacilities to
`capture address and data ranges, and to provide interrupts
`upon capture. This capability is useful for debugging opera-
`tions to trap rouge address or data cycles.
`CENTRAL PROCESSING UNIT 44
`
`Referring now also to FIGS. 2 and 3, the central process-
`ing unit 44 (FIG. 2) of the director 20, includes, as noted
`above, a microprocessor 46; a main memory 48; a CPU
`decoder 50; and a microprocessor interface 52, here an
`ASIC. The data rebuffering section 56 of the microprocessor
`interface 52 is adapted to couple data from a one of a
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`plurality of data ports, here data port A, data port B, and data
`from the embedded SRAM 60 and interrupt request con-
`troller 58, to a data port of the microprocessor 46 selectively
`in accordance with a control signal supplied by the CPU
`decoder 50. The main memory interface 54 is adapted for
`providing control signals to the main memory48 section for
`enabling data transfer between the main memory 48 and the
`microprocessor 50 through the data rebuffering section 56.
`As noted above, the main memory 48 is a selected one of
`a plurality of memory types, such as an SDRAM or an
`RDRAM. Each memory type has a different data transfer
`protocol. The main memory interface 54 is configured in
`accordance with the selected one of the plurality of memory
`types to provide a proper memory protocol to data being
`transferred between the microprocessor 46 and the main
`memory 48 through the main memoryinterface 54. As noted
`above, one main memory type is an SDRAM andanother
`main memorytype is a RDRAM,it being understood that
`other types may also be used.
`Referring now also to FIG. 3, the main memory interface
`54 is shown in more detail to include a main memory API
`controller 64 and a microprocessor memory interface
`control/EDAC section 66. The data to and from the main
`memory 48 passes through the EDACportion 70 of section
`66, such EDAC portion being described in more detail in
`connection with FIG. 5. The main memory APDIcontroller
`64 is responsible for translating upper-level read, write, and
`refresh requests into the appropriate low-Level control sig-
`nals for the type of main memory48 being used. Thatis, the
`data to and from the main memory 48 is through the main
`memory API controller 64 in response to control signals
`supplied by the microprocessor memoryinterface control/
`EDACsection 66. The control signals are generic, that is
`they are independent of the type of main memory 48 being
`used.
`
`The main memory API control 64 is hard-wireda priori to
`translate the generic control signals into the proper protocol
`for the particular type of main memory48, i.c., SDRAM or
`RDRAM,etc. If for example, the original application for the
`CPUcontroller 44 is with an SDRAM,and at some future
`time the application will use an RDRAM,the only portion
`of the microprocessor interface 52 which must be reconfig-
`ured is the main memory API controller 64; the design of the
`microprocessor memory interface control/EDAC section 66
`may remain unchanged.
`the microprocessor memory
`Referring also to FIG. 4,
`interface 66 has three sections;
`the EDAC section 70, a
`memory interface control section 72 and an address/data
`register and control section 74. The microprocessor memory
`interface 66 manages the interface between the micropro-
`cessor 46 and the main memory API controller 64.
`It
`performs the required address decoding, burst address
`counter, Opcode generation, refresh generation, and main
`memory API controller 64/micrcprocessor 46 synchroniza-
`tion.
`
`The address/data register and control section 74 contains
`the necessary input and output registers to properly align the
`microprocessor 46 data stream with the main memory 48
`data stream after it has passed through the EDACsection 70.
`The EDACsection 70 is described in more detail below
`in connection with FIG. 5.
`Various control and data lines and busses are shown in
`FIG. 4:
`
`CPU_DatlOp<71.0>
`DBBnp
`CPU__Adrp<35.0>
`
`

`

`US 6,581,137 B1
`
`TSnop
`ABBnp
`TT1p
`TIT3p
`TBSTnp
`TSIZp<2.0>
`AACKnp
`TAnp
`which are defined in the Motorola MIPC 750 microproces-
`sor Manual;
`Clock
`Reset and
`Pulse
`
`which are system inputs;
`the following EDACsignals:
`SBE—-single bit error indication
`MBE—nultiple bit error indication
`PErr—parity error indication
`Syndrome—XORreduced syndromebits
`Err_Cnt—error count (number of EEACerrors detected
`during a datatransfer)
`Config__Data—indicates where the EDACiseither an error
`detect or error correct mode and whether the EDAC is in
`even or odd parity;
`the following are defined by Rambus Application Guide
`(www.rambus.com):
`WD
`
`Op
`Start
`
`and the following which are conventional SDRAM interface
`signals:
`DQ<71.0>
`Adr<13.0>
`CSn
`RASn
`CASn
`WEn
`DQM
`CKE
`
`Referring again to FIG. 3, the data rebuffering section 56
`1s responsive to a control signal from the CPU decoder 50,
`for coupling data between a selected one of the data ports,
`Le., port A, port B, or the embedded SRAM orthe interrupt
`request controller 58, and the data port of the microprocessor
`48. More particularly, the data rebuffering section 56 has a
`port A transceiver (XCVR), a port B XCVR 82, an XCVR
`core 83, and a CPU XCVR84,arrangedas indicated in FIG.
`3. The XCVRcore 83 is a selector which, in response to a
`control signal from the CPU decoder 50, couples data/
`address between port 86 of the CPU XCVR 84 andeither:
`the port A XCVR 80;or, the port B XCVR 82; the embedded
`SRAM 60,or the interrupt request controller 58 selectively
`
`8
`in accordance with a control signal fed to the XCVR core 83
`from the CPU decoder 50. (As noted from FIG. 2, here the
`port A XCVRis connected at port A to the cache memory 18
`(FIG. 1) through the DMA 42 and the port B XCVR 82 is
`coupled at port B to the shared resources 40.
`The CPU XCVRis a data distribution unit having a
`plurality of ports each one of the ports being coupled to a
`corresponding one of: (i) the XCVR (selector) 83; (ii) the
`Synchronous DRAM 69;(iii) the interrupt requestcontroller
`58; (iv) the microprocessor 46 data port; and (v) the main
`memory interface 54.
`Moreparticularly, the XCVR core 83 and CPU XCVR 84
`are shownin FIGS. 3A, 3B, and 3C. The data rebuffering
`section 56 mechanism that supports microprocessor write
`operations is shown in FIG. 3A. The here 72 bit data from
`the microprocessor 46 data transfer bus transfers to the
`microprocessorinterface 52 at the CPU__DatalIOpinterface.
`The microprocessor interface 52 has two registers 87, 89,
`one for the upper data word and onefor the lower data word,
`respectively. The CPU_DatWUCIkEnp and DatWLClkEnp
`are used to enable data registering into the upper and lower
`word lanes on the rising edge of a clock, not shown,
`respectively. Parity is stored along with the corresponding
`word’s data. CPU_DatSyncSelp, when clear, causes the
`input registers 87, 89 to be by-passed providing an asyn-
`chronous data path to Port A and Port EB. CPU_ULSelp
`determines whether th