`
`
`
`UNIFIED PATENTS
`
`EXHIBIT 1001
`
`UNIFIED PATENTS
`
`EXHIBIT 1001
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 1
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 1
`
`

`

`US007197662B2
`
`United States Patent
`(12)
`(10) Patent No.:
`US 7,197,662 B2
`
`Bullen et a].
`(45) Date of Patent:
`Mar. 27, 2007
`
`(54) METHODS AND SYSTEMS FOR A STORAGE
`SYSTEM
`
`(75)
`
`Inventors: Melvin James Bullen, Reston, VA
`(US); Steven Louis Dodd, Reston, VA
`(US); William Thomas Lynch, Apex,
`NC (US); David James Herbison,
`Arvada, CO (US)
`
`(73) Assignee: Ring Technology Enterprises, LLC,
`Reston, VA (US)
`
`4,538,174 A
`4,646,270 A
`4,672,602 A
`4,709,418 A
`4,763,317 A
`4,790,418 A
`4,796,231 A
`4,980,857 A
`4,995,078 A
`5,003,591 A
`
`8/1985 Gargini et a1.
`2/1987 Voss
`6/1987 Hargrave et a1.
`11/1987 Fox et al.
`8/1988 Lehman et al.
`12/1988 Brown et a1.
`1/1989 Pinkham
`12/1990 Walter et a1.
`2/1991 Monslow et a1.
`3/1991 Kauffman et al,
`
`( * ) Notice:
`
`Subject. to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 866 days.
`
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`(21) Appl. No.: 10/284,199
`
`(22)
`
`Filed:
`
`Oct. 31, 2002
`
`W0 W0 2004/025476 A1
`
`3/2004
`
`OTHER PUBLICATIONS
`
`(65)
`
`Prior Publication Data
`
`US 2004/0088393 A1
`
`May 6, 2004
`
`_
`_
`_
`_
`Farley, Marc, “Building Storage Networks, Second Edition”,
`Osbourne/McGraw-Hill, 2001, entire book.
`
`(51)
`
`Int. Cl.
`(200601)
`G06F 1”00
`(52) US. Cl.
`................................. 714/6; 714/7; 714/42
`(58) Field of Classification Search .................... 714/6,
`714/7, 42
`See application file for complete search history.
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`3,713,096 A
`3,735,362 A
`3,748,647 A
`3,812,476 A
`4,064,556 A
`4,065,756 A
`4,193,121 A
`4,302,632 A
`4,334,305 A
`4,363,125 A
`4,506,387 A
`4,510,599 A
`
`1/1973 Comfort et a1.
`5/1973 Ashany et a1.
`7/1973 Ashany et a1.
`5/1974 Cragon
`12/1977 Edelberg et a1.
`12/1977 Panigrahi
`3/1980 Fedida et a1.
`11/1981 Vicari et a1.
`6/1982 Girardi
`12/1982 Brewer et a1.
`3/1985 Walter
`4/1985 Ulug
`
`(Continued)
`Primary ExamineriMichael Maskulinski
`(74) Attorney, Agent, or FirmiFinnegan, Henderson,
`Farabow, Garrett & Dunner, L.L.P.
`
`(57)
`
`ABSTRACT
`
`A storage system that may include one or more memory
`sections, one or more switches, and a management system.
`The memory sections include memory devices and a section
`controller capable of detecting faults with the memory
`section and transmitting messages to the management sys-
`tem regarding detected faults. The storage system may
`include a management system capable of receiving fault
`messages from the section controllers and removing from,
`service the faulty memory sections. Additionally, the man-
`agement system may determine routing algorithms for the
`one or more switches.
`
`21 Claims, 15 Drawing Sheets
`
`MIT
`
`
`Mmagemenl
`‘
`Switch J
`
` Complex
`,1
`
`
`Seclion
`mmmller
`
`
`
`
`ToSeNeIS(12)
`
`
`AT ,.__..._.A
`H22
`
`
`Swim/Sewer If:
`
`
`
`
`I Imama]:
`
`
`
`:15?
`
`
`
`
`
`
`
`
`
`
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 2
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 2
`
`

`

`US 7,197,662 B2
`
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`5/1991 Pocock et al.
`5,014,125 A
`6/1991 Baji et a1.
`5,027,400 A
`10/1991 Lindstrom
`5,060,068 A
`10/1991 Youngblood et al.
`5,062,059 A
`1/1992 Young
`5,084,839 A
`6/1992 Frank et a1.
`5,119,481 A
`7/1992 Tindell et a1.
`5,130,792 A
`7/1992 Yurt et a1.
`5,132,992 A
`7/1992 Ballantyne et al.
`5,133,079 A
`10/1992 Lucak et a1.
`5,153,884 A
`11/1992 Heilveil et a1.
`5,163,024 A
`3/1993 McCalley et a1.
`5,191,410 A
`4/1993 Morouka et al.
`5,200,925 A
`9/1993 Litteral et a1.
`5,247,347 A
`10/1993 Rozmanith et al.
`5,253,341 A
`11/1993 Raasch et a1.
`5,261,114 A
`11/1994 Nelson et al.
`5,369,784 A
`12/1994 Gelman et a1.
`5,371,532 A
`12/1994 Flohr
`5,374,952 A
`3/1995 Lucak et a1.
`5,400,331 A
`9/1996 McLaughlin et al.
`5,553,311 A
`12/1996 McLaughlin et a1.
`5,581,479 A
`2/1997 McLaughlin et al.
`5,604,682 A
`6/1997 McLaughlin et al.
`5,636,139 A
`3/1999 Lopez et al.
`5,883,831 A
`6/1999 Perino et al.
`5,908,333 A
`6/1999 Alexander et al.
`5,909,564 A
`9/1999 Farmwald et al.
`5,953,263 A
`9/1999 Farmwald et al.
`5,954,804 A
`10/1999 Wentka et a1.
`5,968,114 A
`11/1999 Pomet et a1.
`5,978,295 A
`2/2000 Farmwald et al.
`6,032,214 A
`3/2000 Farmwald et al.
`6,034,918 A
`2/2001 Farmwald et al.
`6,185,644 B1
`11/2001 Kobayashi
`6,317,377 B1
`3/2002 McLaughlin et a1.
`6,356,973 B1
`3/2002 Barth et a1.
`6,356,975 B1
`12/2002 Matsudera et al.
`6,498,741 B2
`7/2003 Olarig et a1.
`............... 710/302
`6,587,909 B1 *
`1/2004 Piccirillo et a1.
`........... 711/106
`6,684,292 B2 *
`2/2004 Chang et a1.
`6,697,368 B2
`4/2004 Perner et al.
`6,728,799 B1
`4/2005 Lynch et a1.
`6,879,526 B2
`6,981,173 B2 * 12/2005 Ferguson et al.
`2003/0018930 A1
`1/2003 Mora et a1.
`2003/0187945 A1
`10/2003 Lubbers et a1.
`2004/0044744 A1
`3/2004 Grosner et al.
`2004/0168101 A1*
`8/2004 Kubo ............................ 714/6
`
`.............. 714/5
`
`OTHER PUBLICATIONS
`
`Clark, Tom, “Designing Storage Area NetworksiA Practical Ref-
`erence for Implementing Fibre Channel SANs”, Addison-Wesley,
`1999.
`US. Appl. No. 10/284,198 including Specification, Claims and
`Figures.
`U.S. Appl. No. 10/284,278 including Specification, Claims and
`Figures.
`U.S. Appl. No. 10/284,268 including Specification, Claims and
`Figures.
`Malavalli, Kumar, “Fibre Channel Classes of Service for Data
`Transport,” Brocade Communications Services,
`Inc. 1997,
`15
`pages.
`“HyperTransportTM I/O Link Specification,” HyperTransport Tech-
`nology Consortium, Rev. 1.03, Oct. 10, 2001, 217 pages.
`Tran, Jennifer, “Synthesizable 1.6 Gbytes/s DDR SRAM Control-
`ler” Xilinx Application Note XAPP200, Mar. 21, 2000, 12 pages.
`“Using Block SelectRAM+ Memory in Spartan-II FPGAs,” Xilinx
`Application Note XAPP173, Dec. 11, 2000, 15 pages.
`“200-MHz SDRAM Controller Core Product Specification” Alli-
`ance Core, Jan. 10, 2000, 4 pages.
`
`Bapat, Sheker, “Synthesizable 200 MHz ZBT SRAM Interface”
`Xilinx Application Note XAPP136, Jan. 10, 2000, 6 pages.
`“Synthesizable High Performance SDRAM Controller” Xilinx
`Application Note XAPP134, Feb. 1, 2001, 16 pages.
`Ma, Alex, “Synchronous DRAM Controller” Powerpoint slides,
`EE527 Spring 1998, 21 pages.
`PCT Search Report dated Jan. 14, 2005 for International Application
`No. PCT/USO3/33665.
`Office Action for US. Appl. No. 10/284,278, dated Mar. 18, 2005,
`(12 pages).
`Office Action for US. Appl. No. 10/284,278 dated Sep. 26, 2005,
`(13 pages).
`Office Action for US. Appl. No. 10/284,268 dated Mar. 18, 2005,
`(14 pages).
`Office Action for US. Appl. No. 10/284,268 dated Sep. 21, 2005,
`(13 pages).
`Office Action for US. Appl. No. 11/030,881 dated Jan. 11, 2006, (5
`pages).
`Office Action for US. Appl. No. 11/030,881 dated Jul. 27, 2005, (8
`pages).
`PCT Search Report dated Jan. 14, 2005, for International Applica-
`tion No. PCT/USO3/33665.
`Office Action for US. Appl. No. 11/030,881 dated Jul. 10, 2006, (5
`pages).
`Office Action for US. Appl. No. 10/284,278 dated Apr. 24, 2006,
`(12 pages).
`Office Action for US. Appl. No. 10/284,268 dated Apr. 24, 2006,
`(16 pages).
`Preston, W. Curtis, “Using SANs and NAS,” O’Reilly & Associates,
`Inc., Feb. 2002.
`Mauro, Douglas R., et al., “Essential SNMP,” O’Reilly & Associ-
`ates, Inc., Jul. 2001.
`Prince, Betty, “High Performance Memories New architecture
`DRAMs and SRAMsgevolution and function Revised Edition,”
`John Wiley & Sons, Ltd., 1996, entire book.
`RDRAM® Memory: Leading Performance and Value Over
`SDRAM and DDR, Document WP0001-R, Version 1.2, 2001.
`RDRAM® : Maximizing the Value of PCs and Workstations,
`Document WP0003-R, Version 1.0, 2001.
`The Economist, Jun. 22nd, 2002, 28th edition, A Match for Flash?,
`pp. 22-23.
`Office Action for US. Appl. No. 10/284,278 dated Mar. 18, 2005,
`(12 pages).
`Office Action for US. Appl. No. 10/284,278 dated Sep. 26, 2005,
`(13 pages).
`Office Action for US. Appl. No. 10/284,268 dated Mar. 18, 2005,
`(14 pages).
`Office Action for US. Appl. No. 10/284,268 dated Sep. 21, 2005,
`(13 pages).
`Office Action for US. Appl. No. 11/030,881 dated Jan. 11, 2006, (5
`pages).
`Office Action for US. Appl. No. 11/030,881 dated Jul. 27, 2005, (8
`pages).
`PCT Search Report dated Jan. 14, 2005, for International Applica-
`tion No. PCT/USO3/33665.
`“170 MHz FIFOs Using the Virtex Block SelectRAM+ Features”
`Xilinx Application Note XAPP131, Jun. 5, 2001, 6 pages.
`“Using the Virtex Block SelectRAM+ Features” Xilinx Application
`Note XAPP130, Dec. 18, 2000, 11 pages.
`generation
`“API Networks,
`enabling
`technology for next
`product .
`.
`. HyperTransport technology licensed by HP” (retrieved
`Mar.
`18,
`2002)
`<http://www.api-networks.com/pressreleases/
`pr121001.shtml>, API Networks, Dec. 10, 2001, 2 pages.
`Richmond, Robert, “AMD 64-Bit K8 Platform Preview” (retrieved
`Mar. 11, 2002) Sep. 14, 2000, 4 pages.
`“InfiniBridge MT21108 Infiniband Channel Adapter and Eight Port
`Switch” Product Specification, Mellanox Technologies,
`(date
`unknown) 4 pages.
`“Block SelectRAM Overview” (retrieved Mar. 18, 2002) <http://
`www.xilinx.com/xiliprodcatisystemsolution.
`jsp?title:xam,memoryiembedded, blockram,pag>, 2 pages.
`McComas, Bert “PCI-X or InfiniBand Complementary New Tech-
`nologies Go Head to Head” (retrieved Mar. 18, 2002) <httpzwww.
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 3
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 3
`
`

`

`US 7,197,662 B2
`Page 3
`
`Inquest
`
`Market
`
`Information”
`
`inqst.com/articles/pcixvib/pciarticle.htm>,
`Research, Jan. 19, 2001, 10 pages.
`“API Networks Accelerates Use of HyperTransportTM Technology
`With Launch of Industry’s First HyperTransport-to-PCI Bridge
`Chip ” API Networks Press Release, Apr. 2, 2001, 2 pages.
`“HyperTransport-to-PCI Bridge Chip from API Networks” Cahners
`Business Information 2002, 1 page.
`“API Networks Unveils Industry’s First HyperTransportTM Switch
`to Bring Products to Market Quickly and Cost-Effectively” API
`Network Press Release, Nov. 5, 2001, 4 pages.
`“74F1763 Intelligent DRAM Controller Product
`Philips Semiconductors 2002, 2 pages.
`“Sections 5.4.1 - 5.4.5 of document describing DRAM and DRAM
`Controller” (author and date unknown).
`“Basic DRAM Cell” (retrieved Mar. 17, 2002) <http://www.mit.
`edu:8001/people/tairan/6371/node59.html>, MIT, Jun. 26, 1996, 1
`page.
`“Row Address Decoder and Row Driver” (retrieved Mar. 17, 2002)
`<http://www.mit.edu:8001/people/tairan/6371/node60.html>, MIT,
`Jun. 26, 1996, 1 page.
`“Column Decode and Refresh Control Logic” (retrieved Mar. 17,
`2002)
`<http://www.mit.edu:8001/people/tairan/6371/node61.
`html>, MIT, Jun. 26, 1996, 1 page.
`“Refresh Circuit”
`(retrieved Mar.17, 2002) <http://www.mit.
`edu:8001/people/tairan/6371/node62.html>, MIT, Jun. 26, 1996, 1
`page.
`“Refresh Address Control” (retrieved Mar. 17, 2002) <http://www.
`mit.edu:8001/people/tairan/6371/node63.html>, MIT,
`Jun.
`26,
`1996, 1 page.
`“Data-In Tristate Buffer” (retrieved Mar. 17, 2002) <http://www.
`mit.edu:8001/people/tairan/6371/node64.html>, MIT,
`Jun.
`26,
`1996, 1 page.
`
`“Data-out Precharging Circuits and Control Circuits” (retrieved
`Mar.
`17,
`2002) <http://www.mit.edu:8001/people/tairan/6371/
`node65.html>, MIT, Jun. 26, 1996, 1 page.
`“Output Multiplexer” (retrieved Mar. 17, 2002), <http://www.mit.
`eduz8001/people/tairan/6371/node66.html>, MIT, Jun. 26, 1996, 1
`page.
`17, 2002) <http://www.mit.
`“Other Circuits” (retrieved Mar.
`eduz8001/people/tairan/6371/node67.html>, MIT, Jun. 26, 1996, 1
`page.
`“Simulations” (retrieved Mar. 17, 2002) <http://www.mit.edu:8001/
`people/tairan/6371/node68.html>, MIT, Jun. 26, 1996 1 page.
`“Comments” (retrieved Mar. 17, 2002) <http://www.mit.edu:8001/
`people/tairan/6371/node70.html>, MIT, Jun. 26, 1995, 1 page.
`“HHSPICE Verification of Latch” (retrieved Mar. 17, 2002), <http://
`www.mit.edu:8001/people/tairan/6371/node73.html>, MIT,
`Jun.
`26, 1996, 1 page.
`http://www4.
`2002)
`17,
`(retrieved Mar.
`“Introduction”
`tomshardware.com/mainboard/98q4/981024/index.html, 2 pages.
`“Basic DRAM Operation” (retrieved Mar. 17, 2002) <http://www4.
`tomshardware.com/mainboard/98q4/981024/ram-01.html>,
`2
`pages.
`“Asynchronous Operation” (retrieved Mar. 17, 2002), <http://
`www4.tomshardware.com/mainboard/98q4/981024/ram-02.html>,
`2 pages.
`“Synchronous Operation” (retrieved Mar. 17, 2002) <http://www4.
`tomshardware.com/mainboard/98q4/981024/ram-07.html>,
`2
`pages.
`17, 2002), <http://www4.
`(retrieved Mar.
`“DDR SDRAM”
`tomshardware.com/mainboard/98q4/981024/ram-10.html>,
`2
`pages.
`
`* cited by examiner
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 4
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 4
`
`

`

`U.S. Patent
`
`Mar. 27, 2007
`
`Sheet 1 of 15
`
`US 7,197,662 B2
`
`12—1
`
`12-2
`
`Server
`
`
`
`
`
`External
`
`
`
`
`
`
`
`
`Managemeni
`System
`
`
`
` Storage Hub
`
`14-2
`
`External
`
`Management
`System
`
`Figure 1
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 5
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 5
`
`

`

`U.S. Patent
`
`Mar. 27, 2007
`
`Sheet 2 of 15
`
`US 7,197,662 B2
`
`122
`
`12—1
`
`28-1
`
`28—2
`
`
`
`IJO
`
`Conxroller
`
`
`
`
`
`Management
`Complex
`
`
`
`
`
`
`
`
`
`
`Memory
`Memory
`
`Seclinn
`Section
`
`
`Slo7ra9_eI-_lub
`
`_
`
`7
`
`_
`
`7
`
`‘
`7
`Syslem
`
`External
`Management
`
`Figure 2
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 6
`
` 24
`
`
`
`
`7
`7
`7
`7
`_
`_
`_
`7
`
`. |I I. Il |l Ir II |.
`
`Memory
`
`Section
`
`
`Memory
`_
`Sectron
`
`30-n
`
`EI
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 6
`
`

`

`U.S. Patent
`
`Mar. 27, 2007
`
`Sheet 3 of 15
`
`US 7,197,662 B2
`
`
`
`
`Administration Processors
`
`Control Processors
`
`
`TCP/IP
`
`Figure 3
`
`36
`
`
`32m
`32—1
`
`,4
`7
`,J
`
`Administration
`Processors
`
`____________
`
`Administration
`Processors
`
`
`Shared Memory
`
`I
`
`,7 ,
`
`34—1
`
`Control Processor
`
`7
`
`
`J
`
`I_
`
`
`
`26
`
`38
`
`Figure 4
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 7
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 7
`
`

`

`U.S. Patent
`
`Ddar.27,2007
`
`Sheet4 of15
`
`US 7,197,662 B2
`
`mm
`
`2050955.0..
`
`xEnEoo
`
`I.M.25262.3325558
`
`aIIBI.aIIm33.35
`
`.ILFI'IIIII
`
`Baum—om
`
`
`
`:ozoom30505.
`
`I!N_r
`
`.2mN.mmwnNF_3Wm£3258.<5336ahI
`
`_
`
`_I
`
`
`
`:mtoaé
`
`
`
`Eanth
`
`wmfloum
`
`mm
`
`9.hmIm
`
`
`
`
`
`
`
`
`m0.59“.
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 8
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 8
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Mar. 27, 2007
`
`Sheet 5 of 15
`
`US 7,197,662 B2
`
`To Servers (I 2)
`
`22
`
`
`
` fl
`
`Switch/Server
`Communications
`Interface
`
`
`
`Switch Fabric
`
`
`206
`
`
`
`
`
`Switch/Memory
`208 Section Interfaces
`
`
`
`Switch
`
`
`
`
`
`Communications
`Channel Interface
`
`
`
` Memory Section
`
`30
`
`Figure 6
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 9
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 9
`
`

`

`U.S. Patent
`
`Mar. 27, 2007
`
`Sheet 6 of 15
`
`US 7,197,662 B2
`
`To Servers (I2)
`
`SwitchIServer
`Communications
`Interfaces
`
`206
`
`Fabrlc
`
`lllllll
`
`IIIIIII
`
`Communications
`Channel Interface
`
`4‘3 IIIIIII
`
`Selector
`
`Memory
`
`Switch
`
`,
`
`:
`
`Management
`Complex
`
`26
`
`
`
`
`
`Figure 7
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 10
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 10
`
`

`

`U.S. Patent
`
`Mar. 27, 2007
`
`Sheet 7 of 15
`
`US 7,197,662 B2
`
`12—1
`
`12—2
`
`12—M
`
`device
`
`
`driver
`
`28-1
`
`SSCI
`
`1204- -
`
`
`
`
`
`
`
`
`SSCI
`_
`
`204-
`
`22
`[J
`
`[302
`Switch
`Controller
`
`
`
`
`
`
`
`
`
`
`
`
`ME
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`{FOB-1'
`'
`I
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Memory Section
`
`
`
`Figure 8
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 11
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 11
`
`

`

`U.S. Patent
`
`Mar. 27, 2007
`
`Sheet 8 of 15
`
`US 7,197,662 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`Switch
`Controfler
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Figure 9
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 12
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 12
`
`

`

`U.S. Patent
`
`Mar. 27, 2007
`
`Sheet 9 of 15
`
`US 7,197,662 B2
`
`1002-1
`
`1004-1
`
`1002—1
`
`1004-1
`
`
`
`
`
`
`
`
`
`Figure 10
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 13
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 13
`
`

`

`U.S. Patent
`
`Mar. 27, 2007
`
`Sheet 10 of 15
`
`US 7,197,662 B2
`
`.D“K0:OH
`
`Ezom
`
`
`
`4
`
`.m:3:oh
`
`Bram
`
`o
`
`
`
`m-‘0Bzobcoflw.8203
`
`59c2.2mm3E8.550E;x020
`
`
`
`.p--..u.
`
`___-_'Q______.L___..._(D__..___J
`h
`
`t.
`
`:9:9“.
`
`868Bigroam:mo:5oh
`
`:68830D@0802?35oh
`
`
`
`lIIIIIIIIIIIIIIW
`
`
`
`QR..llr.ILILIIfIL-lLIrll_l.rlL_.II.II.II.rIl_IIIL
`
`
`
`
`
`
`
`
`
` _5:05:00550,»..2802qu58m
`
`____fIllr
`
`
`
`
`
`
`
`
`EatEmcmz35:00.550man“.620
`
`Tvm
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 14
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 14
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Mar. 27, 2007
`
`Sheet 11 of 15
`
`US 7,197,662 B2
`
`3120
`~
`
`Initiate Write
`Request
`
`Send Dev. Driver
`
`81220
`
`Ready
`
`81222
`
`8120 Map DBI to l/F
`-
`address
`
`Dev. Driver xmit
`data
`
`
`
`
`3120
`
`-
`
`est. path thru
`switch
`
`3120'
`
`Fwd rqst to CC!
`
`5121. send rqstt toI sect.
`con r0
`
`
`
`No
`
`NAK to server
`
`8121 Server rcv NAK
`
`S121.-
`
`81224
`
`Mem
`Dev Busy /'
`?
`
`N0
`
`Yes
`
`3 226
`
`Put data in Temp
`Storage
`
`
`
`81228
`
`Mem Dev signal
`when available
`
`
`
`
`
`
`
`clock data thru
`Mem l/F
`
`Store data
`
`send succesful
`
`msg to sect contr.
`
` S1232
`
`S1234
`
`
`
`S1236
`
`
`
`
`81238
`
`81240
`
`
`
`Figure 12
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE15
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 15
`
`

`

`U.S. Patent
`
`Mar. 27, 2007
`
`Sheet 12 of 15
`
`US 7,197,662 B2
`
`
`
`Mem Dev elgnal
`when available
`
`Reserve Mem Dev 31326
`
`S1312 //In \\ Y55
`Mem Sect
`?
`
`
`
`
`
`S1328
`
`
`
`
`
`
`
`S1314
`
`8131.
`
`-
`
`8131:
`
`ID to
`Mem IIF
`
`81332
`
`Append data to ID 81338
`
`Send data to
`selector
`
`send data to
`server
`
`
`
`Figure 13
`
`81340
`
`81342
`
`
`
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 16
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 16
`
`

`

`U.S. Patent
`
`Mar. 27, 2007
`
`Sheet 13 of 15
`
`US 7,197,662 B2
`
`'K
`
`Partition 1
`
`K Partition 2
`
`66-1
`
`
`
`M
`Partition 1
`
`Partition 2
`
`
`
`_.
`Partition 4
`
`Partition 3
`
`
`
`Figure 14
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 17
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 17
`
`

`

`aeum
`
`669791,7
`
`2
`
`m
`
`
`
`tEcogMEEQmn.9:3:oh3:chw836ohIa23.8%P-mpxmca.5623
`
`.-HEQE
`
`mm
`
`
`rlrILILIIflP'lI-lullrllILIIfILIII—II'mau2nuM5:23.09
`
`screamN-‘0v-10Ezifiomv55%
`
`mn__._________,0rlrllrlluall—IIFILllull—IlLlLllflL-‘I—Illall-IIIIIIIIIIIall-IIIIIIIIIIII7m____.__w_____
`
`
`
`
`Eatgamma35:8“was.38V320EocmagmaBEE”.350ESV320
`
`
`
`
`
`HtttRf...A0.
`
`
`
`:___:__-_________.___._:_________:___—__n.
`
`
`1IrrLLI._.|_|_.._.L1r._.|_.._.r»thurL:
`
`.mww
`
`m893mm
`
`
`
`
`
`
`
`mout/DD\CogmzmomcwmOH.out/MD\COEUE(MD9.3%OH.
`
`
`
`
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 18
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 18
`
`
`

`

`U.S. Patent
`
`Mar. 27, 2007
`
`Sheet 15 of 15
`
`US 7,197,662 B2
`
`30\'
`
`46-1
`
`46m
`
`
`
`Communications
`Communications
`
`
`
`Channel Interface
`Channel Interface
`
`
`
`
`T0 Management
`Complex
`
`
`
`Section
`
`Com to II or
`
`
`
`
`
` Elésfltmlle
`1"
`
`PC &
`DMA
`Control
`
`1610
`
`
`
`Figure 16
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 19
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 19
`
`

`

`US 7,197,662 B2
`
`1
`METHODS AND SYSTEMS FOR A STORAGE
`SYSTEM
`
`RELATED APPLICATIONS
`
`The present application relates to the US. patent appli-
`cation Ser. No. 10/284,278 by M. James Bullen, Steven L.
`Dodd, David J. Herbison, and William T. Lynch, entitled
`“Methods and Systems for a Storage System Including and
`Improved Switch,” and the US. patent application Ser. No.
`10/284,268 by M. James Bullen, Steven L. Dodd, David J.
`Herbison, and William T. Lynch, entitled “Methods and
`Systems for a Memory Section,” both of which are incor-
`porated by reference herein in their entireties.
`
`BACKGROUND
`
`The present invention relates to data storage, and more
`particularly, to methods and systems for a high throughput
`storage device.
`A form of on-line transaction processing (OLTP) appli-
`cations requiring a high number of data block reads or writes
`are called H-OLTP applications. A large server or mainframe
`or several servers typically host an H-OLTP application.
`Typically, these applications involve the use of a real time
`operating system, a relational database, optical fiber based
`networking, distributed communications facilities to a user
`community, and the application itself. Storage solutions for
`these applications use a combination of mechanical disk
`drives and cached memory under stored program control.
`The techniques for the storage management of H-OLTP
`applications can use redundant file storage algorithms on
`multiple disk drives, memory cache replications, data coher-
`ency algorithms, and/or load balancing.
`Abrief overview of the storage management technologies
`of cached disk arrays (CDAs) and solid-state disk storage
`systems (SSDs) follows.
`Cached disk arrays (CDAs) combine disk drives and
`solid-state memory systems under common program con-
`trol. The disk drives in CDAs are servo-mechanical devices.
`
`2
`
`It would be desirable for large capacity storage to provide
`sufficient throughput for high-volume, real-time applica-
`tions, especially, for example in emerging applications in
`financial, defense, research, customer management, and
`homeland security areas.
`
`SUMMARY
`
`Accordingly, the present invention is directed to methods
`and systems that address the problems of prior art.
`In accordance with the purposes of the invention, as
`embodied and broadly described herein, methods and sys-
`tems for an apparatus are provided including one or more
`memory sections, one or more switches, and a management
`system. The one or memory sections include one or more
`memory devices capable of storing data in storage locations,
`and a memory section controller capable of detecting faults
`in the memory section and transmitting a fault message in
`response to the detected faults. The one or more switches
`include one or more interfaces for connecting to one or more
`external devices, and a switch fabric connected to one or
`more memory sections and the external device interfaces
`and interconnecting the memory sections and the external
`device interfaces based on an algorithm. A management
`system is provided capable of receiving fault messages from
`the memory section controllers and removing from service
`the memory section from which the fault message was
`received, and wherein the management system is further
`capable of determining an algorithm for use by a switch
`fabric in interconnecting the memory sections and the exter-
`nal device interfaces, and instructing the switch to execute
`the determined algorithm.
`The summary and the following detailed description
`should not restrict the scope of the claimed invention. Both
`provide examples and explanations to enable others to
`practice the invention. The accompanying drawings, which
`form part of the description for carrying out the best mode
`of the invention, show several embodiments of the inven-
`tion, and together with the description, explain the principles
`of the invention.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`Advances in motor technology currently allow the platters of
`the disk drives to spin at 15,000 revolutions per minute;
`advanced systems may spin their platters at 18,000 revolu-
`tions per minute.
`CDAs combine several racks of rotating disks with a
`common memory cache in an architecture where capacity
`may be added through the addition of more racks of devices,
`more cache, or both. CDAs often are used by companies to
`provide storage services in their mission critical applica- 50
`tions, including H-OLTP applications.
`The on-board cache of a CDA stores frequently used data
`because access times for data in cache memory can be short
`relative to access times for data on the drives. Such high-end
`storage system devices with rotating media, such as CDAs,
`include less than ideally desirable characteristics in terms of
`total throughput and memory cache size.
`A solid-state disk (SSD) is a storage device corresponding
`to the solid-state memory attached to a computer’s central
`processing unit through its internal bus structure. To an 60
`external computer (server or mainframe) the SSD appears as
`a very fast disk drive when it is directly attached to the
`computer over a fast communications link or network.
`Operating under stored program control, SSDs store fre-
`quently used information like transaction logs, database 65
`indices, and specialized data structures integral to the effi-
`cient execution of a company’s mission critical applications.
`
`55
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a block diagram of a storage hub environment,
`in accordance with methods and systems provided;
`FIG. 2 is a more detailed block diagram of a storage hub,
`in accordance with methods and systems provided;
`FIG. 3 illustrates a logical architecture for a management
`complex, in accordance with methods and systems provided;
`FIG. 4 is a block diagram of a physical architecture for a
`management complex,
`in accordance with methods and
`systems provided;
`FIG. 5 is a block diagram of a exemplary memory section,
`in accordance with methods and systems provided;
`FIG. 6 illustrates a functional diagram of a switch and
`memory section, in accordance with methods and system
`consistent with the invention;
`FIG. 7 illustrates an alternative functional diagram of a
`switch and memory section, in accordance with methods and
`systems provided;
`FIG. 8 illustrates a diagram of an alternative exemplary
`switch, in accordance with methods and systems provided.
`FIG. 9 illustrates a diagram of an alternative switch, in
`accordance with methods and systems provided;
`FIG. 10 illustrates an exemplary pipeline shift register, in
`accordance with methods and systems provided;
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 20
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 20
`
`

`

`US 7,197,662 B2
`
`3
`FIG. 11 includes a more detailed block diagram of an
`exemplary embodiment of a memory interface device, in
`accordance with methods and systems provided;
`FIG. 12 illustrates a flow chart for an exemplary writing
`operation,
`in accordance with methods and systems pro-
`vided;
`FIG. 13 illustrates a flow chart for an exemplary reading
`operation,
`in accordance with methods and systems pro-
`vided;
`FIG. 14 illustrates a logical diagram of partitioned
`memory devices, in accordance with methods and systems
`provided;
`FIG. 15 illustrates an alternative embodiment of a
`
`memory interface devices, in accordance with methods and
`systems provided; and
`FIG. 16 illustrates an alternative memory section,
`accordance with methods and systems provided.
`
`in
`
`DETAILED DESCRIPTION
`
`to exemplary
`Reference will now be made in detail
`embodiments, examples of which are illustrated in the
`accompanying drawings. Wherever possible, the same ref-
`erence numbers will be used throughout the drawings to
`refer to the same or like parts.
`FIG. 1 is a block diagram of one embodiment storage hub
`environment, in accordance with methods and systems pro-
`vided. As illustrated, the storage hub environment includes
`a storage hub 10, servers 12-1 and 12-2, external manage-
`ment systems 14-1 and 14-2, a non-volatile storage device
`16, an IP network 18 and a connection to another network
`20. The storage hub 10 may include a large amount of
`storage (not shown) and stores the data in data blocks.
`Although the data may be stored in data blocks, any other
`mechanism for storing the data may be used without depart-
`ing from the scope of the invention. The non-volatile storage
`device 16 may be a magnetic storage device, such as a CDA
`as described above. The non-volatile storage device 16 may
`be used to store back-up versions of the data stored by the
`storage hub 10.
`The description below is organized in the following
`manner. First, a brief overview of the storage hub 10
`environment illustrated in FIG. 1 is presented. Then, more
`detailed descriptions of the components of the storage hub
`10 are presented, after which a more detailed description of
`exemplary methods for writing data to the storage hub,
`reading data from the storage hub 10, and a testing operation
`for the storage hub 10 are presented. Then, exemplary
`alternatives to these components are presented. It should,
`however, be understood that these are all exemplary descrip-
`tions regarding example methods and systems for imple-
`menting the invention. As such, one of skill will recognize
`that there are other methods and systems that may be used
`for practicing the invention that is defined by the claims of
`this application.
`The servers 12-1 and 12-2 are, for example, standard
`commercially available servers or farms of servers that can
`be connected to internal or external networks (not shown).
`For example, the servers 12-1 and/or 12-2 may be connected
`to an internal network such as an Ethernet for receiving
`requests for the retrieval or storage of information from end
`users connected to the network. Alternatively, the servers
`12-1 and/or 12-2 could be connected to external networks,
`such as the Internet, for receiving requests for retrieval or
`storage of information from end users connected to the
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`
`external network. Further, although two servers 12-1 and
`12-2 are illustrated, the storage hub 10 may be connected to
`any number of servers 12.
`When an application being executed by the server 12
`requires data, the server 12 determines if the storage hub 10
`stores the data. The servers 12 may store a record showing
`whether the data their applications require is on the storage
`hub 10. The server 12 then sends a data request to the storage
`hub 10 requesting the data. The storage hub 10 reads the data
`from the location in which it is stored and sends it to the
`
`server requesting the data 12-1 or 12-2. The server may run
`different types of applications and database management
`systems that may require data from the storage hub 10.
`Examples of typical applications
`include, by way of
`example only, billing systems, customer relationship man-
`agement systems, reservations systems, ordering systems,
`security systems, etc. Examples of database management
`systems include ORACLE, DB2, Sybase, Informix, etc.
`Additionally, the storage hub 10 may receive a request
`from a server 12-1 or 12-2 to store data. Thereafter, the
`storage hub 10 preferably provides the server 12 with either
`an acknowledgement that the write occurred (i.e., the storage
`of the data) or a failure message. Such messages could
`include, for example, an acknowledgement that the data
`block was safely stored on both the storage (not shown) in
`the storage hub 10 and on the CDA 16 when a CDA 16 is
`used as backup for the storage hub 10, an acknowledgement
`that the data block is safely stored in the storage hub’s 10
`storage (not shown), no acknowledgement of any sort, or a
`failure message.
`The external management system 14 may be directly
`connected to the storage hub 10, such as external manage-
`ment system 14-2. Or, the external management system 14
`may be connected to the storage hub 10 via a network, such
`as external management systems 14-1 that is connected to
`the storage hub 10 via network 18. Network 18 may be any
`type of network, such as an internal Ethernet network, an IP
`network, or the Internet. Although FIG. 1 illustrates both
`external management systems 14-1 and 14-2 connected to
`the storage hub 10, in other implementations there may be
`only one or any number of external management systems, or
`an external management system 14 need not be included.
`For example, in other implementations it may be desirable
`to have 3 or more external management systems. Addition-
`ally, the external management system may be a computer
`running proprietary or commercially available software,
`such as, for example, HP Openview. The storage hub 10 may
`provide surveillance and administration information to the
`external management system 14, such as the status and
`location of stored data blocks.
`
`FIG. 2 illustrates a more detailed block diagram of the
`storage hub 10, in accordance with methods and systems
`provided. As illustrated, the storage hub 10 includes a switch
`or switches 22-1 and 22-2, a management complex 26, and
`memory sections 30-1 thru 30-n. In this embodiment, both
`switches 22-1 and 22-2 may be active or one of the switches
`may be active while the other is a redundant switch for use
`in the event the active switch suffers a problem. Although
`FIG. 2 illustrates two switches, the storage hub 12 may
`include only one switch or any number of switches.
`In FIG. 2, server 12-2 connects to the storage hub 10 via
`a network 20 thru an input/output (I/O) controller 24. The
`network may be any type of internal or external network,
`such as an Ethernet network or the Internet. The I/O con-
`
`troller 24 preferably is an appropriate I/O controller for
`connecting to the particular network 20. Preferably, the I/O
`controller 24 converts signals between a native protocol of
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 21
`
`UNIFIED PATENTS EXHIBIT 1001
`PAGE 21
`
`

`

`US 7,197,662 B2
`
`5
`the network 20 and a local protocol used by the storage hub
`10. Potential protocols include, but are not
`limited to,
`Telecommunications Control Protocol/Intemet Protocol
`
`(TCP/IP), System Network Architecture (SNA)-based pro-
`tocols, Serial Communications Control Interface (SCCI),
`Intelligent Serial Communications Interface (ISCI), Fibre
`Channel, Infiniband, and other third generation input/output
`(3GIO) protocols
`The memory sections 30 preferably include the storage
`for the storage hub 10 along with other hardware for
`accessing the storage. As used herein, the term “memory
`section” refers to any subsystem including one or more
`memory devices that may be used for storing information.
`This architecture is applicable to any device that can store
`data. Thus, when the storage hub 10 receives a request to
`store data, the data is forwarded to a memory section 30,
`which stores the data. L