(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2007/0050571 A1
`(43) Pub. Date:
`Mar. 1, 2007
`Nakamura et al.
`
`US 20070050571 A1
`
`(54) STORAGE SYSTEM, STORAGE DEVICE,
`AND CONTROL METHOD THEREOF
`
`(76) Inventors: Shuji Nakamura, Kanagawa (JP);
`Kazuhisa Fujimoto, Tokyo (JP); Akira
`Fujibayashi, Kanagawa (JP)
`Correspondence Address:
`MATTINGLY, STANGER, MALUR &
`BRUNDIDGE, P.C.
`1800 DAGONAL ROAD
`SUTE 370
`ALEXANDRIA, VA 22314 (US)
`(21) Appl. No.:
`11/247,161
`
`(22) Filed:
`(30)
`
`Oct. 12, 2005
`Foreign Application Priority Data
`
`Sep. 1, 2005 (JP)...................................... 2005-25298.9
`
`Publication Classification
`
`(51) Int. Cl.
`(2006.01)
`G06F 3/00
`(2007.01)
`G06F 2/4
`(52) U.S. Cl. ............................................ 711/154; 711/163
`
`(57)
`
`ABSTRACT
`
`A storage system including a storage device 1 which
`includes: media 50 for storing data from a host computer 2:
`a medium controller 14 for controlling the media; a plurality
`of channel controllers 11 for connecting to the host computer
`2 through a channel; and a cache memory 13 for temporarily
`storing data from the host computer 2, wherein the media 50
`have a restriction on a number of writing times. The storage
`device 1 includes a bus 123 for directly transferring data
`from the medium controller 14 to the channel controller 11.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`DESTAGEISTAGE PROCESSING
`
`M
`
`WATYPE OF MEDIUMS
`WRITE DESTINATION?
`HDD
`
`
`
`S12O2
`DESTAGE THE OLDEST SLOTUSING HDD usen
`
`SEARCHFOR
`M WITH THESMALEST NUMBER OF ESTAGES
`
`
`
`S1203
`
`YES
`
`STHERE ASLOT FOR USEBY HE FM?
`(ISLIST LENGTH NOTZERO2)
`
`NO
`SEARCHFOR
`FM WIHHENEX SMALLES NUMBER
`
`41
`
`S 206
`DESTAGE HE OLDESTSLOT OF THE FM-h-
`
`A1 TO
`NUMBER OF DESAGES OF THE MEDIUM 1.
`
`("gles REMOVE
`
`HE SLOT FROM DIRECTORY INFORMATION
`AND ACCESS SEQUENCEST
`
`ADD DATAOBESTAGED TO
`RY INFORMATON RECO
`
`MRU OF Ol
`AND ACCESS SEQUENCELIST
`
`STAGE NEW DATA TO THE SLOT
`
`END
`
`S1208
`
`S1209
`
`S1210
`
`HPE, Exh. 1025, p. 1
`
`

`

`Patent Application Publication Mar. 1, 2007 Sheet 1 of 17
`
`US 2007/0050571 A1
`
`F.G. 1
`
`SANSWITCH
`4
`
`
`
`
`
`11
`//
`
`77
`|
`CHANNEL
`CHANNEL
`E
`CONTROLLER CONTROLLER
`
`ZZ
`
`//
`
`ZZ
`
`corol
`UNIT
`
`DSK
`DISK
`ONTROLLER CONTROLLER
`
`
`
`
`
`
`
`
`
`
`
`V
`
`n
`
`S
`
`N
`
`1.
`
`7.
`
`1
`
`12
`
`2- 2 As.
`
`
`
`CACHE
`MEMORY
`
`STORAGE CONTROLLER
`
`CONTROL
`MEMORY M.
`
`HPE, Exh. 1025, p. 2
`
`

`

`Patent Application Publication
`
`Mar. 1, 2007 Sheet 2 of 17
`
`US 2007/0050571 A1
`
`SL
`
`TANNVYHD
`TANNVHO
`
`JOOOLOWd
`1OD0LOdd
`
`vib
`
`WHSHdluad
`
`LINNTOMLNOD
`
`Clb
`
`TANNVHO'"TANNVHO:FTANNVHO-%7S4NNVHO'F
`
`
`
`TOOOLOYd||TOOOLOYd
`
`TSNNVHOTANNVHO
`
`
`AOVAYSLNIMHOMLANTWNYALNI
`HOSSAOONd||HOSSAIOUd|YOSSAIONd||YOSSAOONd
`
`YHATIOULNOOTANNVHO
`
`
`
`
`
`LLL
`
`LL
`
`¢Old
`
`XOdTVW
`
`YSsSNVAL
`
`
`
`HPE, Exh. 1025, p. 3
`
`HPE, Exh. 1025, p. 3
`
`
`
`
`
`
`
`

`

`Patent Application Publication Mar. 1, 2007 Sheet 3 of 17
`
`US 2007/0050571 A1
`
`
`
`HPE, Exh. 1025, p. 4
`
`

`

`Patent Application Publication Mar. 1, 2007 Sheet 4 of 17
`
`US 2007/0050571 A1
`
`FG4
`
`
`
`FM CONTRO UNIT
`
`INTERNAL NETWORK iNTERFACE
`
`MEMORY
`CONTROLLER
`
`DMA CONTROLLER
`
`16
`7
`
`167
`
`164
`
`FM PROTOCOL.
`PROCESSOR
`
`FM PROTOCOL
`PROCESSOR
`
`FM PROTOCOL.
`PROCESSOR
`
`CONNECTORS
`
`FM PROTOCOL. MESSEY
`PROCESSOR
`ULE
`TRANSFER
`LIST
`
`64 CONNECTOR
`
`CONNECTOR
`
`ScONNECTOR
`
`coNNECTOR
`
`SCONNECTOR
`
`FM DEVICE
`
`FM DEVICE
`
`FM DEVICE
`
`FM DEVICE
`
`HPE, Exh. 1025, p. 5
`
`

`

`Patent Application Publication Mar. 1, 2007 Sheet 5 of 17
`
`US 2007/0050571 A1
`
`FG.5
`
`
`
`FM CONTROL UNIT
`
`NTERNAL NETWORK iNTERFACE
`
`DMA CONTROLLER
`
`MEMORY
`CONTROLLER
`
`164
`
`FMPROTOCOL
`PROCESSOR
`
`FM PROTOCOL
`PROCESSOR
`
`FM PROTOCOL
`PROCESSOR
`
`FM PROTOCOL MES
`PROCESSOR
`MODULE
`TRANSFER
`LIST
`
`1610
`
`1610
`
`1610
`
`1610
`
`1641
`
`FM DEVICE
`
`FM DEVICE
`
`FM DEVICE
`
`169
`
`169
`
`169
`
`169
`
`f
`
`N
`
`Y - - - - -
`
`169
`
`HPE, Exh. 1025, p. 6
`
`

`

`Patent Application Publication Mar. 1, 2007 Sheet 6 of 17
`
`US 2007/0050571 A1
`
`FIG.6
`
`1.
`
`16
`
`14
`
`CHANNEL
`CONTROLLER
`
`FM
`CONTROL
`UN
`
`DSK . .
`CONTROLLER
`
`121
`
`121
`
`121
`
`
`
`INTERNAL NETWORK
`NTERNAL NETWORK
`INTERFACE
`NERFACE
`123:CONNECTION BETWEEN
`CHANNEL CONTROLLER
`AND FM CONTROL UNIT
`
`NTERNAL NETWORK
`INTERFACE
`
`NTERNAL SWITCH
`
`INTERNAL NETWORK
`NTERFACE
`
`INTERNAL NETWORK
`NTERFACE
`
`12
`
`13
`
`
`
`17
`
`CACHE
`MEMORY
`
`CONTROL
`MEMORY
`
`HPE, Exh. 1025, p. 7
`
`

`

`Patent Application Publication Mar. 1, 2007 Sheet 7 of 17
`
`US 2007/0050571 A1
`
`FG.7
`
`13: CACHE MEMORY
`11: CHANNEL CONTROLLER7 CONTROL MEMORY
`
`14: DSK CONTROLLER
`
`SO1
`
`S702
`
`
`
`STO3
`s
`
`S705
`
`
`
`s 704
`
`S7O6
`
`S708
`
`STO
`
`S711
`
`HPE, Exh. 1025, p. 8
`
`

`

`Patent Application Publication Mar. 1, 2007 Sheet 8 of 17
`
`US 2007/0050571 A1
`
`FIG.8
`
`11: CHANNEL CONTROLLER 17, CONTROL MEMORY
`
`16: FM CONTROL UNIT
`
`S802 is S803
`
`S804
`
`8O
`S
`
`
`
`S805
`
`S806
`
`S807
`
`HPE, Exh. 1025, p. 9
`
`

`

`Patent Application Publication Mar. 1, 2007 Sheet 9 of 17
`
`US 2007/0050571 A1
`
`FIG.9
`
`
`
`CACHE MEMORY
`
`READ CACHE AREA
`
`WRITE CACHE AREA
`
`CONTROL
`MEMORY .
`
`CONFIGURATION COMMUNICATION
`INFORMATION
`
`DIRECTORY
`INFORMATION
`
`ACCESS
`SECUENCEST
`
`HPE, Exh. 1025, p. 10
`
`

`

`Patent Application Publication Mar. 1, 2007 Sheet 10 of 17
`
`US 2007/0050571 A1
`
`FIG.10A
`1001
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FG1 OB
`
`1713:
`
`174:
`
`MEDIUMPOINTER
`LBA
`0000-1ff HDDO -
`
`
`
`
`
`LBA1,
`CMslot 1
`
`LBA2,
`CMslot2
`
`LBA3, a ...
`CMslot3
`
`LBA n,
`2- CMslot m
`
`CMslot4
`
`CMslotS
`
`CMslot6.
`
`Nil(LRU)
`
`HPE, Exh. 1025, p. 11
`
`

`

`Patent Application Publication Mar. 1, 2007 Sheet 11 of 17
`
`US 2007/0050571 A1
`
`F.G. 1 OC
`
`O61
`
`1067
`
`CMslot/
`
`CMslots
`
`CMslot2
`
`Nil(LRU)
`
`
`
`1063
`1064
`1065
`
`NUMBER OF
`DESTAGES
`NUMBER OF
`
`CAPACITY
`
`100GB
`
`
`
`O6
`
`1067
`
`
`
`CMslots
`
`CMslot3
`
`CMslot4
`
`Nil(LRU)
`
`NUMBER OF
`
`NUMBER OF
`
`
`
`O 63
`1064
`1065
`
`1062
`
`
`
`1067
`
`CMSOt1
`
`... 2 > CMslotg
`
`Nil(LRU)
`
`DESTAGE
`RESTRICTION
`NUMBER OF
`DESTAGES
`NUMBER OF
`ACCESSES
`CAPACTY
`
`NONE
`
`10524
`
`113223
`600GB
`
`1066
`1063
`1064
`1065
`
`
`
`HPE, Exh. 1025, p. 12
`
`

`

`Patent Application Publication
`
`Mar. 1, 2007 Sheet 12 of 17
`
`US 2007/0050571 A1
`
`
`
`90 || S.
`
`HPE, Exh. 1025, p. 13
`
`

`

`Patent Application Publication Mar. 1, 2007 Sheet 13 of 17
`
`US 2007/0050571 A1
`
`FG 11B
`
`17: CONTROL MEMORY
`11 : CHANNEL CONTROLLER
`S1121
`S123
`s
`S 1122
`s
`
`HOST WRITE 2
`
`
`
`13: CACHE MEMORY
`
`16: FM CONTROL UNIT/
`14: DISKCONTROLLER
`
`
`
`
`
`2Ls 1124
`
`S1 34
`
`HPE, Exh. 1025, p. 14
`
`

`

`Patent Application Publication Mar. 1, 2007 Sheet 14 of 17
`
`US 2007/0050571 A1
`
`FG12A
`
`DESTAGE/STAGE PROCESSING
`
`
`
`FM
`
`WHAT TYPE OF MEDIUMS
`WRITE DESTINATION?
`HDD
`
`DESTAGE THE OLDEST SLOT USING HDD LIST
`
`
`
`SEARCH FOR
`FM WITH THE SMALLEST NUMBER OF DESTAGES
`
`S 120
`
`S12O2
`
`S 1203
`
`
`
`
`
`
`
`IS THERE A SLOT FOR USE BY THE FM?
`(IS LIST LENGTH NOTZERO?)
`
`
`
`
`
`
`
`
`
`
`
`4.
`SEARCH FOR
`FM WITH THE NEXT SMALLEST NUMBER
`
`NO
`
`DESTAGE THE OLDEST SLOT OF THE FM
`
`ADD 1 TO
`NUMBER OF DESTAGES OF THE MEDIUM
`
`REMOVE
`THE SLOT FROM ORECTORY INFORMATION
`AND ACCESS SECRUENCE LIST
`
`ADD DATA TO BE STAGED TO
`MRU OF DIRECTORY INFORMATION
`AND ACCESS SEQUENCE LIST
`
`STAGE NEW DATA TO THE SLOT
`
`END
`
`S 1208
`
`S1209
`
`
`
`S 1210
`
`HPE, Exh. 1025, p. 15
`
`

`

`Patent Application Publication Mar. 1, 2007 Sheet 15 of 17
`
`US 2007/0050571 A1
`
`FIG.12B
`
`
`
`
`
`
`
`
`
`
`
`DESTAGEISTAGE PROCESSING
`
`SEARCH FOR MEDIUM WITH THE SMALLEST NUMBER
`ox NUMBER OF DESTAGES + B + NUMBER OF ACCESSES)
`(
`
`S1241
`
`S THERE A SLOT FOR THE MEDIUM7
`(ISLIST LENGTHZERO?)
`
`NO
`
`SEARCH FOR
`FM WITH THE NEX SMALLEST NUMBER
`
`
`
`
`
`DESTAGE THE OLDEST SLOT OF THE FM
`
`
`
`ADD 1 TO
`NUMBER OF DESTAGES OF THE MEDUM
`
`
`
`REMOVE
`THE SLOT FROM DIRECTORY INFORMATION
`AND ACCESS SEOUENCE LIST
`
`S 1244
`
`S1245
`
`S 1246
`
`ADD DATA TO BE STAGED TO DIRECTORY INFORMATION
`AND MRU OF ACCESS SECRUENCE LIST
`
`S 1247
`
`STAGE NEW DATA TO THE SLOT
`
`
`
`S 1248
`
`END
`
`HPE, Exh. 1025, p. 16
`
`

`

`Patent Application Publication Mar. 1, 2007 Sheet 16 of 17
`
`US 2007/0050571 A1
`
`FG.13
`
`I
`
`169
`
`69
`
`50
`
`CNJ
`
`. 50
`
`G
`
`review W. J.
`
`C
`
`4
`
`4
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`11
`ZZ
`WI-A
`A/
`AV
`CONTROLLER CONTROLLER FM conTROL UNIT FM conTROL UNIT CONTROLLER
`
`VS TS
`L.
`N N SA
`
`/
`
`
`
`
`
`
`
`CACHE
`MEMORY
`
`CONTROL
`UNIT
`
`STORAGE CONTROLLER
`
`6
`
`CONTROL
`ONT
`MEMO
`RY
`
`
`
`MEMORY
`
`CONTROL
`UNT
`
`-16
`
`
`
`M
`
`CONTROL
`MEMORY
`
`HPE, Exh. 1025, p. 17
`
`

`

`Patent Application Publication Mar. 1, 2007 Sheet 17 of 17
`
`US 2007/0050571 A1
`
`FIG.14
`
`160
`
`HIGHLY-FUNCTIONAL FM CONTROL UNIT
`1 11
`
`161
`
`PROCESSORPROCESSORPROCESSORPROCESSOR
`
`NTERNAL NETWORK iNTERFACE
`
`
`
`
`
`
`
`112
`
`- 113
`PERPHERAL
`CONTROL UNIT
`
`MAIL BOX
`
`1122
`TRANSFER
`LIST
`
`
`
`1641
`
`DMA CONTROLLER
`
`162
`
`67
`
`FMPROTOCOL.
`PROCESSOR
`
`FM PROTOCOL
`PROCESSOR
`
`1610
`
`FM DEVICE
`
`1 610
`
`device
`move
`
`FM DEVICE
`
`FM DEVICE
`
`FM DEVICE
`
`FM DEVICE
`
`more
`
`FM DEVICE
`
`169
`
`FM DEVICE
`
`FM DEVICE
`
`HPE, Exh. 1025, p. 18
`
`

`

`US 2007/0050571 A1
`
`Mar. 1, 2007
`
`STORAGE SYSTEM, STORAGE DEVICE, AND
`CONTROL METHOD THEREOF
`0001. The present application is based on and claims
`priority of Japanese patent applications No. 2005-252989
`filed on Sep. 1, 2005, the entire contents of which are hereby
`incorporated by reference.
`
`BACKGROUND OF THE INVENTION
`
`0002)
`1. Field of the Invention
`0003. The present invention relates to a storage device
`and the control method thereof.
`0004 2. Description of the Related Art
`0005. In recent years, the reduction of the total cost of
`ownership (TCO) of a storage system becomes increasingly
`important in information business sites such as a data center.
`On the other hand, demand for recording data reliably for a
`long period of time is increasing. As an example of this fact,
`the document data of financial institutions, medical institu
`tions, etc., are required to be stored without being erased by
`law. Under these circumstances, there is much demand for
`highly reliable storage systems having a large capacity.
`However, in general, in a large-scale storage system using
`hard-disk drives (in the following, referred to as “HDDs),
`the amount of power consumption increases in proportion to
`the storage capacity. That is to say, the possession of a
`storage system with a large capacity means an increase in the
`total cost of ownership including electricity charges. In view
`of Such situations, a technique for reducing the power
`consumption of HDDs by a cache management algorithm
`has been proposed (Nonpatent document; ZHU, Q., DAVID,
`F., ZHOU, Y, DEVARA.J. C., AND CAO, P, “Reducing
`Energy Consumption of Disk Storage Using Power-Aware
`Cache Management”. InProc. of the 10th Intl. Symp. on
`High Performance Computer Architecture (HPCA-10) (Feb
`ruary 2004)). Also, the problem is not limited to the elec
`tricity charges. In general, the floor area for installation
`increases as the capacity of a storage system increases. This
`also increases the total cost of ownership.
`0006 Incidentally, a flash memory attracts attention as a
`nonvolatile medium in recent years. A flash memory com
`monly consumes less than one several-tenth of power when
`compared with an HDD, and can be read at a high speed.
`Also, a flash memory is small sized unlike an HDD having
`a mechanically driven part.
`0007. However, a flash memory has a restriction on the
`number of writing times because of the physical constitution
`of a cell for holding information. Against Such a restriction,
`the number of writing times of a flash memory has been
`improved by a technique called wear leveling, in which the
`number of writing times to each cell is controlled to be
`averaged by having correspondence between an address to
`be shown to the upper apparatus and a cell position. In this
`regard, in the following, an element for holding information
`is simply called a “flash memory’, and a device including a
`mechanism for performing the above-described wear level
`ing, protocol processing for the upper apparatus, etc., is
`called a “flash memory device'. Although some improve
`ments have been made against the restriction on the number
`of writing times as a flash memory device by Such a
`technique, there still exists a restriction on the number of
`writing times of a flash memory device. Also, in addition to
`
`that, there is a drawback in that the throughput of a flash
`memory becomes similar to an HDD when an operation
`called erase becomes necessary at the time of writing to a
`flash memory.
`0008. As a technique for constituting a storage system
`using such a flash memory, there is, for example, Japanese
`Unexamined Patent Application Publication No. 6-324815.
`In this patent document, a technique in which frequently
`accessed parity data is stored in a semiconductor memory
`Such as a flash memory in a RAID configuration, etc., in
`order to improve performance of a storage system is
`described. However, means for preventing a restriction on
`the number of writing times as a storage system has not been
`disclosed. Also, one RAID group is constructed by mixing
`an HDD and a flash memory, that is to say, one virtual device
`is constituted, and thus a virtual device is not controlled in
`consideration of the characteristics of individual media.
`
`SUMMARY OF THE INVENTION
`0009 Under these circumstances, it is desirable to pro
`vide a storage system which is low in power consumption,
`has a small installation area, and is capable of constructing
`a large-scale System having a large capacity.
`0010 Also, it is desirable to provide a high system
`performance in accordance with the medium for storing
`data.
`0011 Moreover, it is also desirable to improve reliability
`and availability as a storage system. For a medium having a
`restriction of the number of writing times, it is necessary to
`alleviate the restriction as a storage system.
`0012. In the present invention, a storage system includes
`a plurality of channel controllers connecting to a host
`computer through channels and a cache memory containing
`a plurality of Volatile memories for temporarily storing data
`from the host computer. The storage system includes a
`plurality of first media having a restriction of the number of
`writing times and a plurality of first medium controllers for
`controlling the first media, and stores data from the host
`computer to the first media.
`0013 That is to say, according to the present invention,
`there is provided a storage system including a storage
`controller including: one medium or more for storing data
`from a host computer, a medium controller for controlling
`the medium; a channel controller for connecting to the host
`computer through a channel; and a cache memory including
`a volatile memory for temporarily storing data from the host
`computer, wherein the media at least partially includes a first
`medium having a restriction on a number of writing times.
`0014. In the storage system of the present invention, the
`power consumption is low, the installation area is Small, and
`it is possible to construct a large-scale system. Also, it is
`possible to provide a high system performance in accor
`dance with the medium for storing data. Also, there is an
`advantage in that the number of writing times to each
`medium is reduced, and thus it is possible to increase the
`reliability and the availability of the storage system even for
`the medium having a restriction on the number of writing
`times.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`0015 FIG. 1 is a block diagram of a configuration of a
`storage system according to a first embodiment;
`
`HPE, Exh. 1025, p. 19
`
`

`

`US 2007/0050571 A1
`
`Mar. 1, 2007
`
`0016 FIG. 2 is a block diagram of a detailed configura
`tion of a channel controller 11;
`0017 FIG. 3 is a block diagram of a detailed configura
`tion of an FM control unit 16;
`0018 FIG. 4 is a block diagram of another detailed
`configuration of the FM control unit 16:
`0019 FIG. 5 is a block diagram of another detailed
`configuration of the FM control unit 16:
`0020 FIG. 6 is a block diagram of a detailed configura
`tion of an internal Switch 12;
`0021
`FIG. 7 is a diagram illustrating the processing flow
`when a read request comes from a host computer 2 to an
`HDD 50 area;
`0022 FIG. 8 is a diagram illustrating the processing flow
`when a read request comes from a host computer 2 to a flash
`memory area;
`0023 FIG. 9 is a block diagram illustrating detailed data
`stored in a cache memory 13 and a control memory 17:
`0024 FIG. 10A is a block diagram illustrating details of
`read-cache directory information 1711 and write-cache
`directory information 1172;
`0.025
`FIG. 10B is a block diagram illustrating details of
`known access-sequence lists 1713 and 1714;
`0026 FIG. 10C is a block diagram illustrating details of
`access-sequence lists 1713 and 1714 for performing prefer
`able cache control;
`0027 FIG. 11A is a diagram illustrating the processing
`flow when a write request comes from the host computer 2
`and there is already the data of the address in the cache
`memory 13;
`0028 FIG. 11B is a diagram illustrating the processing
`flow when a write request comes from the host computer 2
`and there is not the data of the address in the cache memory
`13 or there is already no empty slot:
`0029 FIG. 12A is a diagram illustrating the processing
`for determining a slot to be destaged;
`0030 FIG. 12B is a diagram illustrating the processing
`for determining a slot in consideration of not only the
`number of stages but also the number of accesses;
`0031
`FIG. 13 is a block diagram of a storage system
`according to a second embodiment; and
`0032 FIG. 14 is a block diagram of a highly-functional
`FM control unit 160.
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENTS
`0033. A description will be given of a best mode of
`embodiment for carrying out the present invention.
`0034. In the following, a description will be given of a
`storage system, a storage device, and the control method
`thereof according to embodiments of the present invention
`based on the drawings.
`
`First Embodiment
`0035 A description will be given of a first embodiment.
`FIG. 1 is a block diagram of a configuration of a storage
`system according to the first embodiment of the present
`invention. The storage system includes a storage controller
`1, hard disk drives (HDDs) 50, and a flash memory device
`(in the figure, an example of an FM control unit 16 internally
`including flash memory devices is shown). The storage
`controller 1 is connected to a host computer 2 via a channel
`4 through a SAN (Storage Area Network) including a SAN
`Switch 3. Also, the storage controller 1 is connected to a
`plurality of HDDs 50 which are media storing data through
`a disk-side channel 60. The storage controller 1 includes a
`plurality of channel controllers 11, a plurality of cache
`memories 13, a control memory 17, a plurality of disk
`controllers 14, a plurality of FM control units 16, and
`internal Switches 12 connecting these through an internal
`bus 15. The channel controller 11 receives an input/output
`request from the host computer 2 through the channel 4.
`interprets the request type (for example, a read request and
`a write request) of this input/output request, the target
`address, etc., and performs the processing as described in
`FIG. 7 and below. The cache memories temporarily stores
`data to be stored in an HDD and a flash memory and data to
`be returned to the host computer 2. The control memory 17
`stores the directory information of the cache memories 13
`and the configuration information of the storage system. The
`disk controllers 14 control the HDDs 50 through a disk-side
`channel 60 based on a request of the channel controller 11,
`etc., and fetches and stores the data requested from the host
`computer 2. At this time, the disk controllers 14 may
`perform RAID control on the HDDs 50 in order to improve
`the reliability, availability, and performance of the storage
`system. The FM control units 16 perform the control of flash
`memories or the flash memory devices. The FM control
`units 16 fetch and store the data requested from the host
`computer 2 to the flash memories or the flash memory device
`based on a request of the channel controllers 11, etc. At this
`time, the FM control units 16 may perform RAID control on
`the flash memory devices in order to improve the reliability,
`availability, and performance of the storage system. In this
`regard, in the present embodiment, the storage system is
`connected to the HDDs 50. However, the storage system
`may have a configuration without having the HDDs 50 and
`the disk controllers 14. Also, the information stored in the
`control memory 17 may be physically located in the same
`memory as the cache memories 13.
`0036 FIG. 2 is a block diagram of the detailed configu
`ration of the channel controller 11. The channel controller 11
`includes a plurality of processors 111, a memory module
`112, a peripheral control unit 113, a plurality of channel
`protocol processors 114, and an internal network interface
`117. The processors 111 are connected to the peripheral
`control unit 113 through a bus, etc. The peripheral control
`unit 113 is connected to the memory module 112 and
`controls the memory module. Also, the peripheral control
`unit 113 is connected to the channel protocol processors 114
`and the internal network interface 117 through a control
`system bus 115. The peripheral control unit 113 receives a
`packet from the connected processors 111, the channel
`protocol processors 114, and the internal network interface
`117. If the destination address indicated by the packet falls
`on the memory module 112, the peripheral control unit 113
`performs the processing, and returns data if necessary. Also,
`
`HPE, Exh. 1025, p. 20
`
`

`

`US 2007/0050571 A1
`
`Mar. 1, 2007
`
`if the destination address falls outside, the peripheral control
`unit 113 performs appropriate forwarding. Also, the periph
`eral control unit 113 has a mail box 1131 for the other
`processors 111 to perform communication with the proces
`sor 111 connected to the peripheral control unit 113. The
`processors 111 access the memory module 112 through the
`peripheral control unit 113, and performs processing based
`on the control program 1121 stored in the memory module
`112. Also, the memory module 112 stores a transfer list 1123
`for the channel protocol processors 114 performing DMA
`(Direct Memory Access). The channel protocol processor
`114 performs protocol control on the channel 4, and converts
`into a protocol which can be processed in the storage system
`1. When the channel protocol processor 114 receives an
`input/output request from the host computer 2 through the
`channel 4, the channel protocol processor 114 notifies the
`host computer number, the LUN (Logical Unit Number), the
`access destination address of the input/output request, etc.,
`to the processor 111. The processor 111 accesses directory
`information 1323 based on the notification. If there is an
`address to which the input/output data is to be stored or the
`input/output data, the processor 111 creates the transfer list
`1123 in the memory module 112, and causes the channel
`protocol processor 114 to perform transfer based on the list.
`When the data requested for reading by the host computer 2
`is not in the cache memory 13, if the data is stored in the
`HDD 50, the processor 111 instructs the disk controller 14
`to store the requested data stored in the HDD 50 into the
`cache memory 13 (this operation is called 'staging”), and
`then causes to transfer the data by the transfer list 1123. If
`the data is stored in the flash memory, the address of the flash
`memory is set in the transfer list. The transfer list is an
`address list in the cache memory 13 or the flash memory. If
`the input/output request is writing, the data from the host
`computer is written into the address described in the list
`through the internal network interface 117 connected the
`data-transfer system bus 115. Also, if the request is reading,
`the data is similarly read from the address described in the
`list, and that data is returned to the host computer. The
`details of these operations are described using FIG. 7 and
`subsequent figures. The internal network interface 117 is a
`part to be an interface when communication is performed
`between the inside of the channel controller 11 and the inside
`of the other storage systems 1 through an internal bus 15.
`0037. In this regard, the disk controller 14 has a substan
`tially similar structure to the channel controller 11. However,
`the contents of the control program 1121 is different and the
`channel protocol processor 114 performs communication
`with the HDD 50 (the protocol of the channel 4 and that of
`the disk-side channel 60 may be different. However, the
`processing is the same as the channel protocol processors
`114 in the channel controller 11 in the sense that protocol
`processing of the disk-side channel 60 is performed and the
`conversion is performed in order to be processed in the
`storage system 1). The processor 111 writes the data in the
`cache memory 13 into a hard disk drive 50 by a request from
`the channel controller 11 or at regular time intervals. Also,
`if the data requested from the host computer is not in the
`cache memory 13, the processor 111 receives an instruction
`from the channel controller 11, reads data from the HDD 50,
`and writes the data into the cache memory 13. At these
`times, the processor 111 accesses the directory information
`stored in the control memory 17, and investigates the
`address of the cache memory from or to which the requested
`
`data by the host computer 2 is read or stored. When there is
`not the requested data in the cache memory 13, if there is no
`empty area in the cache memory 13, the existent data is
`stored into the HDD 50 for making an empty area in order
`to store the requested data (this operation is called destage).
`In the operation of the HDD 50, the disk controller 14
`controls the HDD 50 through the disk-side channel 60. At
`this time, in order to improve the availability and the
`performance as the entire HDDs 50, the disk controller 14
`performs the RAID control on the HDD 50 group.
`0038 FIG. 3 is a block diagram of the detailed configu
`ration of the FM control unit 16, in which flash memories are
`integrated. The FM control unit 16 includes an internal
`network interface 161, a DMA controller 162, a memory
`module 164 which is a volatile memory, a memory control
`ler 163 for controlling the memory module, flash memories
`166 (FM in the figure), FM controllers 165 which control the
`flash memories. The internal network interface 161 is a part
`to be an interface between the inside of the FM control unit
`16 and the inside of the other storage controller 1 through the
`internal bus 15. The DMA controller 162 in the FM control
`unit 16 performs data transfer from the cache memory 13 to
`the flash memory 166 using the transfer list 1641 set by the
`processor 111 of the channel controller 11 in the case of
`creating an empty area in the cache memory when process
`ing a write request from the host computer 2. The FM
`controller 165 controls the flash memory 166 to exchange
`data by a read request made from the channel controller 11
`through the internal network and a write request of the DMA
`controller 162. In FIG. 3, as an embodiment of the flash
`memory 166, it is possible to directly dispose the memory on
`a printed-circuit board. At this time, parts such as the
`connectors, the FM protocol processors in FIG. 4, and
`FM-side channels become unnecessary, and thus it is pos
`sible to achieve more compact storage system. Also, the
`wear leveling for each of the flash memories 166 may be
`performed by the FM controller 165.
`0039 FIG. 4 is a block diagram of another detailed
`configuration of the FM control unit 16. Here, a flash
`memory device 169 is used as a memory element. The flash
`memory device 169 is disposed separately. The flash
`memory device 169 is connected to the FM control unit 16
`through a connector 168, and thus the flash memory device
`is made detachable. Accordingly, when the flash memory
`device 169 has broken down, it can be replaced (in order to
`do this, the transfer list 1641 should be set such that the
`processor 111 of the channel controller 11 has a redundant
`configuration among the flash memory device 169 in
`advance). Also, it is possible to replace the flash memory
`device 169 itself by a flash memory device having a large
`capacity. The reliability and performance of this flash
`memory device 169 have been improved by employing a
`technique. Such as wear leveling, etc., inside the device.
`Data is exchanged with the outside using a special protocol.
`Thus, a FM protocol processor 167 converts the data into a
`format that can be processed in the storage controller 1.
`0040 FIG. 5 is a block diagram of another detailed
`configuration of the FM control unit 16. Here, the flash
`memory devices 169 are connected through FM-side chan
`nels 1610. By using this configuration, a larger number of
`flash memory devices 169 can be connected in order to
`achieve a storage system with a large capacity in addition to
`the features of the FM control unit 16 described in FIG. 4.
`
`HPE, Exh. 1025, p. 21
`
`

`

`US 2007/0050571 A1
`
`Mar. 1, 2007
`
`Also, part of the area of the flash memory devices 169 may
`be used as an urgent destage area 1690 described below.
`0041
`FIG. 6 is a block diagram of a detailed configura
`tion of the internal switch 12. The internal switch 12
`includes an internal network interface 121 and a plurality of
`selectors 122. The selector 122 analyzes the destination of
`the request sent from each part Such as the internal channel
`controller 11 of the storage controller 1, and transfers the
`request to the internal network interface 121 controlling the
`internal bus 15 connected to the request destination. At that
`time, each of the selectors performs the contention of the
`internal network interface 121 of the request-transfer desti
`nation. The internal switch 12 makes it possible for the
`channel controller 11 to directly exchange data with the
`cache-memory 13, the control memory 17, and the FM
`control unit 16. The FM control unit 16 can exchange data
`with the channel controller 11, the cache memory 13, and the
`control memory 17. Also, the disk controller 14 can directly
`exchange data with the cache memory 13 and the control
`memory 17. The difference on the connection of the FM
`control unit 16 and that of the disk controller 14 is that the
`internal Switch 12 has a connection 123 between the channel
`controller and the FM control unit, and thus the FM control
`unit 16 can directly exchange data with the channel con
`troller 11.
`0.042
`FIG. 7 is a diagram illustrating the processing flow
`when a read request comes from the host computer 2 to an
`HDD 50 area. The channel controller 11 receives a read
`request from the host computer 2 through the channel 4 (step
`s701). The processor 111 of the channel controller 11
`analyzes the received request and obtains an LUN and a
`target logical block address. Here, the processor 111 knows
`that the data is in an area to be stored in the HDD 50 (step
`s702). Furthermore, the processor 111 of the channel con
`troller 11 accesses the directory information of the write
`cache area and the read cache area, which is stored in the
`control memory 17, and checks whether there is the data
`stored in the cache memory 13 (steps s703 and s704. In the
`figure, accessed once. However, sometimes accessed a plu
`rality of times in reality. This is the same in the following).
`If there is already the data in cache memory 13, the
`processor 111 responds to the host computer 2 by the
`processing of steps715 and after. Here, the data is assumed
`not to be in the cache memory 13. In this case, the disk
`controller is caused to transfer the data to the cache memory
`13 (staging). However, is there is no empty area in the cache
`memory 13, the cache area for storing the data needs to be
`created before the staging. In steps s705 and s706, a deter
`mination operation is performed on which area is used for
`the empty area. After the area is obtained, a staging request
`is made to the disk controller 14 by writing a message into
`a communication area 173 of the control memory 17 (step
`s707). The disk controller 14 knows that there is a request
`from the channel controller 11 by reading the communica
`tion area 173 of the control memory 17 periodically or at
`each completion of a series of processing (steps s708 and
`s709