US007464221B2
`
`(12) United States Patent
`Nakamura et a].
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 7,464,221 B2
`Dec. 9, 2008
`
`(54) STORAGE SYSTEM, STORAGE DEVICE, AND
`CONTROL METHOD THEREOF
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`(75) 1
`nventors:
`
`JP
`Sh " N k
`K
`;
`up a amura, anagaWa
`Kazuhisa Fujimoto, Tokyo (JP); Akira
`Fujibayashi, KanagaWa (JP)
`
`(73) Assignee: Hitachi, Ltd., Tokyo (JP)
`
`2005/0172074 Al* 8/2005 Sinclair .................... .. 7ll/ll4
`
`FOREIGN PATENT DOCUMENTS
`
`JP
`
`06-324815
`
`ll/l994
`
`OTHER PUBLICATIONS
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 114 days.
`
`“Reducing Energy Consumption of Disk Storage Using Power
`Aware Cache Management” by Zhu, et al.
`European Patent Search to corresponding EP 06 250 654.8 dated
`Nov. 21, 2007.
`
`(21) App1.No.: 11/247,161
`
`(22) Filed:
`
`Oct. 12, 2005
`
`(65)
`
`Prior Publication Data
`
`* cited by examiner
`
`Primary ExamineriTuan V. Thai
`(74) Attorney, Agent, or FirmiMattingly, Stanger, Malur &
`Brundidge, PC.
`
`US 2007/0050571 A1
`
`Mar. 1, 2007
`
`(57)
`
`ABSTRACT
`
`(30)
`
`Foreign Application Priority Data
`
`Sep. 1,2005
`
`(JP)
`
`........................... .. 2005-252989
`
`(51) Int. Cl.
`(2006.01)
`G06F 13/00
`(2006.01)
`G06F 12/14
`(52) US. Cl. ...................... .. 711/113; 711/154; 711/163
`(58) Field of Classi?cation Search ............... .. 711/103,
`711/100,118,135,111,112,113,154
`See application ?le for complete search history.
`
`A storage system including a storage device 1 Which includes:
`media 50 for storing data from a host computer 2; a medium
`controller 14 for controlling the media; a plurality of channel
`controllers 11 for connecting to the host computer 2 through
`a channel; and a cache memory 13 for temporarily storing
`data from the host computer 2, Wherein the media 50 have a
`restriction on a number of Writing times. The storage device 1
`includes a bus 123 for directly transferring data from the
`medium controller 14 to the channel controller 11.
`
`3 Claims, 17 Drawing Sheets
`
`DESTAGE/STAGE PROCESSING
`
`M
`
`WHAT TYPE OF MEDIUM IS
`WRITE DESTINATION?
`
`$1201
`
`s1202
`
`DES‘I'AGE THE OLDEST SLOT usms HDD LIST
`
`L—____—
`51203
`
`SEARCH FOR
`FM WITH THE SMALLEST NUMBER OF DESTAGES
`
`ES
`
`LOT FOR USE BY THE F 7
`IS THERE A S
`ENGTH NOT ZERO?)
`(IS LIST I.
`
`No
`SEAR CH FOR
`FM WITH THE NEXT SMALLEST NUMBER
`
`I’
`
`51204
`
`SI 205
`
`DESTAGE THE OLDEST SLOT OF THE FM
`
`51206
`
`@
`$1207
`
`ADD 1 TO
`NUMBER OF DESTAGES OF THE MEDIUM
`
`REMOVE
`THE SLOT FROM DIRECTORY INFORMATION
`AND ACCESS SEQUENCE LIST
`
`sI208
`
`ADD DATA TO BE STAGED TO
`MRU OF DIRECTORY INFORMADON
`AND ACCESS SEQUENCE LIST
`
`STAGE NEW DATA TO THE SLOT
`
`s I 21 0
`
`HPE, Exh. 1004, p. 1
`
`

`

`US. Patent
`
`Dec. 9,2008
`
`Sheet 1 or 17
`
`US 7,464,221 B2
`
`CZJSZGQ CZEZCQ
`
`1 4
`/1/
`/
`
`14
`’
`1/1]
`
`A/
`/
`
`/
`
`1 6
`n/
`7
`
`16
`A/
`7
`
`CHANNEL
`CHANNEL
`CONTROLLER CONTROLLER
`
`FM
`60ml?“
`
`FM
`806$?‘-
`
`DISK
`DISK
`CONTROLLER CONTROLLER
`
`I \
`
`T
`
`L7
`
`\ T
`
`\
`
`1
`
`12
`
`12
`
`INTERNAL sWiTCH
`
`INTERNAL sWlTCH q/
`
`1 5
`
`CACHE
`MEMORY
`
`13
`
`1 T
`CONTROL 4/
`MEMORY
`
`1 3
`CACHE 4/
`MEMORY
`
`17
`CONTROL /
`MEMORY 1
`
`STORAGE CONTROLLER
`
`\
`
`HPE, Exh. 1004, p. 2
`
`

`

`U.S. Patent
`
`Dec. 9, 2008
`
`Sheet 2 of 17
`
`US 7,464,221 B2
`
` XO8TIVA
`TOIOLONd||TODOLOYd||TODOLOed|)TODOLOYd
`
`
`
`TANNVHOTANNYHDJANNVYHOTANNVHS
`
`
`
`TANNVHO-FTANNVHO:FTSNNVHO-7qJaNNVWHO:P
`
`
`AJOVAYSALNIMYHOMLANTWNYSLNI
`YOSS3AIONd||HOSSAIONd|}HOSSADONd|}HOSSADOUd
`
`LINNIOYLNOD
`WdaHdldad
`
`YATIOULNOOTANNVHO
`
`WVed0ud
`
`éOld
`
`HPE, Exh. 1004, p. 3
`
`FIL
`
`LLL
`
`Lb
`
`HPE, Exh. 1004, p. 3
`
`

`

`U.S. Patent
`
`Dec. 9, 2008
`
`Sheet 3 of 17
`
`US 7,464,221 B2
`
`YSASNVYL
`
`vOL
`
`YATIONLNOO
`
`COL
`
`YATIONLNOOVWG
`
`AYOWSW
`
`aInNaowWSWaW4Wa
`
`GOLGOLGOLGOL
`
`
`
`
`
`FOVAYSLNIMYOMLINTWNUYALLNI
`
`LVOL|Wa||Wa|W4
`isnSQSOSQLo
`"NLSLSILNL
`
`991991991991
`
`
`
`LOLLINNTONLNOOWa
`
`GL
`
`OL
`
`€Sls
`
`HPE, Exh. 1004, p. 4
`
`HPE, Exh. 1004, p. 4
`
`
`

`

`US. Patent
`
`Dec. 9, 2008
`
`Sheet 4 0f 17
`
`US 7,464,221 B2
`
`F 16.4
`
`/I/16
`
`FM CONTROL UNIT
`
`/7//1 5
`
`INTERNAL NETWORK INTERFACE
`
`I
`T
`DMA CONTROLLER
`
`'1 61
`
`1 63
`1/
`MEMORY
`162 CONTROLLER
`1/
`
`167
`1/
`FM PROTOCOL
`PROCEssOR
`
`167
`T[
`FM PROTOCOL
`PROCESSOR
`
`167
`1/
`FM PROTOCOL
`PROcEssOR
`
`164
`167
`K
`1/"
`FM PROTOCOL ‘£382’
`PROCESSOR
`
`168
`I
`T
`163
`l
`ONNECTOR
`[CONNECTOBT
`[CONNECTOKF
`PLOONNEOTOR]
`I
`I
`168
`; 165k
`I
`_[CONNECTOR1_ QOONNECTORL‘ TEONNECTOR1]__1 _%ONNECTOR
`
`TRANSFER
`LUST
`\ I
`
`1 641
`
`FM DEVICE
`
`FM DEVICE
`
`FM DEVICE
`
`FM DEVICE
`
`)
`&‘I69
`
`2
`&169
`
`\
`
`D
`K169
`
`D
`(\169
`
`HPE, Exh. 1004, p. 5
`
`

`

`US. Patent
`
`Dec. 9, 2008
`
`Sheet 5 0f 17
`
`US 7,464,221 B2
`
`F1G.5
`
`/,/16
`
`FM CONTROL UNIT
`
`4/15
`
`INTERNAL NETWORK INTERFACE
`
`T
`
`DMA CONTROLLER
`
`161
`/1/
`/
`
`163
`7IVIEIVIORY
`
`167
`f
`FM PROTOCOL
`PROCESSOR
`
`167
`f
`FM PROTOCOL
`PROCESSOR
`
`167
`If
`FM PROTOCOL
`PROCESSOR
`
`1 54
`167
`/
`f
`FM PROTOCOL 121E118‘?
`PROCESSOR
`OD ‘
`
`16105
`
`5
`1510
`
`5
`1610
`
`5
`16m
`
`TRANSFER
`LIST
`
`x H Yn1
`1641
`
`5 FM DEVICE 1 FM DEVICE — FM DEVICE 1 FM DEVICE 7
`
`1 59
`
`— FM DEVICE
`
`FM DEVICE — FM DEVICE - FM DEVICE
`
`1 69
`
`- FM DEVICE ~ FM DEVICE ~ FM DEVICE - ,FM DEVICE
`
`1 6 9
`
`-—1 FM DEVICE — FM DEVICE — FM DEVICE 5 FM DEVICE
`
`1 69
`
`I _ _ _ ."L \
`
`l 711
`
`I
`
`I
`
`\ _. ._ _ _ -
`
`1 59
`
`HPE, Exh. 1004, p. 6
`
`

`

`US. Patent
`
`Dec. 9, 2008
`
`Sheet 6 0f 17
`
`US 7,464,221 B2
`
`FIG.6
`
`MI I
`
`/
`
`CHAN NEL
`CONTROLLER
`
`I6
`
`M14
`
`FM
`CONTROL
`UNH.
`
`DISK
`CONTROLLER
`
`121
`//
`
`INTERNAL NETWORK
`INTERFACE
`
`INTERNAL NETWORK
`INTERFACE
`
`121
`J/
`
`121
`W
`
`INTERNAL NETWORK
`INTERFACE
`
`123:00NNE0TI0N BETWEEN
`CHANNEL CONTROLLER
`AND FM CONTROL UNIT
`
`N
`
`N
`
`122
`
`A
`
`122
`
`122
`
`121 122
`/I/
`INTERNAL NETWORK’
`INTERFACE
`
`121
`n/
`INTERNAL NETWORK’
`INTERFACE
`
`INTERNAL SWITCH
`
`I
`
`/
`
`CACHE
`MEMORY
`
`/
`
`CONTROL
`MEMORY
`
`HPE, Exh. 1004, p. 7
`
`

`

`US. Patent
`
`Dec. 9, 2008
`
`Sheet 7 0f 17
`
`US 7,464,221 B2
`
`13: CACHE MEMORY
`11.‘ CHANNEL CONTROLLER”: CONTROL MEMORY
`
`_
`14. DISK CONTROLLER
`
`57ml S702
`
`/s708
`
`$710
`
`$711
`
`HPE, Exh. 1004, p. 8
`
`

`

`US. Patent
`
`Dec. 9, 2008
`
`Sheet 8 0f 17
`
`US 7,464,221 B2
`
`F168
`
`11: CHANNEL CONTROLLER 1'7: CONTROL MEMORY
`
`162 FM CONTROL UNIT
`
`$802
`
`HPE, Exh. 1004, p. 9
`
`

`

`US. Patent
`
`Dec. 9, 2008
`
`Sheet 9 0f 17
`
`US 7,464,221 B2
`
`FIG .9
`//I 3
`
`CACHE MEMORY
`
`READ CACHE AREA
`
`I 31
`r/V
`
`WRITE CACHE AREA
`
`I 32
`l/l/
`
`'
`
`II
`\ A \
`V I
`I
`V
`SLOT
`
`/I/ \
`MEMORY
`\
`
`IT
`
`CONTROL
`
`I
`
`I
`/
`
`I---
`
`I
`
`I
`/
`
`I
`
`I
`
`I--
`
`/ / M172
`
`M173
`
`I711
`
`I 712
`
`IIIIIIII
`
`1713
`kgjj __
`
`/
`/
`CONFIGURATION COMMUNICATION
`INFORMATION
`AREA
`
`DIRECTORY
`INFORMATION
`
`ACCESS
`SEQUENCE LIST
`
`HPE, Exh. 1004, p. 10
`
`

`

`U.S. Patent
`
`Dec. 9, 2008
`
`Sheet 10 of 17
`
`US 7,464,221 B2
`
`FIG.10A
`
`
`
`1001
`
`
`
`soooet]HDD
`
`0000-1fff}| HDDO =
`
`FiG.10B
`
`1713:
`
`1714:
`
`CMslot4
`
`CMslot5
`
`CMslot6
`
`Nil(LRU)
`
`HPE, Exh. 1004, p. 11
`
`HPE, Exh. 1004, p. 11
`
`

`

`US. Patent
`
`Dec. 9,2008
`
`Sheet 11 or 17
`
`US 7,464,221 B2
`
`FIG.1OC
`
`1061
`
`1057
`
`.1 FM’!
`
`ICMsl0t7
`
`I CMslotS
`
`CMBIOtZ % Ni|(|_RU)
`
`NUMBER OF
`10635- DESTAGES
`NUMBER OF
`10645- AGCEssEs
`CAPACITY
`
`1065
`
`256
`2203
`
`0 G
`1 0 B
`
`1061
`
`1057
`
`IFMZ
`
`I
`
`CMsIotS T2» CMslotS —z_> CMSIOM
`
`NMLRU)
`
`NUMBER OF
`1063_§— DESTAGES
`NUMBER OF
`10645- ACCEssEs
`CAPACITY
`
`1065_5-
`
`1024
`
`3223
`0 G
`1 0 B
`
`1062
`
`7
`HOUR
`
`1067
`[11/
`CMslotB Y9 CMs1ot1
`
`7+ CMslotQ
`
`Nil(LRU)
`
`DESTAGE
`1D66_'>—— RESTRICTION
`NUMBER OF
`10635- DESTAGES
`NUMBER OF
`1064;- ACCESSES
`CAPACITY
`
`1 065
`
`NONE
`
`10524
`
`113223
`600GB
`
`HPE, Exh. 1004, p. 12
`
`

`

`U.S. Patent
`
`Dec. 9, 2008
`
`Sheet 12 of 17
`
`US 7,464,221 B2
`
`YATIONLNOOUSIC-71/LIND1ONLNOOWa91
`SHOVO“E+
`
`
`
`
`
`
`
`
`
`YATIONLNOOTANNVHD‘+1
`
`ViOla
`
` AYOWAWN
`cOLls|SLIMMLSOH
`
`AYOWASWTON.LNOS‘21
`
`EOLLS
`
`LOLLS
`
`SOLES
`
`9OLLS
`
`HPE, Exh. 1004, p. 13
`
`HPE, Exh. 1004, p. 13
`
`

`

`US. Patent
`
`Dec. 9, 2008
`
`Sheet 13 0f 17
`
`US 7,464,221 B2
`
`F|G.1 1B
`
`17: CONTROL MEMORY
`11: CHANNEL CONTROLLER
`51
`
`13: CACHE MEMORY
`
`16'. FM CONTROL UNIT}
`DISK CONTROLLER
`
`HOST WRITE 2
`
`51 131
`
`$1133
`
`$1130
`
`$1132 11 S1135
`
`S1134
`
`HPE, Exh. 1004, p. 14
`
`

`

`US. Patent
`
`Dec. 9 ,2008
`
`Sheet 14 0f 17
`
`US 7,464,221 B2
`
`FIG.12A
`
`i/HDD
`
`C DESTAGE/STAGE PROCESSING 3
`‘I
`51201
`WHAT TYPE OF MEDIUM IS
`WRITE DESTINATION?
`
`FM
`
`51202
`
`ESTAGE THE OLDEST SLOT USING HDD LIST/
`
`I
`51203
`SEARCH FOR
`//I/
`FM WITH THE SMALLEST NUMBER OF DESTAGES
`
`YES
`
`IS THERE A SLOT FOR USE BY THE FM?
`(IS LIST LENGTH NOT ZERO?)
`
`$1204
`
`A 51205
`
`51206
`
`NO
`SEARCH FOR
`FM WITH THE NEXT SMALLEST NUMBER
`I
`DESTAGE THE OLDEST SLOT OF THE FM/
`I
`ADD1TO
`$1207
`NUMBER OF DESTAGES OF THE MEDIUM/ %
`I
`
`REMOVE
`THE SLOT FROM DIRECTORY INFORMATION
`AND ACCESS SEQUENCE LIST
`
`S1208
`
`II
`
`ADD DATA TO BE STAGED TO
`MRU OF DIRECTORY INFORMATION /
`AND ACCESS SEQUENCE LIST
`
`SI209
`
`I
`
`51210
`
`STAGE NEW DATA TO THE SLOT /
`
`C END I 3
`
`HPE, Exh. 1004, p. 15
`
`

`

`US. Patent
`
`Dec. 9, 2008
`
`Sheet 15 0f 17
`
`US 7,464,221 B2
`
`( DESTAGE/STAGE PROCESSING D
`
`SEARCH FOR MEDIUM WITH THE SMALLEST NUMBER /
`(CI x NUMBER OF DESTAGES + B + NUMBER OF ACCESSES)
`
`$1241
`
`6
`
`s] 242
`
`YES
`
`IS THERE A SLOT FOR THE MEDIUM?
`(IS LIST LENGTH ZERO?)
`
`No
`
`SEARCH FOR
`FM WITH THE NEXT SMALLEST NUMBER
`
`81243
`/\/
`
`51 244
`DESTAGE THE OLDEST SLOT OF THE FM/V
`
`W51 245
`ADD 1 TO
`NUMBER OF DESTAGES OF THE MEDIUM/
`I
`REMOVE
`THE SLOT FROM DIRECTORY INFORMATION/
`AND ACCESS SEQUENCE LIST
`
`81246
`
`ADD DATA TO BE STAGED TO DIRECTORY INFORMATION /I/S1247
`AND MRU OF ACCESS SEQUENCE LIST
`/
`
`/
`
`STAGE NEW DATA To THE SLOT
`
`s1 248
`
`I
`
`c
`
`>
`
`HPE, Exh. 1004, p. 16
`
`

`

`US. Patent
`
`Dec. 9, 2008
`
`Sheet 16 0f 17
`
`US 7,464,221 B2
`
`SAN SWITCH
`
`FMD
`
`E152}
`15m @ 50 _@ 60
`/I/ ELEV/M
`/I/
`W
`
`/I /
`/ V
`
`I
`
`1
`L
`V
`T
`CHANNEL
`CHANNEL
`CONTROLLER CONTROLLER
`
`160
`1
`
`J
`/V
`/
`l
`l
`DISK
`HIGHLY-FUNCTIONAL IGHLY-FUNCTIONALF’
`I=M CONTROL uNIT
`FM CONTROL UNIT ‘ CONTROLLER
`
`14
`/
`/I/
`/
`DISK
`CONTROLLER
`
`L
`
`F
`
`1 2
`INTERNAL SWITCH 4/
`
`INTERNAL SWITCH
`
`1 3
`
`lM
`
`CACHE
`MEMORY
`
`PM
`CONTROL
`uNIT
`\
`L] 6
`STORAGE CONTROLLER
`
`1 7
`
`/
`
`CONTROL
`MEMORY
`
`1 3
`
`?ll
`
`CACHE
`MEMORY
`
`1 7
`
`.I/I/
`
`CONTROL
`MEMORY
`
`PM
`CONTROL
`uNIT
`\
`L] 5
`
`1 LI
`
`HPE, Exh. 1004, p. 17
`
`

`

`US. Patent
`
`Dec. 9, 2008
`
`Sheet 17 0f 17
`
`US 7,464,221 B2
`
`160 f I
`
`HIGHLY-FUNCTIONAL FM CONTROL UNIT
`"1
`f
`
`1'
`
`I 61
`L
`
`j
`
`PROCESSOR PROCESSOR PROCESSOR PROCESSOR
`
`INTERNAL NETWORK INTERFACE
`
`1 1 3
`If
`PERIPHERAL
`CONTROL UNIT
`
`1 1 2
`f
`IVI’EMORY
`MODULE
`
`1 1 21 [
`éONTROL
`
`PROGRAM
`
`MAIL BOX
`
`K 1122
`
`IENSFER
`1H“ 1541
`
`DMA CONTROLLER
`
`1 57
`If-
`
`[1 62
`/
`
`167
`If
`
`FM PROTOCOL
`PROCESSOR
`
`FM PROTOCOL
`PROCESSOR
`
`-1 FM DEVICE
`
`1 610
`
`1 61 1)
`
`— FM DEVICE
`
`- FM DEVICE \
`
`- FM DEVICE
`
`5
`
`I
`
`I 69 ,3
`
`— FM DEVICE
`
`7 FM DEVICE
`
`J FM DEVICE
`
`-— FM DEVICE
`
`HPE, Exh. 1004, p. 18
`
`

`

`US 7,464,221 B2
`
`1
`STORAGE SYSTEM, STORAGE DEVICE, AND
`CONTROL METHOD THEREOF
`
`The present application is based on and claims priority of
`Japanese patent applications No. 2005-252989 ?led on Sep.
`1, 2005, the entire contents of Which are hereby incorporated
`by reference.
`
`BACKGROUND OF THE INVENTION
`
`2
`Patent Application Publication No. 6-324815. In this patent
`document, a technique in Which frequently-accessed parity
`data is stored in a semiconductor memory such as a ?ash
`memory in a RAID con?guration, etc., in order to improve
`performance of a storage system is described. HoWever,
`means for preventing a restriction on the number of Writing
`times as a storage system has not been disclosed. Also, one
`RAID group is constructed by mixing an HDD and a ?ash
`memory, that is to say, one virtual device is constituted, and
`thus a virtual device is not controlled in consideration of the
`characteristics of individual media.
`
`SUMMARY OF THE INVENTION
`
`Under these circumstances, it is desirable to provide a
`storage system Which is loW in poWer consumption, has a
`small installation area, and is capable of constructing a large
`scale system having a large capacity.
`Also, it is desirable to provide a high system performance
`in accordance With the medium for storing data.
`Moreover, it is also desirable to improve reliability and
`availability as a storage system. For a medium having a
`restriction of the number of Writing times, it is necessary to
`alleviate the restriction as a storage system.
`In the present invention, a storage system includes a plu
`rality of channel controllers connecting to a host computer
`through channels and a cache memory containing a plurality
`of volatile memories for temporarily storing data from the
`host computer. The storage system includes a plurality of ?rst
`media having a restriction of the number of Writing times and
`a plurality of ?rst medium controllers for controlling the ?rst
`media, and stores data from the host computer to the ?rst
`media.
`That is to say, according to the present invention, there is
`provided a storage system including a storage controller
`including: one medium or more for storing data from a host
`computer; a medium controller for controlling the medium; a
`channel controller for connecting to the host computer
`through a channel; and a cache memory including a volatile
`memory for temporarily storing data from the ho st computer,
`Wherein the media at least partially includes a ?rst medium
`having a restriction on a number of Writing times.
`In the storage system of the present invention, the poWer
`consumption is loW, the installation area is small, and it is
`possible to construct a large-scale system. Also, it is possible
`to provide a high system performance in accordance With the
`medium for storing data. Also, there is an advantage in that
`the number of Writing times to each medium is reduced, and
`thus it is possible to increase the reliability and the availability
`of the storage system even for the medium having a restriction
`on the number of Writing times.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a block diagram of a con?guration of a storage
`system according to a ?rst embodiment;
`FIG. 2 is a block diagram of a detailed con?guration of a
`channel controller 11;
`FIG. 3 is a block diagram of a detailed con?guration of an
`FM control unit 16;
`FIG. 4 is a block diagram of another detailed con?guration
`of the FM control unit 16;
`FIG. 5 is a block diagram of another detailed con?guration
`of the FM control unit 16;
`FIG. 6 is a block diagram of a detailed con?guration of an
`internal sWitch 12;
`
`1. Field of the Invention
`The present invention relates to a storage device and the
`control method thereof.
`2. Description of the Related Art
`In recent years, the reduction of the total cost of oWnership
`(TCO) of a storage system becomes increasingly important in
`information business sites such as a data center. On the other
`hand, demand for recording data reliably for a long period of
`time is increasing. As an example of this fact, the document
`data of ?nancial institutions, medical institutions, etc., are
`required to be stored Without being erased by laW. Under these
`circumstances, there is much demand for highly reliable stor
`age systems having a large capacity. HoWever, in general, in
`a large-scale storage system using hard-disk drives (in the
`folloWing, referred to as “HDDs”), the amount of poWer
`consumption increases in proportion to the storage capacity.
`That is to say, the possession of a storage system With a large
`capacity means an increase in the total cost of oWnership
`including electricity charges. In vieW of such situations, a
`technique for reducing the poWer consumption of HDDs by a
`cache management algorithm has been proposed (N onpatent
`document; ZHU, Q., DAVID, F., ZHOU, Y., DEVARAJ, C.,
`AND CAO, P., “Reducing Energy Consumption of Disk Stor
`age Using PoWer-AWare Cache Management”. InProc. of the
`10th Intl. Symp. on High Performance Computer Architec
`ture (HPCA-lO) (February 2004)). Also, the problem is not
`limited to the electricity charges. In general, the ?oor area for
`installation increases as the capacity of a storage system
`increases. This also increases the total cost of oWnership.
`Incidentally, a ?ash memory attracts attention as a nonvola
`tile medium in recent years. A ?ash memory commonly con
`sumes less than one several-tenth of poWer When compared
`With an HDD, and can be read at a high speed. Also, a ?ash
`memory is small siZed unlike an HDD having a mechanically
`driven part.
`HoWever, a ?ash memory has a restriction on the number of
`Writing times because of the physical constitution of a cell for
`holding information. Against such a restriction, the number of
`Writing times of a ?ash memory has been improved by a
`technique called Wear leveling, in Which the number of Writ
`ing times to each cell is controlled to be averaged by having
`correspondence betWeen an address to be shoWn to the upper
`apparatus and a cell position. In this regard, in the folloWing,
`an element for holding information is simply called a “?ash
`memory”, and a device including a mechanism for perform
`ing the above-described Wear leveling, protocol processing
`for the upper apparatus, etc., is called a “?ash memory
`device”. Although some improvements have been made
`against the restriction on the number of Writing times as a
`?ash memory device by such a technique, there still exists a
`restriction on the number of Writing times of a ?ash memory
`device. Also, in addition to that, there is a drawback in that the
`throughput of a ?ash memory becomes similar to an HDD
`When an operation called erase becomes necessary at the time
`of Writing to a ?ash memory.
`As a technique for constituting a storage system using such
`a ?ash memory, there is, for example, Japanese Unexamined
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`HPE, Exh. 1004, p. 19
`
`

`

`US 7,464,221 B2
`
`3
`FIG. 7 is a diagram illustrating the processing ?oW When a
`read request comes from a host computer 2 to an HDD 50
`area;
`FIG. 8 is a diagram illustrating the processing ?oW When a
`read request comes from a host computer 2 to a ?ash memory
`area;
`FIG. 9 is a block diagram illustrating detailed data stored in
`a cache memory 13 and a control memory 17;
`FIG. 10A is a block diagram illustrating details of read
`cache directory information 1711 and Write-cache directory
`information 1172;
`FIG. 10B is a block diagram illustrating details of knoWn
`access-sequence lists 1713 and 1714;
`FIG. 10C is a block diagram illustrating details of access
`sequence lists 1713 and 1714 forperforming preferable cache
`control;
`FIG. 11A is a diagram illustrating the processing ?oW
`When a Write request comes from the host computer 2 and
`there is already the data of the address in the cache memory
`13;
`FIG. 11B is a diagram illustrating the processing ?oW
`When a Write request comes from the host computer 2 and
`there is not the data of the address in the cache memory 13 or
`there is already no empty slot;
`FIG. 12A is a diagram illustrating the processing for deter
`mining a slot to be destaged;
`FIG. 12B is a diagram illustrating the processing for deter
`mining a slot in consideration of not only the number of stages
`but also the number of accesses;
`FIG. 13 is a block diagram of a storage system according to
`a second embodiment; and
`FIG. 14 is a block diagram of a highly-functional FM
`control unit 160.
`
`20
`
`25
`
`30
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENTS
`
`A description Will be given of a best mode of embodiment
`for carrying out the present invention.
`In the folloWing, a description Will be given of a storage
`system, a storage device, and the control method thereof
`according to embodiments of the present invention based on
`the draWings.
`
`First Embodiment
`
`35
`
`40
`
`45
`
`50
`
`A description Will be given of a ?rst embodiment. FIG. 1 is
`a block diagram of a con?guration of a storage system accord
`ing to the ?rst embodiment of the present invention. The
`storage system includes a storage controller 1, hard disk
`drives (HDDs) 50, and a ?ash memory device (in the ?gure,
`an example of an FM control unit 16 internally including ?ash
`memory devices is shoWn). The storage controller 1 is con
`nected to a host computer 2 via a channel 4 through a SAN
`(Storage Area Network) including a SAN sWitch 3. Also, the
`55
`storage controller 1 is connected to a plurality of HDDs 50
`Which are media storing data through a disk-side channel 60.
`The storage controller 1 includes a plurality of channel con
`trollers 11, a plurality of cache memories 13, a control
`memory 17, a plurality of disk controllers 14, a plurality of
`FM control units 16, and internal sWitches 12 connecting
`these through an internal bus 15. The channel controller 11
`receives an input/output request from the host computer 2
`through the channel 4, interprets the request type (for
`example, a read request and a Write request) of this input/
`output request, the target address, etc., and performs the pro
`cessing as described in FIG. 7 and beloW. The cache memo
`
`60
`
`65
`
`4
`ries temporarily stores data to be stored in an HDD and a ?ash
`memory and data to be returned to the host computer 2. The
`control memory 17 stores the directory information of the
`cache memories 13 and the con?guration information of the
`storage system. The disk controllers 14 control the HDDs 50
`through a disk-side channel 60 based on a request of the
`channel controller 11, etc., and fetches and stores the data
`requested from the host computer 2. At this time, the disk
`controllers 14 may perform RAID control on the HDDs 50 in
`order to improve the reliability, availability, and performance
`of the storage system. The PM control units 16 perform the
`control of ?ash memories or the ?ash memory devices. The
`PM control units 16 fetch and store the data requested from
`the host computer 2 to the ?ash memories or the ?ash memory
`device based on a request of the channel controllers 11, etc. At
`this time, the FM control units 16 may perform RAID control
`on the ?ash memory devices in order to improve the reliabil
`ity, availability, andperformance of the storage system. In this
`regard, in the present embodiment, the storage system is
`connected to the HDDs 50. HoWever, the storage system may
`have a con?guration Without having the HDDs 50 and the disk
`controllers 14. Also, the information stored in the control
`memory 17 may be physically located in the same memory as
`the cache memories 13.
`FIG. 2 is a block diagram of the detailed con?guration of
`the channel controller 11. The channel controller 11 includes
`a plurality of processors 1 11, a memory module 112, a periph
`eral control unit 113, a plurality of channel protocol proces
`sors 114, and an internal netWork interface 117. The proces
`sors 111 are connected to the peripheral control unit 113
`through a bus, etc. The peripheral control unit 113 is con
`nected to the memory module 112 and controls the memory
`module. Also, the peripheral control unit 113 is connected to
`the channel protocol processors 114 and the internal netWork
`interface 117 through a control system bus 115. The periph
`eral control unit 113 receives a packet from the connected
`processors 111, the channel protocol processors 114, and the
`internal netWork interface 1 17. If the destination address indi
`cated by the packet falls on the memory module 112, the
`peripheral control unit 113 performs the processing, and
`returns data if necessary. Also, if the destination address falls
`outside, the peripheral control unit 113 performs appropriate
`forWarding. Also, the peripheral control unit 113 has a mail
`box 1131 for the other processors 111 to perform communi
`cation With the processor 111 connected to the peripheral
`control unit 113. The processors 111 access the memory
`module 112 through the peripheral control unit 113, and
`performs processing based on the control program 1121
`stored in the memory module 112. Also, the memory module
`112 stores a transfer list 1123 for the channel protocol pro
`cessors 114 performing DMA (Direct Memory Access). The
`channel protocol processor 114 performs protocol control on
`the channel 4, and converts into a protocol Which can be
`processed in the storage system 1. When the channel protocol
`processor 114 receives an input/output request from the host
`computer 2 through the channel 4, the channel protocol pro
`cessor 114 noti?es the host computer number, the LUN
`(Logical Unit Number), the access destination address of the
`input/output request, etc., to the processor 111. The processor
`111 accesses directory information 1323 based on the noti?
`cation. If there is an address to Which the input/output data is
`to be stored or the input/ output data, the processor 111 creates
`the transfer list 1123 in the memory module 112, and causes
`the channel protocol processor 114 to perform transfer based
`on the list. When the data requested for reading by the host
`computer 2 is not in the cache memory 13, if the data is stored
`in the HDD 50, the processor 111 instructs the disk controller
`
`HPE, Exh. 1004, p. 20
`
`

`

`US 7,464,221 B2
`
`6
`5
`
`14 to store the requested data stored in the HDD 50 into the is possible to directly dispose the memory onaprinted-circuit
`cache memory 13 (this operation is called “staging”’), and
`board. At this time, parts such as the connectors, the FM
`then causesto transfer the data by the transfer list 1123. If the
`protocol processors in FIG. 4, and FM-side channels become
`data is stored in the flash memory, the address of the flash
`unnecessary, and thus it is possible to achieve more compact
`memory isset in the transferlist. The transferlist is an address
`storage system. Also, the wear leveling for each ofthe flash
`list in the cache memory 13 or the flash memory. If the
`memories 166 may be performed by the FM controller 165.
`input/output request is writing, the data from the host com-
`FIG.4 is a block diagram of another detailed configuration
`puter is written into the address describedin the list through
`ofthe FM control unit 16. Here, a flash memory device 169 is
`the internal network interface 117 connected the data-transfer
`used as a memory element. The flash memory device 169 is
`disposed separately. The flash memory device 169 is con-
`nected to the FM controlunit 16 through a connector 168, and
`thus the flash memory device is made detachable. Accord-
`ingly, when the flash memory device 169 has broken down,it
`can be replaced (in order to do this, the transfer list 1641
`should be set such that the processor 111 of the channel
`controller 11 has a redundant configuration among the flash
`memory device 169 in advance). Also, it is possibleto replace
`the flash memory device 169 itself by a flash memory device
`having a large capacity. The reliability and performance of
`this flash memory device 169 have been improved by employ-
`ing a technique, such as wear leveling, etc., inside the device.
`Data is exchanged with the outside using a special protocol.
`Thus, a FM protocol processor 167 converts the data into a
`format that can be processed in the storage controller 1.
`FIG. 5 is a block diagram of another detailed configuration
`ofthe FM control unit 16. Here, the flash memory devices 169
`are connected through FM-side channels 1610. By using this
`configuration, a larger numberof flash memory devices 169
`can be connected in order to achieve a storage system with a
`large capacity in addition to the features of the FM control
`unit 16 described in FIG.4. Also, part of the area of the flash
`memory devices 169 may be used as an urgent destage area
`1690 described below.
`
`20
`
`25
`
`30
`
`35
`
`FIG.6 is a block diagram of a detailed configuration of the
`internal switch 12. The internal switch 12 includesan internal
`
`system bus 115. Also, if the request is reading, the data is
`similarly read from the address described in the list, and that
`data is returned to the host computer. The details of these
`operations are described using FIG.7 and subsequentfigures.
`The internal network interface 117 is a part to be an interface
`when communication is performed betweenthe inside of the
`channel controller 11 and the inside of the other storage
`systems 1 through an internal bus 15.
`In this regard, the disk controller 14 has a substantially
`similar structure to the channel controller 11. However, the
`contents of the control program 1121 is different and the
`channel protocol processor 114 performs communication
`with the HDD 50 (the protocol ofthe channel4 andthat of the
`disk-side channel 60 maybe different. However, the process-
`ing is the same as the channel protocol processors 114 in the
`channel controller 11 in the sense that protocol processing of
`the disk-side channel 60 is performed and the conversion is
`performed in order to be processedin the storage system 1).
`The processor 111 writes the data in the cache memory 13
`into a hard disk drive 50 by a request from the channel
`controller 11 or at regular time intervals. Also, if the data
`requested from the host computer is not in the cache memory
`13, the processor 111 receives an instruction from the channel
`controller 11, reads data from the HDD 50, and writes the data
`into the cache memory 13. At these times, the processor 111
`accesses the directory information stored in the control
`memory 17, and investigates the address ofthe cache memory
`from or to which the requested data by the host computer 2 is
`read or stored. When there is not the requested data in the
`cache memory 13, if there is no empty area in the cache
`memory 13, the existent data is stored into the HDD 50 for
`making an emptyarea in orderto store the requested data (this
`operation is called destage). In the operation of the HDD 50,
`the disk controller 14 controls the HDD 50 through the disk-
`side channel 60. At this time, in order to improve the avail-
`ability and the performance as the entire HDDs50, the disk
`controller 14 performs the RAID control on the HDD 50
`group.
`FIG.3 is a block diagram ofthe detailed configuration of
`the FM control unit 16, in which flash memories are inte-
`grated. The FM control unit 16 includes an internal network
`interface 161, a DMA controller 162, a memory module 164
`which is a volatile memory, a memory controller 163 for
`controlling the memory module, flash memories 166 (FM in
`the figure), FM controllers 165 which control the flash memo-
`ries. The internal network interface 161 is a part to be an
`interface between the inside ofthe FM controlunit 16 and the
`
`network interface 121 and a plurality of selectors 122. The
`selector 122 analyzesthe destination of the request sent from
`each part such as the internal channel controller 11 of the
`storage controller 1, and transfers the request to the internal
`network interface 121 controlling the internal bus 15 con-
`nected to the request destination. At that time, each of the
`selectors performs the contention of the internal network
`interface 121 of the request-transfer destination. The internal
`switch 12 makesit possible for the channel controller 11 to
`directly exchange data with the cache-memory 13, the control
`memory 17, and the FM control unit 16. The FM control unit
`16 can exchange data with the channel controller 11, the
`cache memory 13, and the control memory 17. Also, the disk
`controller 14 can directly exchange data with the cache
`memory 13 and the control memory 17. The difference on the
`connection of the FM control unit 16 and that of the disk
`controller 14 is that the internal switch 12 has a connection
`123 between the channel controller and the FM control unit,
`and thus the FM control unit 16 can directly exchange data
`with the channel controller 11.
`FIG. 7 is a diagram illustrating the processing flow when a
`read request comes from the host computer 2 to an HDD 50
`inside of the other storage controller 1 through the internal
`bus 15. The DMA controller 162 in the FM control unit 16
`area. The channel controller 11 receives a read request from
`the host computer 2 through the channel 4 (step s701). The
`performsdata transfer from the cache memory 13to the flash
`processor 111 of the channel controller 11 analyzes the
`memory 166 using the transferlist 1641 set by the processor
`
`111 of the channel controller 11 in the case of creating an received request and obtains an LUN andatarget logical
`empty area in the cache memory when processing a write
`block address. Here, the processor 111 knowsthat the data is
`request from the host computer 2. The FM controller 165
`in an area to be stored in the HDD 50 (step s702). Further-
`controls the flash memory 166 to exchange data by a read
`more, the processor 111 ofthe channel contr