US007882320B2
`
`(12) United States Patent
`Caulkins
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7.882,320 B2
`*Feb. 1, 2011
`
`(54)
`
`(75)
`
`(73)
`(*)
`
`(21)
`(22)
`(65)
`
`(63)
`
`MULT-PROCESSOR FLASH MEMORY
`STORAGE DEVICE AND MANAGEMENT
`SYSTEM
`
`Inventor: Jason Caulkins, West Windsor, NJ (US)
`Assignee: Dataram, Inc., West Windsor, NJ (US)
`Notice:
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 84 days.
`This patent is Subject to a terminal dis
`claimer.
`
`Appl. No.: 12/113,606
`Filed:
`May 1, 2008
`
`Prior Publication Data
`US 2008/0209 11.6 A1
`Aug. 28, 2008
`
`Related U.S. Application Data
`Continuation-in-part of application No. 1 1/439,619,
`filed on May 23, 2006, now Pat. No. 7,464,240, and a
`continuation-in-part of application No. 1 1/439,620,
`filed on May 23, 2006, now Pat. No. 7,461.229, and a
`continuation-in-part of application No. 1 1/439,615,
`filed on May 23, 2006, now Pat. No. 7,424,587.
`
`(51)
`
`(52)
`
`Int. C.
`(2006.01)
`G06F 12/00
`(2006.01)
`G06F 2/16
`U.S. Cl. ............................... 711/165; 711/3: 711/4;
`711/103: 711/113: 711/162
`
`(58) Field of Classification Search ....................... None
`See application file for complete search history.
`References Cited
`
`(56)
`
`U.S. PATENT DOCUMENTS
`
`7, 1992 Kikuchi et al.
`5,129,074 A
`7,103,684 B2 * 9/2006 Chen et al. .................... T10.62
`7,383,362 B2 * 6/2008 Yu et al. ....................... T10/22
`2003. O115282 A1
`6/2003 Rose
`2006, OO31389 A1
`2/2006 Shimozono et al.
`8/2007 Fujibayashi et al.
`2007,0180188 A1
`2007/0276994 A1
`11/2007 Caulkins et al.
`2008/0320214 A1* 12/2008 Ma et al. .................... T11 103
`2009,0193184 A1* 7/2009 Yu et al. ..................... T11 103
`2009/0204732 A1* 8, 2009 Yu et al. ....................... T10/22
`2009, 0240873 A1* 9, 2009 Yu et al. ..................... T11 103
`
`* cited by examiner
`Primary Examiner Jack A Lane
`(74) Attorney, Agent, or Firm—Donald R. Boys; Central
`Coast Patent Agency, Inc.
`
`(57)
`
`ABSTRACT
`
`A data storage device has a host controller interface, a plu
`rality of microprocessor units each having a portion of ran
`dom access memory (RAM) dedicated thereto, a plurality of
`Flash device configurations each having dedicated bus con
`nections to individual ones or multiples of the microprocessor
`units, and a dataflow controller accessible to the host control
`ler interface for managing access to the Flash device configu
`rations.
`
`8 Claims, 6 Drawing Sheets
`
`500
`
`5O1
`
`N.
`Incoming
`Write
`Request
`
`502
`
`503
`
`Request
`For DFC
`
`
`
`HPE, Exh. 1001, p. 1
`
`

`

`U.S. Patent
`U.S. Patent
`
`Feb.1
`
`2011
`
`Sheet 1 of 6
`
`US 7.882,320 B2
`US 7,882,320 B2
`
`(U-T)601
`
`Use
`
`[DolAcd
`
`qSera
`
`ZDIAS,
`
`Uselq
`
`|
`
`WSSTAOG
`
`
`
`SAO]SSeIOISBIL
`
`LOSSIIOIGODTAL
`
`GAY40D)[SLT
`
`901
`
`sopqey,Ws]
`
`Sor
`
`LOT
`
`TayjOIUO7) SORTS WOT
`
`
`
`T0T
`
`001
`
`
`
`
`
`
`
`
`
`2“)
`\
`
`NOcor
`
`^_Z0 I
`
`cOl
`
`JSOP{WISISkG
`
`HPE, Exh. 1001, p. 2
`
`HPE, Exh. 1001, p. 2
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Feb. 1, 2011
`
`Sheet 2 of 6
`
`US 7.882,320 B2
`
`200 y
`
`
`
`Flash Storage Device
`202(1-n)
`
`201 (in
`l y sity
`Flash Config. 1
`Microprocessor 1
`
`Flash Config. 2
`
`Flash Config. 3
`
`i
`R
`
`O
`O
`g
`
`g
`I
`
`H
`
`C
`O
`a
`'t
`s
`A
`
`Microprocessor in
`
`RAM/FMD
`
`Fig. 2
`
`HPE, Exh. 1001, p. 3
`
`

`

`U.S. Patent
`
`Feb. 1, 2011
`
`Sheet 3 of 6
`
`US 7,882,320 B2
`
`Y 009
`
`
`
`"Toonto) MOUeed"
`IoIOIUOO 90ele UI SOH
`
`HPE, Exh. 1001, p. 4
`
`

`

`U.S. Patent
`
`Feb.1, 2011
`
`Sheet 4 of 6
`
`US 7,882,320 B2
`
`_1|
`
`FlashConfi
`
`M-processor1]|
`
`iO400
`
`M-processor2
`
`Qo
`al
`
`|
`
`=
`
`SobjIoIUy] ISO
`
`yetsseeeae __.\saanee—__|_V¥oaeee
`(Giana [yom
`—BpoyuoyMoyeeq
`
`—y_
`
`on
`
`|e
`
`M-processor4
`
`Fig.4
`
`HPE, Exh. 1001, p. 5
`
`HPE, Exh. 1001, p. 5
`
`

`

`U.S. Patent
`
`Feb. 1, 2011
`
`Sheet 5 of 6
`
`US 7.882,320 B2
`
`
`
`
`
`503
`
`Perform
`Address
`Lookup
`
`Access MP
`Fl-Channel
`
`505
`
`504
`
`Write to
`Flash
`
`
`
`- 501
`
`Incoming
`Write
`Request
`
`500 N.
`
`
`
`
`
`
`
`
`
`509
`
`510
`
`
`
`
`
`511
`
`NO
`
`1RAMNYES
`Cache Full
`
`512
`
`
`
`NO
`
`HPE, Exh. 1001, p. 6
`
`

`

`U.S. Patent
`
`Feb. 1, 2011
`
`Sheet 6 of 6
`
`US 7.882,320 B2
`
`60
`
`
`
`602
`
`
`
`
`
`603
`
`Incoming
`Read
`Request
`
`Read from
`Flash
`
`
`
`
`
`Return Read
`Data to DFC
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Access MP
`Fl-Channel
`
`
`
`Address
`Lookup
`
`Read from
`RAM
`
`Fig. 6
`
`HPE, Exh. 1001, p. 7
`
`

`

`US 7,882,320 B2
`
`1.
`MULT-PROCESSOR FLASH MEMORY
`STORAGE DEVICE AND MANAGEMENT
`SYSTEM
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`The present invention is a continuation-in-part (CIP) to a
`U.S. patent application Ser. No. 11/439,619, filed May 23,
`2006 and entitled “Hybrid Solid State Disk Drive with Con
`10
`troller, to a U.S. patent application Ser. No. 1 1/439.620, filed
`May 23, 2006 and entitled “Software Program for Managing
`and Protecting Data Written to a Hybrid Solid State Disk
`Drive', and to a U.S. patent application Ser. No. 1 1/439,615,
`filed on May 23, 2006 and entitled “Methods for Managing
`Data Writes and Reads to a Hybrid Solid State Disk Drive':
`disclosures of which are incorporated in their entireties at
`least by reference.
`
`15
`
`BACKGROUND OF THE INVENTION
`
`2
`drive is, in one embodiment, a hot swappable disk drive that
`is recognized by a host system upon boot as a destination
`drive for reads and writes.
`A controller is provided on the disk drive for managing the
`memory portions as a single non-volatile memory through
`use of at least one integrated circuit Supporting one or more
`sets of machine-readable instructions and a data port and
`buffer circuitry for bi-directional communication of data
`between the controller and a host system such as a computer.
`The system known to the inventor uses a RAM/Flash data
`storage addressing method that prevents continued and
`repetitive writing to Flash to preserve mean time before fail
`ure (MTBF) of the Flash storage device or devices of the
`system. The system uses RAM as a caching memory and only
`writes to Flash when absolutely necessary. Another optimi
`zation of the system is rotation of Flash blocks in and out of
`service to further enhance wear leveling of the Flash device or
`aggregate of devices onboard or plugged into the disk drive.
`Writing to Flash memory is comparatively slower than
`writing to RAM, hence the use of RAM in the above-de
`scribed system to cache data for eventual write to Flash on
`power down, power interruption, or only when the RAM
`cache is full. The system described above provides a practical
`and economical Solution for replacing mechanical hard disk
`drives in computers and other devices.
`It has occurred to the inventor that there is a need for faster
`data management speeds in the computing industry in general
`and in particular in the area of robust servers and other busi
`ness machines. While CPU speeds are at all time highs in
`terms of computing cycles, speeds at which data can be man
`aged relative to disk storage on a Flash memory are still
`relatively slower. This has caused a barrier to extensive use of
`Flash memory in more robust data storage systems.
`Still another disadvantage of using Flash memory as long
`term storage in robust systems is that a number of writes
`performed on the memory is limited on a Flash memory chip
`and the media must be written in a manner, often proprietary,
`as specified by the manufacturer of the Flash memory. Such
`adaptations may not be readily Supported by a particular host
`system sending the data for storage. This fact has been a basis
`for differing approaches to Flash memory management soft
`ware and firmware that deal essentially with how data may be
`rendered and stored on the particular type Flash memory
`implemented.
`Further to the above, current Flash data storage systems
`rely on a single central processing unit (CPU) to manage
`Flash tables and perform other data management tasks. A
`drawback is that such applications place significant perfor
`mance demands on Flash-based storage and caching systems,
`preventing Scaling of those systems to meet enterprise stan
`dards for mass data storage systems.
`RAM and specific data bus contentions or issues effec
`tively prohibit single processor Flash management schemes
`from Scaling to a high-performance level. For example, there
`are many operations performed by and in conjunction with a
`microprocessor that compete with each other on a storage
`device for RAM space. Error Code Correction (ECC) and
`real-time data encryption are just a few of these operations
`that compete with address lookups, read and write requests
`and other important data access functions.
`Current Flash memory research is resulting in faster Flash
`memory types that may be operated much faster than current
`Flash types. But RAM access and data bus contentions
`present problems in current architectures that cause latency
`and prevent full potential for faster computing. Therefore,
`what is needed in the art is a Flash-based storage device and
`data management system that can be scaled up for high
`
`25
`
`30
`
`35
`
`1. Field of the Invention
`The present invention is in the field of data storage devices
`including disk drives and mass storage systems and pertains
`particularly to processor-based data storage devices and sys
`tems for managing host access to and data management on
`those devices.
`2. Discussion of the State of the Art
`In the field of data storage, non-volatile mechanical disk
`drives have been developed for short and long-term data
`storage. Solid-state non-volatile memory has been imple
`mented for specific data storage needs, especially in Small,
`portable electronic computing devices such as cellular tele
`phones, video cameras and the like.
`Volatile memory is a solid-state memory typically used as
`a high-speed temporary memory Such as random access
`memory (RAM) of which there are many known variations.
`Common versions of RAM include Dynamic Random Access
`Memory (DRAM) and Static Random Access Memory
`40
`(SRAM) among other variations such as SDRAM.
`Flash memory, on the other hand, is a Solid-state, high
`speed data storage solution that has, until recently, been used
`mainly for handheld devices like cellphones, personal digital
`assistants (PDAs), cameras, or Universal Serial Bus (USB)
`peripheral storage devices referred to as jump drives orthumb
`drives. Flash memory provides a non-volatile memory for
`storing data with read speeds approaching that of RAM.
`Common memory types that require management include
`Phase Change Memory and NAND Flash.
`When referring to these memory types, the terms volatile
`and non-volatile are blurring as new research in memory
`continues and new memory types are developed. But for
`purpose of this specification, volatile memory shall refer to
`memory in which stored data is lost upon interruption of
`power and non-volatile memory shall refer to memory in
`which no power is required to retain the data stored. Flash
`memory is increasingly being used as primary or secondary
`storage memory in computing systems. Such devices are
`commonly known as solid state disks. Flash is also being used
`as cache memory in Some systems.
`A solid-state disk drive known to the inventor, but not as
`publically available prior art, includes a first portion of solid
`state memory of a Volatile nature, a second portion of Solid
`state memory of a non-volatile nature, a controller for man
`65
`aging the memories, and a power Subsystem for protecting
`data in volatile memory in the event of loss of power. The
`
`45
`
`50
`
`55
`
`60
`
`HPE, Exh. 1001, p. 8
`
`

`

`3
`performance write and read operations without bogging
`down due to RAM and Bus issues.
`
`US 7,882,320 B2
`
`SUMMARY OF THE INVENTION
`
`10
`
`15
`
`25
`
`30
`
`35
`
`One of several problems stated above is that it in computing
`where data storage is involved it is difficult to scale up to
`larger more robust system using a device having a single
`processor for managing all of the Flash data management
`operations over a shared bus system. It is desirable to store
`data in a fast, secure, and reliable manner using Flash memory
`as a preferred Solid state medium. However, existing systems
`use a single complex processor over a shared bus to access the
`Flash media for read and write access. The result is a less than
`desired performance speed for read and write operations due
`to RAM and bus contentions by the various data management
`process components.
`The inventor searched the art of data storage devices and
`systems looking for components that could be leveraged or
`otherwise modified to improve the flexibility, reliability and
`performance speed of a data storage system.
`Every data storage drive depends on a processor and a bus
`system for reading and writing data. Most such systems are
`not flexible enough to be scaled up for enterprise applications
`that might use Flash media as the persistent storage medium
`because of bus and RAM contentions. Moreover, larger more
`complex processors are expensive and although capable of
`the data management tasks required of enterprise systems,
`Suffer degradation of performance speed at levels of higher
`utilization.
`The inventor conceptualized and Subsequently provided a
`Flash data storage device constructed using a distributive
`architecture that was less expensive to implement and had
`fewer RAM and bus contention issues than single processor
`devices. The result was a better performance relative to data
`management speeds including reads and writes. Accordingly,
`in one embodiment of the invention, a data storage device is
`provided, comprising a host controllerinterface, a plurality of
`microprocessor units each having a portion of random access
`40
`memory (RAM) dedicated thereto, a plurality of Flash device
`configurations each having dedicated bus connections to indi
`vidual ones or multiples of the microprocessor units, and a
`dataflow controller accessible to the host controller interface
`for managing access to the Flash device configurations.
`In another aspect of the invention a Flash channel config
`ured for data storage is provided, comprising a coupled con
`figuration of one or more Flash memory devices, a bus struc
`ture connected to the Flash device configuration, the bus
`including an address line, a data line, and a control line, and
`one or more microprocessor units also connected to the bus
`structure, and having a portion of RAM dedicated thereto.
`In another aspect of the invention a Flash channel config
`ured for data storage is provided, comprising two or more
`Flash device configurations comprising one or more Flash
`memory devices, two or more bus structures connected to
`each Flash device configuration, the bus structures each
`including an address line, a data line, and a control line, and a
`single microprocessor unit also connected to the bus struc
`tures, the microprocessor unit having a portion of RAM dedi
`cated thereto.
`
`45
`
`50
`
`55
`
`60
`
`BRIEF DESCRIPTION OF THE DRAWING
`FIGURES
`
`FIG. 1 is a block diagram illustrating a single processor
`Flash storage device according to existing art.
`
`65
`
`4
`FIG. 2 is a block diagram illustrating a multi-processor
`Flash storage device according to an embodiment of the
`present invention.
`FIG. 3 is a block diagram illustrating a multi-processor
`Flash storage device according to another embodiment of the
`present invention.
`FIG. 4 is a block diagram illustrating a multi-processor
`Flash storage device according to a further embodiment of the
`present invention.
`FIG. 5 is a process flow chart illustrating steps for storing
`data in a multi-processor Flash device according to an
`embodiment of the present invention.
`FIG. 6 is a process flow chart illustrating steps for reading
`from a multi-processor Flash device according to an embodi
`ment of the present invention.
`
`DETAILED DESCRIPTION
`
`The inventor provides a multiple processor flash-based
`storage device and a system for managing data relative to use
`of the device for data storage. The invention is enabled in
`detail according to the following embodiments.
`FIG. 1 is a block diagram illustrating a single-processor
`Flash storage device as known to the inventor. In this system
`known to the inventor and briefly described above in the
`background section of this specification, a single processor is
`used to perform all of the functions relative to reading and
`writing data to one or more Flash-based storage devices.
`In the system of FIG. 1 data storage system 100 includes a
`data storage device 101 and a host computing device 102.
`Host computing device 102 may be a personal computer (PC)
`or a hand-held device Such as a personal digital assistant
`(PDA), a Laptop computer, or some other computing device
`that can be coupled to data storage device 101 for the purpose
`of reading data from and writing data to the device.
`Data storage device 101 is a solid state storage device that
`can be hardwired to or can be plugged into the host for use as
`a disk drive in place of a mechanical disk drive. Data storage
`device 101 has a host interface controller 103 for adapting to
`the host system though a computer bus. Data storage device
`101 further includes a microprocessor 104 for processing
`commands from the host. Microprocessor 104 is connected
`by internal bust 107 to a random access memory (RAM) 105
`used as a cache memory for the device.
`Internal bus 107 connects processor 104 to host interface
`controller 103 and to a plurality of flash-based data storage
`devices 109 (1-n). Flash-based storage devices 109(1-n) may
`be flash chips bused in series or parallel. RAM 105 is used for
`all RAM-based functions including caching writes to flash for
`the purpose of lessening the number of actual writes that the
`host system makes to flash to preserve the lifespan of the flash
`storage devices. Data management tables for both flash space
`and RAM space are provided in RAM for mitigating write
`addressing and lookups for reading from the flash devices.
`In this example writes to flash are kept to a minimum and
`writing to flash actually occurs in flash dumps from RAM
`Such as when there is a power interruption, a purposeful
`power-down event, and when RAM space is approaching
`capacity. Using RAM as a fast caching system makes the
`application of flash-based storage more practical. However,
`there are limitations with this exemplary architecture that
`prevent this system from economical application to more
`robust systems like server-based storage on an enterprise
`scale, or mass data storage applications like redundant array
`of independent disk (RAID) systems and other like mass data
`storage systems.
`
`HPE, Exh. 1001, p. 9
`
`

`

`US 7,882,320 B2
`
`10
`
`15
`
`30
`
`5
`The fact that only one processor is active on data storage
`device 101 coupled with a shared data bus produces certain
`performance delays in data management relative to processor
`speed. RAM space 105 is a precious resource on device 101.
`Many processes other than data write and read operations
`compete for available RAM space. Some of the aforemen
`tioned processes that contend for available RAM space
`include data encryption, error correction coding (ECC), and
`address lookups. Successful utilization of RAM 105 by
`microprocessor 104 for all RAM-based data operations suf
`fers some degradation as RAM cache fills with pending Flash
`writes and as the shared data bus becomes increasingly busy
`with more data traffic. Providing more RAM memory is not a
`viable option in this example as the shared data bus is only so
`wide presenting a bottleneck to higher performance required
`for more robust systems.
`FIG. 2 is a block diagram illustrating a multi-processor
`Flash storage device 200 according to an embodiment of the
`present invention. Flash storage device 200 is a solid-state
`data storage system using a distributed architecture and dedi
`cated bus structures. Device 200 includes a host interface
`controller 204 in this example that provides an interface to a
`system host Such as a powerful workstation or an enterprise
`server application. In one embodiment storage device 200
`may be a shared device accessible from more than one com
`25
`puting station or server. Also in one embodiment device 200
`may be part of an aggregation of multiple similar devices to
`form a server data storage rack or array of disks as in a RAID
`array or in a storage area network (SAN).
`Flash storage device 200 may be adapted for use with a
`small computer system interface (SCSI) bus, parallel
`advanced technology attachment (PATA) or serial advanced
`technology attachment (SATA) protocols, integrated Drive
`Electronics/Advanced Technology Attachment (IDE/ATA)
`interface, an Enhanced Small Device Interface (ESDI), a
`Serial Advanced Technology Attachment, (SATA), or a Par
`allel Advanced Technology Attachment (PATA) interface or a
`Peripheral Component Interface (PCI). Disk 200 may also be
`adapted to work with enterprise Fibre Channel data storage
`networks and serial attached SCSI (SAS) networks. In this
`particular embodiment, disk 200 may be thought of as a
`Solid-state mass storage device using the appropriate form
`factors and interfaces.
`Flash storage device 200 includes a distributed processor
`architecture comprising multiple microprocessor units 202
`(1-n). Each microprocessor unit 202 (1-n) includes a micro
`processor and an onboard or bused access to a dedicated
`amount of RAM. The dedicated RAM is used by the micro
`processor in each unit for caching and other data management
`functions. Microprocessor units 202 (1-n) are intended to be
`low cost dedicated processors that function independently of
`one another. Each microprocessor has a dedicated bus to one
`of a plurality of Flash configurations 201 (1-n).
`The illustration of separate RAM/FMD in each processor
`unit 202(1-n) is not meant to indicate that there are com
`55
`pletely separate and autonomous RAM units, but simply that
`each microprocessor unit has a dedicated portion of RAM. As
`described above, the dedicated portions might be all a part of
`a single RAM array. Moreover, Ram portions may be pro
`vided on Flash configurations where the configuration is a
`removable module containing one or more Flash devices and
`the dedicated RAM. In that case, access to RAM would be
`over a dedicated bus.
`A Flash configuration is defined as one or more Flash
`memory devices configured to be accessible through a dedi
`cated bus. A Flash channel is defined for the purpose of
`discussion as a bus connection from a processor, for example,
`
`35
`
`6
`to one or more Flash chips or devices illustrated logically
`herein, defined as a Flash device configuration or simply
`Flash device. Therefore, a plurality of dedicated internal bus
`structures 205 (1-n) is provided to complete the architecture.
`Flash configurations 201 (1-n) may also be referred to as
`Flash channels throughout this specification.
`Microprocessor unit 202(1) is coupled to Flash configura
`tion 1 (Flash device) by dedicated Bus 1. Microprocessor unit
`202(2) is coupled to Flash configuration 2 by dedicated Bus 2,
`and so on for the number of processor units (n) on device 200.
`The ratio of Flash configuration to processor is one-to-one
`over a single bus in this example. However, this is not a strict
`requirement for practice of the present invention as will be
`detailed further below.
`Each microprocessor unit 202 (1-n) has a dedicated bus
`connection to a unique dataflow controller 203. Dataflow
`controller 203 manages the data traffic over all of the Flash
`channels through each of the microprocessor units. Each
`microprocessor unit 202 (1-n) has a base address and is
`responsible for a single Flash channel of multiple channels
`201 (1-n). The microprocessor units are completely indepen
`dent and do not communicate with one another in the archi
`tecture in this particular embodiment. In other embodiments,
`the microprocessors distributed over the architecture may be
`bused for communication with each other and may share data
`and tasks.
`Dataflow controller 203 communicates with host interface
`controller 204 by way of a bus illustrated herein as a bus 206.
`The host system may view Flash storage device 200 as a
`single drive or disk or according to any particular partitioning
`that may be implemented Such as primary storage space and
`backup storage space. Dataflow controller 203 determines
`which Flash channel to use, that is, which microprocessor
`unit to use, according to information received in a request and
`according to a Flash management system implemented in
`RAM in each of the microprocessor units 202 (1-n).
`RAM at each processor unit 202 (1-n) includes Flash man
`agement Data tables (FMD) tracking the local block
`addresses (LBAs) and state for the Flash memory connected
`to the channel to which the processor unit controls access. The
`actual Flash memory devices may be Phase Change Memory
`or NAND Flash or any other variant of Flash memory or
`persistent memory. Such devices may be Flash chips con
`nected in parallel or daisy chained, and that are accessible as
`a configuration through a single dedicated bus. The invention
`may leverage existing Flash memory types and newer Flash
`memory types being developed. The type of RAM used at
`each processor may also vary. Available RAM types include
`SDRAM, MRAM, FRAM, and NRAM. In one embodiment
`Flash memory may instead be a non-volatile RAM that is
`Suitable for use as a persistent storage space.
`Dataflow controller 203 may be a state machine imple
`mented in software or firmware. Also, dataflow controller 203
`may be implemented as processor-controlled hardware. Inte
`gration between host interface controller 204 and dataflow
`controller 203 is also plausible and may be practiced without
`departing from the spirit and scope of the present invention.
`Application as a data storage device for a larger enterprise
`scale system like a server-based system is among the many
`adaptation possibilities for data storage device 200.
`The simple one-to-one correlation between microproces
`sors and Flash channels in this example is exemplary only as
`other ratios between processor and Flash memory may be
`observed in the architecture. Some of these variations are
`explained more fully later in this specification.
`There are several optimization techniques that may imple
`mented relative to Flash memory management in terms of
`
`40
`
`45
`
`50
`
`60
`
`65
`
`HPE, Exh. 1001, p. 10
`
`

`

`US 7,882,320 B2
`
`15
`
`7
`reads, writes, erasures, and wear leveling. One case for using
`RAM has a cache memory for parking Flash data for eventual
`write to Flash, and uses both RAM address and Flash address
`tables in FMD, as is the case for the co-pending application
`referenced in the cross-reference section of this application.
`In one embodiment dataflow controller 203 selects a proces
`Sor unit 202 (1-n) in sequential order for performing data
`access. In this scheme a first request will be filled by processor
`202 (1), a next request by processor 202 (2) and so on. By the
`time the selection process loops back to the first processor, it
`10
`is most likely free again (free of ongoing data access tasks). A
`goal is to have maximum throughput of data while not over
`utilizing or under utilizing any processing resource.
`In one embodiment a random selection approach for pro
`cessors is used. In this approach dataflow controller 203 may
`select a processor for completing a write from the host based
`on a random assignment of addresses. In one embodiment
`wear leveling is practiced in conjunction with all of the Flash
`channels by ensuring that data is evenly distributed over the
`collective Flash memory space.
`Dataflow controller 203 is asynchronous and may simul
`taneously communicate with all microprocessor units 202
`(1-n). Address and state tables (not illustrated) are provided to
`the dataflow controller by each of processor units 202 (1-n).
`In this way the dataflow controller may manage where writes
`occur transparently from the host. The host may view the
`compilation of Flash devices as a single disk according to a
`file system-based view used by the operating system of the
`host. A more primitive view or block view of the Flash
`memory space may also be ordered. It is noted herein that
`storage device 200 may be one of multiple devices compris
`ing a mass storage system accessible from one or more
`machines.
`FIG. 3 is a block diagram illustrating a distributed multi
`processor Flash storage device according to another embodi
`ment of the present invention. Flash storage device 300 is
`illustrated in this embodiment and is implemented using a
`distributed architecture including multiple processor units
`illustrated herein as microprocessor units 304(1-n). Micro
`processor units 304 (1-n) each have onboard or dedicated
`RAM for processing data management functions and for
`caching data before writing to Flash operations.
`A Flash channel is defined as one or more Flash devices (in
`configuration) connected by a dedicated bus to a processor
`unit as described above. Each of Flash devices 306 (1-n)
`represent one or more Flash memory devices bused to a
`processor unit by a dedicated bus, in this example. Flash
`device 1 and Flash device 2 of devices 306 (1-n) in this
`embodiment share microprocessor unit 304 (1). Micropro
`cessor unit 304(1) is bused by a dedicated bus 305 (1) to Flash
`device(s) 306 (1) to form one complete Flash channel. The
`same microprocessor unit is bused by a dedicated bus 305 (2)
`to Flash device(s) 306 (2).
`The same configuration is repeated on the device where
`one microprocessor unit is responsible for two Flash chan
`nels, for example, microprocessor unit 304 (n) is bused by
`dedicated bus 305 (m) to Flash device(s) 306 (m) and by
`dedicated bus 305 (n) to Flash device(s) 306 (n). In another
`embodiment one microprocessor unit may handle four or
`eight Flash channels, or other numbers of Flash channels.
`There are many possibilities. In this case RAM is shared for
`caching writes to both Flash device configurations (Flash 1,
`Flash 2). In this embodiment RAM is not dedicated to a single
`Flash channel but is dedicated to a single microprocessor unit
`and is shared by two Flash channels. While this may introduce
`some contention for RAM between the Flash channels, the
`fact that the channel pair earmarked by sharing one micro
`
`45
`
`8
`processor unit is duplicated over entire device 300 makes any
`performance degradation negligible when compared to the
`performance of a single processor unit managing multiple
`Flash channels over a common bus.
`Each microprocessor unit 304 (1-n) has a single bus con
`nection to a data flow controller 303 integrated with a host
`interface controller 302. In this example dataflow controller
`303 is onboard the host interface controller. Dedicated data
`buses 305 (1-n) may be 32-bit, 64-bit, or 128-bit wide buses,
`or some other bus configuration. The same can be said for all
`dedicated internal (onboard) buses described in the various
`architectures present.
`Single bus connection Bus (1) from microprocessor unit
`304 (1) to dataflow controller 303 may be a 32-bit, 64-bit, or
`128-bit wide bus, or some other. It is possible that Bus (1) may
`be configured to be twice as fast as buses 305 (1-n) to allow for
`possible bottle-necking of data traffic on the host-side of the
`device 300. Other optimizations may be practiced such as
`RAM caching before write where the actual writes to Flash
`over the dedicated buses 305 (1-n) are kept to a minimum
`number as much as is practical. Bus (1) that connects micro
`processor 304 (1) to dataflow controller 303 for communica
`tion may be a duel independent bus (DIB) or some other bus
`architecture that is optimized for speed.
`Microprocessor unit 304(n) is bused to dataflow controller
`303 by a dedicated bus n. Dataflow controller 303 includes an
`onboard processor 307 with a dedicated RAM with dataflow
`controller tables for use in microprocessor communication.
`Dataflow controller 303 is hosted on or integrated with host
`interface controller 302. It is not specifically required that
`dataflow controller 303 be controlled by an onboard proces
`sor to practice the present invention. The dataflow controller
`may be a state machine running in firmware on the host
`controller interface. The dataflow controller may also be con
`trolled by a processor residing in a host system or in a system
`adapter without departing from the spirit and scope of the
`present invention. In this example, each multiprocessor unit
`manages data access to two independent Flash memory con
`figurations. The Flash configuration