(12) Ulllted States Patent
`(10) Patent N0.:
`US 6,683,649 Bl
`
`Anderson
`(45) Date of Patent:
`Jan. 27, 2004
`
`U5006683649B1
`
`(54) METHOD AND APPARATUS FOR CREATING
`A lVlULTlMEDIA PRESEN’IWI‘ION FROM
`HETEROGENEOUS MEDIA OBJECTS INA
`v
`DIGITAL IMAGING DEVICE
`Inventor: Eric C. Anderson, San Jose, CA (US)
`
`(75)
`
`. 358/512
`.
`5,852,502 A * 12/1998 Beckett
`701/208
`6,023,241 A *
`2/2000 Clapper ...........
`
`5/2001 Matsumnto et a]
`. 348/2312
`6,229,566 Bl
`*
`........
`6,237,010 B1 *
`5/2001 Hui et al.
`345/620
`
`6,310,648 B1 * 10/2001 Miller et al.
`........... 348333.05
`FOREIGN PATENT DOCUMENTS
`
`(73) Assignee: FlashPoint Technology, Inc.,
`Petcrborough, NH (US)
`
`,
`( * ) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`21
`
`,
`o.:
`.
`App] N 09/223 962
`
`(22)
`
`Filed:
`
`Dec. 31, 1998
`.
`.
`‘
`Related U.8.Appllcat10n Data
`_
`.
`f
`1.
`. N 08/702 286 fil d
`_
`(63) C
`0nunuanon'm'Part ° app ”anon 0'
`=
`’
`e on
`Aug. 23’ 1996’ now Pat. No. 6249316
`Int. Cl.7 ................................................ H04N 5/222
`51
`(52) US. Cl.
`.............................. 343/333.05; 348/2314;
`348/231.6; 348/239
`(58) Field of Search """""""""""""""""" 348/239, 333.01,
`348,833.02, 333.05, 333.1], 33312, 220-];
`345/730, 731, 732, 864, 723
`
`(56)
`
`References Cited
`
`U‘S‘ PATENT DOCUMENTS
`$020,012 A
`51991 Stockbcrger
`5,742,339 A *
`4/1998 Wakui
`........................ 348/239
`5,828,406 A
`10/1998 Parulski
`5,845,166 A * 12/1998 Fellegara et a].
`
`348/64
`
`1041992
`”1993
`
`4302886
`JP
`514847
`JP
`* cited by examiner
`.
`.
`Pr1’?“er ExamzrteriTuan H0 .
`Asststartt Examiner—Jason Whipkey
`(74) Attorney, Agent, or Firm—Sawyer Law Group LLP
`.
`.
`57
`ABSTRACT
`
`A method and apparatus [or creating a multimedia presen-
`tation from heterogeneous media objects stored in a digital
`imaging device, wherein each one of the media objects
`includes one or more media types associated therewith, such
`as images, video, audio and text. The digital imaging device
`includes a display screen, a navigation control button, and
`y
`P
`P
`one or more function ke s. In a first as ect of the
`resent
`invention, a representation of each one of the media objects
`in the digital imaging device is displayed on the display
`screen. Thereafter, the user navigates to one of the displayed
`media objects using the navigation control button and marks
`the selected media object by pressing a corresponding
`function key. These steps are repeated to provide an ordered
`set of marked media objects. A slide show is then presented
`from the ordered set of marked media objects, whereupon
`initiation of the slide show presentation, each one of the
`media objects is automatically displayed on the display in
`the order marked.
`
`17 Claims, 13 Drawing Sheets
`
`
`
`Huawei v. Maxell - Exhibit No. 1003 - 1/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 1/24
`
`

`

`US. Patent
`
`Jan. 27, 2004
`
`Sheet 1 0f 13
`
`US 6,683,649 B1
`
`FIG.1
`
`Huawei V. Maxell - Exhibit No. 1003 - 2/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 2/24
`
`

`

`US. Patent
`
`Jan. 27, 2004
`
`Sheet 2 0f 13
`
`US 6,683,649 B1
`
`
`
`FIG. 2B
`
`‘-
`
`Huawei V. Maxell — Exhibit No. 1003 — 3/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 3/24
`
`

`

`US. Patent
`
`Jan. 27, 2004
`
`Sheet 3 0f 13
`
`US 6,683,649 B1
`
`Single Still Image
`
`Burst or Time Lapse
`
`Text file
`
`Panorama
`
`Video
`
`Audio Clip
`
`FIG. 3
`
`Huawei V. Maxell - Exhibit No. 1003 - 4/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 4/24
`
`

`

`US. Patent
`
`Jan. 27, 2004
`
`Sheet 4 0f 13
`
`US 6,683,649 B1
`
`306
`
`FIG. 4A
`
`302
`
`
`140
`
`350
`
`356
`
`354
`
`}352
`
`306
`
`}310
`
`Image #314
`12/4/96
`
`
`
`Huawei V. Maxell — Exhibit No. 1003 — 5/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 5/24
`
`

`

`US. Patent
`
`Jan. 27, 2004
`
`Sheet 5 0f 13
`
`US 6,683,649 B1
`
`Press mark
`
`Display mark number
`
`Update mark softkey
`to unmark
`
`
`
`
`Select
`
`next imae
`
`
`
`510
`
`
`
`
`N
`
`N
`
`Remove image
`from group?
`
`Select a function to
`apply to the group
`
`512
`
`500
`
`502
`
`504
`
`506
`
`508
`
`to remove
`
`
`
`
`
`
`
`
`
`514
`
`524
`
`Renumber all higher
`
`N
`
`Remove mark
`number from cell
`
`Highest number
`
`number marks - mark?Y
`
`FIG 5
`
`526
`
`Update unmark
`softkey to mark
`
`Huawei V. Maxell — Exhibit No. 1003 — 6/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 6/24
`
`

`

`US. Patent
`
`Jan. 27, 2004
`
`Sheet 6 0f 13
`
`US 6,683,649 B1
`
`140
`.ddressesl
`-
`E-M.“
`
`
`302
`
`140
`
`
`
`Huawei V. Maxell - Exhibit No. 1003 - 7/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 7/24
`
`

`

`US. Patent
`
`Jan. 27, 2004
`
`Sheet 7 0f 13
`
`US 6,683,649 B1
`
`
` Slide Show 1
`
`Pointer to Media Object 8
`Positionj2)
`Duration oi piay(auto)
`Pointer to audio file
`
`
`
`Pointer to Media Object i
`Positionjaj
`Duration at ptayjs sec.)
`
`Pointer to audio iiie
`
`
`
`
`Meta Data File
`
`Pointer to Media Object 6
`Positionjij
`Duration of play(5 sec.)
`Pointer to audio tile
`
`
`
`
`
`
`
`
`
`
`
`
`Directory
`
`Media Object 6
`
`Media Object 8
`
`Media Object 1
`
`FIG. 9B
`
`Huawei V. Maxell — Exhibit No. 1003 — 8/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 8/24
`
`

`

`US. Patent
`
`Jan. 27, 2004
`
`Sheet 8 0f 13
`
`US 6,683,649 B1
`
`
`
`
`
`140
`
`302
`
`
`
`Slide Show
`Editing
`Screen
`
`m
`
`
`
`
`
`
`
`
`402
`
`
`
`308a
`O~206a
`
`308b
`206b
`
`30m
`szoac
`
`}310
`
`FIG. 11
`
`Huawei V. Maxell - Exhibit No. 1003 - 9/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 9/24
`
`

`

`US. Patent
`
`Jan. 27, 2004
`
`Sheet 9 0f 13
`
`US 6,683,649 B1
`
`<> Brightness
`
`Image
`Edlting
`
`Screen
`
`Huawei V. Maxell - Exhibit No. 1003 - 10/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 10/24
`
`

`

`US. Patent
`
`Jan. 27, 2004
`
`Sheet 10 0f 13
`
`US 6,683,649 B1
`
` Video
`
`Editing
`Screen
`
`fl
`
`Huawei V. Maxell - Exhibit No. 1003 - 11/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 11/24
`
`

`

`US. Patent
`
`Jan. 27, 2004
`
`Sheet 11 0f 13
`
`US 6,683,649 B1
`
`fl
` “-fii’.
`
`440
`
`Video
`Editing
`Screen
`
`Video
`Editing
`Screen
`
`fl
`
`
`
`
`
`402
`
`1310
`
`308a
`0406a
`
`3081:
`3080
`206b Guam
`
`FIG. 17
`
`Huawei V. Maxell - Exhibit No. 1003 - 12/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 12/24
`
`

`

`US. Patent
`
`Jan. 27, 2004
`
`Sheet 12 0f 13
`
`US 6,683,649 B1
`
`
`
`436 g.o_o-—o.o
`404
`-. mnvmmwvmm 1 <» }ssz
`
`452 438
`
`‘
`
`
`
`434
`
`
`
`Audio
`
`Editing
`
`Screen
`
`£9
`
`402
`
`mm m
`
`3083
`308b
`szoéa ©2061) O~206c
`
`FIG. 18
`
`
`
`The Seventh S m-hony shows
`
`
`
`evidence of afiv
`466
`from his earlier ones
`464 -—
`
`
`
`}352
`
`402
`
`Huawei V. Maxell - Exhibit No. 1003 - 13/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 13/24
`
`

`

`US. Patent
`
`Jan. 27, 2004
`
`Sheet 13 0f 13
`
`US 6,683,649 B1
`
`Play List
`
`2006
`
`A
`20021
`Prevrous { @ > Next
`
`20%
`
`200d
`
`V
`
`FIG. 20
`
`140
`
`
`
`Properties
`Screen
`
`404
`
`1 0
`
` Object #8
`
`Object #20
`
`@1310
`
`308a
`308b
`@2061 O~206b O~206c
`
`FIG. 21
`
`Huawei V. Maxell - Exhibit No. 1003 - 14/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 14/24
`
`

`

`US 6,683,649 B1
`
`1
`METHOD AND APPARATUS FOR CREATING
`A MULTIMEDIA PRESENTATION FROM
`HETEROGENEOUS MEDIA OBJECTS IN A
`DIGITAL IMAGING DEVICE
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`The present invention is a continuation-in- part of Ser. No.
`08/702,286 now US. Pat. No. 6,249,316 entitled “Method
`and System For Creating a Temporary Group of Images In
`A Digital Camera" filed on Aug. 23, 1996.
`The present invention is also related to the following US.
`Patents and Applications: US Pat. No. 5,903,309 entitled
`“Method And System For Displaying Images And Associ—
`ated Media Types In The Interface Of A Digital Camera,”
`filed Sep. 19, 1996, and US. Pat. No. 6,317,141 entitled
`“Method and Apparatus for Editing Heterogeneous Media
`Objects In A Digital Imaging Device,” and application Ser.
`No. 09/223,961, entitled “Method And Apparatus For Cre- -
`ating An Interactive Slide Show In A Digital Imaging
`Device”, both filed concurrently herewith.
`FIELD OF THE INVENTION
`
`15
`
`The present invention relates generally to a digital imag—
`ing device and more particularly to a method and apparatus
`for creating and presenting a multimedia presentation com-
`prising heterogeneous media objects in the digital imaging
`device.
`
`BACKGROUND OF THE INVENTION
`
`The use of digital cameras is rapidly proliferating and
`they may one day overtake 35 mm SLRs in terms of
`worldwide sales. There are basically three types of digital
`cameras: digital still cameras, digital video cameras, and
`hybrid digital-video cameras.
`Still digital cameras are used primarily for capturing high
`quality static photographs, and offer a less expensive alter—
`native to digital video cameras. Still digital cameras are
`typically less expensive because they have far less process-
`ing power and memory capacity than digital video cameras.
`Digital video cameras dilfer from digital still cameras in
`a number of respects. Digital video cameras are used to
`capture video at approximately thirty frames per second at
`the expense of image quality. Digital video cameras are
`more expensive than still cameras because of the extra
`hardware needed. The uncompressed digital video signals
`from all the low-resolution images require huge amounts
`memory storage, and high-ratio real-time compression
`schemes, such as MPEG, are essential for providing digital
`video for today’s computers. Until recently, most digital
`video recorders used digital magnetic tape as the primary
`storage media, which has the disadvantage of not allowing
`random access to the data.
`
`Hybrid digital video cameras, also referred to as multi-
`media recorders, are capable of capturing both still JPEG
`images and video clips, with or without sound. One such
`camera, the M2 Multimedia Recorder by Hitachi America,
`Ltd, Brisbane, Calif, stores the images on a PC card hard
`disk (PCMCIA Type III), which provides random access to
`the recorded video data.
`
`All three types of cameras typically include a liquid-
`erystal display (LCD) or other type of display screen on the
`back of the camera. Through the use of the LCD, the digital
`cameras operate in one of two modes, record and play. In
`record mode, the display is used as a viewfinder in which the
`
`35
`
`4O
`
`55
`
`60
`
`65
`
`2
`user may View an object or scene before taking a picture. In
`play mode,
`the display is used as a playback screen for
`allowing the user to review previously captured images
`and/or video. The camera may also be connected to a
`television for displaying the images on a larger screen.
`Since digital cameras capture images and sound in digital
`format, their use for creation of multimedia presentations is
`ideal. However, despite their capability to record still
`images, audio, and video, today’s digital cameras require the
`user to be very technologically proficient in order to create
`multimedia presentations.
`For example, in order to create a multimedia presentation,
`the user first captures desired images and video with the
`camera and then downloads the images to a personal com-
`puter or notebook computer. There, the user may import the
`images and video directly into a presentation program, such
`as Microsoft PowerPoint”. The user may also edit the
`images and video using any one of a number of image
`editing software applications. After the PowerPoint presen-
`tation has been created, the user must connect the PC or
`notebook to a projector to display the presentation. Finally,
`the user typically controls the playback of the presentation
`using a remote control.
`Due to the limitations of today’s digital cameras in terms
`of capabilities and features, the user is forced to learn how
`to operate a computer, image editing software, and a pre-
`sentation program in order to effectively create and display
`the multimedia presentation. As the use of digital cameras
`becomes increasingly mainstream, however, the number of
`novice computer users will increase. Indeed, many users will
`not even own a computer at all. Therefore, many camera
`owners will be precluded from taking advantage of the
`multimedia capabilities provided by digital cameras.
`What is needed is an improved method for creating and
`displaying a multimedia presentation using images and/or
`video from a digital imaging device. The present invention
`addresses such a need.
`SUMMARY OF THE INVENTION
`
`The present invention provides a method and apparatus
`for creating a multimedia presentation from heterogeneous
`media objects stored in a digital imaging device, wherein
`each one of the media objects includes one or more media
`types associated therewith, such as images, video, audio and
`text. The digital imaging device includes a display screen, a
`navigation control button, and one or more function keys. In
`a first aspect of the present invention, a representation of
`each one of the media objects in the digital imaging device
`is displayed on the display screen. Thereafter,
`the user
`navigates to one of the displayed media objects using the
`navigation control button and marks the selected media
`object by pressing a corresponding function key. These steps
`are repeated to provide an ordered set of marked media
`objects. Aslide show is then presented from the ordered set
`of marked media objects, whereupon initiation of the slide
`show presentation, each one of the media objects is auto-
`matically displayed on the display in the order marked.
`In a second aspect of the present invention, each of the
`media objects may be edited before or after incorporation
`into the slideshow, where each media object is edited using
`specialized media types editors designed to edit the media
`types associated with that particular media object.
`In a third aspect of the present invention, the user may
`specify properties for slide show so that the objects in the
`slide show are not displayed linearly, but rather are dis—
`played in an order that is dependent upon user defined
`events, thus creating an interactive slide show.
`
`Huawei v. Maxell - Exhibit No. 1003 - 15/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 15/24
`
`

`

`US 6,683,649 B1
`
`3
`The present invention thus enables a novice user to create,
`edit and present multimedia presentations all
`from the
`camera without the need for downloading images and video
`to a computer and incorporating the images and video into
`complicated presentation software.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a block diagram illustrating of one preferred
`embodiment of a digital Video camera (DVC) for use in
`accordance with the present invention.
`FIGS. 2A and 2B are diagrams depicting an exemplary
`form factor design for the DVC.
`FIG. 3 is a table listing example media types that may be
`captured and stored by the DVC.
`FIG. 4A is a diagram illustrating one preferred embodi—
`ment of the review mode screen.
`
`15
`
`FIG. 4B is a diagram illustrating a second preferred
`embodiment of the review mode screen.
`
`FIG. 5 is a flowchart depicting the process of creating an “
`ordered group of heterogeneous media objects in accordance
`with the present invention.
`FIGS. 6—8 are diagrams illustrating examples of marking
`heterogeneous media objects.
`FIG. 9A is a diagram illustrating
`implemented as a metadata file.
`FIG. 9B is a diagram illustrating
`implemented as a file directory.
`FIG. 10 is a diagram illustrating the DVC connected to
`external projector, and alternatively to a television.
`FIG. 11 is a diagram illustrating the components of the
`slide-show edit screen in accordance with the present inven-
`tion.
`
`a slide show object
`
`a slide show object
`
`35
`
`FIG. 12 is a diagram illustrating the image editing screen.
`FIG. 13 is a diagram illustrating the video editing screen.
`PIGS. 14—17 are diagrams illustrating the process of
`editing a video on the DVC by creating and moving a Video
`clip.
`FIG. 18 is a diagram illustrating an audio editing screen
`for editing audio media types.
`FIG. 19 is a diagram illustrating a text editing screen for
`editing text media types.
`FIG. 20 is a diagram illustrating the mapping of the
`four-way control during slide show presentation.
`FIG. 21 is a diagram illustrating the properties page of a
`media object.
`DETAILED DESCRIPTION OF THE
`INVENTION
`
`The present invention is a method and apparatus for
`creating and presenting a multimedia presentation compris-
`ing heterogeneous media objects stored in a digital imaging
`device. The following description is presented to enable one
`of ordinary skill in the art to make and use the invention and
`is provided in the context of a patent application and its
`requirements. Although the present
`invention will be
`described in the context of a digital video camera, various
`modifications to the preferred embodiment will be readily
`apparent to those skilled in the art and the generic principles
`herein may be applied to other embodiments. That is, any
`digital
`imaging device used to store and display and/or
`video, could incorporate the features described hereinbelow
`and that device would be within the spirit and scope of the
`present
`invention. Thus,
`the present
`invention is not
`
`4O
`
`55
`
`6O
`
`65
`
`4
`intended to be limited to the embodiment shown but is to be
`accorded the widest scope consistent with the principles and
`features described herein.
`
`Referring now to FIG. 1, a block diagram of one preferred
`embodiment of a digital video camera (DVC) is shown for
`use in accordance with the present invention. The DVC 100
`is preferably capable of capturing and displaying various
`types of image data including digital video and high-
`resolution still images.
`The DVC 100 comprises an imaging device [10, a com-
`puter 112, and a hardware user interface 114. The Imaging
`device 110 includes an image sensor (not shown), such as a
`charged coupled device (CCD) or a CMOS sensor, for
`capturing frames of image data in Bayer format. The image
`frames are transferred from the imaging device 110 to the
`computer 112 for processing, storage, and display on the
`hardware user interface 114.
`
`The computer 112 includes an image processing digital—
`signal—processor (DSP) 116, a video codec 118, an audio
`codec 120, a mass storage device 122, a CPU 124, a DRAM
`126, an internal nonvolatile memory 128, a mixer 130, and
`a video control 132. The computer 112 also includes a power
`supply 134, a power manager 136, and a system bus 138 for
`connecting the main components of the computer 112.
`The hardware interface 114 for interacting with the user
`includes a display screen 140 for displaying the digital video
`and still images, an audio subsystem 142 for playing and
`recording audio, buttons and dials 146 for operating the
`DVC I00, and an optional status display 148.
`The CPU 124 may include a conventional microprocessor
`device for controlling the overall operation of camera. In the
`preferred embodiment, the CPU 124 is capable of concur—
`rently running multiple software routines to control
`the
`various processes of camera within a multithreaded envi-
`ronment. In a preferred embodiment, the CPU [24 runs an
`operating system that
`includes a menu—driven GUI. An
`example of such software is the DigitaTM Operating Envi—
`ronment by FlashPoint Technology of San Jose, Calif.
`Although the CPU 124 is preferably a microprocessor, one
`or more DSP 116’s (digital signal processor) or ASIC’s
`(Application Specific Integrated Circuit) could also be used.
`Non-volatile memory 128, which may typically comprise
`a conventional read—only memory or flash memory, stores a
`set of computer readable program instructions that are
`executed by the CPU 124. Input/Output interface (1/0) 150
`is an interface device allowing communications to and from
`computer 112. For example, I/O 150 permits an external host
`computer (not shown) to connect to and communicate with
`computer 112.
`Dynamic Random-Access-Memory (DRAM) 126 is a
`contiguous block of dynamic memory that may be selec-
`tively allocated for various storage functions. DRAM 126
`temporarily stores both raw and compressed image data and
`is also used by CPU 124 while executing the software
`routines used within computer 112. The raw image data
`
`
`received from imaging device 110 is temporarily stored in
`
`several input bu ers (not shown) within DRAM 126. A
`
`frame buffer (no shown) is used to store still image and
`graphics data via the video control 132 and/or the mixer.
`Power supply 134 supplies operating power to the various
`components of camera. Power manager 136 communicates
`via line with power supply 134 and coordinates power
`management operations for camera.
`In the preferred
`embodiment, power supply 134 provides operating power to
`a main power bus 152 and also to a secondary power bus
`154. The main power bus 152 provides power to imaging
`
`Huawei v. Maxell - Exhibit No. 1003 - 16/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 16/24
`
`

`

`US 6,683,649 B1
`
`5,
`
`15
`
`5
`device 110, 1/0 150, Non-volatile memory 128 and remov-
`able memory. The secondary power bus 154 provides power
`to power manager 136, CPU 124 and DRAM 126.
`Power supply 134 is connected to main batteries and also
`to backup batteries. In the preferred embodiment, a camera
`user may also connect power supply 134 to an external
`power source. During normal operation of power supply
`134, the main batteries (not shown) provide operating power
`to power supply 134 which then provides the operating
`power to camera via both main power bus 152 and second-
`ary power bus 154. During a power failure mode in which
`the main batteries have failed (when their output voltage has
`fallen below a minimum operational voltage level)
`the
`backup batteries provide operating power to power supply
`134 which then provides the operating power only to the
`secondary power bus 154 of camera.
`FIGS. 2A and 2B are diagrams depicting an exemplary
`form factor design for the DVC 100, shown here as a
`clam-shell design having a rotatable imaging device 110.
`FIG. 2A is a top view of the DVC 100 in an opened position, ~
`while FIG. 2B is a top View of the DVC 100 in a closed
`position. FIG. 2A shows the display screen 140, a four-way
`navigation control 200, a mode dial 202, a display button
`204, a set of programmable soft keys 206, a shutter button
`208, a menu button 210, and an audio record button 212.
`The mode dial 202 is used to select the operating modes
`for DVC 100, which include a capture mode (C) for record—
`ing video clips and for capturing images, a review mode (R)
`for quickly viewing the video clips and images on the
`display screen 140, and a play mode (P)
`for Viewing
`full-sized images on the display screen 140.
`When the DVC 100 is placed into capture mode and the
`display screen 140 is activated, the camera displays a “live
`View” of the scene viewed through the camera lens on the
`display screen 140 as a successive series of real—time frames.
`If the display screen 140 is not activated, then the user may
`View the scene through a conventional optical viewfinder
`(not shown).
`the
`Referring to FIGS. 1 and 2A, during live View,
`imaging device 110 transfers raw image data to the image
`processing DSP 116 at 30 frames per second (fps), or 60
`fields per second. The DSP 116 performs gamma correction
`and color conversion, and extracts exposure, focus, and
`white balance settings from the image data and converts the
`data into CCIR 650 streaming Video. (CCIR 650 is an
`international standard for digital Video designed to encom—
`pass both NTSC and PAL analog signals, providing an
`NTSC-equivalent resolution of 720x486 pixels at 30 fps. It
`requires 27 MB per second and uses three signals: one 13.5 _
`MB/sec luminance (gray scale) and two 6.75 MB/sec
`chrominance (color)).
`After processing, the streaming Video from the DSP 116
`is transferred to the mixer for the overlay of optional
`graphics and/or images onto the Video. The graphics data
`from the DRAM’s 126 frame buffer is transferred to the
`mixer in synch with streaming video, where the mixer
`combines the graphic data with the Video. After the stream-
`ing video and the graphics are combined,
`the Video is
`displayed on the display screen 140 via the video control
`132. A video out port is also provided to display the video
`on an external display device.
`When the user initiates the Video capture function to
`record the digital Video, the streaming video output from the
`DSP 116 is also transferred to the video codec 118 for
`compression and storage. The Video codec 118 performs
`MPEG-2 encoding on the streaming Video during recording,
`
`35
`
`4O
`
`55
`
`60
`
`65
`
`6
`and performs MPEG-2 decoding during playback. The Video
`codec 118 may include local memory, such as 32 Mbits of
`SDRAM 126 for example, for MPEG-2 motion estimation
`between frames. Such Video codecs 118 are commercially
`available from Sony Electronics (CXD 192200) and Mat-
`sushita Electronics Corp.
`As the Video codec 118 compresses the digital Video, the
`compressed video stream is transferred to a temporary buffer
`in DRAM 126. Simultaneously, audio is recorded by the
`audio subsystem 142 and transferred to the audio codec 120
`for compression into a compressed audio format, such
`MPEG Audio Layer 3 (MP3), which is common internet
`format. In an alternative embodiment, the audio could be
`compressed into AC—3 format, a well—known Dolby Digital
`audio recording technology that provides six surround-
`sound audio channels.
`
`The CPU 124 mixes the compressed video and audio into
`a specified format, such as MPEG-2, for example. After the
`compressed MPEG-2 data is generated, the CPU 124 trans-
`fers the MPEG—2 data to the removable mass—storage device
`122 for storage. In a preferred embodiment, the mass storage
`device 122 comprises a randomly accessible 3-inch record-
`able DVD drive from 'lbshiba/Panasonic, or a one-inch 340
`MB MicroDriveTM from IBM, for example.
`The Video architecture inputs the video stream from the
`DSP 116 directly into the mixer, rather than first storing the
`
`Video in memory and then inputting the video to the mixer,
`
`
`
`in order to save bus bandwidth. However, if su icient bus
`bandwidth is provided (e.g., 100 MHz), the video stream
`could be first stored in memory.
`Although the resolution of the display screen 140 may
`vary, the resolution of display screen 140 is usually much
`less than the resolution of the image data that’s produced by
`imaging device 110 when the user captures a still image at
`full resolution. Typically, the resolution of display screen
`140 is 1/11 the video resolution of a full resolution image.
`Since the display screen 140 is capable of only displaying
`images at '/a resolution, the images generated during the live
`View process are also 1/4 resolution.
`As stated above, the DVC 100 is capable of capturing
`high—resolution still images in addition to video. When the
`user initiates the capture function to capture a still or
`sequential
`image,.the image device captures a frame of
`image data at a resolution set by user. The DSP 116 performs
`image processing on the raw CCD data to convert the frame
`of data into YCC color format, typically YCC 222:2 format
`(YCC is an abbreviation for Luminance, Chrominance-red
`and Chrominance-blue). Alternatively,
`the data could be
`converted into RGB format (Red, Green, Blue).
`After the still image has been processed, the image is
`compressed, typically in JPEG format, and stored as an
`image file on the mass storage device 122. A JPEG engine
`(not shown) for compressing and decompressing the still
`images may be provided in the image processing DSP 116,
`the Video codec 118, provided as a separate unit, or per-
`formed in software by the CPU 124.
`After the image has been compressed and stored, live
`View resumes to allow the capture of another image. The
`user may continue to either capture still images, capture
`Video, or switch to play or review mode to playback and
`View the previously stored Video and images on the display
`screen 140. In a preferred embodiment, the DVC 100 is
`capable of capturing several different media types, as shown
`in FIG. 3.
`
`FIG. 3 is a table listing example media types that may be
`captured and stored by the DVC 100. Also shown are the
`
`Huawei V. Maxell - Exhibit No. 1003 - 17/24
`
`Huawei v. Maxell - Exhibit No. 1003 - 17/24
`
`

`

`US 6,683,649 B1
`
`7
`corresponding icons that are used to indicate to the media
`type. The media types include a single still image, a time
`lapse or burst image, a panorama, a video segment, an audio
`clip, and a text file.
`Astill image is a high-quality, single image that may have
`a resolution of 1536x1024 pixels, for example. Atime-lapse
`image is a series of images automatically captured by the
`DVC 100 at predefined time intervals for a defined duration
`(e.g. capturing a picture every five minutes for an hour). A
`burst image is similar to a time-lapse, but instead of cap-
`turing images for defined period of time,
`the DVC 100
`captures as many images as possible in a brief time frame
`(e.g., a couple seconds). A panorama image is an image
`comprising several overlapping images of a larger scene that
`have been stitched together. A burst image, a time-lapse
`image, and a panorama image are each objects that include
`multiple still images, therefore, they may be referred to as a
`sequential images.
`image-based media
`In addition to capturing different
`types, the DVC 100 can capture other media types, such as ~
`audio clips and text. The user can record a voice message to
`create a stand-alone audio clip, or the user may record a
`voice message and have it attached to an image to annotate
`the image. Audio clips may also be downloaded from an
`external source to add sound tracks to the captured objects.
`Atext media type is created by entering letters through the
`buttons on the user interface. The text along with graphics
`can be overlaid as watermarks on the images or, the text can
`
`be saved in a file to create a text-based media type.
`
`
`
`In a preferred embodiment, one or more of the di erent
`media types can be combined to form a single media object.
`Since various combinations may be formed, such as single
`image with sound, or burst image with text, etc, the DVC
`100 can be described at storing heterogeneous media
`objects, each comprising a particular combination of media
`types, such as images, video, sound, and text/graphics. Some
`types of media objects are formed automatically by the DVC
`100, such as a captured image or an annotated image, others
`are formed manually by the user.
`After media objects are created and stored, the user may
`view the media objects by switching the camera to play
`mode or review mode. In play mode, the camera 100 allows
`the user to view screen-sized images in the display screen
`140 in the orientation that the image was captured. Play
`mode also allows the user to hear recorded sound associated
`with a displayed image, and to play back sequential groups
`of images (time lapse, burst, and panorama images) and to
`view movies from the video.
`
`15
`
`3O
`
`35
`
`4O
`
`In review mode, the DVC I00 enables the user to rapidly
`review the contents of the DVC. In addition,
`the media
`objects may be edited, sorted, printed, and transferred to an
`external source.
`
`Referring now to FIG. 4A, a diagram illustrating one
`preferred embodiment of the review mode screen is shown.
`Moving the mode dial 202 (FIG. 2) to access the review
`mode enables the user to view all the media objects in the
`camera along with the specific media types associated with
`each of the objects.
`The first embodiment of the review mode screen displays
`a series of object cells 300 that represent the media objects
`stored on the DVC 100, and a command bar 310. The
`display screen 140 is shown here as displaying nine object
`cells 300, although other numbers are also suitable.
`The user may navigate through a series of displayed
`object cells 300 in the display screen 140 using the four-way
`navigation control 200. The object cell 300 currently
`
`50
`
`55
`
`60
`
`65
`
`8
`selected by the four-way navigation control 200 is indicated
`by a highlighted area 302, which in this embodiment is
`shown as selection rectangle. Other shapes or indications
`that a object cell 300 is the currently active object cell are
`also suitable.
`
`Each object cell 300 includes an image area 304 and an
`icon/information area 306. In the case of a still image, the
`image area 304 of an object cell 300 displays a thumbnail of
`the media object, which in the case of an image—based media
`object is a small, low-resolution version of the image. In the
`case of sequential images and video segments, the image
`area 304 of an object cell 300 displays a representative
`thumbnail or frame from the image sequence or video,
`respectively, typically the first one.
`The icon/information area 306 displays one or more
`graphical icons and/or text information indicating to the user
`what media types have been associated with the media
`object displayed in the image area 304. The icon/information
`area 306 may be placed in various positions relative to the
`image area 304. However, in a preferred embodiment, the
`icon/information area 306 is displayed on the right—hand side
`of each object cell 300, as shown.
`Referring now to FIG. 4B, a diagram illustrating a second
`preferred embodiment of the review mode screen is shown,
`where like components share like reference numerals. In the
`second preferred embodiment,
`the review mode screen
`includes a filmstrip 352, the icon/information area 306 for
`displaying the media type icons associated with the active
`media object 302, a large thumbnail 354 show