(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2003/0013073 A1
`Jan. 16, 2003
`(43) Pub. Date:
`Duncan et al.
`
`US 2003OO13073A1
`
`(54)
`
`(75)
`
`(73)
`
`ELECTRONIC BOOK WITH MULTIMODE
`I/O
`
`Inventors: Brandon Duncan, San Jose, CA (US);
`Joshua Seth Hodas, Claremont, CA
`(US); Jared Joseph Jackson, San Jose,
`CA (US); Walter Irwin Nissen III,
`Livermore, CA (US); Neelakantan
`Sundaresan, San Jose, CA (US)
`Correspondence Address:
`John L. Rogitz
`Rogitz & Associates
`750 B. Street, Suite 3120
`San Diego, CA 92101 (US)
`Assignee: International Business Machines Cor
`poration
`
`(21)
`(22)
`
`Appl. No.:
`Filed:
`
`09/829,249
`Apr. 9, 2001
`
`Publication Classification
`
`(51)
`(52)
`
`(57)
`
`Int. Cl. ............................................... G09B 5/00
`U.S. Cl. .............................................................. 434/317
`
`ABSTRACT
`
`An electronic book device runs at least two output mode
`threads-one graphics, and one audio-Such that the device
`can both graphically display book content, and play the
`content over a speaker in the device Annotations can be
`made audibly or graphically, and user commands can like
`wise be made audibly or graphically
`
`- 6
`GUDEVICE
`VOICE RECOG.INPUT
`
`
`
`
`
`4
`
`DISPLAY
`SPEAKER
`
`
`
`
`
`24
`
`MODULE
`PROCESSOR
`
`
`
`
`
`SORAGE
`
`22
`
`26
`
`O
`
`18
`
`2
`
`-1-
`
`Amazon v. Audio Pod
`US Patent 10,805,111
`Amazon EX-1027
`
`

`

`Patent Application Publication Jan. 16, 2003 Sheet 1 of 4
`
`US 2003/0013073 A1
`
`FIG. 1
`ELECBOOK
`DEVICE
`
`
`
`2O
`
`
`
`24
`
`-
`6
`GUDEVICE
`VOICE RECOG.INPUT
`
`4.
`
`DISPLAY
`SPEAKER
`
`MODULE
`PROCESSOR
`
`STORAGE
`
`22
`
`26
`
`26
`-
`STORAGE
`(LIBRARY)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`PARSER
`
`
`
`ABSTRACT
`INTERFACE
`
`FIG. 2
`OVERALL
`ARCHITECTURE
`
`O
`/
`TANGIBLE
`INTERFACE
`(GRAPHICS)
`
`TANGIBLE
`INTERFACE
`(AUDIO)
`
`TANGIBLE
`INTERFACE
`(OTHER)
`
`
`
`
`
`
`
`
`
`O
`?
`18
`
`2
`
`32
`
`34
`
`36
`
`-2-
`
`

`

`Patent Application Publication
`
`US 2003/0013073 A1
`
`
`
`
`
`squalue[GI 33euII. Jo 1s??
`
`-3-
`
`

`

`Patent Application Publication Jan. 16, 2003 Sheet 3 of 4
`
`US 2003/0013073 A1
`
`? 50
`LOAD BOOK
`MEADATAN
`BOOK CHOOSER
`
`
`
`
`
`
`
`
`
`52
`
`54
`
`INTIALIZE GUI,
`AU AND OTHER
`TANGIBLE ENT,
`
`
`
`
`
`RECEIVE
`SELECTION
`
`
`
`FIG. 4
`SETUP
`
`
`
`
`
`
`
`
`
`PASS TO
`TANGIBLE
`INTERFACES
`
`
`
`
`
`
`
`LOAD AND
`PARSE
`SELECTED BOOK
`
`? 60
`
`62
`
`64
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`RECEIVE
`NAVIGATION
`COMMANDS FROM
`ETHER ANGBLE
`INTERFACE AND
`NAVIGATE;
`UPDATE OTHER
`INTERFACE
`
`
`
`
`
`
`
`RECEIVE PRESENTATION
`COMMANDS FROM
`ETHER INTERFACE
`AND PRESENT
`UPDATE OTHER
`INTERFACE
`
`
`
`
`
`
`
`
`
`RECEIVE
`ANNOTATON
`COMMANDS
`FROM
`GRAPHICS
`INTERFACE
`
`
`
`
`
`
`
`
`
`
`
`RECEIVE
`ANNOTATION FROM
`GRAPHICS INTER-
`70 FACE; UPDATE AUDIO
`
`
`
`
`
`
`AUDIO
`ANNOTATION
`p
`
`FIG. 5
`OPERATION
`
`
`
`
`
`
`
`RECEIVE ANNOTATION
`FROMAUDIO
`INTERFACE;
`UPDATE GRAPHICS
`
`-4-
`
`

`

`Patent Application Publication
`
`Jan. 16, 2003 Sheet 4 of 4
`
`US 2003/0013073 A1
`
`99
`
`99
`
`9/
`
`09
`
`
`
`
`
`XIONA?UJOJH OID@W OAO^
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`09
`
`-5-
`
`

`

`US 2003/0013073 A1
`
`Jan. 16, 2003
`
`ELECTRONIC BOOK WITH MULTIMODE I/O
`
`BACKGROUND OF THE INVENTION
`0001) 1. Field of the Invention
`0002 The present invention relates generally to elec
`tronic books.
`0003 2. Description of the Related Art
`0004 Replacing paper books with electronic devices
`enables more compact Storage and ease of access to written
`information. For instance, a Single Small electronic device
`can Store many books, including books that might be down
`loaded instantaneously at any time via the Internet. Further,
`electronic books can be backed up, So that physical damage
`to the device does not necessarily correspond to a loss of
`content. Additionally, many electronic books provide the
`capability of annotating the displayed content as desired.
`Even further, to accommodate Sight-impaired readers, Some
`electronic books audibly display content using Synthesized
`Speech.
`0005. Unfortunately, existing electronic book devices do
`not fully exploit many of the advantages they might other
`wise. For instance, while Some books might audibly display
`content, they do not provide a reciprocal means for accept
`ing non-written input from the Visually impaired user. Fur
`ther, electronic devices that are configured for playing
`Synthesized speech typically do not also provide a visual
`display as a convenience for Sighted users, and Vice-versa.
`Not Surprisingly, since existing electronic book devices are
`Single mode, no capability has been provided to manage and
`coordinate plural modes of input and output. Still further, as
`recognized herein Visually impaired users could be accom
`modated by providing the capability to resize text without
`requiring the user to wait until a large print version of the
`book is published, but unfortunately this is not routinely
`done. In any case, existing electronic book devices are
`Simply not as useful to certain users as they might otherwise
`be. The present invention has carefully considered the above
`problems and has provided the Solutions to one or more of
`them as set forth herein.
`
`SUMMARY OF THE INVENTION
`0006 A preferred, non-limiting embodiment of the
`present electronic book device works acroSS a variety of
`modes of input and output. It manages plural modes and
`coordinates the efforts of I/O for consistency. Moreover, the
`invention can extend these capabilities to user-generated
`annotations, Such that the device can be used by disabled
`users and non-disabled users.
`0007 Accordingly, an electronic book device includes a
`portable housing and a processor in the housing. Content
`Stored in a Storage device is displayed in plural output
`modes, and the processor can respond to plural input modes.
`0008. The preferred output modes can include visual
`graphics and Sound. Accordingly, the device can include a
`Visual display and an audio Speaker, both being responsive
`to the processor for outputting content. The processor is
`responsive to user input using one or more input modes to
`Select an output mode. The input modes include graphics
`and Sound, and the device includes a graphics input device
`and an audio input device, both Sending input Signals to the
`
`processor. AS Set forth in greater detail below, the processor
`is responsive to user input to Select an input mode.
`0009 Preferably, the processor responds to a graphics
`input mode by outputting content in a graphics output mode
`using a graphic user interface. The processor also responds
`to an audio input mode by outputting content in an audio
`output mode using an audio user interface. The user inter
`faces run simultaneously with each other. Moreover, the
`processor can receive for Storage annotations from a user
`Selected one of the user interfaces and updates the other user
`interface accordingly Also, the processor is programmed to
`allow a user to navigate through the content using a user
`Selected one of the user interfaces and to update the other
`interface with the navigation result.
`0010. In another aspect, an electronic book includes
`content Stored in a data Storage, an abstract interface acceSS
`ing the data Storage, and an audio user interface communi
`cating with the abstract interface. Also, the book includes a
`graphics user interface communicating with the abstract
`interface. The abstract interface receives user input com
`mands from the audio user interface and updates the graph
`ics user interface in response thereto. Likewise, the abstract
`interface receives user input commands from the graphics
`user interface and updates the audio user interface in
`response thereto.
`0011. In still another aspect, a computer program product
`includes a computer program Storage device and computer
`readable instructions on the Storage device for causing a
`computer to display electronic content in more than one
`mode. The product includes computer readable code means
`for receiving an annotation to content via an audio user
`interface, and computer readable code means for associating
`the annotation with content. Computer readable code means
`display the annotation and associated content using a graphi
`cal user interface.
`0012. In yet another aspect, a method for presenting
`content using an electronic book includes providing a por
`table housing having content electronically Stored therein.
`The method also includes running a first output thread that
`is useful for displaying the content, and, Simultaneously with
`running the first output thread, running at least a Second
`output thread useful for displaying the content.
`0013 In one preferred embodiment, the first output
`thread is a graphics output thread and the Second output
`thread is an audio output thread. The threads are run Simul
`taneously with each other Such that each thread is at Sub
`Stantially the same location in the content as the other thread.
`Each output thread can be associated with a corresponding
`input mode for inputting annotations and user commands
`The method also includes responding to user commands to
`Switch between displaying the content from the first output
`thread and displaying the content from the Second output
`thread.
`0014. The details of the present invention, both as to its
`Structure and operation, can best be understood in reference
`to the accompanying drawings, in which like reference
`numerals refer to like parts, and in which:
`BRIEF DESCRIPTION OF THE DRAWINGS
`0015 FIG. 1 is a schematic diagram of the electronic
`book;
`0016 FIG. 2 is a block diagram showing the architecture
`of the present invention;
`
`-6-
`
`

`

`US 2003/0013073 A1
`
`Jan. 16, 2003
`
`0017 FIG. 3 is a diagram of the main window display on
`the book;
`0.018
`FIG. 4 is a flow chart showing the set up logic;
`0019 FIG. 5 is a flow chart showing the operational
`logic, and
`0020 FIG. 6 is a block diagram showing the detailed
`architecture of one particularly preferred embodiment.
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`Referring initially to FIG. 1, an electronic book
`0021
`device is shown, generally designated 10, that includes a
`portable housing 12 and a visual display 14 on the housing
`12. The display 14 can be a matrix display, touch Screen
`display, or any other visual display device. The display 14
`essentially establishes a first output device that has a corre
`sponding first input device, depicted in FIG. 1 as a graphics
`user interface (GUI) device 16. The GUI 16 can be estab
`lished by the display 14 itself when the display 14 is a touch
`Screen display, or by any other Suitable input device, Such as
`a keyboard or keypad, mouse, tablet with writing Stylus, etc.
`In any case, provision is made for both input and output
`using a graphics device, Such as but not limited to tablets,
`touch Screen displays, etc.
`0022. In addition to the output device established by the
`display 14, the book 10 includes a second output device. The
`Second output device can be, e.g., an audio speaker 18, or
`braille device, or other device A Second input device, Such
`as an audio user interface (AUI) Such as a voice recognition
`module 20 that includes a microphone on the housing 12, is
`asSociated with the Speaker 18. Additional input/output
`device pairs in modes other than audio and graphics can be
`provided in accordance with present principles.
`0023. As shown in FIG. 1, a microprocessor 22 resides
`within the housing 12 and accesses a Software, firmware, or
`hardware module 24 to execute the logic Set forth herein to
`present, in a user-Selected one of at least two modes, content
`that is Stored in a data Storage 26 in the housing 12.
`Alternatively, the Storage 26 can be remote from the housing
`12 and linked to the microprocessor 22 via wired or wireless
`connection. The microprocessor 22 presents content in
`
`graphics form on the display 14 and/or displayS content in
`audio form by playing the content over the Speaker 18.
`Accordingly, the microprocessor 22 accesses the module 24
`to undertake the logic of the present invention, which is
`executed by the microprocessor 22 as a Series of computer
`executable instructions. The instructions may be contained
`on a data Storage device with a computer readable medium,
`having a computer usable medium with code elements
`Stored thereon. Or, the instructions may be Stored on random
`access memory (RAM), on a DASD array, or on magnetic
`tape, conventional hard disk drive, electronic read-only
`memory, optical Storage device, or other appropriate data
`Storage device
`0024. The following description and pseudo-code herein
`illustrate the Structure of the logic of the present invention as
`embodied in computer program Software. Those skilled in
`the art will appreciate that these descriptions illustrate the
`Structures of computer program code elements, including
`logic circuits on an integrated circuit, that function accord
`ing to this invention. Manifestly, the invention is practiced
`in its essential embodiment by a machine component that
`renders the program code elements in a form that instructs
`a digital processing apparatus (that is, a computer) to per
`form a sequence of function Steps corresponding to those
`shown. In an exemplary embodiment, the instructions may
`be written in JAVA 1.3 or JAVA 2.
`0025 FIG. 2 shows the software architecture of the book
`10. AS shown, an abstract interface 28 accesses the Storage
`26, which stores one or more books in electronic form. In
`one preferred, non-limiting example, a book can be stored
`acroSS a number of files that contain the actual text content
`of the book stored as XML files for each chapter or other
`unit, while Some files hold graphics and other inserts as in
`typical HTML documents. In addition to these files, each
`book has a package file. This file contains an XML Speci
`fication of metadata containing information Such as the
`name of the author, the publisher, the date of publication,
`and the type of book. In addition, the package file contains
`a complete manifest of the files that make up the electronic
`book, and a list of the linear order in which the book should
`be presented. The file optionally defines alternative nonlin
`ear navigation paths through the files of the book. A Sample
`of Such a file follows:
`
`&xml version="1.0's
`<!--
`
`Taken from Test Suite for the Open eBook document version 1.0
`Garret Wilson <garretGoglobalmentor.com.>
`Author:
`
`-->
`<! DOCTYPE package PUBLIC “+/ISBN 0-9673.008-1-9//DTD OEB 1.0 Package?/EN”
`<package unique-identifier="oeb10testsuite''>
`<metadata>
`<dc-metadata>
`<dc:Title>Open eBook 1.0 Test Suite</dc:Title>
`<dc:Creator role="aut file-as-"Wilson, Garret's-Garret Wilson.<fdc:Creators
`<dc:Creator role="spin's GlobalMentor, Inc.</dc:Creators
`<dc:Rights>Copyright (C) 1999 Garret Wilson, GlobalMentor, IncY</dc:Rights>
`<dc:Description>Test suite for validating the OEB spec.</dc:Description>
`
`</dc-metadata>
`</metadata>
`
`-7-
`
`

`

`US 2003/0013073 A1
`
`Jan. 16, 2003
`
`-continued
`
`<manifest>
`<!-- OEB Documents.-->
`<item id="foreword href="foreword.html media-type="text/x-oeb1-document f>
`<item id="headings' href="headings.html media-type="text/x-oeb1-document f>
`
`<!-- PNG Images -->
`<item id="mandelbrot.png href="mandelbrot.png media-type="image?.png fs
`
`</manifest>
`<spines
`<item ref idref="foreword f>
`<item refidref="headings' fs
`
`</spines
`<tourse
`<tour id="alphatour title="Alphabetical Tour of Element >
`<site title="Element <as href="links.html#a f>
`<site title="Element <areas href="images.html#area/>
`
`</tours
`</tourse
`<guide>
`<reference type="toc' title="Table of Contents' href="toc.html fs
`<reference type="foreword title="Foreword” href="foreword.html fs
`
`0026. The storage 26 in turn cooperates with a parser 30,
`which sends portions of content Stored in the Storage 26 to
`the abstract interface 28. In one preferred, non-limiting
`implementation, the parser 30 parses each book's package
`file and Stores the metadata into an object representing the
`book. The package files (and all other XML files handled by
`the system) can be parsed using the Document Object Model
`(DOM) and Simple API for XML (SAX) standards. In one
`particularly preferred embodiment, IBM's XML4J 2.0.15
`parser is used.
`0027. In accordance with the present invention, the
`abstract interface 28 manages and coordinates plural tan
`gible interfaces. In the embodiment shown, the tangible
`interfaces include a graphics interface 32 and an audio
`interface 34. Other tangible interfaces 36 can be provided. It
`is to be understood that the graphics interface 32 includes
`Software that outputs content on the display 14 and that
`receives user commands via the GUI device 16. Likewise,
`the audio interface 34 includes Software that outputs content
`by playing the content via the Speaker 18 and that receives
`user commands via the AUI that is established by the voice
`recognition module 20. The tangible interfaces 32, 34, 36
`can each be a combined input/output interface or a pair of an
`input-only interface and output-only interface. In any event,
`each tangible interface 32, 34, 36 can communicate con
`tinuously with the abstract interface 28 and query the
`abstract interface 28 for positional information relative to
`the content being rendered.
`0028. In light of this, for each tangible interface 32, 34,
`36, the abstract interface 28 stores information Such as the
`current position in the content being rendered and/or a
`
`spanning region of the content being rendered (e.g., the start
`and end position of a page of text being displayed). Thus, by
`means of the abstract interface 28, first and Second output
`threads are essentially being run Simultaneously with each
`other AS a consequence, actual output might be undertaken
`using only one output device (display 14 or audio speaker
`18) at a time, depending on user preference, but it might also
`be undertaken in plural modes simultaneously, i.e., by
`displaying the content Simultaneously on both the display 14
`and speaker 18. To this end, the abstract interface 28 also
`determines positions at which tangible interfaces should
`pause when required to coordinate the multiple output
`modes.
`0029. In addition, the tangible interfaces 32, 34, 36
`communicate user commands Such as positional changes (to,
`e.g., change the page being displayed, or Skip to the end of
`the book, etc.) as well as user annotations and mode
`changes. The annotations are associated with user-Selected
`portions of content for Subsequent display of the annotations
`with the associated content. Thus, for example, a user might
`use the GUI device 16 to make an annotation in a page of a
`book being rendered, and if that page is Subsequently played
`back in an alternate mode (e.g., audio), the annotation would
`also be rendered with the page in the alternate mode.
`Moreover, the user can change modes of display by Speaking
`appropriate commands into the Voice recognition device 20
`or writing appropriate commands using the GUI device 16,
`as described more fully below.
`0030) Further details of preferred user commands can be
`appreciated in reference to FIG. 3. It is to be understood that
`while FIG. 3 illustrates certain graphics-related ways to
`
`-8-
`
`

`

`US 2003/0013073 A1
`
`Jan. 16, 2003
`
`input commands, the Same commands can be input using the
`voice recognition module 20, which has a vocabulary for
`recognizing the commands.
`0031. As shown, the display 14 can display graphics
`content 38, including text. Also, command input buttons can
`be provided on the display 14 when it is a touch-screen
`display. The buttons could alternatively be hardware-imple
`mented buttons on a tablet. In any case, the buttons dis
`cussed below communicate Signals to the graphics interface
`32 and thence to the abstract interface 28.
`0032. In one preferred embodiment, the command but
`tons include navigation buttons 40 that can be used to
`change pages or navigate up or down a Single page, go to the
`end or start of the book, and So on. Also, the command
`buttons can include a Search button 42 that can be toggled to
`cause the below-mentioned book chooser to display a list of
`books Stored in the Storage 26, and a Select button 44 that can
`be toggled to Select a book in the list Additionally, the
`command buttons can include annotation buttons 46 that can
`be used to make and erase annotations, with an annotation
`being associated with the content over which the annotation
`has been made.
`0.033
`Still further, presentation mode buttons 48 can be
`provided to instruct the abstract interface 28 to display the
`content in both graphics and audio modes, or only graphics
`mode, or only audio mode. For instance, clicking a “read”
`button Signals the System to begin reading the current page
`from the beginning, and clicking the “read” button again
`pauses the recitation. If the button is clicked a third time
`while the user is viewing the Same page on the display 14,
`recitation resumes where it left off. When the system reaches
`the end of the text of the current page, it automatically
`advances the viewer to the next page (loading the next
`chapter, if necessary) and continues speaking.
`0034 Recitation can also be toggled using voice com
`mands such as “Begin speak” and “stop speak'. Whether the
`toggle is invoked by a click or a voice command, the Visual
`State of the button is always updated appropriately. The
`presentation buttons can also control the size of the fonts
`used to display the text in the display 14.
`0035 FIG. 4 shows the logic for setting up the book 10.
`Commencing at block 50, book metadata is loaded into a
`Software-implemented book chooser. All tangible interfaces
`are initialized at block 52, and then the user's Selection,
`either by means of the select button 44 or by receiving an
`audio command Such as "Select from the Voice recognition
`device 20, is received at block 54. Specifically, the book
`chooser parses the metadata contained in the book files and
`presents the user with a list containing book titles and author
`names. The user can navigate the list using speech com
`mands. Such as “UP and “DOWN to move the Selection
`highlight, and “SELECT to open the document. The user
`can also move the highlight Simply by Speaking the title of
`the book the user wishes to select. The audio interface 34
`will recognize the name and move the highlight to that title.
`The selected book is loaded and parsed by the parser 30, and
`then passed by the abstract interface 28 to the tangible
`interfaces 32, 34, 36 for display in one or more output
`modes.
`0036 FIG. 5 illustrates, in flow chart form, some of the
`functionality discussed above. Commencing at block 60,
`
`navigation commands are received from one or more of the
`tangible interfaces 32, 34, 36. For instance, when the user
`Speaks a command that is recognized as a navigation com
`mand by the audio interface 34, the audio interface 34
`communicates the command to the abstract interface 28 for
`processing. The abstract interface 28 then updates the other
`tangible interface accordingly.
`0037 Also, at block 62 presentation commands are
`received from either one or both of the tangible interfaces
`32, 34. The commands are processed by the abstract inter
`face 28, which updates the other interfaces as required. In
`this way, the graphics thread and audio thread are kept
`coordictaed with each other. Moreover, at block 64 annota
`tions are received graphically (as the user writes across a
`tablet or other GUI established by the GUI device 16), or as
`the user Speaks a command Such as "annotate' followed by
`a spoken annotation.
`0038. In one preferred embodiment, to create an annota
`tion the user Simply clicks on or drags over the text to be
`annotated. The three annotation buttons 46 control which
`type of annotation is created by this action. Each type of
`annotation is represented by a different mark in the text
`window displayed on the display 14. In highlight mode,
`which is the initial Setting (and which can be reselected by
`clicking on the leftmost annotation button 46), dragging
`acroSS a block of text places a yellow background behind the
`Selected text. The highlight can be removed by clicking
`anywhere in the highlighted block. Clicking on the middle
`annotation button 46 puts the System into text-note mode. In
`this mode, clicking on a position in the text causes a dialog
`box to open into which the user can type some text. When
`the box is closed, a green underScore at the position the user
`clicked indicates the presence of the annotation. Subsequent
`clicking on the mark causes the note box to reopen, display
`ing the text of the annotation. When the box is displayed, the
`user can also choose to dictate the annotation rather than
`type it. Last, clicking on the rightmost annotation button 46
`puts the System in audio-note mode. In this mode, clicking
`on a position in the text opens a box in which an audio
`annotation (that is, a stream of uninterpreted digitized audio)
`can be recorded. Recording is started by clicking on the
`record button, or Saying, “begin recording'. For reasons
`explained below in the implementation Section, recording
`can only be terminated by clicking on the Stop button; no
`Voice command can be used for this purpose. The presence
`of an audio annotation is indicated by a red underScore
`beneath that position in the text. AS with text-notes, clicking
`on the marked Spot causes the annotation window to reopen
`So that the recording can be replayed, replaced, or deleted.
`0039 Decision diamond 66 simply indicates that when
`the annotation is made audibly, at block 68 the annotation is
`received and Stored by associating it with the portion of the
`content being displayed. The graphics interface 32 is then
`updated. On the other hand, when the annotation is received
`using the GUI 16, the annotation is passed from the graphics
`interface 32 to the abstract interface 28 for storage and
`asSociation, and the audio interface 34 is updated accord
`ingly at block 70.
`0040 FIG. 6 shows a preferred, non-limiting architecture
`that uses open Standards and off-the-shelf technologies to the
`greatest extent possible, to leverage these technologies to
`reduce implementation time while ensuring that the resulting
`
`-9-
`
`

`

`US 2003/0013073 A1
`
`Jan. 16, 2003
`
`system would not be tied too tightly to any one platform. The
`class structure and method architecture shown in FIG. 6 can
`be specified and maintained using the Unified Modeling
`Language (UML).
`0041. In the exemplary embodiment shown in FIG. 6, the
`graphics interface 32 preferably incorporates a reader Such
`as a reader known as “JEditorPane' Swing component that
`supports HTML 3.2 and a subset of level one of the CSS
`Standard, with the abstract interface 28 communicating
`therewith. In this regard, it is to be understood that while
`books can be thought of as XML files, the tag set used is
`essentially that of HTML 4.0 and level one CSS. Use of the
`above-mentioned reader facilitates the use of off-the-shelf
`Software without having to write a new HTML renderer.
`0.042 AS recognized herein, it is desirable that electronic
`book readers present documents in the familiar page-by
`page manner. Accordingly, the abstract interface 28 com
`municates with a paginator 72, which is a class that is
`responsible for pre-rendering a chapter file, determining
`where to place page breaks, and dividing the document at
`these points. Since changing the font Size used in the viewer
`changes the amount of text that can appear on Screen at one
`time, the paginator 72 can function dynamically at any time.
`The paginator 72 first renders the chapter off-screen in a
`Single Scrolling JEditorPane that is sized to the dimensions
`of the displayed pages. The paginator 72 then Steps through
`the content of the chapter, finding the cutoff position for
`page breaks. This is accomplished by finding the character
`offset of the final character displayed in full in the off-screen
`buffer, then advancing the following line to the top of the
`buffer and repeating the process until the entire chapter has
`been examined. It Stores the resulting character offsets in an
`array Once the page breaks have been determined, the
`System fragments the Scrolling, off-Screen pane into a Series
`of page panes. This can require the creation of new JEdi
`torPane, and the entire chapter file can be re-parsed for each
`page of the chapter. Once the file is paginated, the JEditor
`Pane holding the first page is displayed.
`0.043 AS understood herein, it is important that pagina
`tion occurs after the HTML has already been rendered to
`styled text. Otherwise, the paginator 72 would be forced to
`consider what tags were open at the point that it cuts off a
`page, close them for the current page, and reopen them on
`the next. In the case of complex tag structures, like tables,
`this could inappropriately change the appearance of the
`rendered output.
`0044 As mentioned above, the graphics interface 32 is
`associated with the GUI device 16 on the housing 12, with
`the GUI device 16 including provisions Such as the presen
`tation buttons 48 for changing the fontsize that is displayed.
`Accordingly, a FontSize Changer 74 is provided because, as
`recognized herein, as a result of using the JEditorPane
`component as the renderer, the Seemingly simple task of
`changing the size of the fonts used for display is unduly
`complex Accordingly, a new default CSS Style Sheet is
`applied to the JEditorPane, Setting a new base font Size and
`each other element as well. When changing the font Size, the
`new Style Sheet is applied, and the paginator 72 is called to
`reflow the text and redisplay the pages. Since a new style
`sheet is generated on the fly for each font-Size changing
`event, this technique does not rely on Some fixed number of
`predefined Style sheets.
`
`004.5 To implement the above-described annotations in
`the exemplary architecture shown in FIG. 6, an Open
`Electronic Book (OEB) Annotation module 76 communi
`cates with the abstract interface 28. To associate a location
`in the text being displayed to a location in the document
`when Storing an annotation and thereby establish an anno
`tation Set 78, and then reversing the proceSS when retrieving
`the annotation, character offsets are used to indicate anno
`tation positions in the (rendered) text contained in a JEdi
`torPane. Because these offsets are the values returned from
`JEditorPane text-selection events, they can be used directly
`as returned when the user Selects a block of text to annotate
`The offsets are used in the same way to tell the JEditorPane
`where to put the highlights and annotation marks when
`displaying annotated text. These offsets, which are relative
`to the beginning of the chapter file, are Stable with respect
`to changes of font Size and other rendering characteristics.
`0046. In the architecture shown in FIG. 6, a customized
`XML Schema is used to Store annotations. An annotation file
`is broken into three SubSections corresponding to the three
`types of annotations. A highlight 80 is represented by its
`Starting and ending offset in the chapter text, and a text note
`82 is stored as an offset location 84 together with a string
`containing the text of the note. These Strings are Java-style,
`and So can contain arbitrary Unicode. An audio annotation
`86 consists of the location 88 of the annotation and the name
`of the file containing the digitized audio. One annotation file
`is Stored for each file in an OEB package. The annotation
`file's name is formed by appending the extension annotation
`to the underlying chapter file name. Whenever a chapter file
`is read, the corresponding annotation file, if it exists, is read
`as well. A Sample annotation file is as follows:
`
`&xml version="1.O's
`<annotation sets
`<highlight sets
`<highlight start="1301 end="2984"/>
`</highlight sets
`<text note Set
`<text note offset="66' text="Implementation details?/>
`</text note sets
`<audio note sets
`<audio note offset="13
`path="file:/C:/oeb/RIAO2000/riao0.html/>
`</audio note sets
`<fannotation sets
`
`0047. In the current version of the system, highlights and
`annotations are implemented using the So-called Highlight
`Painter 90 provided by JEditorPane. While text and audio
`notes are denoted with green and red underline markings,
`these are actually implemented as narrow highlights, gen
`erated by a so-called HighlightPainter 90. Ordinary high
`lights are presented using the DefaultHighlightPainter with
`a light yellow color Setting.
`0048. In continued reference t