(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2009/0259711 A1
`(43) Pub. Date:
`Oct. 15, 2009
`Drieu et al.
`
`US 20090259711A1
`
`(54) SYNCHRONIZATION OF MEDIA STATE
`ACROSS MULTIPLE DEVICES
`
`(22) Filed:
`
`Apr. 11, 2008
`
`Publication Classification
`
`(75) Inventors:
`
`Gilles Drieu, San Francisco, CA
`(US); Barry Richard
`Munsterteiger, Belmont, CA (US)
`
`Correspondence Address:
`FSH & RICHARDSON P.C.
`PO BOX 1022
`MINNEAPOLIS, MN 55440-1022 (US)
`
`(73) Assignee:
`
`APPLE INC., Cupertino, CA (US)
`
`(21) Appl. No.:
`
`12/101,896
`
`
`
`(51) Int. Cl.
`(2006.01)
`G06F 5/16
`(52) U.S. Cl. ........................................................ 709/201
`(57)
`ABSTRACT
`Media State synchronization across multiple devices can
`include detecting an event relating to a user's access of con
`tent on a first device, determining state information relating to
`an access state of the content corresponding to the detected
`event, and transmitting the determined State information to a
`remote location for use in accessing the content on a second
`device.
`
`220
`
`225
`
`230
`
`s
`
`A
`
`v.
`
`235
`
`240
`
`
`
`275
`
`270
`
`
`
`285
`
`2O
`
`-1-
`
`Amazon v. Audio Pod
`US Patent 10,091,266
`Amazon EX-1021
`
`

`

`Patent Application Publication
`
`Oct. 15, 2009 Sheet 1 of 8
`
`US 2009/0259711 A1
`
`
`
`S
`IIEN
`
`S
`
`S
`
`r
`N
`N
`
`
`
`S
`
`S
`
`S
`
`S
`
`-2-
`
`

`

`Patent Application Publication
`
`Oct. 15, 2009 Sheet 2 of 8
`
`US 2009/0259711 A1
`
`s
`N
`
`
`
`
`
`
`
`IIDEN
`
`S
`
`S
`
`IIDX
`
`Š
`
`S
`
`S.
`
`S
`
`S
`
`:
`
`
`
`S
`N
`
`O
`
`-3-
`
`

`

`Patent Application Publication
`
`Oct. 15, 2009 Sheet 3 of 8
`
`US 2009/0259711 A1
`
`S 301
`Queue media object for
`display
`
`Monitor Commands
`
`
`
`Play
`-l
`Start/Resume Playback
`
`302
`
`303
`
`
`
`
`
`
`
`Monitor commands and the playback l
`
`304
`
`No
`
`—-
`
`
`
`ACCeSS event
`detected?
`
`
`
`Yes
`
`Determine type o
`access event
`
`305
`
`306
`
`Pause/Stop
`
`End of Playback
`
`Cease playback
`
`307
`
`-
`
`
`
`308
`
`
`
`Send State information
`including current playhead
`position and media object
`identifier to Server
`
`-
`Cease playback
`
`309
`
`r 310
`Send State information
`including completion
`event and media object
`identifier to Server
`
`
`
`
`
`
`
`FIG. 3
`
`-4-
`
`

`

`Patent Application Publication
`
`Oct. 15, 2009 Sheet 4 of 8
`
`US 2009/0259711 A1
`
`
`
`
`
`s
`
`lar
`S
`N
`
`S
`
`|
`
`S
`
`-5-
`
`

`

`Patent Application Publication
`
`Oct. 15, 2009 Sheet 5 of 8
`
`US 2009/0259711 A1
`
`Connect to a synchronization process
`
`Detect content update event
`
`S 502
`
`503
`
`Retrieve metadata for media objects within a
`Collection
`
`s 504
`Create a download list of unwatched and/or
`partially viewed media objects within the
`Collection based On the metadata
`
`s 505
`Download a media object from the download
`list
`
`Yes
`
`
`
`More media
`objects to
`download?
`
`No
`
`FIG. 5
`
`-6-
`
`

`

`Patent Application Publication
`
`Oct. 15, 2009 Sheet 6 of 8
`
`US 2009/0259711 A1
`
`Obtain media object and associated S'
`metadata including a playhead position
`for the media object
`T
`Queue media object for display based
`on the playhead position
`
`S 6O2
`
`Monitor Commands
`
`603
`
`Play
`l
`Resume playback at playhead position
`
`y 604
`
`Monitor commands and playback
`while advancing playhead
`
`605
`
`No
`
`
`
`
`
`-
`Pause/Stop
`
`Access event
`detected?
`
`
`
`Yes
`
`Determine type o
`aCCeSS event
`
`
`
`606
`
`607
`
`S 608
`
`Cease playback
`
`- 609
`Send state information
`including current
`playhead position and
`media objecticentifier to
`SeVer
`
`End of Playback
`
`l 610
`Cease playback
`
`l 67.1
`
`Send state information
`including completion
`event and media object
`identifier to Server
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 6
`
`-7-
`
`

`

`Patent Application Publication
`
`Oct. 15, 2009 Sheet 7 of 8
`
`US 2009/0259711 A1
`
`Monitor incoming
`Connections
`l
`Connect incoming
`Connection to a
`Synchronization process
`
`r 701
`
`702
`
`- 703
`
`Monitor Connection for
`messages
`
`
`
`
`
`
`
`
`
`Metadata
`update?
`
`704
`
`Yes
`- 705
`l
`POCeSS metadata
`update for a media object
`
`706
`Store metadata update
`for the media object
`
`-
`
`N
`
`707
`
`Metadata
`request?
`
`708
`Retrieve metadata
`for the requested
`media object
`
`Send metadata for
`the requested media
`object
`
`710
`Process other
`update or request
`
`FIG. 7
`
`-8-
`
`

`

`Patent Application Publication
`
`Oct. 15, 2009 Sheet 8 of 8
`
`US 2009/0259711 A1
`
`Detect an event relating to a user's access of
`Content On a first device
`
`Determine state information relating to an
`access state of the content corresponding to
`the detected event
`
`
`
`Transmit the determined State information to
`a remote location for use in accessing the
`Content On a SeCond device
`
`SO
`
`SO
`
`FIG. 8A
`
`
`
`Receive from a remote location State
`information relating to an access state of
`Content on a first device
`l
`Use the received state information to manage
`a user's access of Content on a second
`device
`
`71
`
`37.2
`
`FIG. 8B
`
`-9-
`
`

`

`US 2009/0259711 A1
`
`Oct. 15, 2009
`
`SYNCHRONIZATION OF MEDIA STATE
`ACROSS MULTIPLE DEVICES
`
`TECHNICAL FIELD
`0001. The present disclosure relates to synchronizing
`media State across multiple devices.
`
`BACKGROUND
`0002 Content can include media objects, such as movies,
`audio files, digital video, presentations, and documents. The
`media objects can be stored on and accessed over networks.
`Devices such as a laptop, mobile phone, computer, entertain
`ment system., and a mobile media device can be used to
`access the content. A user can Switch devices while viewing a
`media object. For example, a user might view part of a movie
`on a laptop and a second part of the movie on a mobile phone.
`Typically, a user has to reposition the playback of the media
`object to the position where the user was last when Switching
`between devices.
`
`SUMMARY
`0003. This specification describes technologies that,
`among other things, synchronize media state across multiple
`devices.
`0004. In general, the subject matter described can be
`implemented in methods that include detecting an event relat
`ing to a user's access of content on a first device, determining
`state information relating to an access state of the content
`corresponding to the detected event, and transmitting the
`determined State information to a remote location for use in
`accessing the content on a second device. Other implemen
`tations can include corresponding systems, apparatus, and
`computer program products.
`0005. These, and other aspects, can include one or more of
`the following features. The detected event c(an include at
`least one of pause, stop, access complete, power off, and user
`input. The state information can include a playhead position.
`The state information can include an indication of whether
`the content has been accessed completely. The content can
`include video, audio, text, graphics, or a combination thereof.
`Copies of the content can reside on each of the first and
`second devices. The content can reside at a remote location
`and can be streamed to a device during access. The remote
`location can include a computer system accessible via a wide
`area network. The remote location can include a computer
`system within a same local area network as the first device.
`Content can include digital video and accessing the content
`can include playing a digital video file. The transmitted State
`information can be used to update content residing on the
`second device. The transmitted State information can be used
`to position an access point in the, content on the second
`device. The remote location can include a computer system
`within a same local area network as the second device. The
`first device can be located at the remote location. The second
`device can be located at the remote location. The content can
`include one or more media objects.
`0006. The subject matter described can also be imple
`mented in methods that include receiving from a remote
`location state information relating to an access state of con
`tent on a first device and using the received state information
`to managea user's access of content on a second device. Other
`implementations can include corresponding systems, appa
`ratus, and computer program products.
`
`0007. These, and other aspects, can include one or more of
`the following features. The state information can include a
`playhead position. The state information can include an indi
`cation of whether the content has been accessed completely.
`The content can include video, audio, text, graphics, or a
`combination thereof. The content can include one or more
`media objects. The remote location can include a computer
`system within a same local area network as the second device.
`Content can include digital video and accessing the content
`can include playing a digital video file. The transmitted State
`information can be used to update content residing on the
`second device. The transmitted State information can be used
`to position an access point in the content on the second device.
`The remote location can include a computer system within a
`same local area network as the second device.
`0008 Particular implementations of the subject matter
`described in this specification may be implemented to realize
`one or more of the following potential advantages. The Sub
`ject matter described can be implemented Such that content
`access amongst devices can be synchronized. For example, a
`user viewing content on a first device can Switch to a second
`device to continue viewing the content without having to
`manually reposition playback of the content on the second
`device to where the user left off on the first device. The subject
`matter described also can be implemented to retrieve content
`in anticipation of future content access.
`0009. The details of one or more implementations are set
`forth in the accompanying drawings and the description
`below. Other features and advantages will be apparent from
`the description and drawings, and from the claims.
`
`DESCRIPTION OF DRAWINGS
`0010 FIG. 1 shows an example of a distributed content
`viewing environment.
`0011
`FIG. 2 shows another example of a distributed con
`tent viewing environment.
`0012 FIG. 3 shows an example of accessing content.
`0013 FIG. 4 shows an example of a content update event.
`0014 FIG. 5 shows an example of a content synchroniza
`tion flowchart.
`0015 FIG. 6 shows an example of a flowchart of resuming
`playback based on an access state.
`0016 FIG. 7 shows an example of a synchronization pro
`CCSS,
`0017 FIGS. 8AB show examples of synchronization pro
`cesses from a device view point.
`0018. Like reference symbols in the various drawings
`indicate like elements.
`
`DETAILED DESCRIPTION
`0019 Content can be, viewed on multiple devices. View
`ing of content, such as a, digital video, can take place on two
`or more of these devices. In order for a user to continue to
`watch the digital video without having to reposition the play
`back of the digital video on another device, state information
`about the digital video can be distributed to the other device.
`The other device can use the state information to automati
`cally reposition the playback of the digital video.
`0020 FIG. 1 shows an example of a distributed content
`viewing environment. Content can include one or more media
`objects. A media object can include digital video, audio, text,
`and/or graphics. For example, the media object can include a
`movie or presentation. The content can be viewed on multiple
`
`-10-
`
`

`

`US 2009/0259711 A1
`
`Oct. 15, 2009
`
`devices and combination of devices. For example, a mobile
`computing device 110. Such as a laptop, can be used to view
`content and can be used to download content from a server
`130. The mobile computing device 110 and server 130 can be
`connected to a communication network 120 via network con
`nections 115, 125. The communication network 120 can
`include wired and wireless components. For example, the
`communication network 120 can include the Internet.
`0021. A media processing device 135, such as an
`AppleTV, manufactured by Apple Inc. of Cupertino, Calif.,
`can download content from server 130 via network connec
`tion 140. A presentation device such as a monitor 150 can be
`coupled to the media processing device 135 through a media
`connector 145. Such that video and/or audio information gen
`erated by the media processing device 135 can be presented
`through the monitor 150.
`0022. A mobile phone 160 can download content from
`server 130. For example, the mobile phone can be an iPhone,
`manufactured by Apple Inc. of Cupertino, Calif. The mobile
`phone 160 can connect to the network 120 via a wireless link
`155 to download content from server 130.
`0023. A device, such as a mobile computing device 110,
`mobile phone 160, media processing device 135. etc., can be
`used to access content. The device can detect an event relating
`to an access of content. For example, the content can include
`a digital video such as a movie and one type of content access
`can include playing the movie. The access events for a movie
`can include when playback pauses, stops, rewinds, or fast
`forwards and when playback of the content is complete. In
`another example, the content can include a presentation and
`the content access can include advancing to a next slide
`within the presentation. The access events for a presentation
`can include advancing to the next slide or accessing a differ
`ent slide within the presentation. In some implementations,
`access events can also include when a device starts to power
`down, user input, or a periodic firing of a timer.
`0024. The device can determine state information relating
`to an access state of the content corresponding to the detected
`event. In some implementations, state information relating to
`playback of content can include the position of a playhead.
`The playhead position can represent a currently or last dis
`played frame or slide. The determined state information can
`be sent to a remote location, such as a server 130, for use in
`accessing the content on a second device.
`0025. A user can create a play list that includes one or
`more media objects to access. For example, the play list can
`include multiple episodes of a television series. In another
`example, the play list can include one or more movies or
`audio files. A server such as server 130 can store the play list.
`0026. As an example, a user can watch a media object off
`of the play list Such as a television episode on a mobile
`computing device 110. The user can begin playback of the
`episode on the mobile computing device 110. During play
`back, State information relating to the playback of the episode
`can be sent to server 130. A mobile phone 160 can obtain this
`state information then updates to this state information by
`requesting this information from server 130. In some imple
`mentations, the mobile phone 160 can download the episode
`being watched on the mobile computing device 110 in
`response to the state information. Additionally, the mobile
`phone 160 can download the next episode or media object in
`the play list. When playback continues on mobile phone 160,
`playback of the episode can continue at a position related to
`the position where playback was ceased on the mobile com
`
`puting device 110. A similar transition of playback can hap
`pen between any pair of devices including media processing
`device 135 and mobile phone 160.
`0027 FIG. 2 shows another example of a distributed con
`tent viewing environment. The environment can include a
`host location 201, such as a home or office. The host location
`201 can include a mobile computing device 205, mobile
`media player 215, and media processing device 235 con
`nected via a media connector 240 to a presentation device
`such as a monitor 245 for viewing the output from the media
`processing device 235. The mobile computing device 205 and
`the media processing device 235 can be connected to a local
`area network (LAN) 225 via network connections 220, 230.
`Network connections 220, 230 can be wired or wireless. The
`LAN 225 can include wireless access points. LAN 225 can be
`connected to a wide area network (WAN) 255 via a network
`connection 250. A WAN 255 can include the Internet and
`wireless access points. A mobile phone 265 can connect to the
`WAN 255 via a wireless network connection 260.
`0028. A content storage server 275 and a content metadata
`server 285 can be connected to WAN 255 via network con
`nections 270,280. Content storage server 275 can store media
`objects Such as movies, television episodes, music, or presen
`tations. Devices such as mobile computing device 205, media
`processing device 235, and mobile phone 265 can download
`content from the content storage server 275. Additionally,
`mobile media player 215 can receive content from the mobile
`computing device 205 via communication link 210. Metadata
`including state information about the access of content can be
`stored on content metadata server 285. In some implementa
`tions, content storage server 275 and content metadata server
`285 can co-exist on a single server or can be divided amongst
`multiple servers.
`0029. A device within the host location 201, such as
`mobile computing device 205, can act as a local content
`storage server and/or a local content metadata server for other
`devices including media processing device 235 and mobile
`media player 215. When a device. Such as media processing
`device 235, cannot connect or prefers not to connect to server
`275, 285, the device can connect to the mobile computing
`device 205 if the mobile computing device 205 is acting as a
`local content storage server and/or a local content metadata
`SeVe.
`0030 FIG. 3 shows in example of accessing content such
`as a media object and the accessing of the content can include
`playback or viewing of the media object. A content viewer
`running on a device such as devices 110, 135, 160 can queue
`301 a media object for display. The content viewer can moni
`tor 302 commands that include content access commands or
`events. When the content viewer receives a play command,
`the content viewer can start or resume 303 playback of the
`media object. The content viewer can further monitor 304
`commands and the state of the playback. The content viewer
`can detect 305 an access event. If an access event is not
`detected, the content viewer can continue to monitor 304
`commands and continue the playback. If an access event is
`detected, then the type of access event can be determined 306.
`If the access event includes a pause or stop command, then
`playback 307 can cease. State information, that can include a
`current playhead position on the media object and a corre
`sponding media object identifier, can be transmitted 308 to a
`remote location such as a server 130. If the access event
`includes the end of playback of the media object, then play
`back 309 can cease. State information, that can include the
`
`-11-
`
`

`

`US 2009/0259711 A1
`
`Oct. 15, 2009
`
`completion of playback event and a corresponding media
`object identifier, can be transmitted 310 to the remote location
`such as a server 130.
`0031
`FIG. 4 shows an example of a content update event.
`A mobile phone 160 can be position enabled such that either
`the mobile phone 160 or a server can determine the distance
`between the mobile phone 160 and a user's home 400. When
`a user 405 with a mobile phone 160 comes within a user
`configurable range 410 from the user's home 400, a content
`update event can occur. In some examples, range 410 can
`default to 100 feet. The content update event can be sent to a
`server such as server 130. The media processing device 135
`can listen for the content update event or server 130 can send
`the event to the device 135 or other devices requesting to be
`informed of such events. The content update event can trigger
`the media processing device 135 to download content and
`associated metadata. For example, if the user 405 is watching
`a media object on the mobile phone 160 when the content
`update event occurs, the media processing device 136 after
`receiving the event can download the media object so that the
`user 405 can continue to watch the media object on screen
`150.
`0032. Another content update event can be the disconnect
`of a Bluetooth R wireless connection Such as a connection
`between a mobile media device 215 and a mobile computing
`device 205. The shutdown process of a device 110, 135, 160
`can also trigger a content update event. In some implementa
`tions, a periodic timer can be set on a device 110, 135, 160 to
`trigger content update events. For example, a mobile phone
`can be set to trigger a content update event at 8 AM. In some
`implementations, the content update event can be sent to a
`synchronization process.
`0033 FIG. 5 shows an example of a content synchroniza
`tion flowchart. A content update process running on a device
`110, 135, 160 can connect 501 to a synchronization process.
`The synchronization process can be running at a remote loca
`tion such as server 130. The content update process can detect
`502 a content update event via the synchronization process.
`The content update process can retrieve 503 metadata for
`media objects within a collection. The content update process
`can create 504 a download list of unwatched and/or partially
`viewed media objects within the collection based on the meta
`data. The content update process can then download 505 a
`media object from the download list. If more media objects
`are to be downloaded 506, then the download 505 can con
`tinue. If no media objects remain to be downloaded, then the
`content update process can return to detecting 502 content
`update events.
`0034 FIG. 6 shows am example of a flowchart of resum
`ing playback based on an access state. A content viewer
`running on a device 110, 135, 160 can obtain 601 a media
`object and associated metadata including a playhead position
`for the media object. The media object can already be stored
`within a storage medium accessible by the content viewer or
`the media object can be downloaded in response to a content
`update event. The metadata associated with the media object
`can be already present on the device 110, 135, 160 or the
`metadata can be retrieved from server 130 and stored on the
`device 110, 135, 160. The content viewer can queue 602
`media object for display based on the playhead position. In
`Some implementations, the content viewer can queue the
`media object for display at a position before, at, or after the
`playhead position. The content viewer can monitor 603 com
`mands. When the content viewer receives a play command,
`
`the content viewer can resume 604 playback of the media
`object at the playhead position. The content viewer can fur
`ther monitor 605 commands and the state of the playback
`while advancing the playhead.
`0035. The content viewer can detect 606 an access event.
`If an access event is not detected, the content viewer can
`continue to monitor 605 commands and continue the play
`back. If an access event is detected, then the type of access
`event can be determined 607. If the access event includes a
`pause or stop command, then playback 608 can cease. State
`information, that can include a current playhead position and
`an identifier of the media object, can be transmitted 609 to a
`remote location such as a server 130. After transmitting 609,
`the content viewer can continue to monitor 603 monitors. If
`the access event includes the end of playback, then playback
`610 can cease. State information, that can include the comple
`tion of playback event and an identifier of the media object,
`can be transmitted 611 to server 130. After transmitting 611,
`the content viewer can continue to monitor 603 for com
`mands.
`0036 FIG. 7 shows in example of a synchronization pro
`cess. The synchronization process can be running on a server
`such as on server 130. A server process can monitor 701 for
`incoming connections. The server process can connect 702 an
`incoming connection to a synchronization process for a user.
`If the synchronization process does not exist, the server pro
`cess can create a synchronization process for the user. In some
`implementations, if a synchronization process already exists
`for the user, the incoming connection can use the already
`created synchronization process. For example, connections
`from multiple devices can connect to the same synchroniza
`tion process. The synchronization process can monitor 703
`the connection for incoming messages. The messages can be
`a metadata update, metadata request, or other types of updates
`or requests. If the message includes a metadata update 704 for
`a media object, then the metadata included in the message can
`be processed 705. The metadata can include state information
`about the accessing of the media object or information about
`the media object. State information can include a position of
`a playhead associated with the playback of the media object
`or a flag indicating that the media object has been completely
`accessed or viewed. The metadata update for the media object
`can be stored 706 on a local or remote storage device attached
`to the server 130 or the update can be stored in memory or a
`combination thereof. The synchronization process can con
`tinue to monitor 703 the connection for new messages. If the
`message includes a metadata request 707, then the metadata
`for tho requested media object can be retrieved 708. The
`metadata request can include a request for the state informa
`tion associated with a media object. The metadata for the
`media object can be sent 709 over the connection. The syn
`chronization process can continue to monitor 703 the connec
`tion for new messages. If the message contains a different
`type of request or update, the message can be processed 710
`and the synchronization process can continue to monitor 703
`the connection for new messages. For example, other mes
`sages can include content update event notifications and
`requests to receive Such notifications. Further, other messages
`can include requests to receive metadata updates for a user.
`0037 FIG. 8A shows, an example of a synchronization
`process from a device view point. A content viewer on a first
`device can detect 801 an event relating to a user's access of
`content on the first device. The content viewer can determine
`802 state information relating to an access state of the content
`
`-12-
`
`

`

`US 2009/0259711 A1
`
`Oct. 15, 2009
`
`corresponding to the detected event. The content viewer can
`transmit 803 the determined state information to a remote
`location for use in accessing the content on a second device.
`The remote location can include a computer system Such as a
`server or a personal computer. In some implementations, the
`remote location can include either the first or second device or
`both.
`0038 FIG. 8B shows an example of a synchronization
`process from a different device's view point. In a complemen
`tary synchronization process to that shown in FIG. 8A, a
`content viewer on a second device can use the state informa
`tion produced on the first device. The content viewer on the
`second device can receive 811 from a remote location state
`information relating to an access state of content on a first
`device. The content viewer can use 812 the received state
`information to manage a user's access of content on a second
`device.
`0039) Implementations of the subject matter and the func
`tional operations described in this specification can be imple
`mented in digital electronic circuitry, or in computer soft
`ware, firmware or hardware, including the structures
`disclosed in this specification and their structural equivalents,
`or in combinations of one or more of them. Implementations
`of the subject matter described in this specification can be
`implemented as one or more computer program products, i.e.,
`one or more modules of computer program instructions
`encoded on a computer-readable medium for execution by, or
`to control the operation of data processing apparatus. The
`computer-readable medium can be a machine-readable stor
`age device, a machine-readable storage Substrate, a memory
`device, or a combination of one or more of them. The term
`"data processing apparatus' encompasses all apparatus,
`devices, and machines for processing data, including by way
`of example a programmable processor, a computer, or mul
`tiple processors or computers. The apparatus can include, in
`addition to hardware, code that creates an execution environ
`ment for the computer program in question, e.g., code that
`constitutes processor firmware, a protocol stack, a database
`management system, an operating system, or a combination
`of one or more of them. A propagated signal is an artificially
`generated signal, e.g., a machine-generated electrical, opti
`cal, or electromagnetic signal, that is generated to encode
`information for transmission to Suitable receiver apparatus.
`0040. A computer program (also known as a program,
`Software, Software application, Script, or code) can be written
`in any form of programming language, including compiled or
`interpreted languages, and it can be deployed in any form,
`including as a stand-alone program or as a module, compo
`nent, Subroutine, or other unit Suitable for use in a computing
`environment. A computer program does not necessarily cor
`respond to a file in a file system. A program can be stored in
`a portion of a file that holds other programs or data (e.g., one
`or more scripts stored in a markup language document), in a
`single file dedicated to the program in question, or in multiple
`coordinated files (e.g., files that store one or more modules,
`Sub-programs, or portions of code). A computer program can
`be deployed to be executed on one computer or on multiple
`computers that are located at one site or distributed across
`multiple sites and interconnected by a communication net
`work.
`0041. The processes and logic flows described in this
`specification can be performed by one or more programmable
`processors executing one or more computer programs to per
`form function S by operating on input data and generating
`
`output. The processes and logic flows car also be performed
`by, and apparatus can also be implementedas, special purpose
`logic circuitry, e.g., an FPGA (field programmable gate array)
`or an ASIC (application-specific integrated circuit).
`0042 Processors suitable for the execution of a computer
`program include, by way of example, both general and special
`purpose microprocessors, and any one or more processors of
`any kind of digital computer. Generally, a processor will
`receive instructions and data from a read-only memory or a
`random access memory or both. The essential elements of a
`computer are a processor for performing instructions and one
`or more memory devices for storing instructions and data.
`Generally, a computer will also include, or be operatively
`coupled to receive data from or transfer data to, or both, one
`or more mass storage devices for storing data, e.g., magnetic,
`magneto-optical disks, or optical disks. However, a computer
`need not have such devices. Moreover, a computer can be
`embedded in another device, e.g., a mobile telephone, a per
`sonal digital assistant (PDA), a mobile audio player, a Global
`Positioning System (GPS) receiver, to name just a few. Com
`puter-readable media Suitable for storing computer program
`instructions and data include all forms of non-volatile
`memory, media and memory devices, including by way of
`example semiconductor memory devices, e.g., EPROM,
`EEPROM, and flash memory devices; magnetic disks, e.g.,
`internal hard disks or removable disks; magneto-optical
`disks; and CD-ROM and DVD-ROM disks. The processor
`and the memory can be Supplemented by, or incorporated in,
`special purpose logic circuitry.
`0043. To provide for interaction with a user, implementa
`tions of the Subject matter described in this specification can
`be implemented on a computer having a display device, e.g.,
`a CRT (cathode ray tube) or LCD (liquid crystal display)
`monitor, for displaying information to the user and a key
`board and a pointing device, e.g., a mouse or a trackball, by
`which the user can provide input to the computer. Other kinds
`of devices can be used to provide for interaction with a user as
`well; for example, feedback provided to the user can be any
`form of sensory feedback, e.g., visual feedback, auditory
`feedback, or tactile feedback; and input from the user can be
`received in any form, including acoustic, speech, near-touch
`input, or tactile input.
`0044) Implemen