`
`
`
`(21) International Application Number:
`
`PCT/US99/O4506
`
`(22) International Filing Date:
`
`1 March 1999 (01.03.99)
`
`
`
`(81) Designated States: CN, JP, European patent (AT, BE, CH, CY,
`DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, NL, PT,
`SE).
`
`Published
`With international search report.
`Before the expiration of the time limit for amending the
`claims and to be republished in the event of the receipt of
`amendments.
`
`(30) Priority Data:
`09/039,019
`
`13 March 1998 (13.03.98)
`
`US
`
`INC.
`SIEMENS CORPORATE RESEARCH,
`(71) Applicant:
`[US/US]; 755 College Road East, Princeton, NJ 08540
`(US).
`
`(72) Inventors: LIOU, Shih—Ping; 3 Orly Court, Robbinsville, NJ
`08691 (US). HJELSVOLD, Rune; 507 Plainsboro Road,
`Plainsboro, NJ 08536 (US).
`
`(74) Agents: AHMED, Adel, A. et al.; Siemens Corporation,
`Intellectual Property Dept., 186 Wood Avenue South, lselin,
`NJ 08830 (US).
`
`
`
`
`
`
`
`PCT
`
`International Bureau
`WORLD INTELLECTUAL PROPERTY ORGANIZATION
`
`
`
`
`
`(51) International Patent Classification 6 1
`
`INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT)
`
`
`
`(11) International Publication Number:
`WO 99/46702
`G06F 17/30, H04N 7/15
`
`
`(43) International Publication Date:
`16 September 1999 (16.09.99)
`
`
`(54) Title: APPARATUS AND METHOD FOR COLLABORATIVE DYNAMIC VIDEO ANNOTATION
`
`
`
`COLLEAGUES‘ HIXEO VOICE COMMENTS
`
`
`
`SYSTEM TESSAG‘ES
`
`
`FRAME MJMBER
`
` PERSED VCR
`PLAY COVHAMJS
`
`
`ATTACl-EO
`TOOLS
`
`
`
`OWN ANNOTATION
`
`RECOPOS
`
`
`COLLEAGUES'
`ACTIVE POINTER
`848
`
`
`POSITIONSI
`COW
`
`
`UNIFIED
`SYSTEM HESSAGES
`COMPUTER/INTERNET
`
`COLLABORATIVE
`TELEPHONY INTERFACE W
`
`VIDEO ANNOTATION
`
`semen
`
`
`815
`
`
`
`
`
`D
`
`
`
`W
`
`(57) Abstract
`
`A method for dynamic video annotation among a plurality of users utilizes a programmable computer and comprises the steps of:
`selecting a network service coupled to the computer; performing one of (a) starting and (b) joining a collaborative session among the users;
`loading one of (a) a video and (b) a recorded annotation file; performing annotation of at least one of graphical, text, and audio annotation;
`and storing said collaborative session.
`
`I_
`
`
`
`TikTok TTI-1021, Page 1
`|PR2021-00476 (TikTok Inc. V. 10Tales Inc.)
`
`

`

`
`
`FOR THE PURPOSES OF INFORMATION ONLY
`
`Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT.
`SI
`Slovenia
`LS
`ES
`Lesotho
`FI
`SK
`LT
`Lithuania
`Slovakia
`SN
`FR
`LU
`Senegal
`Luxembourg
`GA
`LV
`Latvia
`SZ
`Swaziland
`TD
`Chad
`MC
`Monaco
`GB
`MD
`TC
`GE
`Togo
`Republic of Moldova
`GH
`MG
`1‘1
`Tajikistan
`Madagascar
`MK
`TM
`Turkmenistan
`GN
`The former Yugoslav
`TR
`GR
`Turkey
`Republic of Macedonia
`TT
`HU
`Mali
`Trinidad and Tobago
`UA
`[E
`Ukraine
`Mongolia
`UG
`[L
`Mauritania
`Uganda
`US
`United States of America
`IS
`Malawi
`IT
`UZ
`Uzbekistan
`Mexico
`VN
`Viet Nam
`JP
`Niger
`KE
`YU
`Netherlands
`Yugoslavia
`ZW
`Zimbabwe
`KG
`Norway
`KP
`New Zealand
`Poland
`Portugal
`Romania
`Russian Federation
`Sudan
`Sweden
`Singapore
`
`
`
`M L
`MN
`MR
`MW
`MX
`NE
`NL
`NO
`NZ
`PL
`PT
`RO
`RU
`SD
`SE
`SG
`
`AL
`AM
`AT
`AU
`AZ
`BA
`BB
`BE
`BF
`BG
`3.]
`BR
`BY
`CA
`CF
`CG
`CH
`CI
`CM
`CN
`CU
`CZ
`DE
`DK
`EE
`
`
`
`Albania
`Armenia
`Austria
`Australia
`Azerbaijan
`Bosnia and Herzegovina
`Barbados
`Belgium
`Burkina Faso
`Bulgaria
`Benin
`Brazil
`Belarus
`Canada
`Central African Republic
`Congo
`Switzerland
`Gate d‘lvoire
`Cameroon
`China
`Cuba
`Czech Republic
`Gemiany
`Denmark
`Estonia
`
`Spain
`Finland
`France
`Gabon
`United Kingdom
`Georgia
`Ghana
`Guinea
`Greece
`Hungary
`Ireland
`Israel
`Iceland
`Italy
`Japan
`Kenya
`Kyrgyzstan
`Democratic People’s
`Republic of Korea
`Republic of Korea
`Kazakstan
`Saint Lucia
`Liechtenstein
`Sri Lanka
`Liberia
`
`KR
`KZ
`LC
`LI
`LK
`LR
`
`l_______—____._______#
`
`TTI-1021, Page 2
`
`

`

`WO 99/46702
`
`PCT/US99/04506
`
`APPARATUS AND METHOD FOR
`
`COLLABORATIVE DYNAMIC VIDEO AN NOTATION
`
`FIELD OF THE INVENTION
`
`The present invention relates to the field of collaborative video annotation, and
`
`more specifically, to apparatus for enabling multiple users to share their views about a
`
`video content.
`
`10
`
`BACKGROUND OF THE INVENTION
`
`Reference is made to a patent application entitled Method and Apparatus for
`
`Creating Dynamic Object Markers in a Video Clip being filed on even date herewith,
`
`and assigned to the same assignee as the present application, and whereof the disclosure
`
`is herein incorporated by reference to the extent it is not incompatible with the present
`
`15
`
`application.
`
`A situation can arise wherein two or more users wish to communicate in
`
`reference to a common object, for example, in reference to a video. An example of this
`
`could be where a soccer team coach wishes to consult with a colleague to seek advice.
`
`The soccer team coach might wish to show a taped video of a game and ask the
`
`colleague to explain, using the video, why one team failed to score in a given attack
`
`situation.
`
`In addition, the coach might wish to record this discussion and show it later
`
`to other coaches to get more opinions.
`
`In another scenario, a student could be taking a training course being given at a
`
`remote location from where a course instructor is located.
`
`It may be that the student
`
`cannot understand a procedure being taught in the course. The student can then call the
`
`instructor over the Internet phone to find out how such a procedure should be
`
`performed. The instructor can first browse through the training video together with the
`
`student to find the clip where the difliculty can be identified. The student may then ask
`
`various questions of the instructor about that procedure. For example, the instructor
`
`may then decide to show the student another video, which offers more detailed
`
`20
`
`25
`
`30
`
`TTI-1021, Page 3
`
`

`

`WO 99/46702
`
`2
`
`PCT/US99/04506
`
`information. The instructor may then annotate this video using collaborative video
`
`annotation tools to explain to the student how this procedure should be performed.
`
`A need exists for systems and products to provide services such as those
`
`described above. One such product is the Sprint’s Drums system which allows two
`
`users to view video simultaneously by using the Shared Movie Player that runs on
`
`Silicon Graphics, Inc. computers. The shared video playback starts with one of the
`
`users sending the video file in SGI Movie Player format to be shared with the other
`
`user. Once the complete video has been transferred, either of the two users can
`
`initiate video playback. The playback control is also shared. Either of the two users
`
`can pause the video, jump to a random position in the video by use of a scrollbar,
`
`or playback video in reverse direction.
`
`However,
`
`the Shared Movie Player generally does not provide certain
`
`features such as graphical annotation on top of the video window.
`
`In order to add
`
`graphical annotations, the user will have to pause the video, copy and paste the
`
`current frame to an ordinary shared whiteboard application.
`
`A Tele—Pointer, which is a device for controlling the appearance and
`
`position of a cursor or pointer on computer displays from a remote location,
`
`is
`
`also typically not provided; the video window itself is not shared and the users do
`
`not have any means for sharing a pointing device in either play or pause mode.
`
`Generally an integrated audio conferencing mixing conference audio with
`
`video sound-track is not provided: a regular telephone connection is typically used
`
`for user-to-user dialogues.
`
`In such system, recording/playback of shared playback session is typically
`
`not provided nor is Multi-user conferencing: the Shared Movie Player only works
`
`for point—to-point conferencing.
`
`Yet another product, Creative Partner from eMotion[2,3], contains three video
`
`annotation tools, but annotation has to be recorded off-line there being no on-line
`
`collaboration support and during a period when the video playback is paused. These
`
`three annotation tools include one for graphical annotation, one for text annotation, and
`
`one for audio annotation. The Creative Partner video player allows the user to control
`
`video playback and to invoke one of the three annotation tools provided. Annotation
`
`can only be attached to the video in pause mode. The user selects the appropriate
`
`10
`
`15
`
`20
`
`25
`
`30
`
`TTI-1021, Page 4
`
`

`

`WO 99/46702
`
`b)
`
`PCT/U 899/04506
`
`annotation tool, points to a given image coordinate on the video frame to which the
`
`annotation is attached. The user will then be able to record the given annotation. The
`
`annotation is not related to a video segment, it being only related to the frame to which
`
`it was attached. During playback the Creative Partner video player will pause at any
`
`video flame having annotations attached. The user will have to activate playback of
`
`audio annotations and to resume video playback The annotations are removed from the
`
`video window, once video playback is resumed.
`
`Helpful background information can be found in US. Patent No. 5,600,775,
`
`issued February 4, 1997 in the names of King et al. and ENTITLED METHOD AND
`
`APPARATUS FOR ANNOTATING FULL MOTION VIDEO AND OTHER
`
`lNDEXED
`
`DATA
`
`STRUCTURES,
`
`and
`
`at
`
`Internet
`
`site
`
`http://www.emotion.com/html/creativepartner product pagehtml.
`
`W
`
`It is herein recognized that there is a continuing need for an apparatus to
`
`provide the following functional features which relate to aspects of the
`
`present
`
`invention.
`
`On-line multi-point group discussions on video content over heterogeneous
`
`networks with Tele-Pointer support;
`
`synchronized video playback, overlaid with voice comments as well as
`
`dynamic graphical annotation during group discussion;
`
`dynamic adjustment of playback speed during synchronized video playback
`
`and recording of group discussion sessions;
`
`in response to an existing recorded annotation session, conduct subsequent
`
`on-line multi-point group discussions and annotate during synchronized playback
`
`of recorded annotations; and
`
`attachment of any tool to help browsing of video content and creation of
`
`dynamic markers for static as well as dynamic objects.
`
`In collaborative dynamic video annotation applications,
`
`it
`
`is generally
`
`considered unlikely that the participants in the group discussion will either own the
`
`same type of computer equipment or be physically be present in the same building. It is
`
`also considered in general unlikely that each participant can be required to have a
`
`1O
`
`15
`
`20
`
`25
`
`30
`
`TTI-1021, Page 5
`
`

`

`WO 99/46702
`
`4
`
`PCT/US99/04506
`
`connection of equal data-rate to the Internet.
`
`It is herein recognized that a desirable solution to the above problems should
`
`provide enough flexibility to overcome problems due to the existence of heterogeneous
`
`environments For example. a desirable solution,
`
`in accordance with an aspect of the
`
`present invention, can allow people to use a Public Switched Telecommunications
`
`Network (PSTN), if quality of audio is of concem, and the Internet Phone (IP), if cost is
`
`a greater concern.
`
`Tele-Pointers are an important part of a group discussion. Without them, it is
`
`practically impossible to know what each participant
`
`is pointing at.
`
`It
`
`is herein
`
`recognized that since video is a dynamic document, it is more helpful,
`
`in accordance
`
`with an aspect of the present invention, if each participant can make their own cursor
`
`visible on the screens of other participants.
`
`Since video content is difficult to describe verbally,
`
`it is herein recognized, in
`
`accordance with an aspect of the present
`
`invention,
`
`that
`
`it
`
`is important for all
`
`participants to see the same video fi'ame at the same time. It is also required that any
`
`participant be able to annotate, such as with graphical drawings or texts, on top of a
`
`video frame independently of the state of the video player and for all participants to see
`
`the same annotations on each of their screen at the same time,
`
`In addition, since
`
`participants are usually located in respective remote locations, the ability to have full
`
`duplex multi—point voice communication is considered essential. The system should
`
`preferably also be able to mix the audio track in the video with the voice of all
`
`participants all the time.
`
`Sometimes, annotating simultaneously with voice and graphical drawings while
`
`the video is being played is not a very straightforward task.
`
`It is herein recognized that,
`
`in accordance with an aspect of the present invention, any participant should be able to
`
`dynamically adjust the shared video playback speed during the group discussion. The
`
`entire group discussion should preferably be able to be recorded and played back in the
`
`same sequence in which it happened. The recording should preferably include all VCR
`
`commands, graphical drawings/texts, and voice comments,
`
`time-stamped for later
`
`10
`
`15
`
`20
`
`25
`
`30
`
`playback.
`
`Playback of a recorded annotation can occur in a stand-alone mode or an on-
`
`line collaboration mode. In the second case, the playback of the recorded annotation
`
`TTI-1021, Page 6
`
`

`

`WO 99/46702
`
`U1
`
`PCT/US99/04506
`
`should be synchronized among all participants. In accordance with an aspect of the
`
`present invention, any participant should be able to annotate while the playback is
`
`going on and record new annotations in a separate record.
`
`In accordance with an aspect of the present invention, during a collaborative
`
`dynamic video annotation, any participant is able to use add—on tools to facilitate the
`
`discussion. One such tool is a video browser which allows a user to jump to random
`
`points in the video. While the video is being played, proper places in the tool are
`preferably; be highlighted to reflect the frame that is currently being played on the
`
`screen of each participant. Another tool that may be implemented is the tool that allows
`
`any participant to create dynamic object markers. A dynamic object marker is a
`
`graphical drawing (usually assembled from polygons) that highlights the location of an
`
`object of interest in a video frame. The dynamic marker indicates dynamic objects and
`
`dynamic parameters relating to a steady object. Since the location of a dynamic object
`
`generally changes from frame to frame,
`
`this tool is provided in accordance with the
`
`invention to help locate this object in all frames between a selected pair of video
`
`frames. This tool, when invoked, will create a marker for all frames between a selected
`
`pair of video frames. The system preferably also provides a tool that creates dynamic
`
`markers for steady objects. This is useful when the object of interest contains dynamic
`
`information parameters, for example, current flow in an electrically conductive wire.
`
`In accordance with the present invention, a computer based system or apparatus
`
`provides collaborative dynamic video annotation, recording of such a collaborative
`
`session, synchronized playback of such a recorded annotation, and annotation/recording
`
`during playback of a recorded annotation.
`
`The apparatus comprises a computer
`
`readable storage medium having a computer program stored thereon performing the
`
`steps of: (a) choosing a network service and starting or joining a conference; (b) loading
`
`a video or a recorded annotation file; (c) performing simultaneous graphical, text, and
`
`audio annotation, with the support of tell-pointers, VCR controls, video browses, and
`
`dynamic marker creation tools; and (d) recording of a collaborative session.
`
`The system in accordance with the invention is a client/server model. The client
`
`essentially comprises a shared multi-media player with synchronized multi-point VCR
`
`control. The window that displays live video frames is also a drawing board where
`
`different people can place graphical objects from different clients at the same time.
`
`1O
`
`15
`
`20
`
`25
`
`30
`
`TTI-1021, Page 7
`
`

`

`WO 99/46702
`
`6
`
`PCT/USQ9/04506
`
`In addition, the system provides multi-point full duplex voice connection and the joint
`
`voice comments are mixed with the audio track in the video currently being played.
`
`The network service in Step (a) includes any of the Internet TCP/IP, IPX,
`
`Modem, and Serial connection. The video file described in Step (b) is located locally
`
`with the client. Step (c) can be executed independently of whether the video player is in
`
`the play or pause modes and the playback among multiple clients is
`
`loosely
`
`synchronized, as explained below, while the speed of the video playback can be
`
`adjusted dynamically during synchronized playback. Step (d)
`
`records all visible
`
`activities in the collaborative session, such as lines, texts, Tele—Pointers, markers, as
`
`well as voice comments exchanged during the session.
`
`It is important for any shared applications to serialize all events that occurred
`
`during a session. In order to implement a shared but synchronized multimedia player
`
`while giving all participants an equal access to the VCR control, the player action is
`
`delayed while the VCR button is first pressed.
`
`Instead of interpreting a VCR command
`
`immediately on a client machine, a VCR command is sent to the sewer, serialized
`
`sequentially, and sent back to all client machines. Only after having received merged
`
`VCR commands fiom the server does the player then take action.
`
`In order
`
`to synchronize video playback among all participants,
`
`some
`
`cooperation is required between all video players. This can be done on a frame-by-
`
`frame basis, which is very costly and difficult to realize in reality without sacrificing the
`
`playback quality.
`
`In accordance with the present invention, it is considered preferable
`
`to synchronize on a VCR command-by-command basis. The current frame at the time
`
`the VCR control button is first pressed is recorded and each traditional VCR command
`
`is converted into one of the two types of new VCR commands. namely “JUMP-PLAY
`
`frame-number frame-rate” and “JUMP-PAUSE frame-number”. For example, suppose
`
`a PLAY button is pressed while the player is on frame 300. The message sent to the
`
`server and eventually received by all clients will be “JUMP-PLAY 300 10” instead of
`
`10
`
`15
`
`20
`
`25
`
`TTI-1021, Page 8
`
`

`

`WO 99/46702
`
`7
`
`PCT/US99/04506
`
`“PLAY”, that is, the VCR will seek to frame 300 and then start playing the video at
`
`normal speed.
`
`Another component of the present invention is the ability to mix audio signals
`
`and to overlay graphics with video frames. Some known audio/graphic hardware, such
`
`as Parallax board offers such a functionality. Even if a graphic overlay is supported by
`
`hardware, the system still needs to handle the drawing of Tale-Pointers and graphical
`
`annotations
`
`separately. This
`
`is because Tele—Pointers may occlude graphical
`
`annotations and graphical annotations need to be restored once Tele-Pointers move
`
`away from their current position.
`
`In order for the system to play a recorded annotation session synchronously on
`
`the screens of all participants, it is only necessary to record; with timestamp, all the
`
`messages that reached the server during the discussion session and send them to all
`
`clients according to the timestarnp during playback. This will also allow each
`
`participant to annotate during playback of recorded annotation.
`
`In accordance with an aspect of the invention, a method for dynamic video
`
`annotation among a plurality of users at respective locations, utilizing programmable
`
`computer apparatus with information storage and retrieval capability,
`
`the method
`
`comprises the steps of: selecting a network service coupled to the computer; performing
`
`one of (a) starting and (b) joining a collaborative session among the users; loading one
`
`of (a) a video and (b) a recorded annotation file; performing annotation of at least one
`
`of graphical, text, and audio annotation; and storing the collaborative session.
`
`In accordance with another aspect of the invention a method for dynamic video
`
`annotation among a plurality of users, utilizing programmable computer apparatus with
`
`information storage and retrieval capability,
`
`the method comprises the steps of:
`
`10
`
`15
`
`20
`
`‘25
`
`selecting a network service coupled to the computer; joining a collaborative session
`
`among the users;
`
`loading one of (a) a video and (b) a recorded annotation file;
`
`performing annotation of at least one of graphical, text, and audio annotation; and
`
`storing the collaborative session.
`
`In accordance with another aspect of the invention apparatus for dynamic video
`
`30
`
`annotation among a plurality of users; the apparatus comprises programmable computer
`
`apparatus with information storage and retrieval capability; a user interface coupled to
`
`the computer apparatus for performing selection of a network service, a user interface
`
`TTI-1021, Page 9
`
`

`

`WO 99/46702
`
`8
`
`PCT/US99/04506
`
`coupled to the computer apparatus for performing one of (a) starting and (b) joining a
`
`collaborative session among the users; a user interface coupled to the computer
`
`apparatus for loading one of (a) a video and (b) a recorded annotation file; a user
`
`interface coupled to the computer apparatus for performing annotation of at least one of
`
`graphical,
`
`text, and audio annotation; and a user interface coupled to the computer
`
`apparatus for storing the collaborative session.
`
`In accordance with another aspect of the invention apparatus for enabling a
`
`plurality of users at respective locations to participate in a collaborative session
`
`regarding content of a video; to record such a collaborative session; to annotate/record
`
`during playback of a recorded session; and to play back synchronously such a recorded
`
`annotated session, wherein the apparatus comprises a shared video player/recorder
`
`fitnction (VCR) available to each of the users, with multi-point VCR control exhibiting
`
`dynamic speed adjustment; and an ability to show dynamic markers; a function by
`
`which any of the users can play or stop the video; jump to a different location in the
`
`video; dynamically change video play speed; the shared video player/recorder function
`
`available to each of the users being synchronized at the same video frame whenever
`
`any VCR activity occurs; and apparatus for displaying a dynamic marker when a frame
`
`to which such a marker is attached is displayed.
`
`In accordance with another aspect of the invention, apparatus for enabling a
`
`plurality of users at respective locations to participate in a collaborative session
`
`regarding content of a video; to record such a collaborative session; to annotate/record
`
`during playback of a recorded session; and to play back synchronously such a recorded
`
`annotated session, wherein the apparatus comprises a shared video player/recorder
`
`function (VCR) exhibiting a window available to each of the users, with multi-point
`
`VCR control exhibiting dynamic speed adjustment, and an ability to show dynamic
`
`markers; a function by which any of the users can play or stop the video; jump to a
`
`different location in the video; dynamically change video play speed; the shared video
`
`player/recorder function available to each of the users being synchronized at the same
`
`video frame whenever any VCR activity occurs; apparatus for displaying a dynamic
`
`marker when a frame to which such a marker is attached is displayed; and the shared
`
`video player/recorder function window acting as a shared whiteboard with Tele-Pointer
`
`support for supporting free-hand drawing and text,
`
`10
`
`15
`
`20
`
`25
`
`30
`
`TTI-1021, Page 10
`
`

`

`WO 99/46702
`
`9
`
`PCT/US99/04506
`
`In accordance with another aspect of the invention a method for dynamic video
`
`annotation among a plurality of users utilizes a programmable computer and comprises
`
`the steps of: selecting a network service coupled to the computer; performing one of (a)
`
`starting and (b) joining a collaborative session among the users; loading one of (a) a
`
`video and (b) a recorded annotation file; performing annotation of at least one of
`
`graphical, text. and audio annotation; and storing the collaborative session.
`
`10
`
`15
`
`20
`
`BfiEEDEflJflEflQEQEIHED$flflEE§
`
`The invention will be better understood from the following detailed description
`
`in conjunction with the Drawing, in which
`
`Fig.1 is a diagram illustrating a collaborative dynamic annotation session over
`
`Public Switched Telecommunications Network (PSTN)
`
`in accordance with the
`
`invention;
`
`Fig. 2 is a diagram illustrating a collaborative dynamic annotation session over
`
`Internet Phone (IP) in accordance with the invention;
`
`Fig. 3 is a diagram illustrating a front-end user interface in accordance with the
`
`invention;
`
`Fig. 4 is a diagram illustrating a main client user interface in accordance with
`
`the invention;
`
`Fig. 5 is a diagram illustrating a user interface in accordance with the invention
`
`for deleting dynamic markers by name;
`
`Fig. 6 is a diagram illustrating a video browser interface in accordance with the
`
`invention;
`
`Fig. 7 is a diagram illustrating a dynamic object marker creation tool interface
`
`25
`
`in accordance with the invention;
`
`Fig. 8 is a diagram illustrating the system architecture in accordance with the
`
`invention;
`
`Fig. 9 is a diagram illustrating the system message flow in accordance with the
`
`invention; and
`
`30
`
`Fig. 10 is a diagram illustrating an example that shows how event messages are
`
`communicated in accordance with the invention.
`
`TTL1021,Page11
`
`

`

`WO 99/46702
`
`10
`
`PCT/US99/04506
`
`DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
`
`The invention features a shared “video player” with multi-point VCR control,
`
`dynamic speed adjustment, and the ability to show dynamic markers.
`
`It is emphasized
`
`however that the “video player” is contemplated in the present invention to be either (a)
`
`a simulated video player function provided by a computer with stored video in its
`
`memory and simulated video player controls or (b) an actual VCR operated in
`
`is herein intended for
`it
`conjunction with the rest of the system. Accordingly,
`simplicity and convenience to refer in the present patent application to a video player,
`
`video player/recorder, video recorder/player or VCR by which is meant either a
`
`simulated or an actual VCR or video player/recorder, or as a VCR fimction or a video
`
`recorder/player function, according as a particular embodiment may be constituted.
`
`It
`
`is also understood that
`
`the apparatus may utilize a television receiver
`
`apparatus for the video display fiinction, in association with a computer and a simulated
`
`or actual hardware to provide the functions herein disclosed.
`
`Fig.1 shows an embodiment
`
`illustrating
`
`how the invention is utilized in
`
`conjunction with a public switched telephone network. A telephone switch 100 is
`
`coupled to a computer 130, which in turn is coupled by way of a local area network
`
`(LAN) to each of a client set-up, 102 and l04. Client set-up5102 and 104 are each
`
`equipped with a microphone, 125, respectively. Telephone switch 100 is coupled to a
`
`collaborative dynamic video annotation server 120 by way of a computer modem 115.
`
`Telephone switch 100 is also coupled a modem l 15, Telephone switch 100 is coupled
`
`to telephones 110 and a monitor.
`
`Further computers monitors, speakers, and
`
`telephones similar to those shown in Fig.
`
`1 may be coupled to the system as shown,
`
`although not illustrated in Fig. 1.
`
`Telephone switch 100 is responsible for handling a telephone conference. In
`
`this setup, participants in the conference can also use an IP-PSTN (Internet phone -
`
`Public switched telephone network). bridge 130 to allow Intranet Phone users to use the
`
`public switched telephone network. The collaborative dynamic video annotation server
`
`120,
`
`in accordance with the invention. handles all system messages, such as those
`
`marked by reference numeral 1030 in Fig. 10, sent by clients over the Internet. Server
`
`120 also connects to the telephone conference switch 100 by a computer modem 115.
`
`There are two types of
`
`clients or participants. One utilizes a regular telephone
`
`10
`
`15
`
`20
`
`25
`
`30
`
`TTI-1021, Page 12
`
`

`

`WO 99/46702
`
`l I
`
`PCT/U 899/04506
`
`connection 110 and the other utilizes LAN coupled Intranet Phone connections. Both
`
`types of clients are equipped with a mouse 140, a keyboard 155, a pair of speakers 126,
`
`a monitor 160, and a main computer 108, coupled as shown in Fig. 1. For a computer
`
`equipped with an Intranet phone connection. a microphone 125 is required.
`
`An embodiment in Fig.2 shows the invention as utilized in an Internet Phone
`
`environment. In this setup, the 200, providing the functionalities herein described and
`
`coupled to the Internet, handles all event messages sent by clients over the Internet.
`This server also acts like a digital phone switch, mixing all or part of voice comments
`
`spoken by conference participants and broadcasting the mixed signals back to all
`
`clients. All clients are each equipped with a mouse 220, a keyboard 215, a pair of
`
`speakers 230, a microphone 225, a monitor 205, and a main computer 210 coupled to
`
`the Internet.
`
`Fig.3 illustrates an embodiment of a front-end interface in accordance with the
`
`invention,
`
`including optionally a telephone dialer button.
`
`In accordance with an
`
`embodiment of the invention, network service differences are hidden under a software
`
`layer through the use of Microsoft’s DirectPlay or any sofiware that implements the
`
`T.123 standard. 300 shows an interface that asks the user to select which network
`
`service to use. The front-end server side user interface is 310 which prompts the
`
`user/operator to select a service provider. The client side interface is 320, which selects
`
`a service provider and eventually the collaborative dynamic video annotation server.
`
`Fig.4 is shows an embodiment of a main client user interface in accordance
`
`with the invention. A user first selects a video file or a recorded annotation file to load
`
`using interface 400, while interface 410 gives brief summary instructions on its usage.
`
`Video flames are displayed in a window 498. A button 408 (stop), 412(play),
`
`416(pause), 420(fast forward), 424(fast rewind), 428(step forward), and 432 (step
`
`backward) provide the basic VCR control functions for playing the video. A slider 436
`
`shows the current video position and also provides a means for a user to randomly jump
`
`to any frame in the video. Text display 440 displays the current frame number over the
`
`total frame number comprised in the video. A user can dynamically adjust the play
`
`speed by moving a slider 444 or by pressing the key + , normally provided on the
`
`standard keyboard, for increasing current frame rate by 0.1 frame/sec and the key — for
`
`decreasing current frame rate by 0.1 frame/sec. The current frame rate is displayed in
`
`10
`
`15
`
`20
`
`25
`
`30
`
`TTI-1021, Page 13
`
`

`

`WO 99/46702
`
`l2
`
`PCT/US99/04506
`
`448.
`
`In order to allow a user to control the playback of a recorded annotation, while
`
`permitting them to change the course of video playback, a second set of VCR control,
`
`452(record), 456(stop), 460(play), 464(pause),
`
`is provided on the main client user
`
`interface. These two sets of VCR controls are active simultaneously. However, when
`
`any of the first set of buttons (408, 412, 416, 420, 424, 428, 432) are pressed, the
`
`playback of the recorded annotation will automatically be stopped. Similarly, if a user
`
`starts playing a recorded annotation while the first set of VCR control is active, the
`
`system will also stop the current video playback activity. An exception is the recording
`
`activity, button 452, which will not be stopped even if the user presses any of the first
`
`set of buttons.
`
`In accordance with the invention, a user can make a free-hand drawing by
`
`holding down the left mouse button while moving the mouse cursor on the video frame
`
`window 498. The lines will be shown on all clients’ video frames in the same color as
`
`selected by the system or by the user on a per-user basis. A user can also type a text
`
`string on the video frame window 498 by first right clicking the mouse button. When
`
`this happens, the system will pop up a text edit window only on that particular client’s
`
`screen. The user can then type any single or multiple line text strings and click on the
`
`OK button when ready. Only at that time, the typed string(s) will appear on all clients’
`
`video frames at the place where the right mouse click occurred. Note that during the
`typing process, the conference activities can continue. A user can also turn on their own
`
`Tele-Pointer to enable other participants to know where they are pointing the mouse.
`
`This is accomplished by pressing the button 480. In order not to confuse a user with too
`
`many graphical objects on screen, all graphical annotations, text annotations, and Tele—
`
`Pointers relating to the same user are drawn in the same color.
`
`The system also allows a user to erase the content of all graphical annotations
`
`by pressing button 472, the content of all text annotations by pressing button 468 in the
`
`10
`
`15
`
`20
`
`25
`
`TTI-1021, Page 14
`
`

`

`wo 99/46702
`
`13
`
`PCT/US99/04506
`
`described embodiment, and the record of certain dynamic markers by pr