PCT
`
`WORLD INTELLECTUAL PROPERTY ORGANIZATION
`International Bureau
`
`
`
`INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT)
`(51) International Patent Classification © :
`(11) International Publication Number:
`WO 99/46702
`GO06F 17/30, HO4N 7/15
`_
`
`(43) International Publication Date:
`
`
`
`
`
`
`
`
`16 September 1999 (16.09.99)
`
`(21) International Application Number: PCT/US99/04506|(81) Designated States: CN, JP, Europeanpatent (AT, BE, CH,CY,
`DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, NL, PT,
`
`
`SE).
`
`
`
`
`
`
`Published
`With international search report.
`Before the expiration of the time limit for amending the
`claims and to be republished in the event of the receipt of
`amendments.
`
`(22) International Filing Date:
`
`1 March 1999 (01.03.99)
`
`(30) Priority Data:
`09/039,019
`
`13 March 1998 (13.03.98)
`
`US
`
`INC.
`SIEMENS CORPORATE RESEARCH,
`(71) Applicant:
`[US/US]; 755 College Road East, Princeton, NJ 08540
`(US).
`
`(72) Inventors: LIOU, Shih-Ping; 3 Orly Court, Robbinsville, NJ
`08691 (US). HJELSVOLD, Rune; 507 Plainsboro Road,
`Plainsboro, NJ 08536 (US).
`
`(74) Agents: AHMED, Adel, A. et al.; Siemens Corporation,
`Intellectual Property Dept., 186 Wood Avenue South,Iselin,
`NJ 08830 (US).
`
`
`
`
`
`(54) Title: APPARATUS AND METHOD FOR COLLABORATIVE DYNAMIC VIDEO ANNOTATION
`
`COLLEAGUES’ MIXED VOICE COMMENTS
`
`SYSTEM MESSAGES
`
`
`
`
`rencen ven THE
`PLAY COHMANDS
`
`
`
`COLLEAGUES*
`ACTIVE POINTER
`
`
`
`
`
` UNIFIED
`
`
`
`(57) Abstract
`
`A method for dynamic video annotation among a plurality of users utilizes a programmable computer and comprises the steps of:
`selecting a network service coupled to the computer; performing one of(a) starting and (b) joining a collaborative session among the users;
`loading one of (a) a video and (b) a recorded annotation file; performing annotation ofat least one of graphical, text, and audio annotation;
`and storing said collaborative session.
`
`
`L_
`
`TikTok TTI-1021, Page 1
`IPR2021-00476 (TikTok Inc. v. 10Tales Inc.)
`
`

`

`
`
`FOR THE PURPOSES OF INFORMATION ONLY
`
`Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT.
`SI
`Slovenia
`LS
`ES
`Lesotho
`FI
`SK
`LT
`Lithuania
`Slovakia
`SN
`FR
`LU
`Senegal
`Luxembourg
`GA
`LV
`Latvia
`SZ
`Swaziland
`TD
`Chad
`MC
`Monaco
`GB
`MD
`TG
`GE
`Togo
`Republic of Moldova
`GH
`MG
`TJ
`Tajikistan
`Madagascar
`MK
`T™
`Turkmenistan
`GN
`The former Yugoslav
`TR
`GR
`Turkey
`Republic of Macedonia
`TT
`HU
`Mali
`Trinidad and Tobago
`UA
`TE
`Ukraine
`Mongolia
`UG
`IL
`Mauritania
`Uganda
`US
`United States of America
`IS
`Malawi
`IT
`UZ
`Uzbekistan
`Mexico
`VN
`Viet Nam
`JP
`Niger
`KE
`YU
`Netherlands
`Yugoslavia
`ZW
`Zimbabwe
`KG
`Norway
`KP
`New Zealand
`Poland
`Portugal
`Romania
`Russian Federation
`Sudan
`Sweden
`Singapore
`
`Spain
`Finland
`France
`Gabon
`United Kingdom
`Gcorgia
`Ghana
`Guinca
`Greece
`Hungary
`Treland
`Tsrael
`Teeland
`Ttaly
`Japan
`Kenya
`Kyrgyzstan
`Democratic People’s
`Republic of Korea
`Republic of Korea
`Kazakstan
`Saint Lucia
`Liechtenstein
`Sri Lanka
`Liberia
`
`KR
`KZ
`Le
`LI
`LK
`LR
`
`ML
`MN
`MR
`MW
`MX
`NE
`NL
`NO
`NZ
`PL
`PT
`RO
`RU
`SD
`SE
`8G
`
`AL
`AM
`AT
`AU
`AZ
`BA
`BB
`BE
`BF
`BG
`BJ
`BR
`BY
`CA
`CF
`CG
`CH
`ci
`CM
`CN
`cu
`CZ
`DE
`DK
`EE
`
`
`
`Albania
`Armenia
`Austria
`Australia
`Azerbaijan
`Bosnia and Herzegovina
`Barbados
`Belgium
`Burkina Faso
`Bulgaria
`Benin
`Brazil
`Belarus
`Canada
`Central African Republic
`Congo
`Switzerland
`Céte d'Ivoire
`Cameroon
`China
`Cuba
`Czech Republic
`Germany
`Denmark
`Estonia
`
`Ld
`
`TTI-1021, Page 2
`
`
`
`

`

`WO 99/46702
`
`PCT/US99/04506
`
`APPARATUS AND METHOD FOR
`
`COLLABORATIVE DYNAMIC VIDEO ANNOTATION
`
`FIELD OF THE INVENTION
`
`Thepresent inventionrelates to the field of collaborative video annotation, and
`
`morespecifically, to apparatus for enabling multiple users to share their views about a
`
`video content.
`
`10
`
`BACKGROUND OF THE INVENTION
`
`Reference is madeto a patent application entitled Method and Apparatusfor
`
`Creating Dynamic Object Markers in a Video Clip beingfiled on even date herewith,
`
`and assigned to the sameassignee as the present application, and whereofthe disclosure
`
`is herein incorporated by reference to the extentit is not incompatible with the present
`
`15
`
`application.
`
`A Situation can arise wherein two or more users wish to communicate in
`
`reference to a commonobject, for example, in reference to a video. An exampleofthis
`
`could be where a soccer team coach wishes to consult with a colleague to seek advice.
`
`The soccer team coach might wish to show a taped video of a game and ask the
`
`colleague to explain, using the video, why one team failed to score in a given attack
`situation.
`In addition, the coach might wish to record this discussion and showit later
`
`to other coaches to get more opinions.
`
`In another scenario, a student could be taking a training course being givenat a
`
`remote location from where a course instructor is located.
`
`It may be that the student
`
`cannot understand a procedure being taught in the course. The student can then call the
`instructor over the Internet phone to find out how such a procedure should be
`
`performed. Theinstructor can first browse throughthe training video together with the
`studentto find the clip wherethe difficulty can be identified. The student may then ask
`various questions ofthe instructor about that procedure. For example, the instructor
`may then decide to show the student another video, which offers more detailed
`
`20
`
`25
`
`30
`
`TTI-1021, Page 3
`
`

`

`WO 99/46702
`
`2
`
`PCT/US99/04506
`
`information. The instructor may then annotate this video using collaborative video
`
`annotation tools to explain to the student how this procedure should be performed.
`
`A need exists for systems and products to provide services such as those
`
`described above. One such productis the Sprint’s Drums system which allows two
`
`users to view video simultaneously by using the Shared Movie Player that runs on
`
`Silicon Graphics, Inc. computers. The shared video playback starts with one ofthe
`
`users sendingthe videofile in SGI Movie Player format to be shared with the other
`user. Once the complete video has been transferred, either of the two users can
`initiate video playback. The playback control is also shared. Either of the two users
`
`10
`can pause the video, jump to a random position in the video by use ofascrollbar,
`
`15
`
`20
`
`or playback video in reverse direction.
`
`However,
`
`the Shared Movie Player generally does not provide certain
`
`features such as graphical annotation on top of the video window.
`
`In order to add
`
`graphical annotations, the user will have to pause the video, copy and paste the
`
`current frame to an ordinary shared whiteboard application.
`
`A Tele-Pointer, which is a device for controlling the appearance and
`
`position of a cursor or pointer on computer displays from a remotelocation, is
`
`also typically not provided; the video windowitself is not shared and the users do
`
`not have any meansfor sharing a pointing device in either play or pause mode.
`
`Generally an integrated audio conferencing mixing conference audio with
`
`video sound-track is not provided: a regular telephone connectionis typically used
`
`for user-to-user dialogues.
`
`In such system, recording/playback of shared playback session 1s typically
`
`not provided nor is Multi-user conferencing: the Shared Movie Player only works
`
`25
`
`for point-to-point conferencing.
`
`Yet another product, Creative Partner from eMotion([2,3], contains three video
`
`annotation tools, but annotation has to be recorded off-line,
`
`there being no on-line
`
`collaboration support and during a period when the video playback is paused. These
`
`three annotation tools include one for graphical annotation, one for text annotation, and
`
`30
`
`one for audio annotation. The Creative Partner video player allows the user to control
`
`video playback and to invoke one of the three annotation tools provided. Annotation
`can only be attached to the video in pause mode. The user selects the appropriate
`
`TTI-1021, Page 4
`
`

`

`WO 99/46702
`
`us
`
`PCT/US99/04306
`
`annotation tool, points to a given image coordinate on the video frame to which the
`
`annotation is attached. The user will then be able to record the given annotation. The
`
`annotation is not related to a video segment, it being only related to the frame to which
`
`it was attached. During playback, the Creative Partner video player will pause at any
`
`video frame having annotations attached. The user will have to activate playback of
`
`audio annotations and to resume video playback. The annotations are removed from the
`
`video window, once video playback is resumed.
`
`Helpful background information can be found in U.S. Patent No. 5,600,775,
`
`issued February 4, 1997 in the names of King et al. and ENTITLED METHOD AND
`
`10
`
`APPARATUS FOR ANNOTATING FULL MOTION VIDEO AND OTHER
`
`INDEXED
`
`DATA
`
`STRUCTURES,
`
`and
`
`at
`
`Internet
`
`site
`
`http://www.emotion.com/html/creativepartner product page.html.
`
`
`
`SUMMARYOFTHEINVENTION
`
`15
`
`It is herein recognized that there is a continuing need for an apparatus to
`
`provide the following functional features which relate to aspects of the
`
`present
`
`invention.
`
`On-line multi-point group discussions on video content over heterogeneous
`
`networks with Tele-Pointer support;
`
`20
`
`synchronized video playback, overlaid with voice comments as well as
`
`dynamic graphical annotation during group discussion:
`
`dynamic adjustment of playback speed during synchronized video playback
`
`25
`
`30
`
`and recording of group discussion sessions;
`
`in response to an existing recorded annotation session, conduct subsequent
`on-line multi-point group discussions and annotate during synchronized playback
`
`of recorded annotations; and
`
`attachment of any tool to help browsing of video content and creation of
`
`dynamic markersforstatic as well as dynamic objects.
`In collaborative dynamic video annotation applications,
`it
`is generally
`considered unlikely that the participants in the group discussion will either own the
`
`same type of computer equipmentorbe physically be present in the samebuilding.Itis
`also considered in general unlikely that each participant can be required to have a
`
`TTI-1021, Page 5
`
`

`

`WO 99/46702
`
`4
`
`PCT/US99/04506
`
`connection of equal data-rate to the Internet.
`
`It is herein recognized that a desirable solution to the above problems should
`
`provide enough flexibility to overcome problemsdue to the existence of heterogeneous
`
`environments. For example, a desirable solution,
`
`in accordance with an aspect of the
`
`present invention, can allow people to use a Public Switched Telecommunications
`
`Network (PSTN),if quality of audio is of concern, and the Internet Phone(IP),if cost is
`
`a greater concern.
`
`Tele-Pointers are an important part of a group discussion. Without them, it is
`
`practically impossible to know what each participant
`
`is pointing at.
`
`It
`
`is herein
`
`10
`
`recognized that since video is a dynamic document, it is more helpful,
`
`in accordance
`
`with an aspect of the present invention, if each participant can make their own cursor
`
`visible on the screensof other participants.
`
`Since video contentis difficult to describe verbally,
`
`it is herein recognized, in
`
`accordance with an aspect of the present
`invention,
`that
`it
`is important for all
`participants to see the same video frame at the same time.It is also required that any
`participant be able to annotate, such as with graphical drawingsor texts, on top of a
`
`video frame independently ofthe state ofthe video player andforall participants to see
`
`the same annotations on each of their screen at the same time.
`
`In addition, since
`
`participants are usually located in respective remote locations, the ability to have full
`duplex multi-point voice communication is considered essential. The system should
`preferably also be able to mix the audio track in the video with the voice ofall
`
`participantsall the time.
`
`Sometimes, annotating simultaneously with voice and graphical drawings while
`
`the video is being played is not a very straightforward task.
`It is herein recognized that,
`in accordancewith an aspect of the present invention, any participant should beableto
`dynamically adjust the shared video playback speed during the group discussion. The
`entire group discussion should preferably be able to be recorded and played back in the
`same sequence in which it happened. The recording should preferably include all VCR
`commands, graphical drawings/texts, and voice comments,
`time-stamped for later
`
`playback.
`Playback of a recorded annotation can occur in a stand-alone mode or an on-
`
`line collaboration mode. In the second case, the playback of the recorded annotation
`
`15
`
`20
`
`25
`
`30
`
`TTI-1021, Page 6
`
`

`

`WO 99/46702
`
`Nn
`
`PCT/US99/04506
`
`should be synchronized amongall participants. In accordance with an aspect of the
`
`present invention, any participant should be able to annotate while the playback is
`going on and record new annotationsin a separate record.
`In accordance with an aspect of the present invention, during a collaborative
`
`dynamic video annotation, any participant is able to use add-on tools to facilitate the
`
`discussion. One such tool is a video browser which allows a user to jump to random
`
`points in the video. While the video is being played, proper places in the tool are
`preferably be highlighted to reflect the frame that is currently being played on the
`screen of eachparticipant. Another tool that may be implementedis the tool that allows
`
`any participant to create dynamic object markers. A dynamic object marker is a
`graphical drawing (usually assembled from polygons) that highlights the location of an
`object ofinterest in a video frame. The dynamic markerindicates dynamic objects and
`dynamic parametersrelating to a steady object. Since the location of a dynamic object
`
`generally changes from frameto frame,
`
`this tool is provided tn accordance with the
`
`10
`
`15
`
`invention to help locate this object in all frames between a selected pair of video
`
`frames. This tool, when invoked, will create a marker for all frames between a selected
`
`pair of video frames. The system preferably also provides a tool that creates dynamic
`markers for steady objects. This is useful when the object ofinterest contains dynamic
`information parameters, for example, current flow in an electrically conductive wire.
`In accordance with the presentinvention, a computer based system or apparatus
`provides collaborative dynamic video annotation, recording of such a collaborative
`session, synchronized playback of such a recorded annotation, and annotation/recording
`during playback of a recorded annotation.
`The apparatus comprises a computer
`readable storage medium having a computer program stored thereon performing the
`steps of: (a) choosing a networkservice and starting or joining a conference, (b) loading
`a video or a recorded annotation file; (c) performing simultaneous graphical, text, and
`audio annotation, with the support oftell-pointers, VCR controls, video browses, and
`dynamic marker creation tools; and (d) recording ofa collaborative session.
`The system in accordance with the inventionis a client/server model. The client
`essentially comprises a shared multi-media player with synchronized multi-point VCR
`control. The window that displays live video frames is also a drawing board where
`different people can place graphical objects from different clients at the same time.
`
`20
`
`25
`
`30
`
`TTI-1021, Page 7
`
`

`

`WO 99/46702
`
`6
`
`PCT/US99/04506
`
`In addition, the system provides multi-point full duplex voice connection and the joint
`
`voice comments are mixed with the audio track in the video currently being played.
`
`The network service in Step (a) includes any of the Internet TCP/IP, IPX,
`
`Modem, and Serial connection. The video file described in Step (b) is located locally
`
`with the client. Step (c) can be executed independently of whether the video playeris in
`
`the play or pause modes and the playback among multiple clients is
`
`loosely
`
`synchronized, as explained below, while the speed of the video playback can be
`
`adjusted dynamically during synchronized playback. Step (d)
`
`records all visible
`
`activities in the collaborative session, such as lines, texts, Tele-Pointers, markers, as
`
`well as voice comments exchanged during the session.
`
`It is important for any shared applications to senalize all events that occurred
`
`during a session. In order to implement a shared but synchronized multi-media player
`
`while giving all participants an equal access to the VCR control, the player action is
`
`delayed while the VCR buttonis first pressed.
`
`Instead of interpreting a VCR command
`
`immediately on a client machine, a VCR command is sent to the server, serialized
`
`sequentially, and sent back to all client machines. Only after having received merged
`
`VCR commandsfrom the server does the player then take action.
`
`In order
`
`to synchronize video playback among all participants,
`
`some
`
`cooperation is required between all video players. This can be done on a frame-by-
`
`20
`
`frame basis, whichis very costly and difficult to realize in reality without sacrificing the
`
`playback quality.
`
`In accordance with the present invention, it is considered preferable
`
`to synchronize on a VCR command-by-commandbasis. The current frameat the time
`
`the VCR control button is first pressed is recorded and each traditional VCR command
`
`is converted into one of the two types of new VCR commands, namely “JUMP-PLAY
`
`25
`
`frame-numberframe-rate” and “JUMP-PAUSEframe-number”. For example, suppose
`
`a PLAYbutton is pressed while the player is on frame 300. The message sent to the
`
`server and eventually received byall clients will be “JUMP-PLAY 300 1.0” instead of
`
`TTI-1021, Page 8
`
`

`

`WO 99/46702
`
`7
`
`PCT/US99/04506
`
`“PLAY”, that is, the VCR will seek to frame 300 and then start playing the video at
`
`normalspeed.
`Another componentof the present invention is the ability to mix audio signals
`and to overlay graphics with video frames. Some known audio/graphic hardware, such
`as Parallax board offers such a functionality. Even if a graphic overlay is supported by
`
`hardware, the system still needs to handle the drawing of Tele-Pointers and graphical
`
`is because Tele-Pointers may occlude graphical
`separately. This
`annotations
`annotations and graphical annotations need to be restored once Tele-Pointers move
`
`away from their currentposition.
`In order for the system to play a recorded annotation session synchronously on
`
`the screens ofall participants, it is only necessary to record, with timestamp,all the
`messages that reached the server during the discussion session and send them to all
`clients according to the timestamp during playback. This will also allow each
`participant to annotate during playback of recorded annotation.
`
`In accordance with an aspect of the invention, a method for dynamic video
`
`annotation amonga plurality of users at respective locations, utilizing programmable
`
`computer apparatus with information storage and retrieval capability,
`the method
`comprises the stepsof: selecting a network service coupled to the computer, performing
`oneof(a) starting and (b) joining a collaborative session amongthe users; loading one
`of (a) a video and (b) a recorded annotation file; performing annotation ofat least one
`
`of graphical, text, and audio annotation; and storing the collaborative session.
`In accordance with another aspectof the invention a method for dynamic video
`
`annotation among a plurality ofusers, utilizing programmable computer apparatus with
`information storage and retrieval capability,
`the method comprises the steps of:
`selecting a network service coupled to the computer; joining a collaborative session
`among the users;
`loading one of (a) a video and (b) a recorded annotation file,
`performing annotation of at least one of graphical, text, and audio annotation, and
`
`storing the collaborative session.
`In accordance with another aspectof the invention apparatus for dynamic video
`annotation amonga plurality of users; the apparatus comprises programmable computer
`apparatus with information storage andretrieval capability; a user interface coupled to
`the computer apparatus for performing selection of a network service, a user interface
`
`10
`
`15
`
`20
`
`25
`
`30
`
`TTI-1021, Page 9
`
`

`

`WO 99/46702
`
`8
`
`PCT/US99/04506
`
`coupled to the computer apparatus for performing oneof(a) starting and (b) joining a
`
`collaborative session among the users, a user interface coupled to the computer
`
`apparatus for loading one of (a) a video and (b) a recorded annotation file; a user
`interface coupled to the computer apparatus for performing annotation ofat least one of
`graphical,
`text, and audio annotation; and a user interface coupled to the computer
`apparatusfor storing the collaborative session.
`In accordance with another aspect of the invention apparatus for enabling a
`
`plurality of users at respective locations to participate in a collaborative session
`regarding content of a video; to record such a collaborative session, to annotate/record
`during playback of a recorded session; and to play back synchronously such a recorded
`annotated session, wherein the apparatus comprises a shared video player/recorder
`
`function (VCR)available to each of the users, with multi-point VCR control exhibiting
`
`dynamic speed adjustment, and an ability to show dynamic markers, a function by
`which any ofthe users can play or stop the video; jump to a different location in the
`video; dynamically change video play speed; the shared video player/recorder function
`available to each of the users being synchronized at the same video frame whenever
`
`any VCRactivity occurs; and apparatus for displaying a dynamic marker when a frame
`
`to which such a markeris attachedis displayed.
`
`In accordance with another aspect of the invention, apparatus for enabling a
`
`plurality of users at respective locations to participate in a collaborative session
`regarding content of a video; to record such a collaborative session, to annotate/record
`during playback of a recorded session; andto play back synchronously such a recorded
`annotated session, wherein the apparatus comprises a shared video player/recorder
`function (VCR) exhibiting a window available to each of the users, with multi-point
`VCR contro! exhibiting dynamic speed adjustment, and an ability to show dynamic
`markers; a function by which any ofthe users can play or stop the video; jump to a
`different location in the video; dynamically change video play speed; the shared video
`
`player/recorder function available to each of the users being synchronized at the same
`video frame whenever any VCR activity occurs, apparatus for displaying a dynamic
`
`20
`
`25
`
`30
`
`marker when a frame to which such a marker is attached is displayed; and the shared
`
`video player/recorder function window acting as a shared whiteboard with Tele-Pointer
`support for supporting free-hand drawing andtext.
`
`TTI-1021, Page 10
`
`

`

`WO 99/46702
`
`9
`
`PCT/US99/04506
`
`In accordance with another aspect of the invention a method for dynamic video
`
`annotation amonga plurality of users utilizes a programmable computer and comprises
`
`the stepsof: selecting a network service coupled to the computer, performing one of(a)
`
`starting and (b) joining a collaborative session among the users; loading one of(a) a
`
`video and (b) a recorded annotation file; performing annotation of at least one of
`graphical, text, and audio annotation; andstoring the collaborative session.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The invention will be better understood from the following detailed description
`
`10
`
`in conjunction with the Drawing, in which
`
`Fig.1 is a diagram illustrating a collaborative dynamic annotation session over
`
`Public Switched Telecommunications Network (PSTN)
`
`in accordance with the
`
`invention;
`
`15
`
`20
`
`Fig. 2 is a diagram illustrating a collaborative dynamic annotation session over
`
`Internet Phone (IP) in accordance with the invention;
`Fig. 3 is a diagram illustrating a front-enduser interface in accordance with the
`
`invention;
`
`Fig. 4 is a diagram illustrating a main client user interface in accordance with
`
`the invention;
`
`Fig. 5 is a diagram illustrating a user interface in accordance with the invention
`
`for deleting dynamic markers by name;
`
`Fig. 6 is a diagram illustrating a video browser interface in accordance with the
`
`invention;
`
`Fig. 7 is a diagram illustrating a dynamic object marker creation tool interface
`
`25
`
`in accordancewith the invention;
`
`Fig. 8 is a diagram illustrating the system architecture in accordance with the
`
`invention;
`
`Fig. 9 is a diagram illustrating the system message flow in accordance with the
`
`invention; and
`
`30
`
`Fig. 10 is a diagram illustrating an example that shows how event messages are
`
`communicated in accordance with the invention.
`
`TTI-1021, Page 11
`
`

`

`WO 99/46702
`
`10
`
`PCT/US99/04506
`
`DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
`
`The invention features a shared “video player’ with multi-point VCR. control,
`
`It is emphasized
`dynamic speed adjustment, andthe ability to show dynamic markers.
`howeverthat the “video player” is contemplated in the present inventionto be either (a)
`a simulated video player function provided by a computer with stored video in its
`
`memory and simulated video player controls or (b) an actual VCR operated in
`conjunction with the rest of the system. Accordingly,
`it
`is herein intended for
`simplicity and convenienceto refer in the present patent application to a videoplayer,
`video player/recorder, video recorder/player or VCR, by which is meant either a
`
`10
`
`simulated or an actual VCR or video player/recorder, or as a VCR function or a video
`
`recorder/player function, according as a particular embodiment maybeconstituted.
`
`It
`
`is also understood that
`
`the apparatus may utilize a television receiver
`
`apparatusfor the video display function, in association with a computerand a simulated
`
`or actual hardwareto provide the functionsherein disclosed.
`Fig.l shows an embodimentillustrating how the invention 1s utilized in
`conjunction with a public switched telephone network. A telephone switch 100 is
`coupled to a computer 130, which in turn is coupled by way of a local area network
`
`(LAN) to each of a client set-up, 102 and 104. Client set-ups102 and 104 are each
`equipped with a microphone, 125, respectively. Telephone switch 100 is coupled to a
`collaborative dynamic video annotation server 120 by way of a computer modem 115.
`Telephone switch 100 is also coupled a modem !15. Telephone switch 100 is coupled
`
`to telephones 110 and a monitor.
`Further computers, monitors, speakers, and
`telephones similar to those shown in Fig.
`1 may be coupled to the system as shown,
`
`although notillustrated in Fig. 1.
`Telephone switch 100 is responsible for handling a telephone conference. In
`this setup, participants in the conference can also use an IP-PSTN (Internet phone -
`Public switched telephone network). bridge 130to allow Intranet Phoneusers to use the
`public switched telephone network. The collaborative dynamic video annotation server
`120,
`in accordance with the invention, handles all system messages, such as those
`
`marked by reference numeral 1030 in Fig. 10, sent by clients over the Internet. Server
`120 also connects to the telephone conference switch 100 by a computer modem 115.
`There are two types of
`clients or participants. One utilizes a regular telephone
`
`15
`
`20
`
`25
`
`30
`
`TTI-1021, Page 12
`
`

`

`WO 99/46702
`
`Il
`
`PCT/US99/04506
`
`connection |10 and the other utilizes LAN coupied Intranet Phone connections. Both
`
`types ofclients are equipped with a mouse 140, a keyboard 155, a pair of speakers 126,
`a monitor 160, and a main computer 108, coupled as shown in Fig.
`|. For a computer
`
`equipped with an Intranet phone connection, a microphone125 is required.
`
`An embodiment in Fig.2 shows the invention as utilized in an Internet Phone
`
`environment. In this setup, the 200, providing the functionalities heredescribed and
`
`coupled to the Internet, handles all event messages sent by clients over the Internet.
`Thisserver aiso acts like a digital phone switch, mixingall or part of voice comments
`spoken by conference participants and broadcasting the mixed signals back to all
`
`clients. All clients are each equipped with a mouse 220, a keyboard 215, a pair of
`
`speakers 230, a microphone 225, a monitor 205, and a main computer 210 coupled to
`
`the Internet.
`
`Fig.3 illustrates an embodimentofa front-end interface in accordance with the
`
`invention,
`
`including optionally a telephone dialer button.
`
`In accordance with an
`
`embodimentof the invention, network service differences are hidden under a software
`
`layer through the use of Microsoft’s DirectPlay or any software that implements the
`
`T.123 standard. 300 shows an interface that asks the user to select which network
`
`service to use. The front-end server side user interface is 310 which prompts the
`
`user/operatorto select a service provider. The client side interface is 320, whichselects
`
`10
`
`15
`
`20
`
`a service provider and eventually the collaborative dynamic video annotation server.
`
`Fig.4 is shows an embodiment of a main client user interface in accordance
`
`with the invention. A userfirst selects a video file or a recorded annotationfile to load
`
`using interface 400, while interface 410 gives brief summary instructions onits usage.
`
`Video frames are displayed in a window 498. A button 408 (stop), 412(play),
`
`25
`
`416(pause), 420(fast forward), 424(fast rewind), 428(step forward), and 432 (step
`backward) provide the basic VCR control functions for playing the video. A slider 436
`showsthe current video position and also provides a meansfor a user to randomly jump
`
`to any framein the video. Text display 440 displays the current frame number overthe
`
`total frame number comprised in the video. A user can dynamically adjust the play
`
`30
`
`speed by moving a slider 444 or by pressing the key + , normally provided on the
`
`standard keyboard, for increasing current frame rate by 0.1 frame/sec and the key — for
`decreasing current frame rate by 0.1 frame/sec. The current frame rate is displayed in
`
`TTI-1021, Page 13
`
`

`

`WO 99/46702
`
`12
`
`PCT/US99/04506
`
`448.
`
`In orderto allow a user to control the playback of a recorded annotation, while
`
`permitting them to change the course of video playback, a second set of VCR control,
`452(record), 456(stop), 460(play), 464(pause),
`is provided on the main client user
`
`interface. These two sets of VCR controls are active simultaneously. However, when
`
`any of the first set of buttons (408, 412, 416, 420, 424, 428, 432) are pressed, the
`playbackof the recorded annotation will automatically be stopped. Similarly, if a user
`starts playing a recorded annotation while the first set of VCR control is active, the
`system will also stop the current video playback activity. An exception is the recording
`activity, button 452, which will not be stopped even if the user presses any ofthefirst
`
`set of buttons.
`
`In accordance with the invention, a user can make a free-hand drawing by
`
`holding downthe left mouse button while moving the mouse cursor on the video frame
`
`window 498. Thelines will be shown onall clients’ video frames in the same color as
`
`selected by the system or by the user on a per-user basis. A user can also type a text
`
`string on the video frame window 498byfirst right clicking the mouse button.. When
`this happens, the system will pop up a text edit window only onthatparticular client’s
`screen. The user can then type anysingle or multiple line text strings and click on the
`
`OK button when ready. Only at that time, the typed string(s) will appear onall clients’
`video frames at the place where the right mouse click occurred. Note that during the
`typing process, the conferenceactivities can continue. A user can also tum ontheir own
`Tele-Pointer to enable other participants to know where they are pointing the mouse.
`
`This is accomplished by pressing the button 480. In order not to confuse a user with too
`many graphical objects on screen, all graphical annotations, text annotations, and Tele-
`
`Pointers relating to the same user are drawnin the same color.
`The system also allows a user to erase the content of ali graphical annotations
`by pressing button 472, the contentofall text annotations by pressing button 468 in the
`
`20
`
`25
`
`TTI-1021, Page 14
`
`

`

`WO 99/46702
`
`13
`
`PCT/US99/04506
`
`described embodiment, and the record of certain dynamic markers by pressing button
`
`476.
`
`Three attached tools are shown in the mainclient user interface in Fig. 4. They
`
`are a video browser 486, a dynamic marker creation tool for dynamic objects 490, a
`
`dynamic markercreation tool for steady (or static) objects 494. Tostart the first two
`
`tools, just press the corresponding buttons. To start the third tool, a user has to first
`
`pause the video, then click on button 494, and finally draw a free-hand curve on the
`video frame window 498.
`
`Fig. 5 illustrates an embodiment of a user inte