ttachmnent id: Copy of Document 1 from the Springer boo
`
`Multimodal Maps: An Agent-Based Approach
`
`121
`
`Bolt, R. (1980) Put that there: Voice and Gesture at the Graphic Interface, Computer
`Graphics, 14(3), pp. 262-270.
`Cohen, M., Murveit, H., Bernstein, J., Price, P., and Weintraub, M. (1990) The DE-
`CIPHER Speech Recognition System. In 1990 IEEE ICASSP, pp. 77-80.
`Cohen, P. (1992) The role of natural language in a multiniodal interface. In Proceedings
`of UIST'92, pp. 143-149.
`Cohen, P.R., Cheyer, A _, Wang, M. and Baeg, S.C. (1994) An Open Agent Architecture.
`In Proceedings AAAI'94 - SA, Stanford, pp. 1-8.
`Dauphin DTR-l User's Manual, Dauphin Technology, Inc., Lombard, Ill 60148.
`Faure, C. and Julia, L. (1994) An Agent-Based Architecture for a Multimodal Interface.
`In Proceedings AAA!'94 - IM4S, Stanford, pp. 82-86.
`Genesercth, NI, and Singh, N.P. (1994) A knowledge sharing approach to software inter-
`operation, unpublished manuscript, Computer Science Department, Stanford Uni-
`versity.
`Telescript Product Documentation (1995), General Magic Inc.
`Koons, D.B., Sparrell, C.J., and Thorisson. K.R. (1993) Integrating Simultaneous In-
`put from Speech, Caze and Hand Gestures. In Intelligent Multimedia Interfaces,
`Maybury, M.T. (ed.), Menlo Park: AAAI Press/MIT Press.
`Maybury, M.T. (ed.)
`(1993) Intelligent Mulimedia Interfaces, Menlo Park: AAAI
`Press/MIT Press,
`Neal, J.G., and Shapiro, S.C. (1991) Intelligent Multi-media Interface Technology.
`In Intelligent User Interfaces, Sullivan, J.W. and Tyler, S.W. (eds.), Reading:
`Addison-Wesley Pub. Co., pp. 11-43.
`Nigay, L. and Coutaz,
`. (1993) A Design Space for Multimodal Systems: Concurrent
`Processing and Data Pusion. In Proceedings InterCHI'93, Amsterdam, ACM Press,
`pp. 172-17.
`Object Management Group (1991) The Common Object Request Broker: Architecture
`and Specification, OMG Document Number 91.12.1.
`Oviatt, S. (1994) Toward Empirically-Based Design of Multimodal Dialogue Systems.
`In Proceedings of AAAt'94 - IM4S, Stanford, pp. 30-36.
`Oviatt, S. and Olsen, E. (1994) Integration Themes in Multimodal Human-Computer
`Interaction. In Proceedings of ICSLP'94, Yokohama, pp. 551-554.
`Park, S.K, Choi J.M., Myeong-Wuk J., Lee G.L., and Lim Y.H. (submitted for publi-
`cation), MASCOS : A Multi-Agent System as the Computer Secretary.
`Rhyne J. (1987) Dialogue Management for Gestural Interfaces, Computer Graphics,
`21(2), pp. 137-142.
`Schwartz, D.G. (1993) Cooperating heterogeneous systems: A blackboard-based meta
`approach, Technical Report 93-112, Center for Automation and Intelligent Systems
`Research, Case Western Reserve University, Cleveland Ohio, (unpublished PhD.
`thesis).
`Sullivan, J. and Tyler, S. (eds.) (1991) Intelligent User Interfaces, Reading: Addison-
`Wesley Pub. Co,
`Warren, D. and Pereira, F. (1982) An Efficient Easily Adaptable System for Interpret-
`ing Natural Language Queries, American Journal of Computational Linguistics,
`8(3), pp. 110-123.
`
`DISH, Exh. 1030, p. 230
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 2748
`
`

`

`kttachment le: University of Illinois at Urbana-Champaign Library catalog record for
`the SDrinaer book
`
`UNIVERSITY LIBRARY
`
`UNIRSITYO r ILLINOIS AT A RANA KAMPAIGN
`
`Library Catalog
`
`ASK A LIBRARIAN
`;-ANI
`FOR IrC .P55
`
`H W
`
`hat happnerd to the Library Cata log?
`
`Tell u what you think of the Library Catalog
`
`--
`
`-Local
`
`-ala-og
`
`e
`
`oro-
`
`Only
`
`i
`
`Advanced Search I Classic Search I Course Reserves I E-Reserves
`
`I Search History
`
`Your Account I Log Out
`Feedback
`Logedl in as Helen Sullivan
`(Not Helen?)
`
`. Back to Search Results
`
`Cite this
`
`0I Emal this
`
`i Add to favo
`
`o
`te
`
`Staff view L1miRod&i-P
`
`Gougle
`
`Multimodal human-computer communication : systems,
`techniques, and experiments /
`Harry Bunt, Robberlan Ben, Tijn Borhguis (eds.).
`Bunt, Harry C. I Boon, Robberc-Jan, I Borghuis, tijn.
`Names:
`pehlshed: Berlin ; Springer, c1998.
`Lecture notes in computer science 1374.
`Series:
`Lecture notes in computer science Lecture notes in artificial
`intelligence.
`Human-computer interaction. I Interactive multimedia.
`No Tags, Be the first to tag this record!
`
`Topics:
`Tags:
`
`,j Add
`
`More
`Detai Is
`
`Location &
`Availability
`
`Table of
`Contents
`
`User
`Reviews
`
`Published
`Revi ews
`
`Request
`Item
`
`Copy:
`Notes:
`
`University of Illinois at Urbana Champaign
`Location:
`Oak Street Facility Frequest only]
`Call Number:
`001.64 L497
`Text me this call number
`2
`Copy 2 has v.1 1973) to date
`Copy 3 has v.3 (1973) to date
`Copy 1,4,5 withdrawn per Engineering library
`Analyzed
`Library Has (Volumes): v.1374 (199B) c 2
`Status:
`v.1374(1998) c.2 - Callslip Request
`
`L
`
`-l
`
`-ird
`
`Lc
`
`o O nl
`
`Advanced Search I Classic Search I Course Reserves I F Reserves I Search History
`
`DISH, Exh. 1030, p. 231
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 2749
`
`

`

`Attachment if: Statewide Illinois Library Catalog record for the CMC/95 proceedings
`
`i n
`
`taewd
`Ini
`WorldCat Detailed Record
`Click on a checkbox to marka record to be e-ma led or printed in Marked Records.
`
`Lirr
`
`aao
`
`UNIV OF ILLINOIS
`IAsk A Liera nar
`n
`
`H001e
`
`Dataliare
`
`Sealrdlna
`
`S toff View I My Accoue IOptionsI Comments Em-it I±....oa
`
`1b
`
`Proceedings of the International Conference on Cooperative Multimodal Communication i
`CMC 95 ; Eindhoven, May 24-26,1995.51 f
`
`p-
`
`Ne-
`Add/vew
`
`M
`
`HPtry C Hurt
`
`1995
`
`Englis h9 BookVl, 159S dl, graphi Carat
`Tilburg: Kaltrolilke Univ. Brabant u a.,
`
`G. TIos ITov
`Availability: Check tIe caaloogs in your library.
`I b ranes world wde hat ro m ten
`* 9 Search the cooalio atthe Lbrary of .iniversrtc of Ilinois at Urbana-lhampa on
`External Resources: *
`[iscover ULUC IFull Text
`i nterlibrary Loan Reoet
`* CteThis Item
`FIND RaEtDriE
`More LikeThis: Search for versions with same title and author Advanced ontions
`rile: Proceedings of tire International Conference on Cooperative Multimodal Communication:
`CMC 95 ; Eindloven, May 24-26, 1995. 1 1
`Author(s): Bun harry C
`Corp Author(s): Intemaional Conference an Cooperative Mallimodal Cmmunicatron , 1995, Eindhoverl
`Publication: Tilburc : Katloieke Unv_ Brabant [u a.],
`Year: 195
`Descrption: VI, 159S. ill, graph Darst
`Language: English
`Series: Samenwerngs Orgaan Brabantse Ur versileter,
`Responsibility: Harry Bunt - edo }
`Material Type: Cornerence pubhlicaon (cnp)
`Document Type: Book
`Entry: 19951)t04
`Update: 20120&05
`Aeion No: (OCLC: 63366326
`Database: WarldCat
`
`o9 e L3 a a a
`
`hthe-L:hr.re
`
`(Eneil
`Nib
`
`rent (cantO
`E
`
`O
`
`5 T 9922017 0C
`T}rCLC
`Tax I,
`
`ru5 far: r : prOceeings and ((kw: intnationail and o :
`dat r
`1
`el 15.of15
`
`ucand tw- opnebe and kw. muftndal and tw communication)) and it
`
`*bks
`
`.Reard
`
`DISH, Exh. 1030, p. 232
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 2750
`
`

`

`Attachment lq: Statewide Illinois Library Cataloq record for the Sprinqer book
`
`Worldcat Detailed Record
`Click on a cheskbo to m rk a record to be e maied or printed in larked Records.
`f
`
`Diatabaseso
`
`Seaicnun
`
`fiun
`
`l
`
`Stf Vi'ewo fly Accwii CIcOtionsI Comments IExit a Kdaso,
`
`UNIV OF ILLINOIS
`I Ask ALibrarian
`
`[.
`
`Detiled Record[ Mrked Record
`Lst of Recrds
`S " L3 W S 4W
`a
`[-oel dR
`e alp
`5bpen i
`t
`Fee
`ah
`
`15-:ed Records Go to 0ae
`
`Worldeat results fo. c. -International Conference o Cooperatie MultinidNl Cormmumcation 1, 1995, Endimee. Record 1 of 3.
`
`fl I
`Pr_
`f
`*'ii
`
`fl2 Mari.o
`
`N_~
`AdVine
`Comrments
`
`l
`
`Multimodal human computer communication:
`systems, techniques, and experiments; [selected papers from the First International Conference on Cooperative Mutlimodal Communication (CMCI95), held in Eindhoven,
`the Netherlands, in May 199] 1
`
`Harry C Bunt
`
`199a
`B (o Internet Resource ViII, 343 S. graloh. Etal
`English
`[
`- Spriger ; ISBN 35406430X 9783540643807
`Berlin, Heidelberg n a
`
`Gis Toss ][o
`
`f llrois at Urbar-ChaoamDi
`
`More
`
`Access: hit/svplusbsz- debsz Chh561h ltcovhtm
`
`Availability: Check the catalogs in your library.
`L ibraries worldwide that ow
`item 10
`* S Search fir cataloi a tire [ibrary of Universi
`External Resources:.
`kscover LILJIC Full Text
`I Irterhrr Loan hecoest
`* Cit This itel
`FrNo RJLaTED
`imke This: Search for versions with sare tile aod authnor I Adanced options
`Title: Multimodal human computer communication :
`systems, techniques, and experiments; [selected papers from the First International Coniference on Cooperative Multimodal Communication (CMCIS9), held in Eindhoven, the
`Netherlands, in May 1995] 1
`Authorrs): B ot Harm C (Editor)
`Corp Authorcs): International
`Conference on Cooperatve MLlitmodal Coommuication 1 1995. Eindhoven,
`Publication: Berlin Heidelberg ua ]:Spriger,
`Year: 1908
`Description: Vii, 3433 grai Drist
`In: Bunt, Harry MUitimoal Human-Computer Communiation
`Language: English
`
`Standard No: ISBN: 354014dtX 9783540543-07; National Library: 953101 t0
`Access: File transler mode: textt rilrthhfolswbolus cEsz-bw debzo6561348cvhtm Matenials specified: Cover
`General Info: National biblography no: f8 N1t,0445
`Class Descriptors: Dewey: 00410 1t9, 004 019
`Responsibility: Harry Bunt .. (eds
`Material Type: Conference publicahorn (crp) Internet resource (u)
`Dcumet Type: Book I terel Resou rce
`Date of Entry: 199B0429
`Update: 20160211
`Arcession No: OCLC: 31283800
`Database: WorldCOat
`
`Series: Lecture notes in computer science, 1374 Lecture notes in arral inteligence; Variason: Lecture notes in computerscience; 1374 Lecture notes in artificial ntellerce
`
`C 5
`&b,-c heme
`
`i
`-oi S
`ub
`
`Sr
`LoWs eoip
`
`Tl T.- & C-ed -
`
`Wnlilt
`
`results for: s= -ntenational Conferenae o Cooperainve Mldtinndal Conmmunialo
`
`.1,1995,
`
`ind oeri Record I of 3.
`
`Endse
`
`I E-i I F
`
`IOptionIt
`
`i
`
`Comments
`
`DISH, Exh. 1030, p. 233
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 2751
`
`

`

`Attachment 1a: Online copy-of CMC/95 from a Technishe Universiteit Endhcven We-
`site
`
`or to spatial disposition in the sector), the radar, the telephone, the displays. The strips,
`the radar display, and listening to radio and telephone communications, appear to represent
`important communicative resources.
`
`5 Methodology
`
`We have analysed how air traffic controllers solve conflicts. A simulated situation of air traffic
`
`control was composed of six conflicts evolving over a period of about 30 minutes. We video-
`taped four teams of two controllers who had to regulate this traffic (the executive controller
`in charge of the traflic regulation of communication with pilots and the planning controller).
`Verbal and non verbal communications were analysed in terms of exchanges. We consider
`as exchanges one or many interventions focusing on the same object. Each intervention
`corresponding to turn talking or non-verbal action,
`is considered here as a basic unit of
`communication. In this analysis, we distinguished four types of interventions corresponding
`to (1) verbal communication between controllers (2) verbal communication between executive
`controller and pilots (3) non-verbal communication related to strip handling (writing, moving
`or pointing to strips) and (4) non-verbal communication related to radar scope (pointing to
`radar scope). Three principles guide the collection of observations concerning the non-verbal
`element of the interaction, those are:
`(1) the nature of the observations selected. (2) their
`description in reference to the activity, and (3) their processing in relation to the action
`under way. Taking into account the extreme richness of non-verbal or para-verbal elements,
`we focused on the behaviour related to the use of available environmental resources in the
`
`control environment, and on the content of verbal communications (for example, the handling
`of strips, pointing to the radar scope). In addition, all these elements are described with a
`set of action verbs (e.g.
`"write", ”give”, "place”, ”point", "take”, ”shif ”, ”move", etc.)
`The characteristics of this list are the observable nature of the selected behaviour (actions
`which can be described using the professional vocabulary) and the coherence of the list vis-
`a—vis the control activity (e.g.
`”shift” the strip out of line on the board has a distinctly
`different meaning to ”move” the strip). Amongst the non-verbal actions taken into account
`are (a) deictics (indicating an object or some specific information such as flight level on the
`strip, possibly asssociated with comments such as ”this one”, ”here”, ”th! ere”, and other
`illustrative gestu res, (b) writing (various annotations on the strip are made by controllers),
`and (c) handling of objects (the strip board is organized by moving the strips on the columns
`or between them).
`
`6 Results
`
`We now intend to develop the points developed above through the analysis of several examples.
`The following abbreviations will be used: (Cp (planning controller), Ce (executive controller).
`Pi (pilot).
`
`6.1 Non-verbal resources as a specific means of communication
`
`Several examples illustrate the specific role of non-verbal resources in communicating: non-
`verbal resources are revealed to be a specific vector of c0mmunication for some types of
`information which are not verbally expressed (e.g. urgency of the situation).
`
`69
`
`Petitioner Microsoft Corporation — Ex. 1008, p. 2752
`DISH, Exh. 1030, p. 97
`
`
`DISH, Exh. 1030, p. 97
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 2752
`
`

`

`Attachment 1a: Online copy of CMC/95 from a Technishe Universiteit Endhoven We-
`site
`
`The planning controller receives the strips (several minutes before the aircraft effectively
`enters the sector) which he transmits to the executive controller The gestures associated with
`the transmission of the strip is ”multiform" (passing the strip from hand to hand, placing it
`on the edge of the board, using the strip to point to the radar scope, etc.). The strip can
`also be used for annotations (marking the evolution of the flight, or circling the destination
`airport).
`
`Transmission of strip accordin; to the non-routine character of the information
`Team 4 - V
`
`(11)
`Cp-Pi:”IBERIA takeofi', IBERIA .913 to Amboise 0K?" (Cp writes strip, places strip on edge
`of board)
`Ce-Cp:”OK” (Ce writes strip, keeps strip in his hand)
`Ce -Cp:”How many does he want that man there, he wants 330 too.”
`
`
`
`(...)
`C -Pi: ”IBERIA 913 initial level 280 ” (Ce writes strip, holds Iberia913 strip)
`(...)
`(Cp presents Other strip)
`(12)
`(---)
`(Ce holds Iberia913 strip)
`(Cp holds Other strip)
`(...)
`(Ce puts IBE.913 strip down, picks up Other strip, places on right board)
`(...)
`(Oe picks up Iberia913 strip again) Ce-Cp "IBERIA..."
`Cp—Ce: ”hey?”
`Ce-Cp: ”913 I'll put him straight onto Amboise, why not? He is with us”
`Cp—Ce: ”From now on you'll be all right”
`Ce—Cp: ”He is with us” (Ce points to Scope)
`Ce—Pi: “IBERIA 619 maintain level”
`
`(Ce points to Iberia913 strip. Ce underlines Iberia613 strip.)
`Ce-Pi: ”IBE913 turn left to AME” (Ce holds Iberia913 strip)
`(13)
`Ce—Cp: ” and in any case he can only have 290 huh? En 330 it’s not possible?"
`Cp—Ce: "who‘s that, the 913? why not?”
`Ce-Cp: "330 on amboise, 290 is not possible”
`(Ce points to scopeX2) (Cp places Iberia913 strip)
`Cp—Ce: ”there’s nothing there... I’ll sort you out, Airbus 300.. and the other at 330, all right
`we'll sort it out he said... Bordeaux is taking everything for once"
`(Cp points to strips, Iberi3913 strip and Monarque 598 strip)
`(...)
`
`In this example, it would appear that the planning controller transmits Iberia913 strip by
`placing it sideways on the edge of the board in front of the executive controller. The planning
`controller associates a verbal intervention with two non-verbal ones (underlining the strip and
`
`70
`
`Petitioner Microsoft Corporation — Ex. 1008, p. 2753
`DISH, Exh. 1030, p. 98
`
`DISH, Exh. 1030, p. 98
`
`
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 2753
`
`

`

`Attachment 1a: Online copy-of CMC/95 from a Technishe Universiteit Endhcven We-
`site
`
`placing it on the board). The results of these actions are first to improve memorization of
`the requested level, second to highlight for his/her colleague, a conflict with another aircraft
`in the same sector at the same level. The act of it placing the strip sideway on the board can
`be interpreted by the executive controller as a mark of non-routine information which has to
`be taken into account quickly (the strip is not placed outside the board which forces him to
`pick it up to visualise the strips which are integrated in the board).
`In this same example, it appears that gestures associated with strip handling can be
`interpreted by co—workers; here the executive controller’s activity is constrained by commu-
`nication with the pilots (not reported here), he keeps the strip in his hand several minutes
`(approximately three minutes) before looking to solve this problem which is the purpose of an
`exchange with the planning controller. A second strip is proposed by the planning controller
`(Other strip), he will have to delay transmission to the executive controller who kept the
`Iberia 913 strip without putting it on the board.
`
`In the following example, non-verbal resources appears to manifest pieces of information
`which are not verbally expressed. First, the executive controller points to the scope it with
`a turning movement which represents a regulation strategy.
`
`Conflict solvin; exressed b
`
`Team 2-VI
`
`vesture
`
`(20)
`Cp-Ce: "Yes, he’s coming, he’s here" (Cp points to ACF5111 strip)
`Ce—Pi: "AIR CHARTER 5111, good morning, 26 at Orly maintain 270 call back for descent.”
`(Ce writes ACF5111 strip)
`Ce—Cp: ”430 knots, 430 knots, 430 knots, it's going to be easy"
`(Ce points to X3 scope)
`
`Cp—Ce: ”You know it’s an Airbus”
`
`(---)
`Cp-Ce: "It’s..."
`Ce-Cp: ”One fare... as quiet as that, great”
`(Ce points to scope, turning movement)
`Cp-Ce: "yes”
`Ce-Cp: ”Go on, climb"
`(...)
`
`6.2 Complementary role in communication regulation
`
`Non-verbal resources are used in giving a context of interpretation for what is verbally ex-
`pressed. In this analysis, our second interest is to examine the role they play in the success
`of communication. The study of multimodal aspects of communication leads us to under-
`line the importance of non-verbal acts to understand communications, as part of the context
`activated to interpret communications. In the following example. the exchange is based on
`the establishment of the referenced flight,
`two aircraft are present in the sector, SWROll
`and SWR012. The executive controller’s hypothesis is that SWR concerned is SWR012, the
`planning controller’s interventions, both indicating the strip and commenting verbally, allow
`
`7]
`
`Petitioner Microsoft Corporation — Ex. 1008, p. 2754
`DISH, Exh. 1030, p. 99
`
`
`DISH, Exh. 1030, p. 99
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 2754
`
`

`

`Attachment 1a: Online copy-of CMC/95 from a Technishe.Universiteit Endhoven Wer
`site
`
`him to make his intention explicit.
`
`Strip pointin; in communication re-ulation
`
`Team 3 -III
`
`(13)
`Cp-Ce: ”watch out for the SWR eh?”
`(...)
`(14
`
`(---)
`
`Cp—Ce: " We can transfer the S WR on frequency already"
`(Ce wants to take SWR012 strip) Ce.Cp : the SWR012”
`(Cp points to SWR012 strip) Cp—Ce: ”no, not that one”
`Ce—Cp: ”all right,the SWR011 you mean at 280”
`Cp—Ce:"no, no ...011“(Cp points to SWRDII strip)
`
`This example underlines the richness of the multimodal communication which allows de-
`termination of the object which is referred to and to ensure success of communication (the
`refrence SW ” is first based on a misunderstanding between controllers; in the second part,
`reference to SWROll is expressed four times).
`In the following example, two fights are re-
`ferred to, Monarque 598 and Monarque 1789, both are the subject of a conflict solving, What
`is interesting here is that the controllers never say explicitly ”598” or "1789”, but pointing to
`the radar scope allows them to confirm that the planning controller‘s first hypothesis is not
`relevant.
`
`Pointin; at sco e in communication re
`Team 4-111
`
`(10)
`(...)
`Ce—Cp: ”I turned the Monarque a bit too for”
`Cp-Ce: ”the Monarque? no, it’s fine"
`Ce-Cp: ”no, him”
`(Ce points to the scope)
`Cp—Ce:”ah, that one”
`Ce-Cp: ”the MON is just right.
`
`(...)
`
`As shown in the above two examples, several misunderstandings arose in communication
`between controllers. Verbal and non-verbal resources appear complementary to ensure the
`establishment of a mutual cognitive environment. This complementarity is not only used in
`the case of misunderstandings as we will see in the following examples.
`In the next example, the executive controller clarifies the reference by pointing to aircraft
`AFR022 on the radar scope. The planning controller confirms his identification of the aircraft
`in both a verbal and non-verbal way.
`
`72
`
`Petitioner Microsoft Corporation — Ex. 1008, p. 2755
`DISH, Exh. 1030, p. 100
`
`DISH, Exh. 1030, p. 100
`
`
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 2755
`
`

`

`Attachment 1a: Online copy of CMC/95 from a Technishe-Universiteit Endhoven We
`site
`
`Verbal interventions combined with stri o movin and pointin- at radar scope in dia- nosis.
`Team 2-1
`
`(04)
`Ce-Cp: "Right, there ’5 one there.” (Ce points at scope)
`Cp-Ce: ”Yes, it’s AFR 022 I think” (Cp points to Afr022 strip)
`(...)
`C picks up Afr022 strip and puts it in 5 shifted out of line on the left)
`
`Ce : ”things are getting heated there, it’s not reasonable!"
`
`In this example, reference is made twice, first by moving the strip on the board and sec-
`ondly by pointing at the radar scope. The request concerns remembering an aircraft to be
`transferred to an adjacent sector. The name of the aircraft is not verbally expressed .
`
`Verbal interventions combined with movin- the strio and pointin; at the radar scoe.
`Team 3-11
`
`(--~)
`(2 )
`(---)
`Ce-Cp: ”that one there, I’ll transfer his frequency a bit later, will you remind me? he’s there"
`(Cp removes Monarque598 strip, places it on right hand board, points to scope)
`Cp—Ce: ”all right”
`
`(...)
`
`In a similar way, overbearing the radio frequency allows the planning controller to ini-
`tiate a request based on the reference made to the communication between the executive
`controller and the pilot. This next example shows the way in which controllers use environ-
`mental resources (here overhearing a radio communication) to initiate an exchange. What is
`interesting here is that this is an economical way to communicate (reference made to radio
`communication). This would probably have consequences on mutual understanding in the
`sense that the cognitive effort required to interpret the meaning of the utterance(0p -Ce:
`”which. way are you sending him?”) stays less than if the controller had referred to another
`aircraft in the sector, indeed, in respect to the relevance principle (Sperber, Wilson, 1989),
`the utterance expressed by the planning controller is relevant in the cognitive environment of
`the interlocutor.
`
`To conclude on this point, it would appear that communications between controllers are
`usually composed of both verbal and non-verbal resources in one single exchange. The results
`(obtained through videotaping of four teams) reveal that the exchanges where non-verbal
`resources are used, are the most numerous. Exchanges composed of only verbal interven-
`tions are (percentage): 24.61 (team 1), 25.73 (team 2), 21.55 (team 3) and 28.63 (team 4).
`Exchanges composed of interventions which are either verbal and non-verbal, or only non—
`verbal, are 75.37 (team 1), 74.51 (team 2), 78.42 (team 3), and 71.34 (team 4). This second
`category of exchanges includes three different types of exchange: (a) use of only non-verbal
`communication (between 6 and 16 depending on the team, considering the total number of
`exchanges); (b) use of non—verbal communication supported by strip handling combined with
`use of verbal communication that is here radio-communication with the pilot (between 15 and
`35); (c) use of both verbal and non-verbal communication (between 28 and 51).
`
`73
`
`Petitioner Microsoft Corporation — Ex. 1008, p. 2756
`DISH, Exh. 1030, p. 101
`
`DISH, Exh. 1030, p. 101
`
`
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 2756
`
`

`

`Attachment 1a: Online capy of CMC/95 from a Technishe Univarsiteit Endhoven We
`site
`
`Overhearin_ radio frequency with verbal intervention
`Team 4 - III
`
`(09)
`Ce-Pi: ”L UFT 498 say heading..."
`
`Cp-C'e: "which way me you sending him?”
`Ce-Cp: ”to the right”
`Ce-Pi: ”LUFT498 turn right heading 270”
`Ce-Cp:"where did I put it?”
`
`(...) Cp—Ce:”haven't you got it?”
`
`(...)
`C -Ce: ”You’ve got the LUFT which is there”
`Cp points to DLH498 strip
`Ce-Cp: ”270”
`(Ce writes DLH strip, holds it)
`(etc...)
`
`6.3 Verbal and non-verbal resources in dialogue management
`
`This analysis of non-verbal acts leads us to consider, on the one hand, the action with com-
`municative intention (for example, handling the strip and simultaneously asking something
`to notify relevant information), and on the other hand the action considered as a means
`of organizing information for him/herself (these actions can be interpreted by co-workers
`as intention recognition).
`In other words, informative actions and utterances (versus com-
`municative) serve as a basis for inferences that agents make about their mutual cognitive
`environment, but are not necessarily intended to communicate. The co-presence in the same
`working location allows the construction of mutual beliefs in ways which are efficient, in the
`sense that they are not intrusive in comparison with the verbal channel.
`In the following example, the controllers initiate a common diagnosis of the situation, the
`planning controller acts on the strip in writing and moving the strip on the column on the
`flight progress board. Two minutes later, the executive controller‘s first intervention can be
`interpreted by the planning controller as confirmation of his diagnosis concerning a problem
`at level 330 including DAN4446. One minute before the planning controller has given the
`strip for AFR022 which is descending in the opposite direction. In analysing the situation,
`the executive controller shows that he does not find the flight on the radar scope, by tapping
`with his finger on the strip he is referring to, which is followed by the planning controller’s
`engagement in this exchange. Therefore, such a non-verbal act can be seen as an implicit
`request in the sense that the planning controller is supposed to answer only if he is not engaged
`in an activity which is difficult to interrupt.
`
`74
`
`Petitioner Microsoft Corporation — Ex. 1008, p. 2757
`DISH, Exh. 1030, p. 102
`
`DISH, Exh. 1030, p. 102
`
`
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 2757
`
`

`

`Attachment 1a: Online capy of CMC/95 from a Technishe UniVersiteit Endhoven We
`site
`
`Strip pointin
`
`
`Team 2-1
`
`while formin- a common dia- nosis of the situation
`
`(00)
`Cp—Ce: ”There’s the 330” (Cp underlines strip Dan1116)
`Ce-Cp: ”Well yes... it’s going to be hard”
`(Cp shifts Dan1116 to the right)
`Cp-Ce: ”We’ll have to see”(...)
`(01)
`(Cp gives two strips, Afr022 strip and FBJMG strip)
`Cp -Ce: ”Here’s a problem.”
`Ce: ”There are lots of problems.” (Ce takes slips, places one in pos.4 AFR022, shifts it to the
`left and keeps the other in his hand)
`Ce ”Ah yes”
`(02) (---)
`Ce-Cp: ”Yes there seems to be a lot of climbing to 330”
`(Ce points to Dan4446 strip, tapping it with his finger)
`Ce-Cp: "That’s the one I can’t see.”
`
`m
`
`Cp-Ce: ”Just a moment, yes, he is behind.”
`(Cp adjusts scope, points to scape)
`
`In the following example, it appears that the executive controller does not explicitly ask
`the planning controller for assistance and does not verbally express the name of the aircraft
`concerned. The planning controller listens to the pilot and is able to point to the flight
`progress strip. After the pilot’s call, the executive controller looks for the DAN4446 strip,
`this strip had been put on the board a few minutes before, further to the analysis made
`previously. This position is no longer in accordance with the current situation. The planning
`controller, hearing the radio-communication, can infer who ”this one” is, and points to the
`associated strip. Thus, the reference is made through radio-communication and confirmed
`by the planning controller twice by pointing and verbal intervention. We note that the
`executive controller never named the aircraft explicitly. Radio-communications between the
`executive controller and pilots, preceded or followed by strip handling can be considered as
`an important part of the controller’s work. Strip handling allows the controller to organize
`and process information related to each flight according to his/ her diagnosis or actions on
`conflicts.
`
`75
`
`Petitioner Microsoft Corporation — Ex. 1008, p. 2758
`DISH, Exh. 1030, p. 103
`
`DISH, Exh. 1030, p. 103
`
`
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 2758
`
`

`

`Attachment 1a: Online COPY of CMC/95 from a Technishe Univarsiteit Endhoven We
`site
`
`Overhearin_ radio frequency with verbal interventions and movin strios
`Team 2-1
`
`(06)
`Pi-Ce: ”... "
`
`Dan 4446 strip)
`
`Ce: "1 can’t find him any more, that one”
`Ce-Pi: ”Who was calling?” Pi-Ce: ”DAN4446 Good morning”
`(Cp points to Dan4446 strip)
`Ce -Pi: ”please maintain level 290 I’ll call you back
`
`it’s not right time not” (Ce underlines
`
`Ce~sz ”Wrong place” (Ce places Dan 4446 strip shifted out of line to the right)
`Cp—Ce: ”Yes, you’d put it there because of the conflict...”
`(07)
`Just a minute ...”
`Ce—Cp : ”Shhh!
`Ce-Pi: "AFR022 Good morning it‘s facing west at Roissy - please descend to flight level 200
`two zero zero.” (Ce writes Afr022 strip)(...)
`
`7 Discussion
`
`The study of cooperation between human agents in such face-to-face situations, emphasizes
`the role of external artifacts in cooperation processes because they allow the cooperative
`agents to organize their own cognitive processes (external artifacts are used as support for
`memory and problem solving) at the same time as updating their mutual cognitive envi-
`ronment through intention recognition processes.
`Intention recognition through non-verbal
`communication can probably be seen as essential for cooperation for several reasons. First,
`non—verbal communication allows agents to communicate elements like urgency, etc. which
`are not explicity verbalized. Second, it appears that a large amount of verbal communications
`are associated to non-verbal ones; in this sense they constitute a context of interpretation of
`what is verbally expressed, which is used to regulate misunderstandings. Third, its non-
`interruptive property appears important(instead of making explicit verbal requests!
`, each
`agent notices non-verbal ac ts by other agents as part of their cognitive activities).
`In the
`perspective of the design of cooperative tools, difliculties arise from developing methodologies
`to anticipate and evaluate the implications of a new environment on the cooperation processes
`between agents. The design of complex working environments is conducted today by using
`the following methodologies; (1) validation, based on empirical approaches, use of prototypes
`and iterative testing (Gaillard and Leroux, 1994), (2) simulation, but methodological tools
`for the anticipation of cognitive activities from actual situations have to be investigated. The
`first step of this is to assess the cooperative nature of the working environment. This is based
`on the analysis of how people use external artifacts not only in order to support their own
`cognition, but also to cooperate on implicit modes. The methodology is based on assessment
`of the capacity of external artifacts to support intention recognition and includes two stages:
`(1) identifying in a working situation, the ! supports used for intention recogn ition and for
`cognition (this analysis highlights how all external artifacts are used in diagnosis, problem
`solving, etc.), in nominal and degraded situations (Bressolle, 1992); (2) anticipate or evaluate
`how these cognitive supports are transferred during the introduction of new technologies in
`relation to the cognitive activities in the domain. In this second stage, in order to anticipate
`
`76
`
`Petitioner Microsoft Corporation — Ex. 1008, p. 2759
`DISH, Exh. 1030, p. 104
`
`DISH, Exh. 1030, p. 104
`
`
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 2759
`
`

`

`Attachment 1a: Online capy of CMC/95 from a Technishe Universiteit Endhoven We
`site
`
`cognitive properties of new working environments, a simulation tool is being developed based
`on the formalisation and simulation of the communications between cooperative agents and
`on conceptual specifications of the application (Zorola-Villarreal et al. 1995).
`
`References
`
`Benchekroun H., Pavard B., Salembier P. [1993] Design of cooperative systems in complex
`dynamic environments. In J .M. Hoc, P.C. Cacciabue and E. Hollnagel (eds.) Expertise and
`Technology - Cognition and Human-Computer Cooperation. 16p.
`
`Bressolle M. C. [1992] Perception de l’intention et cooperation dans le cas du controle de la
`navigation aerienne. Premiers elements d’analyse. Internal report.
`
`Cadoz