I lllll llllllll Ill lllll lllll lllll lllll lllll 111111111111111111111111111111111
`US007881441B2
`
`c12) United States Patent
`Engelke et al.
`
`(IO) Patent No.:
`(45) Date of Patent:
`
`US 7,881,441 B2
`Feb. 1, 2011
`
`(54) DEVICE INDEPENDENT TEXT CAPTIONED
`TELEPHONE SERVICE
`
`(75)
`
`Inventors: Robert M. Engelke, Madison, WI (US);
`Kevin R. Colwell, Middleton, WI (US);
`Troy D. Vitek, Wanakee, WI (US); Kurt
`M. Gritner, Madison, WI (US)
`
`(73) Assignee: Ultratec, Inc., Madison, WI (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1303 days.
`
`(21) Appl. No.: 11/391,141
`
`(22) Filed:
`
`Mar. 28, 2006
`
`4,012,599 A
`4,039,768 A
`4,126,768 A
`4,151,380 A
`4,160,136 A
`4,188,665 A
`4,191,854 A
`4,201,887 A
`4,254,308 A
`D259,348 s
`4,268,721 A
`4,289,931 A
`
`3/1977 Meyer
`8/1977 O'Maley
`1111978 Grenzow
`4/1979 Blomeyer et al.
`7/1979 McGough
`2/1980 Nagel et al.
`3/1980 Coles
`5/1980 Burns
`3/1981 Blomeyer et al.
`5/1981 Sakai et al.
`5/1981 Nielson et al.
`9/1981 Baker
`
`(Continued)
`
`(65)
`
`Prior Publication Data
`
`FOREIGN PATENT DOCUMENTS
`
`US 2007 /0036282 Al
`
`Feb. 15, 2007
`
`DE
`
`2647097
`
`4/1978
`
`Related U.S. Application Data
`
`(60) Provisional application No. 601695,099, filed on Jun.
`29, 2005.
`
`(51)
`
`Int. Cl.
`H04M 11100
`(2006.01)
`(52) U.S. Cl. ...................................... 379/52; 379/90.01
`(58) Field of Classification Search ................... 379/52,
`379/93.15, 93.07; 370/352
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`(Continued)
`
`OTHER PUBLICATIONS
`
`PCT Search Report PCT/US2006/025236.
`
`(Continued)
`
`Primary Examiner-Stella L Woo
`(74) Attorney, Agent, or Firm-Quarles & Brady
`
`(57)
`
`ABSTRACT
`
`3,372,246 A
`3,507,997 A
`3,515,814 A
`3,585,303 A
`3,598,920 A
`3,800,089 A
`3,896,267 A
`3,959,607 A
`3,976,995 A
`
`3/1968 Knuepper
`4/1970 Weitbrecht
`6/1970 Morgan
`6/1971 Chieffo
`8/1971 Fischer
`3/1974 Reddick
`7/1975 Sachs
`5/1976 Vargo
`8/1976 Sebestyen
`
`Text captioned telephony, in which a telephone conversation
`is accompanied by text captions to aid in hard-of-hearing
`users, is implemented without the use of specialized text
`captioned telephone terminals by using a combination of an
`Internet appliance such as a computer and a telephone or by
`generalized voice over Internet protocol (VOIP) telephones.
`
`42 Claims, 5 Drawing Sheets
`
`Ultratec Exhibit 1004
`Ultratec v Sorenson IP Holdings Page 1 of 14
`
`

`

`U.S. PATENT DOCUMENTS
`
`4,302,629 A
`4,307,067 A
`4,307,266 A
`4,354,252 A
`4,415,065 A
`4,426,555 A
`D273,l 10 s
`4,451,701 A
`4,471,165 A
`D275,857 s
`4,490,579 A
`4,503,288 A
`D278,435 s
`4,524,244 A
`D280,099 s
`4,533,791 A
`4,568,803 A
`4,569,421 A
`D283,421 s
`4,625,080 A
`RE32,365 E
`4,650,927 A
`4,659,876 A
`4,713,808 A
`4,754,474 A
`D296,894 s
`4,777,469 A
`4,799,254 A
`4,817,135 A
`4,839,919 A
`4,849,750 A
`4,868,860 A
`4,897,868 A
`D306,727 s
`4,918,723 A
`4,926,460 A
`4,951,043 A
`4,959,847 A
`D312,457 s
`4,995,077 A
`5,025,442 A
`D322,785 s
`5,081,673 A
`5,086,453 A
`5,091,906 A
`5,095,307 A
`5,099,507 A
`5,121,421 A
`5,128,980 A
`5,134,633 A
`5,163,081 A
`5,192,947 A
`5,210,689 A
`5,214,428 A
`5,216,702 A
`5,249,220 A
`5,280,516 A
`5,289,523 A
`5,307,399 A
`5,311,516 A
`5,325,417 A
`5,327,479 A
`5,339,358 A
`5,343,519 A
`5,351,288 A
`D351,185 s
`5,359,651 A
`5,377,263 A
`5,393,236 A
`5,396,650 A
`D357,253 s
`
`1111981 Foulkes et al.
`12/1981 Tagawaetal.
`12/1981 Messina
`10/1982 Lamb et al.
`1111983 Sandstedt
`111984 Underkoffler
`3/1984 Genaro et al.
`5/1984 Bendig
`9/1984 Defino et al.
`10/1984 Moraine
`12/1984 Godoshian
`3/1985 Kessler
`4/1985 Hikawa
`6/1985 Faggin et al.
`8/1985 Topp
`8/1985 Read et al.
`2/1986 Frola
`2/1986 Sandstedt
`4/1986 Brier
`1111986 Scott
`3/1987 Sebestyen
`3/1987 James
`4/1987 Sullivan et al.
`12/1987 Gaskill et al.
`6/1988 Feinson
`7/1988 Chen
`10/1988 Engelke et al.
`111989 Dayton et al.
`3/1989 Winebaum
`6/1989 Borges et al.
`7/1989 Andros et al.
`9/1989 Andros et al.
`111990 Engelke et al.
`3/1990 Fritzsche
`4/1990 Iggulden et al.
`5/1990 Gutman et al.
`8/1990 Minami
`9/1990 Engelke et al.
`1111990 Inatomi
`2/1991 Malinowski
`6/1991 Lynk et al.
`12/1991 Wu
`111992 Engelke et al.
`2/1992 Senoo et al.
`2/1992 Reed et al.
`3/1992 Shimura et al.
`3/1992 Mukai et al.
`6/1992 Alheim
`7/1992 Choi
`7/1992 Werner
`1111992 Wycherley et al.
`3/1993 Neustein
`5/1993 Baker et al.
`5/1993 Allen
`6/1993 Ramsden
`9/1993 Moskowitz et al.
`111994 Jang
`2/1994 Vasile et al.
`4/1994 Dai et al.
`5/1994 Kuznicki et al.
`6/1994 Engelke et al.
`7/1994 Engelke et al.
`8/1994 Danish et al.
`8/1994 Feldman
`9/1994 Engelke et al.
`10/1994 Matsuda et al.
`10/1994 Dragan off
`12/1994 Bazemore et al.
`2/1995 Blackmer et al.
`3/1995 Terauchi
`4/1995 Wong
`
`US 7,881,441 B2
`Page 2
`
`4/1995 Hotto
`5,410,541 A
`6/1995 Kidrin
`5,423,555 A
`7/1995 Engelke et al.
`5,432,837 A
`10/1995 Richardson et al.
`5,459,458 A
`D364,865 s
`12/1995 Engelke et al.
`12/1995 Eisdorfer et al.
`5,475,733 A
`111996 Shpiro et al.
`5,487,671 A
`4/1996 Stentiford
`5,503,560 A
`5/1996 Engelke et al.
`5,517,548 A
`5/1996 Kikinis et al.
`5,522,089 A
`5,574,784 A
`1111996 LaPadula et al.
`12/1996 Engelke et al.
`5,581,593 A
`2/1997 Engelke et al.
`5,604,786 A
`D379,181 s
`5/1997 Sawano
`10/1997 Kasdav et al.
`5,680,443 A
`12/1997 Leyen et al.
`5,701,338 A
`111998 Lee et al.
`5,710,806 A
`5,712,901 A
`111998 Meermans et al.
`3/1998 Engelke et al.
`5,724,405 A
`6/1998 Shpiro
`5,766,015 A
`7/1998 August
`5,787,148 A
`8/1998 Mitchell et al.
`5,799,273 A
`9/1998 Ryan
`5,809,112 A
`9/1998 Colwell et al.
`5,809,425 A
`10/1998 Escobar et al.
`5,826,102 A
`12/1998 Gould et al.
`5,850,627 A
`2/1999 Engelke et al.
`D405,793 S
`2/1999 Bernstein
`5,870,709 A
`5/1999 McLaughlin et al.
`5,905,476 A
`6/1999 Engelke
`5,909,482 A
`10/1999 Engelke et al.
`5,974,116 A
`1111999 Colwell et al.
`5,978,654 A
`1111999 Duffin
`5,991,723 A
`5,995,590 A
`1111999 Brunet et al.
`612000 VanBuskirk et al.
`6,075,534 A
`612000 Engelke et al.
`6,075,841 A
`612000 Engelke et al.
`6,075,842 A
`112001 VanAlstine
`6,175,819 Bl
`6,188,429 Bl
`2/2001 Martin et al.
`6,233,314 Bl
`5/2001 Engelke
`10/2001 Engelke et al.
`6,307,921 Bl
`6,314,396 Bl
`1112001 Monkowski
`6,324,507 Bl
`1112001 Lewis et al.
`12/2002 Engelke et al.
`6,493,426 B2
`112003 Engelke et al.
`6,504,910 Bl
`112003 Engelke et al.
`6,510,206 B2
`4/2003 Engelke et al.
`6,549,611 B2
`5/2003 Engelke et al.
`6,567,503 B2
`7/2003 Engelke
`6,594,346 B2
`8/2003 Engelke
`6,603,835 B2
`6/2004 Engelke et al.
`6,748,053 B2
`7/2004 Feigenbraum
`6,763,089 B2
`6,816,468 Bl* 1112004 Cruickshank ............... 370/352
`412005 Engelke et al.
`6,885,731 B2
`8/2005 Engelke et al.
`6,934,366 B2
`7,003,082 B2
`212006 Engelke et al.
`7,006,604 B2
`212006 Engelke
`1112006 McClelland et al.
`7,142,642 B2
`1112006 Brooksby
`7,142,643 B2
`7,164,753 B2
`1/2007 Engelke et al.
`7,233,655 B2 *
`6/2007 Gailey et al. .................. 379/52
`7,313,231 B2 * 12/2007 Reid .......................... 370/352
`7,315,612 B2
`1/2008 McClelland
`2004/0083105 Al
`412004 Jaroker
`8/2005 Bas son et al. .................. 72514
`2005/0183109 Al*
`2005/0226394 Al
`10/2005 Engelke et al.
`2005/0226398 Al
`10/2005 Bojeun
`2006/0140354 Al
`612006 Engelke
`2006/0285652 Al * 12/2006 McClelland et al.
`2008/0152093 Al
`6/2008 Engelke et al.
`
`. ......... 379/52
`
`Ultratec Exhibit 1004
`Ultratec v Sorenson IP Holdings Page 2 of 14
`
`

`

`US 7,881,441 B2
`Page 3
`
`2008/0187108 Al
`
`8/2008 Engelke et al.
`
`FOREIGN PATENT DOCUMENTS
`
`WO
`WO
`WO
`
`PCT/US93/04760
`PCT/US95/00015
`2005/081511 Al
`
`1111993
`7 /1995
`912005
`
`DE
`DE
`DE
`DE
`EP
`EP
`FR
`FR
`FR
`GB
`GB
`GB
`JP
`JP
`JP
`JP
`JP
`
`2749923
`3410619
`3632233
`103 28 884 Al
`0029246
`1330046
`2403697
`2432805
`2538978
`2183880
`2335109
`2339363
`55-044283
`57-055649
`58-134568
`60-259058
`63-198466
`
`5/1979
`10/1985
`4/1988
`212005
`5/1981
`7/2003
`4/1979
`2/1980
`7 /1984
`6/1987
`9/1999
`1/2000
`3/1980
`4/1982
`8/1983
`12/1985
`8/1988
`
`OTHER PUBLICATIONS
`
`PCT/US2009/040523 Search Report and Written Opinion; 14 pages.
`Radio Electronics, Mar. 1976, pp. 51-58, "ASCII to Baudot" Smith,
`Roger L.
`R.J. Cooper, "Break feature for half-duplex modem", IBM Technical
`Disclosure Bulletin, vol. 17, No. 8, pp. 2386-2387 Jan. 1975.
`Gopalakrishnan, "Effective Set-up for Performing Phone Conversa(cid:173)
`tions by the Hearing Impaired", IBM Technical Disclosure Bulletin,
`pp. 423-426, (Dec. 1991).
`Moskowitz, J., "Telocator Alphanumeric Protocol'', Version 1.8, Jan.
`1997.
`AU Pat. App. No. 2006263680, Amendment, Jun. 17, 2010.
`AU Pat. App. No. 2006263680, Examination Report, Jun. 29, 2009.
`EP Pat. App. No. 06785768.0, Communication, Aug. 9, 2010.
`* cited by examiner
`
`Ultratec Exhibit 1004
`Ultratec v Sorenson IP Holdings Page 3 of 14
`
`

`

`U.S. Patent
`
`Feb. 1, 2011
`
`Sheet 1of5
`
`US 7,881,441 B2
`
`10~
`
`FIG. 1
`
`60
`
`62
`
`FILE
`
`16
`_ox
`
`75
`
`70
`
`CAPTION
`
`HELLO
`
`73
`
`FIG. 3
`
`Ultratec Exhibit 1004
`Ultratec v Sorenson IP Holdings Page 4 of 14
`
`

`

`U.S. Patent
`
`Feb. 1, 2011
`
`Sheet 2 of 5
`
`US 7,881,441 B2
`
`58
`
`J
`
`64
`
`66
`
`68
`
`77
`
`CONNECT TO
`RELAY
`
`FORWARD VOICE
`ON VOiP
`
`RECEIVE AND
`DISPLAY TEXT
`
`DISCONNECT
`
`FIG. 4
`
`14'\
`
`FIG. 5
`
`76
`
`78
`
`80
`
`82
`
`94
`
`96
`
`CALL
`RELAY
`
`ENTER PHONE
`NUMBER
`
`RECEIVE
`PIN
`
`CONNECT
`TO SERVER
`WITH PIN
`
`RECEIVE
`AND DISPLAY
`TEXT
`
`DISCONNECT
`
`FIG. 6
`
`TEXT
`
`54
`
`56
`
`RELAY
`
`72
`
`55
`
`,,......n PSTN
`T
`
`2a
`
`74
`
`26
`
`Ultratec Exhibit 1004
`Ultratec v Sorenson IP Holdings Page 5 of 14
`
`

`

`U.S. Patent
`
`Feb. 1,2011
`
`Sheet 3 of 5
`
`US 7,881,441 B2
`
`,-sG
`
`_ox
`I PIN ®- 90
`
`88 -H
`
`FIG. 7
`
`HELLO
`
`-92
`
`:...- """
`
`16
`
`CAPTEL ID
`
`110
`
`_ox
`
`608 123-4567 J. DOE
`r-116
`~114
`CONTACTS
`
`[i] rn rn
`rn m I M60 I
`[i] rn I w; 1
`
`CALL
`
`CAPTION
`
`118
`
`120
`
`122
`
`130
`
`HELLO
`
`FIG. 9
`
`92
`
`Ultratec Exhibit 1004
`Ultratec v Sorenson IP Holdings Page 6 of 14
`
`

`

`U.S. Patent
`
`Feb. 1, 2011
`
`Sheet 4 of 5
`
`US 7,881,441 B2
`
`28
`
`14~
`
`26
`
`FIG. 8
`
`MAKE VOiP
`CONNECTION
`
`112 108
`
`J
`
`140
`
`14
`
`\
`
`142
`VIDEO
`TEXT 56
`
`124
`
`126
`
`132
`
`CONNECT
`TO RELAY
`
`RECEIVE AND
`DISPLAY TEXT
`
`DISCONNECT
`
`FIG. 10
`
`52
`
`FIG. 11
`
`Ultratec Exhibit 1004
`Ultratec v Sorenson IP Holdings Page 7 of 14
`
`

`

`U.S. Patent
`
`Feb. 1, 2011
`
`Sheet 5 of 5
`
`US 7,881,441 B2
`
`136
`
`CAPTEL ID
`
`608 123-4567 J. DOE
`
`116
`
`,r--114
`
`~ 144
`rn rn
`rn
`[i]
`
`FIG. 12
`
`16
`
`92
`
`Ultratec Exhibit 1004
`Ultratec v Sorenson IP Holdings Page 8 of 14
`
`

`

`US 7,881,441 B2
`
`1
`DEVICE INDEPENDENT TEXT CAPTIONED
`TELEPHONE SERVICE
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This Application claims the benefit of U.S. Provisional
`Application 60/695,099 filed Jun. 29, 2005, and hereby incor(cid:173)
`porated by reference.
`
`STATEMENT REGARDING FEDERALLY
`SPONSORED RESEARCH OR DEVELOPMENT
`
`BACKGROUND OF THE INVENTION
`
`2
`tern allows text captioning to be easily used on incoming calls
`by allowing the text captioned telephone to dial out to the
`relay when the call is received and forward the necessary
`voice signal.
`Many users of text-captioned telephony have jobs requir(cid:173)
`ing significant use of the telephone as made possible by
`text-captioning. These users face a significant problem when
`they need to make use of telephone systems outside of their
`normal work environment where text captioning telephones
`10 may not be available or equipment such as cellular telephones
`that does not support text captioning must be used. Many
`workplaces use proprietary private branch exchange (PBX)
`telephone lines that do not support analog text captioning
`telephones, the latter which require standard telephone con-
`15 nections.
`
`SUMMARY OF THE INVENTION
`
`The present application relates generally to telephone sys(cid:173)
`tems and specifically to telephone systems that provide for
`real-time text captioning for the hard of hearing.
`Modem telecommunication services provide features to
`assist those who are deaf or hard of hearing. One such feature
`is the text telephone (TTY) also known as a telecommunica(cid:173)
`tion device for the deaf (TDD). TDDs allow for text commu(cid:173)
`nication through the telephone system by generating tones 25
`that may be transmitted by analog telephone lines in response
`to a user typing letters on a keyboard.
`Hearing users may communicate with deaf users who have
`TDD devices through so-called "relays." A relay is a service
`funded by telephone communication surcharges that provides 30
`a "call assistant" who intermediates between a deaf user and
`a hearing user. The call assistant communicates with the deaf
`user using a TDD and communicates with the hearing user by
`voice over a standard telephone line.
`A relay service may also be used to help those who are not 35
`deaf, but hard of hearing, by providing a captioned telephone.
`With a captioned telephone, a user who is hard of hearing may
`carry on a normal telephone conversation with a hearing user
`while a text transcription of the words spoken by the hearing
`user is displayed on the telephone. The text transcription 40
`allows the hard of hearing user to confirm his or her under(cid:173)
`standing of the spoken words. Unlike with a conventional
`relay service, captioned telephone requires extremely fast
`text transcription using a computer executed voice recogni(cid:173)
`tion system (typically withrevoicing by a trained operator) so 45
`that the text captioning is as nearly as possible contempora(cid:173)
`neous with the spoken words.
`Text captioned telephones of this type are marketed under
`the service mark CAPTEL and described in U.S. Pat. Nos.
`6,307,921 and 6,075,842 assigned to the assignee of the 50
`present invention and hereby incorporated by reference.
`A text captioned telephone system employs a relay service
`in a mode where the relay transmits both the voice of the
`hearing user and a text stream of the spoken words by that
`user. A telephone of the receiving hard of hearing user pro- 55
`vides a text display for displaying the captions.
`In a single-line text captioned telephone, text and voice
`signals may be received over a single telephone line and
`separated electronically. The text captioned telephone
`includes software that automatically dials the relay, and pro- 60
`viding the relay with the telephone number of the ultimate
`destination of the call. For incoming telephone calls, the hard
`of hearing user must normally hang up and redial the caller
`through the relay.
`In a two-line text captioned telephone, a first telephone line 65
`communicates with the caller, and a second telephone line
`communicates exclusively with the relay. The two-line sys-
`
`The present inventors have recognized that increased
`20 accessibility of workers to the Internet and general purpose
`computers makes it possible to provide text-captioned tele(cid:173)
`phony independently of specialized telephone terminal hard(cid:173)
`ware intended narrowly for the deaf, hard of hearing, or
`assisted users. In a first embodiment, a standard telephone
`may be used in tandem with an Internet-connected computer
`to allow the conversation on the telephone to be augmented
`with text displayed on the computer. The voice signal is
`provided to a captioning service which transmits text captions
`over the Internet to the computer in near real time. In this way,
`standard office equipment can provide text-captioning capa(cid:173)
`bilities without the need for specialized telephone equipment.
`Specifically, in a first embodiment of the invention, text
`captioning is provided to callers by a combination of a tele(cid:173)
`phone terminal and an Internet appliance such as a browser on
`a computer. A first caller communicates a voice portion of a
`telephone call over the telephone network with a second
`caller, both callers operating telephone terminals on tele(cid:173)
`phone lines. Text transcribing the telephone call is commu(cid:173)
`nicated over the Internet to one of the first and second callers
`for display on the Internet appliance.
`Thus it is one object of at least one embodiment of the
`invention to use commonly available office equipment to
`provide text-captioned telephony.
`It is another object of at least one embodiment of the
`invention to permit text-captioned telephony using available
`telephone equipment,
`telephone networks, and billing
`arrangements.
`The telephone network may include a PBX network.
`Thus it is an object of at least one embodiment of the
`invention to allow users of PBX phones to participate in the
`benefits of text-captioned telephony normally limited to stan(cid:173)
`dard telephone lines.
`The text may be provided by a relay communicating with
`the Internet.
`It is thus another object of at least one embodiment of the
`invention to allow existing relay services to be used without
`the need to navigate proprietary calling systems such as PBX
`or cellular telephone.
`The relay may receive a portion of the telephone call from
`the telephone network.
`Thus it is an object of at least one embodiment of the
`invention to present an embodiment in which high speed
`Internet is not required.
`The first caller may dial the relay on the telephone network
`to provide the relay with the telephone number of the second
`caller, and the relay may call the telephone number of the
`second caller to complete the call.
`
`Ultratec Exhibit 1004
`Ultratec v Sorenson IP Holdings Page 9 of 14
`
`

`

`3
`Thus it is another object of at least one embodiment of the
`invention to preserve the anonymity or invisibility of the relay
`in the calling process.
`The relay may alternatively receive a portion of the tele(cid:173)
`phone call from the Internet in a voice over Internet protocol.
`It is thus another object of at least one embodiment of the
`invention to eliminate the need for a separate call to the relay
`allowing seamless handling of incoming calls as well as the
`ability to make outgoing calls.
`The portion of the telephone call may be provided by 10
`tapping into the handset of one telephone terminal to com(cid:173)
`municate a portion of the conversation to the Internet appli(cid:173)
`ance.
`It is thus another object of at least one embodiment of the
`invention to provide a wide variety of telephone terminals to 15
`be used, with which the user is convenient and familiar, while
`providing a signal to the Internet appliance without the need
`for cumbersome or specialized microphones and the like.
`The Internet appliance may further transmit a video signal
`to the user over the Internet.
`Thus it is another object of at least one embodiment of the
`invention to take advantage of the Internet connection to
`provide augmented communication between the callers or a
`caller and the relay.
`In an alternative embodiment, the invention may provide
`for text captions to telephone calls by communicating a voice
`portion of the telephone call between the first caller and the
`second caller over an Internet telephone using voice over
`Internet protocol (VOIP). Text captioning of the telephone
`call can be provided by a relay tapping into the Internet 30
`transmission and forwarding captioning information to the
`Internet telephone also over the Internet.
`Thus it is an object of at least one embodiment of the
`invention to take advantage of the ability of the Internet to
`make multiple connections among more than one party (e.g.,
`two callers and the relay) to provide text-captioned telephony.
`It is another object of at least one embodiment of the invention
`to take advantage of the ubiquity of Internet connections in
`the office environment to avoid the problems normally inher-
`ent in PBX proprietary lines.
`The Internet telephone may be a computer communicating
`with the Internet or a dedicated VOIP telephone where the
`captioning is displayed on the VOIP telephone display.
`Thus it is another object of at least one embodiment of the
`invention to take advantage of inherent text communication
`qualities ofVOIP telephone systems, either those using com(cid:173)
`puters or dedicated VOIP phones, to provide for text-cap(cid:173)
`tioned telephony without specialized equipment.
`The user may initiate the communication to the relay to
`promote text captioning.
`Thus it is an object of at least one embodiment of the
`invention to allow on-demand text captioning only as
`required.
`The device may provide for a volume display to one of the 55
`first and second users.
`Thus it is an object of at least one embodiment of the
`invention to allow the hard-of-hearing user to have an addi(cid:173)
`tional dimension of understanding of the conversation, and
`thus some assurance that proper text captioning is occurring. 60
`The Internet telephone may incorporate a serial number
`and may use the serial number to validate the user.
`It is thus another object of at least one embodiment of the
`invention to prevent misuse of relay services as may occur
`with the anonymity of the Internet.
`The Internet telephone may further include a telephone
`network connection and the telephone may use the telephone
`
`FIG. 1 is a representation of a first embodiment of the
`invention providing text-captioned telephony using a PBX
`office telephone and Internet-connected computer;
`FIG. 2 is a perspective view of a voice-tap connecting the
`PBX telephone of FIG. 1 to the computer to allow transfer of
`voice signals over the Internet to a relay;
`FIG. 3 is a caller interface window as may be displayed on
`the computer of FIG. 1 allowing on-demand captioning of the
`20 telephone call;
`FIG. 4 is a flowchart of the principle steps of a program
`executed on the computer of FIG. 1 when captioning is
`demanded by the caller;
`FIG. 5 is a figure similar to that of FIG. 1, showing an
`25 alternative embodiment in which the voice signal is routed
`through a standard telephone line to the relay and the text is
`returned to the computer;
`FIG. 6 is a figure similar to that of FIG. 4 showing the
`flowchart for the embodiment of FIG. 5;
`FIG. 7 is a figure similar to that of FIG. 3 showing a caller
`interface window for entering a PIN number to link caption(cid:173)
`ing text to a particular call;
`FIG. 8 is a figure similar to that of FIGS. 1and5 showing
`implementation of a text-captioning using VOIP transmis-
`sions;
`FIG. 9 is a figure similar to that of FIGS. 7 and 3 showing
`a caller interface window for a VOIP telephone implemented
`on a standard computer;
`FIG. 10 is a flowchart of the principal steps executed by the
`computer in implementing the text-captioned telephony
`using VOIP telephones;
`FIG. 11 is a fragmentary view of the computer of FIG. 8
`showing the addition of a video camera so that voice and
`video may be transmitted to the other caller; and
`FIG. 12 is a fragmentary view of the caller interface win(cid:173)
`dow of FIG. 9 showing a video inset possible with the con(cid:173)
`figuration of FIG. 11.
`
`40
`
`45
`
`US 7,881,441 B2
`
`4
`network in lieu of the Internet when a telephone number is an
`emergency number such as 911.
`Thus it is another object of at least one embodiment of the
`invention to provide a system that allows for traditional 911
`response.
`These particular objects and advantages may apply to only
`some embodiments falling within the claims and thus do not
`define the scope of the invention.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`35
`
`50
`
`DETAILED DESCRIPTION OF THE PREFERRED
`EMBODIMENT
`
`Referring to FIG. 1, in a first embodiment of the invention,
`a text captioned telephone system 10 may be implemented
`within a typical office having a PBX telephone terminal 12
`and a desk top computer 14. The desk top computer 14
`includes a display screen 16, a base unit 18 (including a
`processor, memory, disk drives and importantly a sound card)
`and a keyboard or other entry device 20.
`As is understood in the art, the PBX telephone terminal 12
`may be connected through an office wall jack 22 to a PBX
`network 24 using a proprietary communication protocol. The
`PBX network 24 communicates with a public switched tele(cid:173)
`phone network 26 that allows the PBX telephone terminal 12
`65 to connect to an external telephone terminal 28 to originate a
`call to the external telephone terminal 28 or to receive a call
`from the external telephone terminal 28.
`
`Ultratec Exhibit 1004
`Ultratec v Sorenson IP Holdings Page 10 of 14
`
`

`

`US 7,881,441 B2
`
`5
`The computer 14 may connect through an Ethernet card to
`an Ethernet network 30, passing to a local router 32 to connect
`to the Internet 34 or may use one of a number of well known
`wireless standards to connect to the local router 32.
`Referring now also to FIG. 2, a voice signal tap 36 has a
`short cable stub 38 terminating in an RJ-12 jack 40 that may
`be received in the handset jack for the main body of the PBX
`telephone terminal 12. The tap 36 also provides a receiving
`socket for the RJ-12 jack 42 associated with the handset cord
`44 so that the tap 36 may be simply interposed between the 10
`handset 46 and the main body of the PBX telephone terminal
`12 to conduct voice signals as analog audio signals there(cid:173)
`through.
`A tap line 48 extending from the tap 36 terminates in a
`subminiature telephone jack 50 that may be received by audio
`input of the sound card of the base unit 18. The tap line
`conducts a portion of the voice signals from the handset 46 to
`the sound card of the base unit 18.
`The voice signals received by the computer 14 may be
`processed by an internal program of a type well known in the
`art to encode the voice signals as VOIP signals 52 that may be
`sent over the Internet to a server 54 associated with a relay
`service 56. Programs for transmitting VOIP signal are com(cid:173)
`mercially available from Skype of Luxembourg and Vonage
`of N.J., USA. At the relay service 56, the encoded voice 25
`signals may be converted back to an analog signal for tran(cid:173)
`scription by a call assistant who produces corresponding cap(cid:173)
`tioning text 55 that may be relayed through conventional
`Internet transfer protocols back to the computer 14 where the
`text may be displayed.
`In an alternative embodiment, a headset microphone and
`earpiece (not shown) may communicate separately with left
`and right sound card charmels. This, along with echo cancel(cid:173)
`ing software in the computer 14 allows the relay service 56 to
`separate the voice of the caption user from the other party for 35
`improved transcription and anonymity.
`The relay service 56 may provide for human operators
`working with speech recognition engines to rapidly translate
`voice signals into text streams. The operation of such a relay
`is described in more detail in U.S. Pat. No. 6,567,503, 40
`assigned to the same assignee as the present invention and
`hereby incorporated by reference.
`By placing the tap 36 in the path of the handset 46, analog
`audio signals may be obtained, greatly simplifying the acqui(cid:173)
`sition of the audio signal without the need to contend with the 45
`PBX standard or the need or awkward or unfamiliar computer
`associated microphones.
`Referring now to FIG. 3, an office user wishing to avail
`themselves of text captioning may start a text-captioning
`program 58 on the computer 14 to provide a caller interface 50
`window 60 on the display screen 16. The caller interface
`window 60 may provide simple mouse or keyboard operated
`controls including a caption button 62 that may be actuated by
`the caller to begin the program's operation.
`Referring to FIG. 4, upon pressing of the caption button 62, 55
`as indicated by process block 64, the computer 14 may initiate
`an Internet connection to the relay service 56 by invoking a
`stored URL of the relay server 54. At this time, the computer
`14 may provide some information to the relay including a
`serial number of the caller to validate the caller's location for 60
`the purpose of identifying the particular public authority
`responsible for the relay call. The caller can be required to
`register the program with a relay service database and the
`serial number can be used to authorize their use of the service,
`thus limiting fraudulent use of the relay service and allowing 65
`for local 911 service by providing information about the
`geographic location of the user.
`
`6
`As indicated by process block 66, the program 58 may then
`forward the tapped voice signals as VOIP signals to the relay
`server 54, and receive text as indicated by process block 68
`which may be displayed in text box 70 of the caller interface
`window 60. The text box 70 includes conventional scroll-type
`controls 73 allowing text to be reviewed after it has been
`received. The program 58 may also provide for normal file
`operations 75 including saving of text files, e-mailing text
`files, and the like.
`Upon completion of the call, the caller may press the cap-
`tion button 62 again to disconnect the call as indicated by
`process block 71. The caption button 62 may include an
`animation visually indicating its state as being depressed or
`released or may change its label from "caption" to "end
`15 caption" indicating its changing function.
`Importantly, this system allows for convenient and intuitive
`voice communication between a PBX telephone terminal 12
`and telephone terminal 28, either for making outgoing calls or
`ingoing calls, while allowing either type of call to be cap-
`20 tioned on demand without interruption of the telephone call
`or the need for specialized telephone equipment. This system
`will also work without a PBX exchange and can work for a
`variety of different telephone types not intended for text cap-
`tioning.
`Referring now to FIG. 5, in a second embodiment, a stan-
`dard desk top computer 14 may work in conjunction with a
`standard telephone (i.e., not text captioned), in this case a cell
`telephone 72, the latter of which communicates through a
`cellular service 74 with the public switched telephone net-
`30 work 26, without a direct connection between the cell tele(cid:173)
`phone 72 and the computer 14.
`In this embodiment, the caller using the cell telephone 72
`first calls the relay service 56 as indicated by process block 76
`ofFIG. 6. The relay service 56, per standard practice, receives
`telephone calls over the public switched telephone network
`26. The relay service 56 may provide an automated answering
`system that requests a telephone number from the caller of the
`ultimate destination of the telephone call, in this case, tele(cid:173)
`phone terminal 28. As indicated by process block 78 of FIG.
`6, the caller may enter the requested telephone number by
`using the caller's keypad on the cell telephone 72.
`In the case where the relay service remains anonymous, in
`the sense that the call assistant does not speak directly with
`the caller, the answering system may provide the caller with a
`personal identification number (PIN) as indicated in process
`block 80. The relay service 56 links the caller's particular
`incoming line to the PIN in an internal database 57.
`The caller, using a conventional browser on the desk top
`computer 14, then enters the URL of the server 54 of the relay
`service 56, as indicated by process block 82 of FIG. 6. This
`may also be done before initiating the call to the relay service
`56.
`As shown in FIG. 7, the relay server 54 provides the caller
`with webpage 86 offering a text box 88 allowing entry of the
`PIN received at process block 80. The caller, by entering that
`PIN and press a captioning initiation button 90, causes the
`relay service 56 to complete the call by connecting the incom(cid:173)
`ing audio from the cell telephone 72 to the telephone terminal
`28. The relay service 56 then forwards captioning text 55 over
`the Internet 34 to the caller through a text box 92 as indicated
`by process block 94. The text box 92 may use, for example, a
`browser plug-in or instant messaging program to provide for
`a consistent updating of the text as it is received, but otherwise
`requires no specialized software on computer 14.
`Alternatively and preferably, the caller may first contact the
`webpage 86 using the computer 14 to provide the relay server
`54 with both the number the caller wishes to dial and also the
`
`Ultratec Exhibit 1004
`Ultratec v Sorenson IP Holdings Page 11 of 14
`
`

`

`US 7,881,441 B2
`
`7
`caller's own telephone number. The relay server 54 then calls
`the caller over the cell phone 72. Once a connection is estab(cid:173)
`lished with captions, the relay server 54 dials to the other
`party. This automatically links the IP address to the telephone
`connections without the need for PIN. The phone number of
`the caller may be entered