( 19 ) United States
`( 12 ) Reissued Patent
`Zhu et al .
`
`USOORE48371E
`
`US RE48,371 E
`( 10 ) Patent Number :
`( 45 ) Date of Reissued Patent :
`Dec. 29 , 2020
`
`( 54 ) MICROPHONE ARRAY SYSTEM
`( 71 ) Applicant : LI Creative Technologies , Inc. ,
`Florham Park , NJ ( US )
`( 72 ) Inventors : Manli Zhu , New City , NY ( US ) ; Qi Li ,
`New Providence , NJ ( US )
`( 73 ) Assignee : VOCALIFE LLC , Plano , TX ( US )
`( 21 ) Appl . No .: 16 / 052,623
`( 22 ) Filed :
`Aug. 2 , 2018
`Related U.S. Patent Documents
`
`Reissue of :
`8,861,756
`( 64 ) Patent No .:
`Oct. 14 , 2014
`Issued :
`13 / 049,877
`Appl . No .:
`Mar. 16 , 2011
`Filed :
`U.S. Applications :
`( 63 ) Continuation of application No. 15 / 293,626 , filed on
`Oct. 14 , 2016 , now Pat . No. Re . 47,049 , which is an
`application for the reissue of Pat . No. 8,861,756 .
`( Continued )
`
`( 51 ) Int . Cl .
`H04R 25/00
`HO3G 3/20
`
`( 2006.01 )
`( 2006.01 )
`( Continued )
`
`( 52 ) U.S. CI .
`GOIS 3/8055 ( 2013.01 ) ; GOIS 3/801
`CPC
`( 2013.01 ) ; GOIS 5/22 ( 2013.01 ) ; H04R 1/406
`( 2013.01 ) ;
`
`( Continued )
`( 58 ) Field of Classification Search
`CPC .. HO4R 3/005 ; H04R 1/406 ; H04R 2201/401 ;
`HO4R 2201/403 ; G01S 3/801 ; G01S
`8/055 ; G01S 5/22 ; H04M 3/568
`( Continued )
`
`( 56 )
`
`5,315,562 A *
`5,825,898 A
`
`References Cited
`U.S. PATENT DOCUMENTS
`5/1994 Bradley et al .
`10/1998 Marash
`( Continued )
`FOREIGN PATENT DOCUMENTS
`
`EP
`KR
`
`6/2005
`1538867 A1
`20090128221 A 12/2009
`( Continued )
`
`367/89
`
`OTHER PUBLICATIONS
`US 9,711,140 B2 , 07/2017 , Ayrapetian et al . ( withdrawn )
`( Continued )
`Primary Examiner Ovidio Escalante
`( 74 ) Attorney , Agent , or Firm — Ashok Tankha
`( 57 )
`ABSTRACT
`A method and system for enhancing a target sound signal
`from multiple sound signals is provided . An array of an
`arbitrary number of sound sensors positioned in an arbitrary
`configuration receives the sound signals from multiple dis
`parate sources . The sound signals comprise the target sound
`signal from a target sound source , and ambient noise signals .
`A sound source localization unit , an adaptive beamforming
`unit , and a noise reduction unit are in operative communi
`cation with the array of sound sensors . The sound source
`localization unit estimates a spatial location of the target
`sound signal from the received sound signals . The adaptive
`beamforming unit performs adaptive beamforming by steer
`ing a directivity pattern of the array of sound sensors in a
`direction of the spatial location of the target sound signal ,
`thereby enhancing the target sound signal and partially
`suppressing the ambient noise signals , which are further
`suppressed by the noise reduction unit .
`20 Claims , 34 Drawing Sheets
`
`Page 1 of 52
`
`GOOGLE EXHIBIT 1001
`
`

`

`US RE48,371 E
`Page 2
`
`( 2006.01 )
`( 2006.01 )
`( 2006.01 )
`( 2006.01 )
`( 2006.01 )
`( 2006.01 )
`
`( 56 )
`
`Related U.S. Application Data
`( 60 ) Provisional application No. 61 / 403,952 , filed on Sep.
`24 , 2010 .
`( 51 ) Int . Cl .
`GOIS 3/805
`GOIS 3/801
`GOIS 5/22
`H04R 1/40
`H04R 3/00
`H04M 3/56
`( 52 ) U.S. Ci .
`H04R 3/005 ( 2013.01 ) ; H04M 3/568
`CPC
`( 2013.01 ) ; H04R 2201/401 ( 2013.01 ) ; H04R
`2201/403 ( 2013.01 )
`( 58 ) Field of Classification Search
`USPC
`381/300 , 57
`See application file for complete search history .
`References Cited
`U.S. PATENT DOCUMENTS
`6,198,693 B1
`3/2001 Marash
`6,236,862 B1 *
`5/2001 Erten
`7,039,199 B2
`7,068,801 B1
`7,970,151 B2
`8,855,295 B1
`8,885,815 B1
`8,953,777 B1
`8,983,057 B1
`9,116,962 B1
`9,229,526 B1
`9,319,782 B1
`9,319,783 B1
`9,332,167 B1
`9,354,731 B1
`9,363,616 B1
`9,373,338 B1
`9,390,723 B1
`9,423,886 B1
`9,431,982 B1
`9,432,768 B1
`9,432,769 B1
`9,456,276 B1
`9,473,646 B1
`9,516,410 B1
`9,521,249 B1
`9,589,575 B1
`9,591,404 B1
`9,614,486 B1
`9,653,060 B1
`9,658,738 B1
`9,659,555 B1
`9,661,438 B1
`9,677,986 B1
`9,678,559 B1
`9,689,960 B1
`9,704,478 B1
`9,734,845 B1
`9,747,899 B2
`9,747,920 B2
`9,754,605 B1
`9,767,828 B1
`9,818,425 B1
`9,820,036 B1
`9,837,099 B1
`9,918,163 B1
`9,940,949 B1
`9,966,059 B1
`9,967,661 B1
`9,973,848 B2
`
`5/2006 Rui
`6/2006 Stinson et al .
`6/2011 Oxford et al .
`10/2014 Chhetri et al .
`11/2014 Velusamy et al .
`2/2015 Chhetri
`3/2015 Deng et al .
`8/2015 Pance
`1/2016 Neglur et al .
`4/2016 Crump et al .
`4/2016 Barton et al .
`5/2016 Pance
`5/2016 Pance et al .
`6/2016 Chu et al .
`6/2016 Gopalan et al .
`7/2016 McDonough , Jr. et al .
`8/2016 Neglur et al .
`8/2016 Yang et al .
`8/2016 O'Neill et al .
`8/2016 Sundaram et al .
`9/2016 Chhetri
`10/2016 Chhetri
`12/2016 Ayrapetian et al .
`12/2016 Chhetri
`3/2017 Ayrapetian et al .
`3/2017 Chhetri
`4/2017 Yang et al .
`5/2017 Hilmes et al .
`5/2017 Park et al .
`5/2017 Hilmes et al .
`5/2017 Yang et al .
`6/2017 Baldwin et al .
`6/2017 Devries et al .
`6/2017 Barton et al .
`7/2017 Vitaladevuni et al .
`8/2017 Liu et al .
`8/2017 Pogue et al .
`8/2017 Ayrapetian et al .
`9/2017 Chhetri
`9/2017 Velusamy et al .
`11/2017 Ayrapetian et al .
`11/2017 Tritschler et al .
`12/2017 Sundaram et al .
`3/2018 Ayrapetian et al .
`4/2018 Vitaladevuni et al .
`5/2018 Ayrapetian et al .
`5/2018 Hilmes et al .
`5/2018 Chhetri et al .
`
`GO1S 3/00
`370/342
`
`9,973,849 B1
`9,978,387 B1
`9,997,151 B1
`10,062,372 B1
`10,109,294 B1
`10,147,439 B1
`10,147,441 B1
`10,229,698 B1
`10,237,647 B1
`10,242,695 B1
`10,244,313 B1
`10,304,475 B1
`2003/0204397 A1 *
`2004/0071284 A1
`2004/0161121 A1 *
`2006/0153360 A1
`2006/0245601 A1
`2006/0269080 A1
`2007/0055505 A1
`2007/0076898 Al
`2008/0112574 Al
`2008/0181430 Al
`2008/0232607 Al
`2009/0067642 Al
`2009/0073040 A1
`2009/0141907 A1 *
`2009/0279714 Al
`2009/0304200 A1
`2010/0150364 Al
`2012/0327115 A1
`2013/0265276 A1
`2015/0006176 Al
`2017/0178662 A1
`2018/0130468 Al
`2018/0182387 Al
`
`5/2018 Zhang et al .
`5/2018 Pogue et al .
`6/2018 Ayrapetian et al .
`8/2018 Barton et al .
`10/2018 Ayrapetian et al .
`12/2018 Kristjansson et al .
`12/2018 Pogue et al .
`3/2019 Chhetri
`3/2019 Chhetri
`3/2019 Velusamy et al .
`3/2019 O'Neill et al .
`5/2019 Wang et al .
`10/2003 Amiri et al .
`4/2004 Abutalebi et al .
`8/2004 Chol et al .
`7/2006 Kellermann et al .
`11/2006 Michaud et al .
`11/2006 Oxford et al .
`3/2007 Doclo et al .
`4/2007 Sarroukh et al .
`5/2008 Brennan et al .
`7/2008 Zhang et al .
`9/2008 Tashev et al .
`3/2009 Buck et al .
`3/2009 Sugiyama
`6/2009 Kim et al .
`11/2009 Kim et al .
`12/2009 Kim et al .
`6/2010 Buck et al .
`12/2012 Chhetri et al .
`10/2013 Obeidat et al .
`1/2015 Pogue et al .
`6/2017 Ayrapetian et al .
`5/2018 Pogue et al .
`6/2018 Chua et al .
`
`704/231
`
`381/92
`
`381 / 71.7
`
`FOREIGN PATENT DOCUMENTS
`
`RS
`SG
`WO
`WO
`WO
`WO
`WO
`
`WO2008041878 A2
`2006006935 A1
`2008041878 A2
`WO2008041878 A2
`2013155098 A1
`2017105998 Al
`2018118895 A2
`
`4/2008
`1/2006
`4/2008
`4/2008
`10/2013
`6/2017
`6/2018
`
`OTHER PUBLICATIONS
`Qi ( Peter ) Li , “ A Portable USB - Based Mirophone Array Device For
`Robust Speech Recognition ” , “ 2009 IEEE International Conference
`on Acoustics , Speech , and Signal Processing ” , Apr. 19-24 , 2009 ,
`Seven pages .
`Osamu Hoshuyama , Akihiko Sugiyama , and Akihiro Hirano , A
`Robust Adaptive Beamformer for Microphone Arrays with a Block
`ing Matrix Using Constrained Adaptive Filters , IEEE Transactions
`on Signal Processing , vol . 47 , No. 10 , Oct. 1999 , 8 Pgs .
`Cha Zhang , Dinei Florencio , Demba E. Ba , and Zhengyou Zhang ,
`Maximum Likelihood Sound Source Localization and Beamform
`ing for Directional Microphone Arrays in Distributed Meetings ,
`IEEE Transactions on Multimedia , vol . 10 , No. 3 , Apr. 2008 , 11
`pages
`Afsaneh Asaei , Mohammad Javad Taghizadeh , Marjan Bahrololum ,
`Mohammed Ghanbari , Verified speaker localization utiiizing voic
`ing level in split - bands Signal Processing 89 ( 2009 ) 1038-1049 , 12
`pages .
`Scott Matthew Griebel , A Microphone Array System for Speech
`Source Localization , Denoising and Dereverberation , Thesis , The
`Division of Engineering and Applied Sciences , Harvard
`University , Cambridge , Massachusetts , Apr. 2002 163 pages .
`Cha Zhang , Dinei Florencio , Demba E. Ba , Zhengyou Zhang ,
`Maximum Likelihood Sound Source Localization and Beamform
`ing for Directional Microphone Arrays in Distributed Meetings ,
`Journal of Latex Class files , vol . 6 , No. 1 , Jan. 2007 , 10 pages .
`
`Page 2 of 52
`
`

`

`US RE48,371 E
`Page 3
`
`( 56 )
`
`References Cited
`OTHER PUBLICATIONS
`Michael Brandstein , Darren Ward Microphone Arrays , Signal Pro
`cessing
`Techniques
`and Applications
`Springer
`Verlag , Berlin , Heidelberg , New York in 2001 , 401 pages .
`Qi ( Peter ) Li , Manli Zhu , and Wei Li , “ A Portable USB - Based
`Mirophone Array Device For Robust Speech Recognition ” , “ 2009
`IEEE International Conference on Acoustics , Speech , and Signal
`Processing ” , Apr. 19-24 , 2009 , 7 pages .
`Ivan J. Tashev , Sound Capture and Processing Practical Approaches ,
`2009 Wiley Publisher , 196 pages .
`Manli Zhu , Qi ( Peter ) Li , Joshua J. Hajicek Circular and Linear
`Microphone Arrays for Robust Speech Recognition and Conference
`Phone , ICASSP 2009 Thursday , Apr. 23 , 2009 , 1 page .
`Harry L. Van Trees Arrays and Spatial Filters , Optimum Array
`Processing : Part IV of Detection , Estimation , and Modulation
`Theory , John Wiley & Sons , Inc. , 73 pages .
`Introducing First Low - cost , Light - weight , and Portable USB Array
`Microphone for Consumer Market , Li Creative Technologies , Inc. ,
`Feb. 2. 2010 , 1 page .
`Crispmic USB - Based Microphone Array for Laptops and PCs LI
`Creative Technologies , Inc. 2 pages .
`Matthias Wolfel and John McDonough Distant Speech Recognition
`A John Wiley and Sons , Ltd. Publication , 2009 , 592 pages .
`Darpa 172 Phase I Selections from the 07.2 Solicitation , 69 pages .
`Doh H. Johnson and Dan E. Dudgeon Array Signal Processing :
`Concepts and Techniques , 1993 Prentice Hall Signal Processing
`Series , 554 pages .
`Group Videoconferencing Systems : Video Made Easy HD5000
`Series , Multimedia Workgroup Conferencing System , Installation &
`Setup Guide , 70 pages .
`Andrea DA - 350 Microphone Performance , 1 page .
`VCON — Solutions — Videoconferencing Group Video Products
`MediaConnect 9000 Sep. 21 , 2003 - Feb . 8 , 2004 , 1 page .
`DA - 350 Hands Free Linear Array Microphone , 1 page .
`DA - 350 Auto Array Feb. 25 , 2006 - Jun . 29 , 2016 ,
`1 page .
`VCON Group Videoconferencing Systems HD4000 Software - only
`Multimedia Videoconferencing Version 3.5 , 50 pages .
`MediaConnect 9000 A workgroup conferencing system for medium
`and large room environments , 1 page .
`VCON Group Videoconferencing Systems HD5000 Series Rol
`labout and Compact Systems Installation & Setup Guide , 74 pages .
`Baruch Berdugo , Miriam A. Doron , Judith Rosenhouse , Haim
`Azhari On direction finding of an emitting source from time delays
`33 pages .
`
`EchoStop , Digital Noise Reduction Technology , 1 page .
`VCON — Hardware Addons — Introducing VoiceFinder , Sep. 23 , 2003
`Feb. 25 , 2004 , 2 pages .
`Digital Super Directional Array ( DSDA® 2.0 ) Far - Field Micro
`phone Technology , 1 page .
`Harry L. Van Trees Optimum Array Processing , Part IV of Detec
`tion , Estimation , and Modulation Theory A John Wiley & Sons ,
`Inc. , Publication , 192 pages .
`Charles H. Knapp and G. Clifford Carter The Generalized Corre
`lation Method for Estimation of Time Delay IEEE transactions on
`acoustics , speech , and signal processing , Vol , ASSP - 24 , No. 4 , Aug.
`1976 , 8 pages .
`Pure Audio 2.0 Noise Reduction Algorithm , 1 page .
`Andrea's Technologies Overview Oct. 21 , 2001 - Sep . 11 , 2011 , 4
`pages .
`VCON — Hardware Addons — VoiceFinder Sep. 23 , 2003 - Feb . 8 ,
`2004 , 1 page .
`Joseph Marash DSDA , Andrea Electronics Corporation Technology ,
`4 pages .
`Joseph Hector Dibiase , A High - Accuracy , Low - Latency Technique
`for Talker Localization in Reverberant Environments Using Micro
`phone Arrays , Thesis , Division of Engineering at Brown University ,
`Providence , Rhode Island , May 2000 , 122 pages .
`Qi Li , Manli Zhu , Wei Li A portable USB - based microphone array
`device for robustn speech recognition IEEE International Confer
`ence on Acoustics , Speech and Signal Processing , Apr. 19-24 , 2009 ,
`2 pages .
`Qi ( Peter ) Li , Manli Zhu , and Wei Li A Portable Usb - Based
`Mirophone Array Device For Robust Speech Recognition IEEE
`International Conference on Acoustics , Speech and Signal Process
`ing Proceedings , Apr. 19-24 , 2009 , 7 pages .
`John Mcdonough , Kenichi Kumatani , Matthias Wolfel , Tobias
`Gehrig , Emilian Stoimenov , Uwe Mayer , Stefan Schacht , and Dietrich
`Klakow To Separate Speech ! A System for Recognizing Simulta
`neous Speech , Jun . 2007 , 13 pages .
`Dmitry N. Zotkin , Ramani Duraiswami Accelerated Speech Source
`Localization via a Hierarchical Search of Steered Response Power
`University of Maryland , MD , USA , 20 pages .
`Jacek Dmochowski , Jacob Benesty , Sofiane Affes Direction of
`Arrival Estimation Using the Parameterized Spatial Correlation
`Matrix , IEEE Transaction on Audio , Speech , and Language Pro
`cessing , vol . 15 , No. 4 , May 2007 .
`* cited by examiner
`
`Page 3 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 1 of 34
`
`US RE48,371 E
`
`Page 4 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 2 of 34
`
`US RE48,371 E
`
`Page 5 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 3 of 34
`
`US RE48,371 E
`
`***
`
`mind
`
`1
`
`Page 6 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 4 of 34
`
`US RE48,371 E
`
`1 ***
`3:41
`
`Page 7 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 5 of 34
`
`US RE48,371 E
`
`Page 8 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 6 of 34
`
`US RE48,371 E
`
`Page 9 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 7 of 34
`
`US RE48,371 E
`
`w
`
`O
`
`S2
`
`Page 10 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 8 of 34
`
`US RE48,371 E
`
`FOR DIRECTION I ( 0 < i < 360 ) , CALCULATE THE DELAY
`Dir BETWEEN THE I ' " PAIR OF THE SOUND SENSORS ( t = 1 :
`ALL PAIRS )
`
`CALCULATE THE CORRELATION VALUE corr ( Dit )
`BETWEEN THE T'H PAIR OF THE SOUND SENSORS
`CORRESPONDING TO THE DELAY OF Dit
`
`FOR THE DIRECTION I ( 0 < i < 360 ) ,
`ALLPAIR
`CORR ;
`corr ( Dit )
`
`t = 1
`
`THE TARGET SOUND SIGNAL COMES FROM DIRECTION
`S
`
`O < i < 360
`
`801
`
`802
`
`803
`
`804
`
`FIG . 8
`
`Page 11 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 9 of 34
`
`US RE48,371 E
`
`30
`
`300
`
`Page 12 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 10 of 34
`
`US RE48,371 E
`
`www
`
`* **
`
`M.TR
`
`Page 13 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 11 of 34
`
`US RE48,371 E
`
`3
`
`motora
`
`Page 14 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 12 of 34
`
`US RE48,371 E
`
`Page 15 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 13 of 34
`
`US RE48,371 E
`
`Page 16 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 14 of 34
`
`US RE48,371 E
`
`ee
`
`Q
`
`Page 17 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 15 of 34
`
`US RE48,371 E
`
`21
`
`Page 18 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 16 of 34
`
`US RE48,371 E
`
`Page 19 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 17 of 34
`
`US RE48,371 E
`
`1502
`
`24
`
`M
`
`}
`
`1
`
`2300
`
`Page 20 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 18 of 34
`
`US RE48,371 E
`
`1.4
`1.2
`1
`0.81
`0.6
`0.4
`0.2
`0 .
`5000
`4000
`3000
`2000
`1000
`
`1.4
`1.2
`1
`0.8
`0.6
`0.4
`0.2
`0
`5000
`4000
`3000
`2000
`1000
`
`1
`
`200
`150
`
`100
`
`50
`
`0
`
`-100 -50
`0 200-150
`FIG . 16C
`
`1
`
`1
`
`1
`
`1
`
`200
`
`150
`50 100
`0
`
`-50
`0-200 -150-100
`
`FIG . 16D
`
`Page 21 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 19 of 34
`
`US RE48,371 E
`
`?
`*
`*
`
`*
`
`*
`*
`*
`
`$
`
`.
`
`?
`?
`
`*
`
`*
`
`>
`
`.
`
`.
`
`3
`**
`
`"
`
`*
`
`id
`
`$
`
`*
`
`** 3
`
`***
`
`1
`
`3
`
`*
`
`2
`**
`**
`
`.
`
`12
`
`. :
`
`3
`
`?
`1 .
`
`*
`-
`
`2
`
`+
`
`+
`
`5
`
`3
`
`**
`
`14
`+
`
`Page 22 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 20 of 34
`
`US RE48,371 E
`
`it :
`
`. :
`
`+
`
`3
`
`*
`$
`
`+
`
`1
`
`2
`*
`
`1
`
`3
`
`2 .
`
`?
`
`3
`
`$
`
`2 .
`
`+
`
`#
`
`***
`
`$
`
`3 .
`
`3
`
`**
`
`*
`
`!
`
`3 .
`
`*
`
`3
`1 .
`
`23
`
`: .
`
`.
`2 :
`
`$
`
`>
`
`2 .
`
`$
`
`5
`
`.
`
`*
`3
`
`2
`
`33
`
`3
`
`**
`
`Page 23 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 21 of 34
`
`US RE48,371 E
`
`*
`
`46
`
`+
`
`!
`
`>
`
`3
`
`1 :
`7
`
`2
`
`2
`
`?
`
`2
`2
`
`*
`5
`
`.
`
`.
`
`3
`
`.
`
`4 .
`
`19
`??? ?? ??
`
`.
`
`25
`1 .
`
`.
`
`***
`
`1 .
`
`1 .
`
`.
`
`**
`
`****
`
`**
`
`sos
`
`>
`
`$
`
`**
`
`2 .
`*
`
`*
`
`Page 24 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 22 of 34
`
`US RE48,371 E
`
`3
`
`.
`
`4
`.
`
`**
`**
`3
`
`>
`
`*
`
`2
`
`?
`
`3
`+
`
`*
`
`1 :
`
`3
`
`+
`
`***
`
`1
`
`24
`
`2
`
`3
`
`*
`3
`
`**
`>
`
`2
`
`.
`
`*
`3 .
`
`1
`
`2
`
`-
`
`ii
`-
`
`3
`1
`
`444444447732
`
`*
`
`i
`$
`$
`$
`
`$
`$
`$
`
`$
`
`$
`
`$
`$
`
`$
`
`#
`
`.
`1
`3
`
`3
`
`Page 25 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 23 of 34
`
`US RE48,371 E
`
`180 °
`M
`
`301
`
`201
`
`302
`
`301
`M3
`-90 °
`
`90 °
`M
`
`301
`
`301
`
`0 ° Mo
`
`FIG . 17A
`
`1.4 :
`1.2
`1
`0.8
`0.6
`0.41
`0.2
`0
`4000
`
`1
`
`200
`100 150
`50
`
`3000
`
`2000
`
`1000
`
`0
`
`-200
`
`0
`-50
`- 150 -100
`
`FIG . 17B
`
`Page 26 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 24 of 34
`
`US RE48,371 E
`
`8
`& N
`S
`NSEN
`N
`& N
`
`83
`N & I
`6:38
`
`SIN
`
`4000
`3500
`
`30001 -
`
`2500
`2000 -
`
`1500
`
`1000
`
`500
`1
`0
`-200 -150 -100 -50
`
`1
`
`1
`0
`
`1
`
`50 100 150 200
`
`1.4
`1.2
`1
`0.84
`0.6
`0.44
`0.2
`0 .
`4000
`3000
`
`FIG . 170
`
`V
`
`2000
`
`1000
`
`0 -200
`
`-100
`
`200
`
`100
`
`0
`
`FIG . 17D
`
`Page 27 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 25 of 34
`
`US RE48,371 E
`
`?
`1
`
`- - -
`
`1
`
`4000
`3500 --
`
`3000
`
`2500 ---
`
`2000
`
`1500
`
`1000
`
`500
`
`ST
`9338
`NEEN
`SURES
`SEEN
`SIREN
`REN
`NLBEN
`USNESS
`INTENN
`SRESN
`NOEN
`SNIEREN
`NREN
`SOTON
`NON
`
`EN
`SOIRESS
`NOTEIN
`SOLSEN
`NE
`B :
`1388
`STESNIS
`ONSENS
`WEESNS
`NOSS
`
`S88
`
`ILAN
`N &
`Y8888ILONS
`RSSIONS
`US8XIONS
`38:00 SS
`&& TUESNS
`
`1
`1
`
`0
`-200 -150 -100 -50
`
`0
`
`50
`
`100 150
`
`200
`
`FIG . 17E
`
`Page 28 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 26 of 34
`
`US RE48,371 E
`
`1
`1
`
`2000
`
`1000
`
`200
`100 150
`50
`100 500
`0
`150
`-200
`
`FIG . 17F
`
`1.4
`1.2
`1
`0.8
`0.6-7
`0.44
`0.2
`al .
`4000
`3000
`
`4000
`
`3500
`
`3000
`
`2500
`
`2000
`
`1500
`
`1000
`
`500
`
`SOINS
`SSRUSS
`SOINS
`SORUNS
`NBIES
`883
`
`NI
`BRES
`SS8 : BIS
`ONESID
`N823
`
`83
`STEN
`
`NISSN
`* N3R8ION
`NISESNY
`XNXIONS
`SI8190S
`UNEXESNI
`NON
`NITION
`NEREGNS
`UNBONS :
`ONIONS :
`NEGOS
`NOS :
`N2152
`NEŠION :
`
`I
`
`0
`
`50
`
`100 150 200
`
`I
`
`FIG . 17G
`
`I
`
`I
`
`0
`-200 -150 -100 -50
`
`Page 29 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 27 of 34
`
`US RE48,371 E
`
`S
`
`3
`
`.
`
`3
`
`?
`
`?
`
`X
`
`$
`
`*
`
`Page 30 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 28 of 34
`
`US RE48,371 E
`
`1 .
`
`Page 31 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 29 of 34
`
`US RE48,371 E
`
`Page 32 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 30 of 34
`
`US RE48,371 E
`
`CM
`
`********
`
`
`
`XXX ********
`
`3
`
`Page 33 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 31 of 34
`
`US RE48,371 E
`
`?
`
`Page 34 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 32 of 34
`
`US RE48,371 E
`
`1
`
`Page 35 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 33 of 34
`
`US RE48,371 E
`
`ve
`14
`
`.
`16
`
`Page 36 of 52
`
`

`

`U.S. Patent
`
`Dec. 29 , 2020
`
`Sheet 34 of 34
`
`US RE48,371 E
`
`M
`
`..
`
`Page 37 of 52
`
`

`

`US RE48,371 E
`
`5
`
`1
`MICROPHONE ARRAY SYSTEM
`
`2
`system that enhances acoustics of a desired sound signal
`while suppressing ambient noise signals .
`Matter enclosed in heavy brackets [ ] appears in the
`SUMMARY OF THE INVENTION
`original patent but forms no part of this reissue specifica
`tion ; matter printed in italics indicates the additions
`This summary is provided to introduce a selection of
`made by reissue ; a claim printed with strikethrough
`concepts in a simplified form that are further described in the
`indicates that the claim was canceled , disclaimed , or held
`detailed description of the invention . This summary is not
`invalid by a prior post - patent action or proceeding .
`intended to identify key or essential inventive concepts of
`10 the claimed subject matter , nor is it intended for determining
`the scope of the claimed subject matter .
`CROSS REFERENCE TO RELATED
`The method and system disclosed herein addresses the
`APPLICATIONS
`above stated need for enhancing acoustics of a target sound
`signal received from a target sound source , while suppress
`This application is a continuation reissue application of
`patent application Ser . No. 15 / 293,626 titled " Microphone 15 ing ambient noise signals . As used herein , the term “ target
`Array System ” , filed on Oct. 14 , 2016 in the United States
`sound signal ” refers to a sound signal from a desired or
`Patent and Trademark Office , which is a re - issue application
`target sound source , for example , a person's speech that
`of U.S. patent application Ser . No. 13 / 049,877 titled " Micro-
`needs to be enhanced . A microphone array system compris
`phone Array System " , filed on Mar. 16 , 2011 in the United
`ing an array of sound sensors positioned in an arbitrary
`States Patent and Trademark Office ( now U.S. Pat . No. 20 configuration , a sound source localization unit , an adaptive
`8,861,756 ) , which claims the benefit of provisional patent
`beamforming unit , and a noise reduction unit , is provided .
`application No. 61 / 403,952 titled “ Microphone array design
`The sound source localization unit , the adaptive beamform
`and implementation for telecommunications and handheld
`ing unit , and the noise reduction unit are in operative
`devices ” , filed on Sep. 24 , 2010 in the United States Patent
`communication with the array of sound sensors . The array of
`25 sound sensors is , for example , a linear array of sound
`and Trademark Office .
`The specification of the above referenced patent applica-
`sensors , a circular array of sound sensors , or an arbitrarily
`distributed coplanar array of sound sensors . The array of
`tion is incorporated herein by reference in its entirety .
`sound sensors herein referred to as a “ microphone array ”
`receives sound signals from multiple disparate sound
`BACKGROUND
`30 sources . The method disclosed herein can be applied on a
`microphone array with an arbitrary number of sound sensors
`Microphones constitute an important element in today's
`having , for example , an arbitrary two dimensional ( 2D )
`speech acquisition devices . Currently , most of the hands - free
`speech acquisition devices , for example , mobile devices ,
`configuration . The sound signals received by the sound
`lapels , headsets , etc. , convert sound into electrical signals by
`sensors in the microphone array comprise the target sound
`using a microphone embedded within the speech acquisition 35 signal from the target sound source among the disparate
`device . However , the paradigm of a single microphone often
`sound sources , and ambient noise signals .
`does not work effectively because the microphone picks up
`The sound source localization unit estimates a spatial
`many ambient noise signals in addition to the desired sound ,
`location of the target sound signal from the received sound
`specifically when the distance between a user and the
`signals , for example , using a steered response power - phase
`microphone is more than a few inches . Therefore , there is a 40 transform . The adaptive beamforming unit performs adap
`need for a microphone system that operates under a variety
`tive beamforming for steering a directivity pattern of the
`of different ambient noise conditions and that places fewer
`microphone array in a direction of the spatial location of the
`constraints on the user with respect to the microphone ,
`target sound signal . The adaptive beamforming unit thereby
`thereby eliminating the need to wear the microphone or be
`enhances the target sound signal from the target sound
`in close proximity to the microphone .
`45 source and partially suppresses the ambient noise signals .
`To mitigate the drawbacks of the single microphone
`The noise reduction unit suppresses the ambient noise
`system , there is a need for a microphone array that achieves
`signals for further enhancing the target sound signal
`directional gain in a preferred spatial direction while sup-
`received from the target sound source .
`pressing ambient noise from other directions . Conventional
`In an embodiment where the target sound source that
`microphone arrays include arrays that are typically devel- 50 emits the target sound signal is in a two dimensional plane ,
`oped for applications such as radar and sonar , but are
`a delay between each of the sound sensors and an origin of
`generally not suitable for hands - free or handheld speech
`the microphone array is determined as a function of distance
`acquisition devices . The main reason is that the desired
`between each of the sound sensors and the origin , a pre
`sound signal has an extremely wide bandwidth relative to its
`defined angle between each of the sound sensors and a
`center frequency , thereby rendering conventional narrow- 55 reference axis , and an azimuth angle between the reference
`band techniques employed in the conventional microphone
`axis and the target sound signal . In another embodiment
`arrays unsuitable . In order to cater to such broadband speech
`where the target sound source that emits the target sound
`applications , the array size needs to be vastly increased ,
`signal is in a three dimensional plane , the delay between
`making the conventional microphone arrays large and bulky ,
`each of the sound sensors and the origin of the microphone
`and precluding the conventional microphone arrays from 60 array is determined as a function of distance between each
`having broader applications , for example , in mobile and
`of the sound sensors and the origin , a predefined angle
`handheld communication devices . There is a need for a
`between each of the sound sensors and a first reference axis ,
`microphone array system that provides an effective response
`an elevation angle between a second reference axis and the
`over a wide spectrum of frequencies while being unobtru-
`target sound signal , and an azimuth angle between the first
`65 reference axis and the target sound signal . This method of
`sive in terms of size .
`Hence , there is a long felt but unresolved need for a
`determining the delay enables beamforming for arbitrary
`broadband microphone array and broadband beamforming
`numbers of sound sensors and multiple arbitrary microphone
`
`Page 38 of 52
`
`

`

`US RE48,371 E
`
`10
`
`15
`
`3
`4
`invention are shown in the drawings . However , the invention
`array configurations . The delay is determined , for example ,
`is not limited to the specific methods and instrumentalities
`in terms of number of samples . Once the delay is deter-
`disclosed herein .
`mined , the microphone array can be aligned to enhance the
`target sound signal from a specific direction .
`FIG . 1 illustrates a method for enhancing a target sound
`The adaptive beamforming unit comprises a fixed beam- 5 signal from multiple sound signals .
`former , a blocking matrix , and an adaptive filter . The fixed
`FIG . 2 illustrates a system for enhancing a target sound
`beamformer steers the directivity pattern of the microphone
`signal from multiple sound signals .
`array in the direction of the spatial location of the target
`FIG . 3 exemplarily illustrates a microphone array con
`sound signal from the target sound source for enhancing the
`figuration showing a microphone array having N sound
`target sound signal , when the target sound source is in
`sensors arbitrarily distributed on a circle .
`motion . The blocking matrix feeds the ambient noise signals
`FIG . 4 exemplarily illustrates a graphical representation
`to the adaptive filter by blocking the target sound signal from
`of a filter - and - sum beamforming algorithm for determining
`the target sound source . The adaptive filter adaptively filters
`output of the microphone array having N sound sensors .
`the ambient noise signals in response to detecting the
`FIG . 5 exemplarily illustrates distances between an origin
`presence or absence of the target sound signal in the sound
`of the microphone array and sound sensor My and sound
`signals received from the disparate sound sources . The fixed
`sensor Mz in the circular microphone array configuration ,
`beamformer performs fixed beamforming , for example , by
`when the target sound signal is at an angle 8 from the Y - axis .
`filtering and summing output sound signals from the sound
`FIG . 6A exemplarily illustrates a table showing the dis
`20 tance between each sound sensor in a circular microphone
`sensors .
`In an embodiment , the adaptive filtering comprises sub-
`array configuration from the origin of the microphone array ,
`when the target sound source is in the same plane as that of
`band adaptive filtering . The adaptive filter comprises an
`analysis filter bank , an adaptive filter matrix , and a synthesis
`the microphone array .
`filter bank . The analysis filter bank splits the enhanced target
`FIG . 6B exemplarily illustrates a table showing the rela
`sound signal from the fixed beamformer and the ambient 25 tionship of the position of each sound sensor in the circular
`noise signals from the blocking matrix into multiple fre-
`microphone array configuration and its distance to the origin
`of the microphone array , when the target sound source is in
`quency sub - bands . The adaptive filter matrix adaptively
`the same plane as that of the microphone array .
`filters the ambient noise signals in each of the frequency
`FIG . 7A exemplarily illustrates a graphical representation
`sub - bands in response to detecting the presence or absence
`of the target sound signal in the sound signals received from 30 of a microphone array , when the target sound source is in a
`three dimensional plane .
`the disparate sound sources . The synthesis filter bank syn
`FIG . 7B exemplarily illustrates a table showing delay
`thesizes
`full - band sound signal using the frequency sub
`between each sound sensor in a circular microphone array
`bands of the enhanced target sound signal . In an embodi
`configuration and the origin of the microphone array , when
`ment , the adaptive beamforming unit further comprises an 35 the target sound source is in a three dimensional plane .
`adaptation control unit for detecting the presence of the
`FIG . 7C exemplarily illustrates a three dimensional work
`target sound signal and adjusting a step size for the adaptive
`ing space of the microphone array , where the target sound
`filtering in response to detecting the presence or the absence
`signal is incident at an elevation angle Y < 22
`of the target sound signal in the sound signals received from
`FIG . 8 exemplarily illustrates a method for estimating a
`the disparate sound sources .
`40 spatial location of the target sound signal from the target
`The noise reduction unit suppresses the ambient noise
`sound source by a sound source localization unit using a
`signals for further enhancing the target sound signal from the
`steered response power - phase transform .
`target sound source . The noise reduction unit performs noise
`FIG . 9A exemplarily illustrates a graph showing the value
`reduction , for example , by using a Wiener - filter based noise
`of the steered response power - phase transform for every 10 ° .
`reduction algorithm , a spectral subtraction noise reduction 45
`FIG . 9B exemplarily illustrates a graph representing the
`algorithm , an auditory transform based noise reduction
`estimated target sound signal from the target sound source .
`algorithm , or a model based noise reduction algorithm . The
`FIG . 10 exemplarily illustrates a system for performing
`noise reduction unit performs noise reduction in multiple
`adaptive beamforming by an adaptive beamforming unit .
`frequency sub - bands employed for sub - band adaptive beam-
`FIG . 11 exemplarily illustrates a system for sub - band
`forming by the analysis filter bank of the adaptive beam- 50 adaptive filtering .
`FIG . 12 exemplarily illustrates a graphical representation
`forming unit .
`The microphone array system disclosed herein compris-
`showing the performance of a perfect reconstruction filter
`ing the microphone array with an arbitrary number of sound
`bank .
`sensors positioned in arbitrary configurations can be imple-
`FIG . 13 exemplarily illustrates a block diagram of a noise
`mented in handheld devices , for example , the iPad® of 55 reduction unit that performs noise reduction using a Wiener
`filter based noise reduction algorithm .
`Apple Inc. , the iPhone® of Apple Inc. , smart phones , tablet
`computers , laptop computers , etc. The microphone array
`FIG . 14 exemplarily illustrates a hardware implementa
`tion of the microphone array system .
`system disclosed herein can furthe