USOO886 1756B2
`
`(12) United States Patent
`Zhu et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,861,756 B2
`Oct. 14, 2014
`
`(54) MICROPHONE ARRAY SYSTEM
`(75) Inventors: Manli Zhu, Pearl River, NY (US); Qi
`Li, New Providence, NJ (US)
`(73) Assignee: LI Creative Technologies, Inc., Florham
`Park, NJ (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 794 days.
`(21) Appl. No.: 13/049,877
`
`(*) Notice:
`
`(22) Filed:
`
`Mar 16, 2011
`
`(65)
`
`Prior Publication Data
`US 2012/007631.6 A1
`Mar. 29, 2012
`
`Related U.S. Application Data
`(60) Provisional application No. 61/403,952, filed on Sep.
`24, 2010.
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`
`(51) Int. Cl.
`H04R 25/00
`HO3G 3/20
`H04R 3/00
`GDIS5/22
`GOIS3/80
`GOIS3/805
`H04R L/40
`HO4M 3/56
`(52) U.S. Cl.
`CPC .............. G0IS3/8055 (2013.01); H04M 3/568
`(2013.01); H04R3/005 (2013.01); G0IS5/22
`(2013.01); H04R 220 1/403 (2013.01); G0IS
`3/801 (2013.01); H04R 220 1/401 (2013.01):
`H04R I/406 (2013.01)
`USPC ............................................. 381/300; 381/57
`(58) Field of Classification Search
`CPC ......... G01S 3/80; G01S 3/801: G01S 3/8055;
`G01S 5/22; H04R 1/406; H04R3/005; H04R
`220 1/401; HO4R 220 1/403
`
`
`
`USPC ............................................. 381/92, 94.1, 93
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5/1994 Bradley et al. .................. 367,89
`5,315.562 A *
`5, 2006 Rui
`7,039,199 B2
`2003/0204397 A1* 10, 2003 Amiri et al. ................... TO4,231
`2004/O161121 A1* 8, 2004 Chol et al. ...................... 381/92
`2007/0076898 A1
`4/2007 Sarroukh et al.
`2009/0141907 A1* 6/2009 Kim et al. .................... 381 71.7
`2009,0279714 A1
`11/2009 Kim et al.
`20090304200 A1
`12/2009 Kim et al.
`
`FOREIGN PATENT DOCUMENTS
`
`4/2008
`
`RS
`WO2O08041878 A2
`* cited by examiner
`Primary Examiner — Fan Tsang
`Assistant Examiner — Eugene Zhao
`(74) Attorney, Agent, or Firm — Ash Tankha; Lipton,
`Weinberger & Husick
`
`ABSTRACT
`(57)
`A method and system for enhancing a target Sound signal
`from multiple sound signals is provided. An array of an arbi
`trary number of sound sensors positioned in an arbitrary
`configuration receives the Sound signals from multiple dis
`parate sources. The Sound signals comprise the target Sound
`signal from a target Sound source, and ambient noise signals.
`A sound source localization unit, an adaptive beam forming
`unit, and a noise reduction unit are in operative communica
`tion with the array of Sound sensors. The Sound source local
`ization unit estimates a spatial location of the target Sound
`signal from the received sound signals. The adaptive beam
`forming unit performs adaptive beam forming by Steering a
`directivity pattern of the array of Sound sensors in a direction
`of the spatial location of the target Sound signal, thereby
`enhancing the target Sound signal and partially suppressing
`the ambient noise signals, which are further Suppressed by the
`noise reduction unit.
`
`21 Claims, 34 Drawing Sheets
`
`Page 1 of 47
`
`GOOGLE EXHIBIT 1003
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 1 of 34
`
`US 8,861,756 B2
`
`ex
`s
`
`:
`
`
`
`2
`& :
`- r
`:
`------------------------------------------------------------------------------------------
`
`w
`
`--
`
`::::::::::
`
`:
`
`s :s
`
`w::
`
`-----------------------------------------------------------------------------------------
`&
`s
`
`i8sa
`
`c
`
`.
`
`:
`
`:
`
`:
`
`:
`
`3.
`
`s
`
`
`
`
`
`it.
`
`Page 2 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 2 of 34
`Sheet 2 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`
`
`POPPLEEOLELEUEOLELEDEELELOUEPDEDEOEELELEDEPLELEDDPLEIDISUte
`
`Sree
`ng
`
`***********
`
`oteatetattateatetattatteetetattatteatetattateatetattatteetetatelatteetetetlatteetetetelatteetetetelatteetetetetatteetetetelatteetetetelatteetetetetatteetetetetatteetetetetatteetetetetattestetetetattes
`
`
`
` “?TRAERAPPADADDONPRPOEADDEDODODEDPDEAEAOEDEODPDEDEDEALEEODLDOIDAAD
`
`
`
`
`
`UOMOLLELOLOLLEEELOLLLEELOLOLDELLOLDELLOLEELLOLDELILEENEly
`
`SNHUELDHeSEBACIVNY
`
`
`
`zry
`Sherrer
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`tHE’7
`
`
`
`CEINTHOSiO}AVERY
`
`SSSNS
`
`Page 3 of 47
`
`:
`Iewnonnonnonnet
`8
`
`Page 3 of 47
`
`
`
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 3 of 34
`Sheet 3 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`
`
`
`
`. .
`Fits.
`
`Page 4 of 47
`
`Page 4 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 4 of 34
`Sheet 4 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`
`
`
`
`
`Filly. 4
`
`... .
`PEs, §
`
`Page 5 of 47
`
`Page 5 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 5 of 34
`
`US 8,861,756 B2
`
`Distance tm}|Delay 1 inember ofsamples}
`
`
`FHS. 64
`
`PH. 68
`
`Page 6 of 47
`
`Page 6 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 6 of 34
`
`US 8,861,756 B2
`
`'88883: S::::::::
`888-88.
`
`
`
`
`
`E. :
`
`Page 7 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 7 of 34
`Sheet 7 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`
`
`PARGETSOUND
`
`SEGUNAE,
`
`éi
`f
`penneneeELEAT Psaatinnnnnnnwvahenannsannecnnninnay
`3
`en
`wen
`
`&.
`
`30prey
`
`;
`Me 4
`
`
`i.
`PHS OO
`
`5
`
`Page 8 of 47
`
`Page 8 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 8 of 34
`
`US 8,861,756 B2
`
`
`
`FOR DIRECTION i (0s is 360), CALCULATE THE DELAY
`D. BETWEEN THE t” PAIR OF THE SOUND SENSORS (t=1:
`ALL PAIRS)
`
`CALCULATE THE CORRELATION VALUE COrr(Di)
`h
`BETWEEN THE f PAIR OF THE SOUND SENSORS
`CORRESPONDING TO THE DELAY OF Di
`
`FOR THE DIRECTION i (0 < is 360),
`ALLPAIR
`CORR, =
`
`THE TARGET SOUND SIGNAL COMES FROM DIRECTION
`
`S = argmax CORR,
`O<ig360
`
`891
`
`802
`
`803
`
`804
`
`FIG. 8
`
`Page 9 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 9 of 34
`Sheet 9 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`ii. 8.
`Pils, OA
`
`:
`Ps. UB
`
`
`
`Page 10 of 47
`
`Page 10 of 47
`
`
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 10 of 34
`
`US 8,861,756 B2
`
`s
`
`~~~~
`
`> • • • • • • • • • • • • • ••••• • • ••••• • • • • • •
`
`&sssssssssssssssssssss
`
`X-X
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`:
`
`------------------------
`
`:
`8
`
`Page 11 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 11 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`TELE
`
`SRV?
`
`BESSELLDANSbg
`
`SESSHANAS
`
`OceanaeaSAANOe a ie ate te
`“+
`seers
`
`Gone
`
`------------------------
`
`seva“e3i’fnneatdpeetettd:¢:
`
`EEage
`
`1D
`
`RGLEVLAVUy
`
`SIATERY
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`&
`
`Iatateataataattatteatt
`
`astattaataattatataatete
`
` HEEVGOHIAVSH
`
`UREdO4EL
`
`AOSERED
`
`
`
`SBLIVIADNECRTTE
`
`Page 12 of 47
`
`Page 12 of 47
`
`
`
`
`
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 12 of 34
`Sheet 12 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`
`
`“SHA
`
`rt
`
`Bee
`
`Page 13 of 47
`
`Page 13 of 47
`
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`US 8,861,756 B2
`
`
`
`sissssssssssssssssssssssssssssssss
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Page 14 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 14 of 34
`Sheet 14 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`
`
`EOS
`
`ii
`8
`i
`SoSRECREATESay
`
`
`
`
`
`
`
`rx
`
`“t
`
`ms3
`
` stCREYY
`
`etetattatteetetattatteetetattatteetetattatteetetattatteetetattattetatetas
`
`
`
`beatenttttldttddd
`
`HENSMEDOVAERS
`DSHACELHAY
`HETREAY
`SiananananannnnntanndhniniaininnninttanndiaininininiatannnngNnnnnnnnAnnPEPERTELECTEDEEEEEOEEEEE
`
`s:
`:
`: 8
`
`8
`8
`8
`8
`8
`8
`8
`8
`8
`8
`8
`8
`8
`8
`8
`8
`8
`8
`8
`8
`8
`
`Page 15 of 47
`
`Page 15 of 47
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 15 of 34
`Sheet 15 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`
`
`8, is:
`Py. ISA
`
`
`
`om SOD
`
`i 7
`
`
`
`1302
`
`is:
`Pits TSB
`
`Page 16 of 47
`
`Page 16 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 16 of 34
`Sheet 16 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`
`
`
`
`, is
`FIG. 18C
`
`Page 17 of 47
`
`Page 17 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 17 of 34
`Sheet 17 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`
`
`“HANDSET
`
`MaLX. 2
`BOE,
`BG
`Ns
`90° Mp SN
`
`
`
`
`
`
`f NaanveeeSa
`we
`|Ms:
`
`... if
`FG. 168
`
`Page 18 of 47
`
`Page 18 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 18 of 34
`
`US 8,861,756 B2
`
`
`
`
`
`FIG, 16C
`
`FIG. 16D
`
`Page 19 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 19 of 34
`Sheet 19 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`
`
`
`i. i.
`BiG. Lak
`
`
`
`
`
`PEGs, BAY
`
`Page 20 of 47
`
`Page 20 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 20 of 34
`Sheet20 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`:g
`
`
`
`38
`i we ae ne
`
`ae Ee ne
`
`ae ne oe ak
`
`Y - 8 - - - - - - - - - - - -3 : •
`4
`£
`
`:
`
`
`
`iii., if:
`
`FR. 16
`
`
`
`ve
`
`&
`
`it... if
`FR. 16H
`
`Page 21 of 47
`
`Page 21 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 21 of 34
`Sheet 21 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`
`
`Bu
`
` Aeeeaeae
` ase ie
`
`PH. 16}
`
`
`
`i., if
`FIG. 16]
`
`Page 22 of 47
`
`Page 22 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 22 of 34
`Sheet 22 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`
`
`
`
`: a£
`
`3
`Meeegeeee+
`
`
`
`:
`
`
`
`
`
`
`
`
`
` oh:
`
`*
`if:
`FG. 161.
`
`Page 23 of 47
`
`Page 23 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 23 of 34
`
`US 8,861,756 B2
`
`FIG, 17A
`
`
`
`O
`
`FIG. 17B
`
`Page 24 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 24 of 34
`
`US 8,861,756 B2
`
`-200 - 150 - 100 -50 0
`
`50
`
`100 150 200
`
`
`
`
`
`FIG. 17C
`
`FIG, 17D
`
`Page 25 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 25 of 34
`
`US 8,861,756 B2
`
`
`
`50
`
`100 150
`
`200
`
`FIG. 17E
`
`Page 26 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 26 of 34
`
`US 8,861,756 B2
`
`
`
`FIG 17F
`
`FIG. 17G.
`
`praeg!!!!!!!!!!!!!!--gºz;
`
`50
`
`100 150 200
`
`Page 27 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 27 of 34
`Sheet 27 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`
`
`
`
`
`
`
`weBeelenebeotheeoeneBoeoe
`
`
`
`
`s
`
`it.
`:
`PHVT
`
`
`
`:
`Py. U71
`
`Page 28 of 47
`
`Page 28 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 28 of 34
`Sheet 28 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`}SR}
`
`BEvorceeenveroce,
`
`ii. 8:
`Pids. IRA
`
`
`
`
`
`{.. 8
`PIG. 8B
`
`Page 29 of 47
`
`Page 29 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 29 of 34
`Sheet 29 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`na
`Mey,
`
`tenonswehetoeeee,
`serene
`
`Aiprovcooovocosose EROE 152 amanmemannemmenen|
`
`--- 8i
`
`i8
`PIG. TSC
`
`
`
`Page 30 of 47
`
`Page 30 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 30 of 34
`Sheet30 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`igs
`B
`
`
`
`
`
`FEMME(SBC
`
`
`
`:38:
`
`it. 8:
`FIG. TSE
`
`Page 31 of 47
`
`a
`
`a
`
`28
`
`2
`
`ag
`
`Ba
`
`TIMEGSES
`
`, is
`Pit. IRD
`
`me
`
`
`
`
`
`o == a
`
`e
`Es
`Pd
`
`Page 31 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 31 of 34
`Sheet 31 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`
`
`eyS
`
`
`
`
`PR
`
`
`
`CEE
`
`w s
`
`TIME ISE
`
`
`
`
`
`
`
`
`
`, is
`Fis. DG
`
`Page 32 of 47
`
`Page 32 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 32 of 34
`Sheet 32 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`teedcpsBe}
`
`
`
`tee DPcoasteg)
`qeDonat ties}
`
`... :
`PH. 198
`
`tadasG40Bos 3
`
`tednosGB.)
`
`qedrossi.)
`
`...
`:
`Pe. TR
`
`Page 33 of 47
`
`Page 33 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 33 of 34
`
`US 8,861,756 B2
`
`as:*sis: 8
`
`.
`
`.
`
`
`
`six-38xxYs:::::
`
`
`
`Page 34 of 47
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 14, 2014
`Oct. 14, 2014
`
`Sheet 34 of 34
`Sheet 34 of 34
`
`US 8,861,756 B2
`US 8,861,756 B2
`
`qdcol)
`
`PEG. Tek
`
`
`
`»y SG,
`a
`wesonnettanaNn
`sp atonsmeinning
`
`Sef
`Nhs
`
`He
`
`teedcentoad st
`’
`.
`
`4*,‘,
`
`i
`
`3
`
`pert+nny2MN,
`
`5
`
`FIG. 198
`
`Page 35 of 47
`
`Page 35 of 47
`
`

`

`1.
`MCROPHONE ARRAY SYSTEM
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`This application claims the benefit of provisional patent
`application No. 61/403,952 titled “Microphone array design
`and implementation for telecommunications and handheld
`devices', filed on Sep. 24, 2010 in the United States Patent
`and Trademark Office.
`The specification of the above referenced patent applica
`tion is incorporated herein by reference in its entirety.
`
`BACKGROUND
`
`Microphones constitute an important element in today's
`speech acquisition devices. Currently, most of the hands-free
`speech acquisition devices, for example, mobile devices,
`lapels, headsets, etc., convert Sound into electrical signals by
`using a microphone embedded within the speech acquisition
`device. However, the paradigm of a single microphone often
`does not work effectively because the microphone picks up
`many ambient noise signals in addition to the desired sound,
`specifically when the distance between a user and the micro
`phone is more than a few inches. Therefore, there is a need for
`a microphone system that operates under a variety of different
`ambient noise conditions and that places fewer constraints on
`the user with respect to the microphone, thereby eliminating
`the need to wear the microphone or be in close proximity to
`the microphone.
`To mitigate the drawbacks of the single microphone sys
`tem, there is a need for a microphone array that achieves
`directional gain in a preferred spatial direction while Sup
`pressing ambient noise from other directions. Conventional
`microphone arrays include arrays that are typically developed
`for applications such as radar and Sonar, but are generally not
`suitable for hands-free or handheld speech acquisition
`devices. The main reason is that the desired Sound signal has
`an extremely wide bandwidth relative to its center frequency,
`thereby rendering conventional narrowband techniques
`employed in the conventional microphone arrays unsuitable.
`In order to cater to Such broadband speech applications, the
`array size needs to be vastly increased, making the conven
`tional microphone arrays large and bulky, and precluding the
`conventional microphone arrays from having broader appli
`cations, for example, in mobile and handheld communication
`devices. There is a need for a microphone array system that
`provides an effective response over a wide spectrum of fre
`quencies while being unobtrusive in terms of size.
`Hence, there is a long felt but unresolved need for a broad
`band microphone array and broadband beam forming system
`that enhances acoustics of a desired sound signal while Sup
`pressing ambient noise signals.
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`SUMMARY OF THE INVENTION
`
`55
`
`This Summary is provided to introduce a selection of con
`cepts in a simplified form that are further described in the
`detailed description of the invention. This summary is not
`intended to identify key or essential inventive concepts of the
`claimed Subject matter, nor is it intended for determining the
`Scope of the claimed Subject matter.
`The method and system disclosed herein addresses the
`above stated need for enhancing acoustics of a target Sound
`signal received from a target Sound source, while Suppressing
`ambient noise signals. As used herein, the term “target Sound
`signal” refers to a sound signal from a desired or target Sound
`
`60
`
`65
`
`US 8,861,756 B2
`
`2
`Source, for example, a person's speech that needs to be
`enhanced. A microphone array system comprising an array of
`Sound sensors positioned in an arbitrary configuration, a
`Sound source localization unit, an adaptive beam forming
`unit, and a noise reduction unit, is provided. The sound source
`localization unit, the adaptive beam forming unit, and the
`noise reduction unit are in operative communication with the
`array of Sound sensors. The array of Sound sensors is, for
`example, a linear array of Sound sensors, a circular array of
`Sound sensors, or an arbitrarily distributed coplanar array of
`Sound sensors. The array of Sound sensors herein referred to
`as a “microphone array' receives Sound signals from multiple
`disparate Sound sources. The method disclosed herein can be
`applied on a microphone array with an arbitrary number of
`Sound sensors having, for example, an arbitrary two dimen
`sional (2D) configuration. The sound signals received by the
`Sound sensors in the microphone array comprise the target
`Sound signal from the target Sound Source among the dispar
`ate Sound sources, and ambient noise signals.
`The Sound source localization unit estimates a spatial loca
`tion of the target Sound signal from the received sound sig
`nals, for example, using a steered response power-phase
`transform. The adaptive beam forming unit performs adaptive
`beam forming for steering a directivity pattern of the micro
`phone array in a direction of the spatial location of the target
`Sound signal. The adaptive beam forming unit thereby
`enhances the target Sound signal from the target Sound source
`and partially Suppresses the ambient noise signals. The noise
`reduction unit Suppresses the ambient noise signals for fur
`ther enhancing the target Sound signal received from the
`target Sound Source.
`In an embodiment where the target sound source that emits
`the target Sound signal is in a two dimensional plane, a delay
`between each of the Sound sensors and an origin of the micro
`phone array is determined as a function of distance between
`each of the sound sensors and the origin, a predefined angle
`between each of the sound sensors and a reference axis, and
`an azimuth angle between the reference axis and the target
`Sound signal. In another embodiment where the target Sound
`Source that emits the target Sound signal is in a three dimen
`sional plane, the delay between each of the Sound sensors and
`the origin of the microphone array is determined as a function
`of distance between each of the Sound sensors and the origin,
`a predefined angle between each of the Sound sensors and a
`first reference axis, an elevation angle between a second
`reference axis and the target Sound signal, and an azimuth
`angle between the first reference axis and the target Sound
`signal. This method of determining the delay enables beam
`forming for arbitrary numbers of Sound sensors and multiple
`arbitrary microphone array configurations. The delay is deter
`mined, for example, in terms of number of samples. Once the
`delay is determined, the microphone array can be aligned to
`enhance the target Sound signal from a specific direction.
`The adaptive beam forming unit comprises a fixed beam
`former, a blocking matrix, and an adaptive filter. The fixed
`beam former steers the directivity pattern of the microphone
`array in the direction of the spatial location of the target Sound
`signal from the target Sound source for enhancing the target
`Sound signal, when the target Sound source is in motion. The
`blocking matrix feeds the ambient noise signals to the adap
`tive filter by blocking the target Sound signal from the target
`sound source. The adaptive filter adaptively filters the ambi
`ent noise signals in response to detecting the presence or
`absence of the target Sound signal in the Sound signals
`received from the disparate sound sources. The fixed beam
`former performs fixed beam forming, for example, by filtering
`and Summing output sound signals from the Sound sensors.
`
`Page 36 of 47
`
`

`

`US 8,861,756 B2
`
`3
`In an embodiment, the adaptive filtering comprises Sub
`band adaptive filtering. The adaptive filter comprises an
`analysis filter bank, an adaptive filter matrix, and a synthesis
`filter bank. The analysis filter bank splits the enhanced target
`sound signal from the fixed beam former and the ambient 5
`noise signals from the blocking matrix into multiple fre
`quency sub-bands. The adaptive filter matrix adaptively fil
`ters the ambient noise signals in each of the frequency Sub
`bands in response to detecting the presence or absence of the
`target Sound signal in the Sound signals received from the 10
`disparate sound sources. The synthesis filter bank synthesizes
`a full-band Sound signal using the frequency Sub-bands of the
`enhanced target Sound signal. In an embodiment, the adaptive
`beam forming unit further comprises an adaptation control
`unit for detecting the presence of the target Sound signal and 15
`adjusting a step size for the adaptive filtering in response to
`detecting the presence or the absence of the target Sound
`signal in the Sound signals received from the disparate Sound
`SOUCS.
`The noise reduction unit Suppresses the ambient noise sig- 20
`nals for further enhancing the target Sound signal from the
`target Sound Source. The noise reduction unit performs noise
`reduction, for example, by using a Wiener-filter based noise
`reduction algorithm, a spectral Subtraction noise reduction
`algorithm, an auditory transform based noise reduction algo- 25
`rithm, or a model based noise reduction algorithm. The noise
`reduction unit performs noise reduction in multiple frequency
`sub-bands employed for sub-band adaptive beam forming by
`the analysis filter bank of the adaptive beam forming unit.
`The microphone array System disclosed herein comprising 30
`the microphone array with an arbitrary number of sound
`sensors positioned in arbitrary configurations can be imple
`mented in handheld devices, for example, the iPad(R) of Apple
`Inc., the iPhone(R) of Apple Inc., Smart phones, tablet com
`puters, laptop computers, etc. The microphone array system 35
`disclosed herein can further be implemented in conference
`phones, video conferencing applications, or any device or
`equipment that needs better speech inputs.
`
`4
`FIG. 6B exemplarily illustrates a table showing the rela
`tionship of the position of each Sound sensor in the circular
`microphone array configuration and its distance to the origin
`of the microphone array, when the target Sound source is in
`the same plane as that of the microphone array.
`FIG. 7A exemplarily illustrates a graphical representation
`of a microphone array, when the target Sound source is in a
`three dimensional plane.
`FIG. 7B exemplarily illustrates a table showing delay
`between each sound sensor in a circular microphone array
`configuration and the origin of the microphone array, when
`the target Sound source is in a three dimensional plane.
`FIG.7C exemplarily illustrates a three dimensional work
`ing space of the microphone array, where the target Sound
`signal is incident at an elevation angle <S2
`FIG. 8 exemplarily illustrates a method for estimating a
`spatial location of the target Sound signal from the target
`Sound Source by a sound Source localization unit using a
`steered response power-phase transform.
`FIG. 9A exemplarily illustrates a graph showing the value
`of the steered response power-phase transform for every 10°.
`FIG. 9B exemplarily illustrates a graph representing the
`estimated target Sound signal from the target Sound Source.
`FIG. 10 exemplarily illustrates a system for performing
`adaptive beam forming by an adaptive beam forming unit.
`FIG.11 exemplarily illustrates a system for sub-band adap
`tive filtering.
`FIG. 12 exemplarily illustrates a graphical representation
`showing the performance of a perfect reconstruction filter
`bank.
`FIG. 13 exemplarily illustrates a block diagram of a noise
`reduction unit that performs noise reduction using a Wiener
`filter based noise reduction algorithm.
`FIG. 14 exemplarily illustrates a hardware implementation
`of the microphone array system.
`FIGS. 15A-15C exemplarily illustrate a conference phone
`comprising an eight-sensor microphone array.
`FIG. 16A exemplarily illustrates a layout of an eight-sen
`Sor microphone array for a conference phone.
`FIG.16B exemplarily illustrates a graphical representation
`of eight spatial regions to which the eight-sensor microphone
`array of FIG. 16A responds.
`FIGS. 16C-16D exemplarily illustrate computer simula
`tions showing the steering of the directivity patterns of the
`eight-sensor microphone array of FIG. 16A in the directions
`of 15° and 60° respectively, in the frequency range 300 Hz to
`5 kHZ.
`FIGS. 16E-16L exemplarily illustrate graphical represen
`tations showing the directivity patterns of the eight-sensor
`microphone array of FIG. 16A in each of the eight spatial
`regions, where each directivity pattern is an average response
`from 300 HZ to 5000 HZ.
`FIG.17A exemplarily illustrates a graphical representation
`of four spatial regions to which a four-sensor microphone
`array for a wireless handheld device responds.
`FIGS. 17B-17I exemplarily illustrate computer simula
`tions showing the directivity patterns of the four-sensor
`microphone array of FIG. 17A with respect to azimuth and
`frequency.
`FIGS. 18A-18E3 exemplarily illustrate a microphone array
`configuration for a tablet computer.
`FIG. 18C exemplarily illustrates an acoustic beam formed
`using the microphone array configuration of FIGS. 18A-18E3
`according to the method and system disclosed herein.
`FIGS. 18D-18G exemplarily illustrate graphs showing
`processing results of the adaptive beam forming unit and the
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`40
`
`50
`
`The foregoing Summary, as well as the following detailed
`description of the invention, is better understood when read in
`conjunction with the appended drawings. For the purpose of
`illustrating the invention, exemplary constructions of the 45
`invention are shown in the drawings. However, the invention
`is not limited to the specific methods and instrumentalities
`disclosed herein.
`FIG. 1 illustrates a method for enhancing a target Sound
`signal from multiple sound signals.
`FIG. 2 illustrates a system for enhancing a target Sound
`signal from multiple sound signals.
`FIG. 3 exemplarily illustrates a microphone array configu
`ration showing a microphone array having N sound sensors
`arbitrarily distributed on a circle.
`FIG. 4 exemplarily illustrates a graphical representation of
`a filter-and-sum beam forming algorithm for determining out
`put of the microphone array having N sound sensors.
`FIG. 5 exemplarily illustrates distances between an origin
`of the microphone array and Sound sensor M and Sound 60
`sensor M in the circular microphone array configuration,
`when the target Sound signal is at an angle 0 from the Y-axis.
`FIG. 6A exemplarily illustrates a table showing the dis
`tance between each Sound sensor in a circular microphone
`array configuration from the origin of the microphone array, 65
`when the target Sound Source is in the same plane as that of the
`microphone array.
`
`55
`
`Page 37 of 47
`
`

`

`US 8,861,756 B2
`
`5
`noise reduction unit for the microphone array configuration
`of FIG. 18B, in both a time domain and a spectral domain for
`the tablet computer.
`FIGS. 19A-19F exemplarily illustrate tables showing dif
`ferent microphone array configurations and the correspond
`ing values of delay t, for the Sound sensors in each of the
`microphone array configurations.
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`6
`partially Suppressing the ambient noise signals. Beamform
`ing refers to a signal processing technique used in the micro
`phone array for directional signal reception, that is, spatial
`filtering. This spatial filtering is achieved by using adaptive or
`fixed methods. Spatial filtering refers to separating two sig
`nals with overlapping frequency content that originate from
`different spatial locations.
`The noise reduction unit performs noise reduction by fur
`ther suppressing 105 the ambient noise signals and thereby
`further enhancing the target Sound signal. The noise reduction
`unit performs the noise reduction, for example, by using a
`Wiener-filter based noise reduction algorithm, a spectral sub
`traction noise reduction algorithm, an auditory transform
`based noise reduction algorithm, or a model based noise
`reduction algorithm.
`FIG. 2 illustrates a system 200 for enhancing a target sound
`signal from multiple sound signals. The system 200, herein
`referred to as a “microphone array system', comprises the
`array 201 of sound sensors positioned in an arbitrary configu
`ration, the sound source localization unit 202, the adaptive
`beam forming unit 203, and the noise reduction unit 207.
`The array 201 of sound sensors, herein referred to as the
`“microphone array' is in operative communication with the
`Sound source localization unit 202, the adaptive beam forming
`unit 203, and the noise reduction unit 207. The microphone
`array 201 is, for example, a linear array of Sound sensors, a
`circular array of Sound sensors, or an arbitrarily distributed
`coplanar array of Sound sensors. The microphone array 201
`achieves directional gain in any preferred spatial direction
`and frequency band while Suppressing signals from other
`spatial directions and frequency bands. The Sound sensors
`receive the Sound signals comprising the target Sound signal
`and ambient noise signals from multiple disparate Sound
`Sources, where one of the disparate sound sources is the target
`Sound source that emits the target Sound signal.
`The sound source localization unit 202 estimates the spa
`tial location of the target Sound signal from the received sound
`signals. In an embodiment, the Sound source localization unit
`202 uses, for example, a steered response power-phase trans
`form, for estimating the spatial location of the target Sound
`signal from the target Sound source.
`The adaptive beam forming unit 203 steers the directivity
`pattern of the microphone array 201 in a direction of the
`spatial location of the target Sound signal, thereby enhancing
`the target Sound signal and partially suppressing the ambient
`noise signals. The adaptive beam forming unit 203 comprises
`a fixed beam former 204, a blocking matrix 205, and an adap
`tive filter 206 as disclosed in the detailed description of FIG.
`10. The fixed beam former 204 performs fixed beam forming
`by filtering and Summing output sound signals from each of
`the sound sensors in the microphone array 201 as disclosed in
`the detailed description of FIG. 4. In an embodiment, the
`adaptive filter 206 is implemented as a set of sub-band adap
`tive filters. The adaptive filter 206 comprises an analysis filter
`bank 206a, an adaptive filter matrix 206b, and a synthesis
`filter bank 206c as disclosed in the detailed description of
`FIG 11.
`The noise reduction unit 207 further suppresses the ambi
`ent noise signals for further enhancing the target Sound signal.
`The noise reduction unit 207 is, for example, a Wiener-filter
`based noise reduction unit, a spectral Subtraction noise reduc
`tion unit, an auditory transform based noise reduction unit, or
`a model based noise reduction unit.
`FIG.3 exemplarily illustrates a microphone array configu
`ration showing a microphone array 201 having NSound sen
`sors 301 arbitrarily distributed on a circle 302 with a diameter
`“d', where “N' refers to the number of sound sensors 301 in
`
`FIG. 1 illustrates a method for enhancing a target Sound
`signal from multiple Sound signals. As used herein, the term
`“target Sound signal” refers to a desired Sound signal from a
`desired or target Sound source, for example, a person’s speech
`that needs to be enhanced. The method disclosed herein pro
`vides 101 a microphone array system comprising an array of
`Sound sensors positioned in an arbitrary configuration, a
`Sound source localization unit, an adaptive beam forming
`unit, and a noise reduction unit. The Sound source localization
`unit, the adaptive beam forming unit, and the noise reduction
`unit are in operative communication with the array of Sound
`sensors. The microphone array system disclosed herein
`employs the array of sound sensors positioned in an arbitrary
`configuration, the Sound source localization unit, the adaptive
`beam forming unit, and the noise reduction unit for enhancing
`a target Sound signal by acoustic beam forming in the direc
`tion of the target Sound signal in the presence of ambient noise
`signals.
`The array of sound sensors herein referred to as a “micro
`phone array comprises multiple or an arbitrary number of
`Sound sensors, for example, microphones, operating in tan
`dem. The microphone array refers to an array of an arbitrary
`number of Sound sensors positioned in an arbitrary configu
`ration. The Sound sensors are transducers that detect Sound
`and convert the Sound into electrical signals. The Sound sen
`sors are, for example, condenser microphones, piezoelectric
`microphones, etc.
`The sound sensors receive 102 sound signals from multiple
`disparate sound Sources and directions. The target Sound
`Source that emits the target Sound signal is one of the disparate
`Sound sources. As used herein, the term "sound signals'
`refers to composite sound energy from multiple disparate
`Sound sources in an environment of the microphone array.
`The sound signals comprise the target Sound signal from the