I 1111111111111111 11111 111111111111111 111111111111111 11111 lll111111111111111
`
`US008861756B2
`
`c12) United States Patent
`Zhu et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,861,756 B2
`Oct. 14, 2014
`
`(54) MICROPHONE ARRAY SYSTEM
`
`(75)
`
`Inventors: Manli Zhu, Pearl River, NY (US); Qi
`Li, New Providence, NJ (US)
`
`(73) Assignee: LI Creative Technologies, Inc., Florham
`Park, NJ (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 794 days.
`
`(21) Appl. No.: 13/049,877
`
`(22) Filed:
`
`Mar. 16, 2011
`
`(65)
`
`Prior Publication Data
`
`US 2012/0076316Al
`
`Mar. 29, 2012
`
`Related U.S. Application Data
`
`(60) Provisional application No. 61/403,952, filed on Sep.
`24, 2010.
`
`(51)
`
`(52)
`
`(58)
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`
`Int. Cl.
`H04R 25100
`H03G3/20
`H04R 3/00
`G0lS 5122
`G0lS 3/801
`G0lS 3/805
`H04R 1140
`H04M 3/56
`U.S. Cl.
`CPC .............. G0lS 3/8055 (2013.01); H04M 3/568
`(2013.01); H04R 3/005 (2013.01); G0lS 5122
`(2013.01); H04R 2201/403 (2013.01); G0lS
`3/801 (2013.01); H04R 2201/401 (2013.01);
`H04R 11406 (2013.01)
`USPC ............................................. 381/300; 381/57
`Field of Classification Search
`CPC ......... G0lS 3/80; G0lS 3/801; G0lS 3/8055;
`G0lS 5/22; H04R 1/406; H04R 3/005; H04R
`2201/401; H04R 2201/403
`
`USPC ............................................. 381/92, 94.1, 93
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,315,562 A *
`7,039,199 B2
`2003/0204397 Al *
`2004/0161121 Al*
`2007 /0076898 Al
`2009/0141907 Al*
`2009/0279714 Al
`2009/0304200 Al
`
`5/1994 Bradley et al. .................. 367/89
`5/2006 Rui
`10/2003 Amiri et al. ................... 704/231
`8/2004 Chol et al.
`...................... 381/92
`4/2007 Sarroukh et al.
`6/2009 Kim et al ..................... 381/71.7
`11/2009 Kim et al.
`12/2009 Kim et al.
`
`FOREIGN PATENT DOCUMENTS
`
`RS
`
`W02008041878 A2
`
`4/2008
`
`* cited by examiner
`
`Primary Examiner - Fan Tsang
`Assistant Examiner - Eugene Zhao
`(74) Attorney, Agent, or Firm -Ash Tankha; Lipton,
`Weinberger & Husick
`
`ABSTRACT
`(57)
`A method and system for enhancing a target sound signal
`from multiple sound signals is provided. An array of an arbi(cid:173)
`trary number of sound sensors positioned in an arbitrary
`configuration receives the sound signals from multiple dis(cid:173)
`parate sources. The sound signals comprise the target sound
`signal from a target sound source, and ambient noise signals.
`A sound source localization unit, an adaptive beamforming
`unit, and a noise reduction unit are in operative communica(cid:173)
`tion with the array of sound sensors. The sound source local(cid:173)
`ization unit estimates a spatial location of the target sound
`signal from the received sound signals. The adaptive beam(cid:173)
`forming unit performs adaptive beamforming by steering a
`directivity pattern of the array of sound sensors in a direction
`of the spatial location of the target sound signal, thereby
`enhancing the target sound signal and partially suppressing
`the ambient noise signals, which are further suppressed by the
`noise reduction unit.
`
`21 Claims, 34 Drawing Sheets
`
`....................................................................................
`:>:::)',; ;J;,f: .-1. ;.,,~l.CJ::.,.i;~~iJJ-;f: .·'.R?.1: \' ;:;ys.;TJ:M C!,1,).f:>.:.::Sf:-:G ).N .-\::::.<.-'.)'
`: 101
`:
`I i:n~/i;;/;i~~!~~~~:~;;:~iti~~t~)~;;t:~rr;,,~\~~l~}~{.;:::i~~,:::}r_ v;
`~_..:
`i .... :0.i)APT:v;__: lU.: ... :..a?({)l;_:.,{i.!'X;, ,.::-::"f; ,\;'{fl--~ ;,.;s.-1s;. I-:.::.;_,,;_:c--:·K:N.!;Nn: ••• j
`
`:· .. ;ii:f:i:J~:;;:·~f;;i :;;;1 ;; ·~,;t:~;;-; ·-~l~;~::~ -; :; f:;:~;~;·;,;;-:;_ ·:;:~ ;.; .LJ"=.J:.~;\·ii .. ·: wi.
`~ ................ ~''::~l:•.~,l.K(:E'.~'.,'."l''.'''.':':~':'.,r'i,<J~>················.r·····
`•
`r··;:::;.;:,;:i.~:/i::·~~:~1-;~.:·,:r~~.~:,:;,:.;:·.:.~::11{,:=·.;.:.~;~.:;f:.r;;.;~;~;·~;~~~-~;:···1 jil3
`~ .... ;i;,;,;. /Hf: RH.ms•s,,::;-'.;~;;:;1~;~_;';-;,~;;. r:-:F ,,.,,,,_,_, S:">!:llC!' ••••• ( '
`
`.......................................... l' ......................................... , ltl4
`
`l,,. :~::~~;~\~~·)t~:t.~~:::~~~~:}~;-:~~ ·s~~~~:.t~:~;;:tr;:~~~t~~~t\~y l,,.---.,
`
`')Hf: :-;t,..;TfA:j u~· ... : 10;-.;(H· 'l:"HE T.,\;a_:{:'i" ::,.(cid:141):,( ·~{: ~v•:-..\i {~\ l r::-:
`.. ~r,,'-vnv::. 8:L:.~.:.5l:<...:::~n"'·: ::::,;f_:
`
`~~-~~~-~~~-~~~-~ 18:'i
`
`I_ S<.":'<·SJ,SS.: ,::, .• ,w,,r,,: _,·,c,::,r r~:~ ·''--':;' .,.,,,._,·,nr,r K:,»c:<:::<;c, . ('
`
`Amazon Ex. 1003
`IPR Petition - US RE47,049
`
`Amazon Ex. 1003, Page 1 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 1 of 34
`
`US 8,861,756 B2
`
`PR()V*DE /-.. NffCROP!-lONE /-..RRA Y SYSTEM CO\lPRIS!t.;G .\N i\RRi\Y
`OF SOUND SENSORS POS!Tl{}NED lN AN ARfHTR/1,RY
`coNriuuRATl(>N, A SOlJND SOURCE U}('ALlZATR)N UNlT,AN
`
`'
`
`~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~
`
`,
`
`,,·:a;cc'!'?\-'~,- \,fl f! 1-'H:,1 ~-, "'01· (?-'Z) _;::-s,;:)-.''..i, 1 --~ I;''R>')\i! !\.,~!? ·r1. t.)I _t.; f'l'"'.l'.s.~R i'-·T· t.;
`l' .• fa /S.
`i'U-, ... !. L, l' , ... , H"L.L --> ••.. ,ih ..
`,.H-...,.~"'<-• d,::-i l . -... ,.
`.. L
`t
`,.H. L
`SOUND SOURCES HY THE SOt'NO SENSORS
`
`, . .,_ L
`
`~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~
`
`1 H}2
`,
`~~_;
`'
`'
`'
`
`l
`\ .UH
`1 J
`r·-
`ADAVUVE HEA!\=ffORt\:lING llNIT, AND A NOfSE REDt=('TlON t=Nff I
`'
`❖""""""""""""""""""""""""""""""""""""""""'" !""""""""""""""""""""""""""""""""""""""""'"',
`l
`i
`'
`'
`'
`❖"""""""""""""""""""""""""""""""""""""""""' l '"""""""""""""""""""""""""""""""""""""', * -~
`
`rt"r·1·~ ~ ~-J'P \ ,-,r, ., 't''I >. 1 ,. ,....,,., .. ''fif'">N' ('"f' 1·ur 'f >-1·•-cr't' 0(··,~ '"!=>'J"' ,., s,.,.,._, '$
`.,.,:,;--,. i::l} ..:'>i:'i\i. A-L,.,,, .. ,,, ... A h . .<i··,, .• ,,. ni::l
`~~.;>
`_;:·\>,,,,.$~-:;:,; ,3·.-.H.-'l~- .• '-.:'>(>,.Ji'i.i-s.L-
`fl{O\.·t THE R!-x::RrvEn sor:,Nn SIGNALS BY THE SOUND sovRcE
`LOCA.UZA,TlON t=NIT
`
`l
`l
`l
`l
`'
`'
`
`1 .s.tt.-,
`\
`..
`t,...,..J
`1
`'
`l
`
`I t
`THE SPATUt .. LOCATlON or THE TA.RGET sot,ND s1cNA,t .. B v THE I
`I
`1ns
`l SL!·1·'RES.S THE A\11-m•NT NOISE SIGNALS HY THE NOlSE REDtJ::T!Ut--1 t.,,,/
`'
`'
`1
`l
`UN1T
`'
`'
`'
`'
`'
`'
`'
`'
`L ... ·.························································································································································································································································································································································i
`
`============= , , , , , , _============== i H:W
`PERFORM ,ADAfTl\lE Bb\MFORiVUNG FOR STEERING A D!RECfiVffY \ ,./
`PATfERN OF THE ARRAY UFSOUND SENSfiRS lN /\. D!RFCT!ON OF
`\,---·
`
`ADAVHVE HEAt:lH}RMlNG UNlT
`:
`~:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-._ .. :-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-·}
`
`' """"""""""""""""""""""""""""""""""""""'"''"""""""""""""""""""""""""""""""""""""""""",
`
`FlG, 1
`
`Amazon Ex. 1003, Page 2 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 2 of 34
`
`US 8,861,756 B2
`
`1
`
`Amazon Ex. 1003, Page 3 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 3 of 34
`
`US 8,861,756 B2
`
`FIG.$
`
`Amazon Ex. 1003, Page 4 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 4 of 34
`
`US 8,861,756 B2
`
`JOl
`
`.FtG.S
`
`Amazon Ex. 1003, Page 5 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 5 of 34
`
`US 8,861,756 B2
`
`Smm<l
`S<:'!bOr
`MO
`Ml
`
`$ ,imido-::::kw·i~e ,,way fri:m1 rnff
`{0:::d,:;::90'':r
`
`HG.6A
`
`1)i.\b.v t fmonber of
`...
`·.
`:,am pk:i)
`
`Amazon Ex. 1003, Page 6 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 6 of 34
`
`US 8,861,756 B2
`
`'FlG. 7A
`
`T,::::::
`1\;""''
`d~fr,!ot:•n~-i.O·Hh,)i,irifFVt d~/i.•tu*,(0-
`111 i ):;,i n{ttiv t
`
`T••;z:;
`-d ~/\•{T,1,(H +(fi2 f,j n{\l1)A.~
`
`T:,c::::::
`-d~JktcoM0-
`41:, W,i n{tl1Vt
`
`FIG. iB
`
`Amazon Ex. 1003, Page 7 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 7 of 34
`
`US 8,861,756 B2
`
`..., .................. S1
`
`FIG. 7C
`
`Amazon Ex. 1003, Page 8 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 8 of 34
`
`US 8,861,756 B2
`
`8
`01
`L-J
`
`8 02
`L-J
`
`8
`03
`L-J
`
`8 04
`l....J
`
`FOR DIRECTION i (0 :S i '.S 360), CALCULATE THE DELAY
`Dit BETWEEN THE lh PAIR OF THE SOUND SENSORS (t=l:
`ALL PAIRS)
`
`l
`
`CALCULATE THE CORRELATION VALUE corr(Dit)
`BETWEEN THE lh PAIR OF THE SOUND SENSORS
`CORRESPONDING TO THE DELAY OF Dit
`
`l
`
`FOR THE DIRECTION i (0 :S i '.S 360),
`ALLPAIR
`'I.corr(Dit)
`t=l
`
`CORR.=
`l
`
`l
`
`THE TARGET SOUND SIGNAL COMES FROM DIRECTION
`S = argmax CORRi
`0~i~360
`
`FIG.8
`
`Amazon Ex. 1003, Page 9 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 9 of 34
`
`US 8,861,756 B2
`
`0.7,....----.....---....--------.------,,----.,...---......,,..--------->
`.........
`
`!}5
`
`0.4 •·
`
`{~
`
`~'. : : {['•.
`
`'•"it?¢
`
`(! l L i
`·--J~ .. L ................ ;···
`
`<r
`
`I '['l,.' '!' n hi Tr'I'l ·r <[l: ir t]>l.'' :"Il J !I'):;[:·::"']' 1.•' <[" r·:
`
`'•'•'•'
`
`'•'•'•'_..'•'•'•'
`
`.,,
`·-'·•·:>.·.·•••y.•.·•·•·•·•·•·•❖•·•·•·•~·.·•·•·•:3i .•.•.• •,,,,• ,,,,• ,,,,• ···v···" ,,,. ··,••''•'•'•' ·•·•·•·•-.·•·•·•·• ·•·•❖X·.·•·•·• '•'•'•'
`5
`.}
`
`_.;;._
`
`'•'•'•' •'•'•'•"'•'•'•'•' J..-.·---
`l
`
`FlG,9A
`
`Amazon Ex. 1003, Page 10 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 10 of 34
`
`US 8,861,756 B2
`
`.... ,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,., •.
`
`' ./
`
`~
`
`·(cid:157)
`
`.'-.., :;.i.;;
`
`;~ - . ~"
`,.
`
`~
`
`,,
`
`"""'
`.... \;.;:::
`?
`.,,
`>.·•······················•.
`
`LNA -UNA~
`
`UNA .UNA •
`
`'
`
`:t ~·--(cid:173)
`
`.. \
`\
`
`""""""""""""""""""""'"',
`
`' ~ ... J
`~
`
`""""""""
`
`i.n -M .. , .. •.•············································'·················································,
`
`... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, .
`.. , .... ,p 2
`
`l
`
`Amazon Ex. 1003, Page 11 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 11 of 34
`
`US 8,861,756 B2
`
`¥:'."
`~ .-.,,,,
`
`f
`
`Amazon Ex. 1003, Page 12 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 12 of 34
`
`US 8,861,756 B2
`
`(.)
`
`m u,.
`~J:}
`()
`~ ......
`
`u.. a (,
`
`"
`
`'
`
`{J}
`)~,!
`(<'.)
`
`jZ::
`
`zy,; - s· .,...p
`
`----
`:~
`,:::-,;:
`¢:!} -Z!::~
`{-('~
`,::.:i <))
`
`·~····· i;: -
`
`{
`
`.
`
`; .
`
`....
`
`:
`d#-~·:»'_..,. .,..
`w
`--~'
`,_.,.. _ _ _ _ ...., _ _ _ _ ..., _ _ _ _ .._ _ _ _ _..__._.:.,:;;:...,·<-,..r_·--:_l..,J_:_:;.-.-._-:--_·.;-:_. «_·:-:-_,,_<:-·:~..,--_·· _ _ _ _., _ _ _ _ ...., _ _ _ _ .,.,......,-- :~
`
`Amazon Ex. 1003, Page 13 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 13 of 34
`
`US 8,861,756 B2
`
`~
`WJ
`~
`,....,
`;....<
`~ ~
`=:.t.:z
`;;.. <
`:: ~
`1
`<
`
`Amazon Ex. 1003, Page 14 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 14 of 34
`
`US 8,861,756 B2
`
`! .•,,1 ... •.•.•.❖•❖•❖ .• ❖•❖•❖•❖•❖•❖•❖•❖•❖•❖•❖•❖•❖•❖•❖•❖•❖•❖•❖•❖•❖•❖•❖.•. l .... •.❖•❖•❖•❖•❖•❖•
`I
`
`Amazon Ex. 1003, Page 15 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 15 of 34
`
`US 8,861,756 B2
`
`FIG. l5A
`
`1502
`
`,....,
`
`..
`
`rl500
`
`1501
`
`Amazon Ex. 1003, Page 16 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 16 of 34
`
`US 8,861,756 B2
`
`'·
`
`1502
`
`~ l_::_J
`D
`
`FIG. lSC
`
`Amazon Ex. 1003, Page 17 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 17 of 34
`
`US 8,861,756 B2
`
`1502
`
`FIG.16A
`
`HG.168
`
`Amazon Ex. 1003, Page 18 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 18 of 34
`
`US 8,861,756 B2
`
`_. -.--
`
`, 1
`
`, -,
`
`+,
`
`_. -,--
`
`__ ,..
`
`' ' J -
`
`,
`
`' ' ~--
`' ' ~--
`-:: --+---+---r ~ -
`
`-
`
`_. -
`
`"1
`
`I
`
`, - ,,
`
`_. -
`
`.-1
`
`I
`
`' -
`... - ...
`'
`'
`, ~
`' '
`
`,
`
`_. - ... ,
`'
`' ' '
`- .- -:--
`'
`
`1.4
`
`1.2
`1
`0.
`
`0.6
`
`0.4
`
`- _. - r--.
`I
`
`_, -
`
`("
`
`,,(cid:173)
`'
`
`.-..-:-.
`
`_. -:-- -.
`
`-~
`
`, . ,
`' ' ' . - ,
`
`~--
`' ,,
`'
`,, .- ._-.
`'
`
`,,
`' ' ..,_
`' ' '
`' '
`' ...
`
`--r- ._
`'
`' ,,
`
`I
`
`-.._
`
`' ~ ' -.
`
`--1 ... -
`._-. ~--
`'
`~-- ._ ._
`'
`
`'
`
`'
`' --... ._
`'
`
`,,
`'
`
`0.2
`0
`5000
`4000
`3000 ~
`2000
`
`'
`
`-100 -50
`0 -200 -150
`
`0
`
`FIG.16C
`
`._
`
`'
`
`.- - T
`I
`I
`I
`
`~ _. -
`
`_.
`
`~ _. -
`1
`I
`
`-
`
`._
`
`I
`I
`-..I.
`
`' -' ' '
`' '
`'~ ' ' '
`'
`'
`' ' 1 -
`
`'
`
`'
`.., '
`
`I
`
`-..._
`
`'' -
`'
`... __
`'~
`
`'
`-. -. ~ ._
`
`...,
`
`'
`'"
`'
`'
`'
`' ' ' -,
`
`'
`
`I
`
`:
`
`'.,J
`
`1.4
`1.2
`I
`0.8
`
`0.6
`0.4
`
`0.2
`0
`
`~ -
`'
`, - ,,
`
`I_. -
`
`-
`
`I
`
`,,
`
`I
`.- - ,
`
`.- -
`
`....
`
`-- "'
`
`.- - ~-.-.
`I
`I
`I
`.- - T ._
`I
`I
`I
`- -- T ......
`
`.., '
`
`I
`
`-. ••
`
`' ' -. ' ' '
`
`' '
`
`' ,
`
`.-1
`
`.- _,. -
`
`I
`
`_. - --.
`
`_. - --~
`
`-~
`'
`
`-
`
`I
`
`- ,("
`
`-~
`'
`
`.- -
`
`.-
`
`I
`
`' -
`- ',
`'
`
`' -
`- ',
`'
`' - --·--
`
`--:- .-
`,, -
`
`I_. -
`
`.- -
`
`.-1
`
`_. - .-,
`
`I_. -
`
`'
`- _. -~
`
`.- -
`
`.-,
`
`, _,.
`'
`
`0 -200
`
`FIG.16D
`
`Amazon Ex. 1003, Page 19 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 19 of 34
`
`US 8,861,756 B2
`
`Amazon Ex. 1003, Page 20 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 20 of 34
`
`US 8,861,756 B2
`
`FIG. l6G
`
`Amazon Ex. 1003, Page 21 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 21 of 34
`
`US 8,861,756 B2
`
`FIG. l6J
`
`Amazon Ex. 1003, Page 22 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 22 of 34
`
`US 8,861,756 B2
`
`PIG, 16K
`
`BG, l6L
`
`Amazon Ex. 1003, Page 23 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 23 of 34
`
`US 8,861,756 B2
`
`)
`
`201
`
`~ 0 2
`
`t
`
`FIG.17A
`
`-- - .. --;
`
`~ _. -
`
`!
`
`~ (""
`
`_..
`
`_. -
`
`- - _.
`
`'
`
`I
`I
`
`-
`
`•• ~·-;
`
`I -
`• _.:
`
`-
`
`-.._
`
`,._
`
`1- -
`I
`
`' _, .- '
`' ' - '
`
`.... -
`
`....
`
`--
`
`-
`
`...
`
`... :·- ...
`' '
`··--~
`' -'
`' ,,
`' -
`
`' - ' .,
`'
`...... : ...
`' --
`'
`,- -
`' '
`--...
`
`••._
`
`I
`
`' ' '
`.• _,
`
`I
`
`....
`
`-- .... :_
`
`' --
`' - ' ·~._ ...
`---: ...
`'
`'
`
`--
`
`:
`I
`
`!
`
`I
`•
`
`1.4"
`1.2
`
`1
`
`0.
`
`0.
`
`0.2..----
`
`'.,
`'
`'
`)- -- --!,---------~--
`- -~~ -
`I
`
`1_ -
`i"J
`
`•
`
`_. -
`
`-
`
`•~
`
`.-
`
`:
`
`I
`
`:. - .- ..... ~ ..
`'
`'
`'
`' !
`- -c
`
`-
`
`FIG.17B
`
`Amazon Ex. 1003, Page 24 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 24 of 34
`
`US 8,861,756 B2
`
`4000 -
`
`3500
`
`3000 -
`
`2500
`
`2000 -
`
`1500
`
`1000 -
`
`500 -
`0 .__ _ _,__ _ __,_ __ ,.__ _ _,__ _ ___._ __ -'--_ _,__ _ __,
`-200 -150 -100 -50
`0
`50 100 150
`200
`
`FIG.17C
`
`, -,
`.- ... -:-- - -
`'
`' - _.•- .-
`'
`' '
`
`, -
`
`'
`
`.- _. i ._
`'
`'
`.- .. - t ._ ._
`'
`
`~ ._ ._
`'
`
`l ·-
`
`-
`
`i
`I
`I
`
`-.
`
`~-. ._ ..
`
`._ ._
`
`i
`-..._:-.._
`
`- ~-.
`I
`
`-.._._
`
`I--._
`:
`
`-
`
`.- : - - ._ ._
`
`I
`
`--~-.
`'
`
`-. -.:._ ._
`
`'
`' --~-.
`
`'
`
`-. ._ ._ ~
`' ' '
`'
`' '' ' '
`'
`
`'
`
`- _. i
`'
`
`I
`,1..--"
`I
`
`-
`
`-
`
`_.
`
`"--.
`
`~
`
`' '
`' ' '
`'~
`' '
`' '
`' ..,
`
`- '
`'
`' ... '
`'
`' ' '
`--...
`'
`("- -
`'
`
`-.._._
`
`I
`
`-,
`'
`'
`'
`'
`._i
`'
`' ' '
`'
`
`- a
`
`,_
`
`,-•"I
`I
`I
`
`.- -
`
`.-
`
`I
`
`' , ,
`_. ---.-
`.- _. -.
`,, -
`' ' _.1- .-
`' _:_. - .- _.
`.- -.-
`
`I
`
`_. -
`
`.-
`
`1.4
`1.2
`1
`0.8
`0.6 ,
`0.4
`0.2· ,
`0
`
`O -200
`
`FIG.17D
`
`Amazon Ex. 1003, Page 25 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 25 of 34
`
`US 8,861,756 B2
`
`4000 --
`
`3500 --
`
`3000 --
`
`2500 --
`
`2000 --
`
`1500 --
`
`1000 --
`
`500
`
`' ' ' '
`' ' ' '
`--~
`' ' '
`-- ~
`' ' '
`0 ~-~-~-~-~-~-~~-~~·
`-200 -150 -100
`-50
`0
`200
`50
`100 150
`
`- - ,
`' ' ' '
`- -.
`' ' ' ' '
`' ' ' '
`- -. ' ' ' ' '
`
`- - "i
`
`- - "i
`
`- - ,
`
`FIG. 17E
`
`Amazon Ex. 1003, Page 26 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 26 of 34
`
`US 8,861,756 B2
`
`_. -
`
`-----~
`
`:
`
`- .- _. "'j
`
`,, _. -
`
`.-
`
`._-.
`
`- .- _. - :
`.- ~
`I
`
`I
`
`:
`
`.J _. - .- _. I
`
`~-----~----- : ----
`
`I
`I
`I
`_. L_
`
`I
`
`I
`
`:
`
`_.L_._.-:
`
`I
`
`_ _.--;.-
`_.--~-----:
`:
`:_ .. --~----:
`~ - -- - ~ -- - - :
`
`:
`
`I
`
`.- _. -,
`
`~ - - - -~ - - - -
`
`-
`
`:
`
`I
`
`_!- .- -
`
`I
`
`.-
`
`I
`
`!
`
`I
`
`:
`
`'
`
`-,-
`'
`'
`'
`'
`' -'
`
`'
`
`' -- ' ,,
`' ' '
`'
`'
`',_
`' -'
`
`'
`
`' -- ' ~,
`' ' '
`'
`' - ~
`' '
`'
`'
`'~
`
`'
`
`'
`
`•
`
`14
`1.2
`
`1
`0.8
`
`0
`
`_.--:------:
`I
`
`:
`I -
`
`~----"i
`•• -I
`I
`
`-
`
`06
`0.4 ---~-
`
`_ _.._.1
`
`I
`
`-
`
`I
`
`. - - - -~ - - - -~
`I
`I
`
`_1.---
`
`_. ~ -
`
`I
`
`.- ..
`
`:
`
`I
`
`_. -
`
`:
`
`-
`
`-
`
`- I
`'
`
`'
`' -
`
`' '.
`
`'
`
`' a
`
`FIG. 17F
`
`4000
`
`3500
`
`3000
`
`2500
`
`2000
`
`1500
`
`1000
`
`500
`
`0
`-200 -150 -100
`
`-50
`
`0
`
`50
`
`100 150 200
`
`FIG.17G
`
`Amazon Ex. 1003, Page 27 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 27 of 34
`
`US 8,861,756 B2
`
`iA-
`
`•. / ........
`
`.,. ....... ···+ .......
`
`!I
`
`,:
`•.•
`... ,•· f
`.. __
`
`... ·'-'·} ❖
`
`.... ~-·
`
`s
`
`., ,·: :: 1 ·:: :: : ~i,-
`
`5
`.S-'-'...
`
`...... :: .......
`:•
`
`HG, 171
`
`Amazon Ex. 1003, Page 28 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 28 of 34
`
`US 8,861,756 B2
`
`FIG.18A
`
`Amazon Ex. 1003, Page 29 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 29 of 34
`
`US 8,861,756 B2
`
`\..,
`\
`\
`\
`\
`\
`\ ,,
`
`FIG, lSC
`
`Amazon Ex. 1003, Page 30 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 30 of 34
`
`US 8,861,756 B2
`
`··-~ ' - : ___ .,__ __ __,_ ___ ..,___ __ __,_ ___ __,._ __ __,,
`
`a
`
`ffl
`
`M
`
`¾
`TIME{SEC}
`
`~
`
`¾
`
`{)
`
`m.
`nilE(SEC)
`
`FIG. l8D
`
`.t .... ___ ..........., _________ ............ ....._ _____ _
`
`~)
`
`TIME{SEO
`
`FIG, l8E
`
`Amazon Ex. 1003, Page 31 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 31 of 34
`
`US 8,861,756 B2
`
`t:
`
`.
`
`Tn. if · "f.(''•
`.t .~\!l ._. { :;j. .; .. : .~
`
`:;N
`T!ME.:SEO
`
`:FIG.18G
`
`Amazon Ex. 1003, Page 32 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 32 of 34
`
`US 8,861,756 B2
`
`FlG, 19A
`
`'f'lG.198
`
`Amazon Ex. 1003, Page 33 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 33 of 34
`
`US 8,861,756 B2
`
`FlG.19:C
`
`F'IG.190
`
`Amazon Ex. 1003, Page 34 of 47
`
`

`

`U.S. Patent
`
`Oct. 14, 2014
`
`Sheet 34 of 34
`
`US 8,861,756 B2
`
`HG,l9E
`
`301
`
`FIG. 19F
`
`Amazon Ex. 1003, Page 35 of 47
`
`

`

`US 8,861,756 B2
`
`1
`MICROPHONE ARRAY SYSTEM
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`This application claims the benefit of provisional patent
`application No. 61/403,952 titled "Microphone array design
`and implementation for telecommunications and handheld
`devices", filed on Sep. 24, 2010 in the United States Patent
`and Trademark Office.
`The specification of the above referenced patent applica(cid:173)
`tion is incorporated herein by reference in its entirety.
`
`BACKGROUND
`
`2
`source, for example, a person's speech that needs to be
`enhanced. A microphone array system comprising an array of
`sound sensors positioned in an arbitrary configuration, a
`sound source localization unit, an adaptive beamforming
`5 unit, and a noise reduction unit, is provided. The sound source
`localization unit, the adaptive beamforming unit, and the
`noise reduction unit are in operative communication with the
`array of sound sensors. The array of sound sensors is, for
`example, a linear array of sound sensors, a circular array of
`10 sound sensors, or an arbitrarily distributed coplanar array of
`sound sensors. The array of sound sensors herein referred to
`as a "microphone array" receives sound signals from multiple
`disparate sound sources. The method disclosed herein can be
`applied on a microphone array with an arbitrary number of
`15 sound sensors having, for example, an arbitrary two dimen(cid:173)
`sional (2D) configuration. The sound signals received by the
`sound sensors in the microphone array comprise the target
`sound signal from the target sound source among the dispar-
`ate sound sources, and ambient noise signals.
`The sound source localization unit estimates a spatial loca-
`tion of the target sound signal from the received sound sig(cid:173)
`nals, for example, using a steered response power-phase
`transform. The adaptive beamforming unit performs adaptive
`beamforming for steering a directivity pattern of the micro-
`25 phone array in a direction of the spatial location of the target
`sound signal. The adaptive beamforming unit thereby
`enhances the target sound signal from the target sound source
`and partially suppresses the ambient noise signals. The noise
`reduction unit suppresses the ambient noise signals for fur-
`30 ther enhancing the target sound signal received from the
`target sound source.
`In an embodiment where the target sound source that emits
`the target sound signal is in a two dimensional plane, a delay
`between each of the sound sensors and an origin of the micro-
`35 phone array is determined as a function of distance between
`each of the sound sensors and the origin, a predefined angle
`between each of the sound sensors and a reference axis, and
`an azimuth angle between the reference axis and the target
`sound signal. In another embodiment where the target sound
`40 source that emits the target sound signal is in a three dimen(cid:173)
`sional plane, the delay between each of the sound sensors and
`the origin of the microphone array is determined as a function
`of distance between each of the sound sensors and the origin,
`a predefined angle between each of the sound sensors and a
`45 first reference axis, an elevation angle between a second
`reference axis and the target sound signal, and an azimuth
`angle between the first reference axis and the target sound
`signal. This method of determining the delay enables beam(cid:173)
`forming for arbitrary numbers of sound sensors and multiple
`50 arbitrary microphone array configurations. The delay is deter(cid:173)
`mined, for example, in terms of number of samples. Once the
`delay is determined, the microphone array can be aligned to
`enhance the target sound signal from a specific direction.
`The adaptive beamforming unit comprises a fixed beam-
`55 former, a blocking matrix, and an adaptive filter. The fixed
`beamformer steers the directivity pattern of the microphone
`array in the direction of the spatial location of the target sound
`signal from the target sound source for enhancing the target
`sound signal, when the target sound source is in motion. The
`60 blocking matrix feeds the ambient noise signals to the adap(cid:173)
`tive filter by blocking the target sound signal from the target
`sound source. The adaptive filter adaptively filters the ambi(cid:173)
`ent noise signals in response to detecting the presence or
`absence of the target sound signal in the sound signals
`65 received from the disparate sound sources. The fixed beam(cid:173)
`former performs fixed beamforming, for example, by filtering
`and summing output sound signals from the sound sensors.
`
`Microphones constitute an important element in today's
`speech acquisition devices. Currently, most of the hands-free
`speech acquisition devices, for example, mobile devices,
`lapels, headsets, etc., convert sound into electrical signals by
`using a microphone embedded within the speech acquisition 20
`device. However, the paradigm of a single microphone often
`does not work effectively because the microphone picks up
`many ambient noise signals in addition to the desired sound,
`specifically when the distance between a user and the micro(cid:173)
`phone is more than a few inches. Therefore, there is a need for
`a microphone system that operates under a variety of different
`ambient noise conditions and that places fewer constraints on
`the user with respect to the microphone, thereby eliminating
`the need to wear the microphone or be in close proximity to
`the microphone.
`To mitigate the drawbacks of the single microphone sys(cid:173)
`tem, there is a need for a microphone array that achieves
`directional gain in a preferred spatial direction while sup(cid:173)
`pressing ambient noise from other directions. Conventional
`microphone arrays include arrays that are typically developed
`for applications such as radar and sonar, but are generally not
`suitable for hands-free or handheld speech acquisition
`devices. The main reason is that the desired sound signal has
`an extremely wide bandwidth relative to its center frequency,
`thereby rendering conventional narrowband techniques
`employed in the conventional microphone arrays unsuitable.
`In order to cater to such broadband speech applications, the
`array size needs to be vastly increased, making the conven(cid:173)
`tional microphone arrays large and bulky, and precluding the
`conventional microphone arrays from having broader appli(cid:173)
`cations, for example, in mobile and handheld communication
`devices. There is a need for a microphone array system that
`provides an effective response over a wide spectrum of fre(cid:173)
`quencies while being unobtrusive in terms of size.
`Hence, there is a long felt but unresolved need for a broad(cid:173)
`band microphone array and broadband beamforming system
`that enhances acoustics of a desired sound signal while sup(cid:173)
`pressing ambient noise signals.
`
`SUMMARY OF THE INVENTION
`
`This summary is provided to introduce a selection of con(cid:173)
`cepts in a simplified form that are further described in the
`detailed description of the invention. This summary is not
`intended to identify key or essential inventive concepts of the
`claimed subject matter, nor is it intended for determining the
`scope of the claimed subject matter.
`The method and system disclosed herein addresses the
`above stated need for enhancing acoustics of a target sound
`signal received from a target sound source, while suppressing
`ambient noise signals. As used herein, the term "target sound
`signal" refers to a sound signal from a desired or target sound
`
`Amazon Ex. 1003, Page 36 of 47
`
`

`

`US 8,861,756 B2
`
`5
`
`3
`In an embodiment, the adaptive filtering comprises sub(cid:173)
`band adaptive filtering. The adaptive filter comprises an
`analysis filter bank, an adaptive filter matrix, and a synthesis
`filter bank. The analysis filter bank splits the enhanced target
`sound signal from the fixed beamformer and the ambient
`noise signals from the blocking matrix into multiple fre(cid:173)
`quency sub-bands. The adaptive filter matrix adaptively fil(cid:173)
`ters the ambient noise signals in each of the frequency sub(cid:173)
`bands in response to detecting the presence or absence of the
`target sound signal in the sound signals received from the
`disparate sound sources. The synthesis filter bank synthesizes
`a full-band sound signal using the frequency sub-bands of the
`enhanced target sound signal. In an embodiment, the adaptive
`beamforming unit further comprises an adaptation control
`unit for detecting the presence of the target sound signal and
`adjusting a step size for the adaptive filtering in response to
`detecting the presence or the absence of the target sound
`signal in the sound signals received from the disparate sound
`sources.
`The noise reduction unit suppresses the ambient noise sig(cid:173)
`nals for further enhancing the target sound signal from the
`target sound source. The noise reduction unit performs noise
`reduction, for example, by using a Wiener-filter based noise
`reduction algorithm, a spectral subtraction noise reduction
`algorithm, an auditory transform based noise reduction alga- 25
`rithm, or a model based noise reduction algorithm. The noise
`reduction unit performs noise reduction in multiple frequency
`sub-bands employed for sub-band adaptive beamforming by
`the analysis filter bank of the adaptive beamforming unit.
`The microphone array system disclosed herein comprising
`the microphone array with an arbitrary number of sound
`sensors positioned in arbitrary configurations can be imple(cid:173)
`mented in handheld devices, for example, the iPad® of Apple
`Inc., the iPhone® of Apple Inc., smart phones, tablet com(cid:173)
`puters, laptop computers, etc. The microphone array system 35
`disclosed herein can further be implemented in conference
`phones, video conferencing applications, or any device or
`equipment that needs better speech inputs.
`
`4
`FIG. 6B exemplarily illustrates a table showing the rela(cid:173)
`tionship of the position of each sound sensor in the circular
`microphone array configuration and its distance to the origin
`of the microphone array, when the target sound source is in
`the same plane as that of the microphone array.
`FIG. 7A exemplarily illustrates a graphical representation
`of a microphone array, when the target sound source is in a
`three dimensional plane.
`FIG. 7B exemplarily illustrates a table showing delay
`10 between each sound sensor in a circular microphone array
`configuration and the origin of the microphone array, when
`the target sound source is in a three dimensional plane.
`FIG. 7C exemplarily illustrates a three dimensional work-
`15 ing space of the microphone array, where the target sound
`signal is incident at an elevation angle W<Q
`FIG. 8 exemplarily illustrates a method for estimating a
`spatial location of the target sound signal from the target
`sound source by a sound source localization unit using a
`20 steered response power-phase transform.
`FIG. 9A exemplarily illustrates a graph showing the value
`of the steered response power-phase transform for every 10°.
`FIG. 9B exemplarily illustrates a graph representing the
`estimated target sound signal from the target sound source.
`FIG. 10 exemplarily illustrates a system for performing
`adaptive beamforming by an adaptive beamforming unit.
`FIG.11 exemplarily illustrates a system for sub-band adap(cid:173)
`tive filtering.
`FIG. 12 exemplarily illustrates a graphical representation
`30 showing the performance of a perfect reconstruction filter
`bank.
`FIG. 13 exemplarily illustrates a block diagram of a noise
`reduction unit that performs noise reduction using a Wiener(cid:173)
`filter based noise reduction algorithm.
`FIG. 14 exemplarily illustrates a hardware implementation
`of the microphone array system.
`FIGS. 15A-15C exemplarily illustrate a conference phone
`comprising an eight-sensor microphone array.
`FIG. 16A exemplarily illustrates a layout of an eight-sen-
`40 sor microphone array for a conference phone.
`FIG.16B exemplarily illustrates a graphical representation
`of eight spatial regions to which the eight-sensor microphone
`array of FIG. 16A responds.
`FIGS. 16C-16D exemplarily illustrate computer simula-
`45 tions showing the steering of the directivity patterns of the
`eight-sensor microphone array of FIG. 16A in the directions
`of 15° and 60° respectively, in the frequency range 300 Hz to
`5kHz.
`FIGS. 16E-16L exemplarily illustrate graphical represen-
`50 tations showing the directivity patterns of the eight-sensor
`microphone array of FIG. 16A in each of the eight spatial
`regions, where each directivity pattern is an average response
`from 300 Hz to 5000 Hz.
`FIG.17A exemplarily illustrates a graphical representation
`55 of four spatial regions to which a four-sensor microphone
`array for a wireless handheld device responds.
`FIGS. 17B-17I exemplarily illustrate computer simula(cid:173)
`tions showing the directivity patterns of the four-sensor
`microphone array of FIG. 17A with respect to azimuth and
`frequency.
`FIGS. lSA-18B exemplarily illustrate a microphone array
`configuration for a tablet computer.
`FIG. 18C exemplarily illustrates an acoustic beam formed
`using the microphone array configuration of FIGS. lSA-18B
`according to the method and system disclosed herein.
`FIGS. 18D-18G exemplarily illustrate graphs showing
`processing results of the adaptive beamforming unit and the
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The foregoing summary, as well as the following detailed
`description of the invention, is better understood when read in
`conjunction with the appended drawings. For the purpose of
`illustrating the invention, exemplary constructions of the
`invention are shown in the drawings. However, the invention
`is not limited to the specific methods and instrumentalities
`disclosed herein.
`FIG. 1 illustrates a method for enhancing a target sound
`signal from multiple sound signals.
`FIG. 2 illustrates a system for enhancing a target sound
`signal from multiple sound signals.
`FIG. 3 exemplarily illustrates a microphone array configu(cid:173)
`ration showing a microphone array having N sound sensors
`arbitrarily distributed on a circle.
`FIG. 4 exemplarily illustrates a graphical representation of
`a filter-and-sum beamforming algorithm for determining out(cid:173)
`put of the microphone array having N sound sensors.
`FIG. 5 exemplarily illustrates distances between an origin
`of the microphone array and sound sensor M 1 and sound 60
`sensor M3 in the circular microphone array configuration,
`when the target sound signal is at an angle 8 from the Y-axis.
`FIG. 6A exemplarily illustrates a table showing the dis(cid:173)
`tance between each sound sensor in a circular microphone
`array configuration from the origin of the microphone array, 65
`when the target sound source is in