c19) United States
`c12) Reissued Patent
`
`Zhu
`
`et al.
`
`I 1111111111111111 1111111111111111 IIII IIIII IIIII 1111111111111111 Ill lllll llll
`
`
`
`
`US00RE4 7049E
`
`(10) Patent Number:
`(45) Date of Reissued
`
`US RE47,049 E
`Sep. 18, 2018
`
`Patent:
`
`(54) MICROPHONE ARRAY SYSTEM
`
`(71) Applicant: LI Creative Technologies, Inc.,
`Florham Park, NJ (US)
`
`(72)
`
`Inventors: Manli Zhu, Pearl River, NY (US); Qi
`Li, New Providence, NJ (US)
`
`(73) Assignee: LI Creative Technologies, Inc.,
`Florham Park, NJ (US)
`
`(21) Appl. No.: 15/293,626
`
`(22) Filed:
`
`Oct. 14, 2016
`
`Related U.S. Patent Documents
`
`Reissue of:
`(64) Patent No.:
`Issued:
`Appl. No.:
`Filed:
`U.S. Applications:
`(60) Provisional application No. 61/403,952, filed on Sep.
`24, 2010.
`
`8,861,756
`Oct. 14, 2014
`13/049,877
`Mar. 16, 2011
`
`(51)
`
`Int. Cl.
`H04R 25100
`H03G 3/20
`G0lS 3/805
`G0lS 3/801
`H04R 1140
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(Continued)
`
`(52) U.S. Cl.
`CPC
`
`G0lS 3/8055 (2013.01); G0lS 3/801
`(2013.01); G0lS 5122 (2013.01); H04R 11406
`(2013.01); H04R 3/005 (2013.01); H04M
`3/568 (2013.01); H04R 2201/401 (2013.01);
`H04R 2201/403 (2013.01)
`
`(58) Field of Classification Search
`CPC .......... G0lS 3/8055; G01S 5/22; G01S 3/801;
`H04R 3/005; H04R 1/406; H04R
`2201/403; H04R 2201/401; H04M 3/568
`............................... 381/300.57, 92, 94.1, 93
`USPC
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,315,562 A *
`7,039,199 B2
`
`5/1994 Bradley et al. ................. 367/89
`5/2006 Rui
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`4/2008
`W02008041878 A2
`WO
`WO 2010020162 Al * 2/2010
`WO
`Primary Examiner - Ovidio Escalante
`(74) Attorney, Agent, or Firm -Ash Tankha; Lipton,
`Weinberger & Husick
`
`H04R 1/403
`
`ABSTRACT
`(57)
`A method and system for enhancing a target sound signal
`from multiple sound signals is provided. An array of an
`arbitrary number of sound sensors positioned in an arbitrary
`configuration receives the sound signals from multiple dis(cid:173)
`parate sources. The sound signals comprise the target sound
`signal from a target sound source, and ambient noise signals.
`A sound source localization unit, an adaptive beamforming
`unit, and a noise reduction unit are in operative communi(cid:173)
`cation with the array of sound sensors. The sound source
`localization unit estimates a spatial location of the target
`sound signal from the received sound signals. The adaptive
`beamforming unit performs adaptive beamforming by steer(cid:173)
`ing a directivity pattern of the array of sound sensors in a
`direction of the spatial location of the target sound signal,
`thereby enhancing the target sound signal and partially
`suppressing the ambient noise signals, which are further
`suppressed by the noise reduction unit.
`
`35 Claims, 34 Drawing Sheets
`
`PROVU>E ,\ ~:!ICRKWI H)NL ARRAY .SYSTEM CO\!PRlS!NGAN" ARRAY
`OF WI :r,.;n SEN,ORS POSITIONED J\'
`AR!l!T!!ARY
`S(:WND SO\!lKT LtJCAUZ.\!'ION \;NIT.AN
`CONHGURA!10N.
`ADWTJ\T. KEAMfORWSG U'¾ff Mil),\ NO[SL REIH.'ClfflN
`
`101
`
`RECEl\"E ,n:LTIN.E sm::-m SIGNALS FRml MCLTJPLE
`SOLND SOllRCES llY THE SOt'ND Sl:NS-ORS
`
`\lFTHE TARGET SOL!¾lTJ SIG"AL
`ESf lM ~TE A SP,\TlAL UX'AllflN
`FROM Tm:: RLCUVED ,onm
`SJCN.\LS !W 'HIE
`L(K'AU7ATJON l'NlT
`
`-------------------~104
`l'ElffORM AD,WflVE BEAl',JHJRMlNG mR
`
`A DlRECTIVITY
`
`)
`
`.. , .... _, ... ~ ... ,-, .. ,- .... ~ ...... ~.----~,--~,--------------~,
`,--~-----~,-------·---------------------,
`.... , 105
`.'>U'l'RE5S Tm, AMK!ENT NOIS[ &lG~A!.S HY n11: I\OlSL n:rn.:cnnt>.
`/
`t:SJT
`
`Page 1 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`US RE47,049 E
`Page 2
`
`(51)
`
`Int. Cl.
`H04R 3/00
`GOJS 5122
`H04M 3/56
`
`(2006.01)
`(2006.01)
`(2006.01)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2003/0204397 Al * 10/2003 Amiri et al. .................. 704/231
`2004/0161121 Al*
`8/2004 Chol et al. ...................... 381/92
`2007/0076898 Al
`4/2007 Sarroukh et al.
`2008/0232607 Al*
`9/2008 Tashev ...................... G0lS 3/86
`381/71.11
`................... 381/71.7
`
`2009/0141907 Al *
`2009/0279714 Al
`2009/0304200 Al
`2010/0241426 Al*
`
`2011/0096915 Al*
`
`2011/0135125 Al*
`
`2011/0317522 Al*
`
`* cited by examiner
`
`6/2009 Kim et al.
`11/2009 Kim et al.
`12/2009 Kim et al.
`9/2010 Zhang ................. Gl0L 21/0208
`704/226
`H04M3/568
`379/158
`H04R 1/403
`381/303
`12/2011 Florencio .............. G0lS 3/8006
`367 /129
`
`4/2011 Nemer
`
`.
`
`6/2011 Zhan.
`
`Page 2 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 1 of 34
`
`US RE47,049 E
`
`PR()V*DE /-.. NffCROP!-lONE /-..RRA Y SYSTEM CO\lPRIS!t.;G .\N i\RRi\Y
`OF SOUND SENSORS POS!Tl{}NED lN AN ARfHTR/1,RY
`coNrlGURATl(>N, A SOlJND SOURCE UJC'ALlZATR)N UNlT,AN
`' ADAVUVE HEA!\=ffORt\:lING llNIT, AND A NOfSE REDt=('TlON t=Nff
`
`l
`i .UH
`! J
`r·-
`I
`,
`
`~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~v
`
`,
`
`\:fl f! J. 'H:,1
`~-, "'01· (?-'Z) .i::',,;:,,._·,.i, 1 --~ t;c·R>')\i! !\.,~'? ·11. t.)I _t.; f'l'"'.1'.s·•R i'-·T· t.;
`,,·:a;cc'!'?\-'~,·
`--> .... ,ih
`.. L
`.. ,.H-...,.~"'<.• d,::-i
`t
`l' .• fa
`l.
`-... ,.
`,.H.
`L
`, • .,_ L
`i'U.,
`...
`!.
`L,
`l' , ... , H"L.L
`/S.
`SOUND SOURCES HY THE SOt'NO SENSORS
`
`~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~v
`
`❖"""""""""""""""""""""""""""""""""""""""""w ! """""""""""""""""""""""""""""""""""""""w
`l
`i
`'
`'
`'
`❖"""""""""""""""""""""""""""""""""""""""""w l """""""""""""""""""""""""""""""""""""""WX
`
`,
`
`i H}2
`,
`~~_;
`'
`'
`'
`
`t. ~
`
`i .s.tt.-,
`\
`..
`t,...,..J
`!
`'
`l
`
`rt"r·1·~ ~ ~'J'P
`'f >-1·••cr't' 0(··,~ '"!=>'J"' ,., ,,.,.,._, '$
`,. ,....,,., .• ·•ru·">N' (·">;f' J'Ur
`\ ,·=r, ., 't'·I >. 1
`.. ,,, ... A h . .<i··,, .. ,,.
`.• '-.:'>(>,.Ji'i.i-s.s.-.
`..:'>i='i\i. A.L,.,,,
`ni::l
`i::l}
`_;:·\>,,,,.$s.":;); ,3·.-.H.-'l~-
`s.~.;>
`., .• ,,,..,.
`fl{O\.·t THE R!-x::RrvEn sor:,Nn SIGNALS BY THE SOUND sovRcE
`LOCA.UZA,TlON t=NIT
`
`l
`l
`l
`l
`'
`'
`
`I t
`THE SPATt=\l .. LOCATl{)N or THE TA.RGET sot,ND s1cNA,t .. B v THE I
`I
`1ns
`l SL!·1·'RES.S THE A\11-m•NT NOISE SIGNALS HY THE NOl:SE REDtJ::T!Ut--1 t.,,,/
`'
`'
`l
`UNIT
`!
`'
`'
`'
`'
`'
`'
`'
`'
`L .•. ·.························································································································································································································································································································································i
`
`=============w,-,...==============i
`PERFORM ,ADAfTl\lE Bb\MFORiVUNG FOR STEERING A D!RECfiV!TY
`PATfERN OF THE ARRAY UFSOUND SENSfiRS lN /\. D!RFCT!ON OF
`
`ADAVHVE HEAtJH}RM!N(li FN!T
`
`H:W
`! ,./
`\,---·
`
`:
`
`~:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-.-.::-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:.):
`
`'
`
`""""""""""""""""""""""""""""""""""""""""''
`
`""""""""""""""""""""""""""""""""""""""""''
`
`FlG, 1
`
`Page 3 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`e •
`
`00
`•
`~
`~
`~
`
`~ = ~
`
`rJJ
`('D
`
`'? ....
`0 ....
`
`~CIO
`N
`
`CIO
`
`('D
`
`rJJ =(cid:173)
`('D .....
`201
`N
`0 ....
`........................................................................................
`
`:.-;,,w".,-(;: .............. ..
`(,H
`.i;...
`
`I
`
`. ZOl
`/.
`
`;
`i
`
`i ·11 R\YCr,··1'Nf
`:=\ '· ;• _.. .) • .. ::,.tJ .}.;. ·= .)
`.sr:Nsrnrn
`I ..... ·.·····················································:;························································
`
`!
`:
`
`202
`J
`
`~ '
`1
`;
`
`., •• j"'~•, ···•,·•
`., •• l°''Ur··
`.\O .,,::.-.:,J SO .l<>'.L,::
`l nc: r !Z Yl'RiN FNIT
`
`,
`►
`r················
`
`
`
`1 ....... :::: ... :::: .. : ... :::.: .... :.::: ...... :.: ........... .
`
`r200
`
`203
`.J
`AD.,:\PT!VE BEAMFORMlNG LN!T
`21M
`
`
`
`-,,;·····································································•·,,.-.,:.:;·l ________ . ._.
`l
`lTXED
`I
`l BEAMF'ORMER
`I
`I
`~
`~ ~~~~~~~ ~ ~~~~~~~~~~✓ 106
`I
`l
`
`1(}5
`
`rs~:xit]
`
`I
`
`-~
`
`..
`
`.
`
`---
`
`•.
`
`•
`
`..
`
`..
`
`..
`
`-~
`
`.
`
`••
`
`..
`
`y
`
`...
`
`,t.,_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._.N•
`
`.
`-,
`A.DAPfiVE HLTER
`/, .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. ,.-,..,.., •. ,..fff)~l
`l ,~NAl .. \Sl.S Hl .. fl·,R g,,,NK r
`~
`L .... ✓i
`r
`
`mm
`
`mmml
`
`206b
`~t ..... ~i
`i ADAP'fl\'[ HUER M \ TRIX Lmmmmmmmm
`l .....................................................................
`J 2J~
`M)lSE- ~<-;,~~~~.'.(TI{·iN
`i •")' .. . ....,
`i Sr NI H±5,S fILU,R BANK
`L .... , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. , .. ,w.,wu.•·•·•~·
`
`"I'..
`
`•
`
`•
`
`•
`
`·,
`
`•
`
`••
`
`~--❖~-
`~······················· ••••••• ••••••••••••
`
`muuuuuuuuuuuuuuuuuuuuuuuummm
`
`FlG.2
`
`d r.,;_
`~
`'-"--...l = ~
`
`~
`
`\0
`~
`
`Page 4 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 3 of 34
`
`US RE47,049 E
`
`Page 5 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 4 of 34
`
`US RE47,049 E
`
`Page 6 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 5 of 34
`
`US RE47,049 E
`
`Smm<l
`S<:'!bOr
`MO
`Ml
`
`HG.6A
`
`n''
`
`1)i.\b.v t fmonber of
`·.
`...
`:,am pk:i)
`
`,,way fri:m1 rnff
`$ ,imido-::::kw·i~e
`{0:::d,:;::90'':r
`
`Page 7 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 6 of 34
`
`US RE47,049 E
`
`'FlG. 7A
`
`T,::::::
`1\;""''
`d~fr,!ot:•n~-i.O·Hh,)i,irifFVt
`d~/i.•tu*,(0-
`111 i ):;,i n{ttiv t
`
`T••;z:;
`-d ~/\•{T,1,(H +(fi2 f,j n{\l1)A.~
`
`T:,c::::::
`-d~JktcoM0-
`4 1:, W,i n{tl 1Vt
`
`FIG. iB
`
`Page 8 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 7 of 34
`
`US RE47,049 E
`
`FIG. 7C
`
`Page 9 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 8 of 34
`
`US RE47,049 E
`
`801
`L-,J
`
`802
`u
`
`803
`l-,J
`
`804
`l-,J
`
`FOR DIRECTION i (0 :S i '.S 360), CALCULATE THE DELAY
`Dit BETWEEN THE lh PAIR OF THE SOUND SENSORS (t=l:
`ALL PAIRS)
`
`l
`
`CALCULATE THE CORRELATION VALUE corr(Dit)
`BETWEEN THE lh PAIR OF THE SOUND SENSORS
`CORRESPONDING TO THE DELAY OF Dit
`
`l
`
`FOR THE DIRECTION i (0 :S i '.S 360),
`ALLPAIR
`'I.corr(Dit)
`t=l
`
`CORR.=
`l
`
`l
`
`THE TARGET SOUND SIGNAL COMES FROM DIRECTION
`S = argmax CORRi
`0~i~360
`
`FIG.8
`
`Page 10 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 9 of 34
`
`US RE47,049 E
`
`0.7,........--
`
`......... ---....---........-----,,.-----,,----......,,..--~
`
`0.4 •·
`
`<r
`
`I 'l'J ';_' '[' i h, n i'I'l ·r <[l: ir t]'1.'' :"Il :r !I'):;[:·::"']' 1.•' <!" r·:
`
`'•'•'•'
`
`'•'•'•' ... '•'•'•'
`
`•-'•••:>.•.••••y.••••••••••••••❖•••••••~••••••••Si•.•••••••••• ,,,,•
`
`,,,,•
`
`••·+·••
`
`• ''''
`
`·•••••''•'•'•'
`
`'•'•'•'• .. '•'•'•'•
`
`•.•.❖X•.•••••• '•'•'•'
`5
`
`'•'•'•' •'•'•'•"'•'•'•'•' J. ..... ,-
`
`__
`
`6
`
`_,
`
`l
`
`Page 11 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 10 of 34
`
`US RE47,049 E
`
`LN•••NA
`
`•
`
`,uN•••N••
`
`'
`
`""""""""""""""""""""'",
`
`"""""""'"
`
`' ~ ... J
`~
`
`t,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
`
`.. , .... ,p 2
`l
`
`.
`
`Page 12 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 11 of 34
`
`US RE47,049 E
`
`f
`
`Page 13 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 12 of 34
`
`US RE47,049 E
`
`¢?.} ~· 8,
`.,,_
`
`Page 14 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 13 of 34
`
`US RE47,049 E
`
`~
`WJ
`~
`,....,
`;....<
`~ ~
`=:.t.:z
`;;.. <
`:: ~
`1
`<
`
`Page 15 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 14 of 34
`
`US RE47,049 E
`
`Page 16 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 15 of 34
`
`US RE47,049 E
`
`FIG. l5A
`
`1502
`
`rl500
`
`1501
`
`FIG. I.SB
`
`Page 17 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 16 of 34
`
`US RE47,049 E
`
`"
`
`1502
`
`□
`
`FIG. lSC
`
`Page 18 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 17 of 34
`
`US RE47,049 E
`
`FIG.16.A
`
`11G. l6B
`
`Page 19 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 18 of 34
`
`US RE47,049 E
`
`_. -.--
`
`__ ,..
`
`.-..-:-.
`
`--+---r ~ -
`
`-
`
`_. - "1
`
`' ' ~--
`
`-:: - - +-
`
`_. -
`
`.-1
`
`,
`
`_. - ... ,
`'
`' '
`
`I
`
`' -
`... - ...
`'
`'
`
`I
`
`, -,,
`
`1.4
`1.2
`1
`0.
`
`0.6
`0.4
`
`0.2
`0
`5000
`4000
`3000 ~
`2000
`
`-
`
`-100 -50
`0 -200 -150
`
`0
`
`1.4
`1.2
`I
`0.8
`
`0.6
`0.4
`
`0.2
`0
`
`FIG.16C
`
`~ _. -
`
`_.
`
`~ _. -
`1
`I
`
`-
`
`-
`
`I
`
`,,
`
`I
`.- - ,
`
`-
`
`.-
`
`-
`
`....
`
`.-
`
`.-
`
`.-
`
`._
`
`'
`
`- T
`I
`I
`I
`
`-
`
`~-.-.
`I
`I
`I
`- T ._
`I
`I
`I
`
`._
`
`I
`I
`-..I.
`
`-- ""
`
`- -- T ......
`
`' ,
`
`.-1
`
`~ -
`'
`, - ,,
`
`I_.
`
`' -
`' - - ',
`- ',
`'
`'
`' - --·--
`
`.- _,. -
`
`I
`
`_. - --.
`
`--:-.-
`
`.- -
`
`.-
`
`I
`
`0 -200
`
`FIG.16D
`
`' -' -'
`' '
`' --'
`'~
`'
`-' 1 -
`'
`' -I
`
`-..._
`
`'
`
`...,
`
`-"
`'
`' - '
`'
`I -- ' -,
`
`Page 20 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 19 of 34
`
`US RE47,049 E
`
`Page 21 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 20 of 34
`
`US RE47,049 E
`
`rn:,
`
`FIG. l6G
`
`Page 22 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 21 of 34
`
`US RE47,049 E
`
`FIG. l6J
`
`Page 23 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 22 of 34
`
`US RE47,049 E
`
`BG, l6L
`
`Page 24 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 23 of 34
`
`US RE47,049 E
`
`180°
`
`201
`
`~02
`
`'
`'
`
`'
`
`... :·-...
`' '
`··--~
`'
`'
`
`''
`
`--.... :_
`' -
`'
`'
`' ·~._ ...
`
`'
`
`---: ...
`'
`'
`
`--
`
`• • .._ -
`
`••._
`
`I
`,''
`' '
`I --...
`
`t
`
`FIG.17A
`
`FIG.17B
`
`1.4"
`1.2
`
`1
`
`0.
`
`0.
`
`0.
`0.2..----
`
`Page 25 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 24 of 34
`
`US RE47,049 E
`
`4000 -
`3500
`
`3000 -
`
`2500 -
`
`2000 -
`
`1500
`
`1000 -
`
`500 -
`0~-~-~--~-~-~--~-~-~
`-200 -150 -100 -50 0
`
`50 100 150 200
`
`FIG.17C
`
`.-•"I
`
`I
`I
`
`' , ,
`_. ---.-
`' ' ,, -
`.- _. -.
`' ' _.1-
`'
`
`.-
`
`-
`
`.-
`
`I
`
`_. -
`
`I
`
`.- -.-
`'
`
`.-
`
`1.4
`1.2
`1
`0.8
`0.6 ,
`0.4
`0.2· ,
`0
`
`- _. i
`'
`
`I
`,1..--"
`_. I
`
`-
`
`-
`
`.-
`
`'
`
`.- _. i
`
`._
`
`'
`'
`.- .. - t ._ ._
`'
`
`-
`
`i
`I
`
`I
`
`-.
`
`~-.
`
`._ ._
`
`._ ._
`
`i
`
`-..._:-.._
`
`-
`
`.-
`
`:--
`
`._ ._
`
`I
`
`-.
`
`.- ... -:----
`'
`' - _.•-.-
`'
`'
`'
`
`, -
`
`l ·-
`
`-
`
`I
`~-.
`
`I
`
`._ ._ ~
`'''
`
`-.._._
`
`I--._
`:
`
`"--.
`
`-. -.:._
`
`._
`
`' '
`
`',
`
`, ',
`
`-...-.._
`
`I
`
`200
`
`100
`
`0
`
`-100
`
`O -200
`
`FIG.17D
`
`Page 26 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 25 of 34
`
`US RE47,049 E
`
`4000 --
`
`3500 --
`
`3000 --
`
`2500 --
`
`2000 --
`
`1500 --
`
`1000 --
`
`500 --
`
`0 L____J
`-200 -150 -100
`
`__J
`
`_
`
`_
`
`__J_
`
`_
`
`__J_
`
`_
`
`___J_
`
`_
`
`___J__-----1...
`
`_
`
`-50
`
`0
`
`50
`
`100 150
`
`FIG. 17E
`
`- - "i
`
`- - "i
`
`- - ,
`
`- - ,
`' ' ' '
`- -.
`' ' ' ' '
`' ' ' '
`- -. ' ' ' ' '
`' ' ' '
`' ' ' '
`--~
`' ' '
`--~
`' ' '
`200
`
`___J'
`
`Page 27 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 26 of 34
`
`US RE47,049 E
`
`,, _. -
`
`.-
`
`._-.
`
`I
`I
`I
`_. L_
`
`- .- _. -:
`.- ~
`I
`
`I
`
`: ~-----~-----: ----
`
`.J _. - .- _. I
`
`I
`
`_. -
`
`-----~
`- .- _. "'j
`:
`
`_ _.--;.- I
`
`•
`
`14
`1.2
`
`1
`0.8
`
`0
`
`:
`
`I
`
`.- _. -,
`
`_.--:------:
`
`_ _.._.
`
`-
`
`1
`
`I
`
`.----~----~
`
`06
`0.4 ---~-
`
`I
`
`_. -
`
`:
`
`-
`
`-
`
`- I
`'
`
`:
`_.--~-----:
`:_ .. --~----:
`~ - --- ~ --- - :
`
`~----~----
`
`I
`
`!
`
`I
`
`I
`
`''
`
`'
`
`'
`' ,,
`' ''
`'
`'
`' ''
`
`',,
`
`'
`' ~,
`' ''
`'
`' - ~
`' '
`'
`'
`
`'~
`
`''
`
`'
`
`'
`
`:
`
`_.L_._.-:
`
`I
`
`-
`
`:
`
`I
`
`_!-
`
`.- -
`
`I
`
`.-
`
`:
`I -
`
`-
`
`•• -I
`
`~----"i
`
`I
`
`_. ~ -
`
`I
`
`.-
`
`..
`
`_1.---
`
`:
`
`I
`
`I
`
`:
`
`I
`
`FIG. 17F
`
`4000
`
`3500
`
`3000
`
`2500
`
`2000
`
`1500
`
`1000
`
`500
`
`0
`-200 -150 -100
`
`-50
`
`0
`
`50
`
`100 150 200
`
`FIG.17G
`
`Page 28 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 27 of 34
`
`US RE47,049 E
`
`Page 29 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 28 of 34
`
`US RE47,049 E
`
`FIG.18A
`
`Page 30 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 29 of 34
`
`US RE47,049 E
`
`FIG, lSC
`
`Page 31 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 30 of 34
`
`US RE47,049 E
`
`rn
`
`m.
`Tri'lH(SEC)
`
`......., _____ _
`... ~• .... _______________
`
`ti
`
`FIG, l8E
`
`Page 32 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 31 of 34
`
`US RE47,049 E
`
`:;N
`T!ME.:SEO
`
`:FIG.18G
`
`Page 33 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 32 of 34
`
`US RE47,049 E
`
`FlG, 19A
`
`'f'lG.198
`
`Page 34 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 33 of 34
`
`US RE47,049 E
`
`F'IG.190
`
`Page 35 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`U.S. Patent
`
`Sep.18,2018
`
`Sheet 34 of 34
`
`US RE47,049 E
`
`HG,l9E
`
`301
`
`Page 36 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`1
`MICROPHONE ARRAY SYSTEM
`
`US RE47,049 E
`
`2
`detailed description of the invention. This summary is not
`intended to identify key or essential inventive concepts of
`the claimed subject matter, nor is it intended for determining
`the scope of the claimed subject matter.
`The method and system disclosed herein addresses the
`above stated need for enhancing acoustics of a target sound
`signal received from a target sound source, while suppress(cid:173)
`ing ambient noise signals. As used herein, the term "target
`sound signal" refers to a sound signal from a desired or
`10 target sound source, for example, a person's speech that
`needs to be enhanced. A microphone array system compris(cid:173)
`ing an array of sound sensors positioned in an arbitrary
`configuration, a sound source localization unit, an adaptive
`beamforming unit, and a noise reduction unit, is provided.
`15 The sound source localization unit, the adaptive beamform(cid:173)
`ing unit, and the noise reduction unit are in operative
`communication with the array of sound sensors. The array of
`sound sensors is, for example, a linear array of sound
`sensors, a circular array of sound sensors, or an arbitrarily
`20 distributed coplanar array of sound sensors. The array of
`sound sensors herein referred to as a "microphone array"
`receives sound signals from multiple disparate sound
`sources. The method disclosed herein can be applied on a
`microphone array with an arbitrary number of sound sensors
`25 having, for example, an arbitrary two dimensional (2D)
`configuration. The sound signals received by the sound
`sensors in the microphone array comprise the target sound
`signal from the target sound source among the disparate
`sound sources, and ambient noise signals.
`The sound source localization unit estimates a spatial
`location of the target sound signal from the received sound
`signals, for example, using a steered response power-phase
`transform. The adaptive beamforming unit performs adap(cid:173)
`tive beamforming for steering a directivity pattern of the
`35 microphone array in a direction of the spatial location of the
`target sound signal. The adaptive beamforming unit thereby
`enhances the target sound signal from the target sound
`source and partially suppresses the ambient noise signals.
`The noise reduction unit suppresses the ambient noise
`40 signals for further enhancing
`the
`target sound signal
`received from the target sound source.
`In an embodiment where the target sound source that
`emits the target sound signal is in a two dimensional plane,
`a delay between each of the sound sensors and an origin of
`45 the microphone array is determined as a function of distance
`between each of the sound sensors and the origin, a pre(cid:173)
`defined angle between each of the sound sensors and a
`reference axis, and an azimuth angle between the reference
`axis and the target sound signal. In another embodiment
`50 where the target sound source that emits the target sound
`signal is in a three dimensional plane, the delay between
`each of the sound sensors and the origin of the microphone
`array is determined as a function of distance between each
`of the sound sensors and the origin, a predefined angle
`55 between each of the sound sensors and a first reference axis,
`an elevation angle between a second reference axis and the
`target sound signal, and an azimuth angle between the first
`reference axis and the target sound signal. This method of
`determining the delay enables beamforming for arbitrary
`60 numbers of sound sensors and multiple arbitrary microphone
`array configurations. The delay is determined, for example,
`in terms of number of samples. Once the delay is deter(cid:173)
`mined, the microphone array can be aligned to enhance the
`target sound signal from a specific direction.
`The adaptive beamforming unit comprises a fixed beam(cid:173)
`former, a blocking matrix, and an adaptive filter. The fixed
`beamformer steers the directivity pattern of the microphone
`
`Matter enclosed in heavy brackets [ ] appears in the
`original patent but forms no part of this reissue specifica(cid:173)
`tion; matter printed in italics indicates the additions 5
`made by reissue; a claim printed with strikethrough
`indicates that the claim was canceled, disclaimed, or held
`invalid by a prior post-patent action or proceeding.
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`This application is a reissue application of U.S. patent
`application Ser. No. 13/049,877, filed Mar. 16, 2011 (now
`U.S. Pat. No. 8,861,756), which claims the benefit of pro(cid:173)
`visional patent application No. 61/403,952 titled "Micro(cid:173)
`phone array design and implementation for telecommunica(cid:173)
`tions and handheld devices", filed on Sep. 24, 2010 in the
`United States Patent and Trademark Office.
`The specification of the above referenced patent applica(cid:173)
`tion is incorporated herein by reference in its entirety.
`
`BACKGROUND
`
`Microphones constitute an important element in today's
`speech acquisition devices. Currently, most of the hands-free
`speech acquisition devices, for example, mobile devices,
`lapels, headsets, etc., convert sound into electrical signals by
`using a microphone embedded within the speech acquisition
`device. However, the paradigm of a single microphone often
`does not work effectively because the microphone picks up 30
`many ambient noise signals in addition to the desired sound,
`specifically when the distance between a user and the
`microphone is more than a few inches. Therefore, there is a
`need for a microphone system that operates under a variety
`of different ambient noise conditions and that places fewer
`constraints on the user with respect to the microphone,
`thereby eliminating the need to wear the microphone or be
`in close proximity to the microphone.
`To mitigate the drawbacks of the single microphone
`system, there is a need for a microphone array that achieves
`directional gain in a preferred spatial direction while sup(cid:173)
`pressing ambient noise from other directions. Conventional
`microphone arrays include arrays that are typically devel(cid:173)
`oped for applications such as radar and sonar, but are
`generally not suitable for hands-free or handheld speech
`acquisition devices. The main reason is that the desired
`sound signal has an extremely wide bandwidth relative to its
`center frequency, thereby rendering conventional narrow(cid:173)
`band techniques employed in the conventional microphone
`arrays unsuitable. In order to cater to such broadband speech
`applications, the array size needs to be vastly increased,
`making the conventional microphone arrays large and bulky,
`and precluding the conventional microphone arrays from
`having broader applications, for example, in mobile and
`handheld communication devices. There is a need for a
`microphone array system that provides an effective response
`over a wide spectrum of frequencies while being unobtru(cid:173)
`sive in terms of size.
`Hence, there is a long felt but unresolved need for a
`broadband microphone array and broadband beamforming
`system that enhances acoustics of a desired sound signal
`while suppressing ambient noise signals.
`
`SUMMARY OF THE INVENTION
`
`This summary is provided to introduce a selection of
`concepts in a simplified form that are further described in the
`
`65
`
`Page 37 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`US RE47,049 E
`
`3
`array in the direction of the spatial location of the target
`sound signal from the target sound source for enhancing the
`target sound signal, when the target sound source is in
`motion. The blocking matrix feeds the ambient noise signals
`to the adaptive filter by blocking the target sound signal from
`the target sound source. The adaptive filter adaptively filters
`the ambient noise signals in response to detecting the
`presence or absence of the target sound signal in the sound
`signals received from the disparate sound sources. The fixed
`beamformer performs fixed beamforming, for example, by
`filtering and summing output sound signals from the sound
`sensors.
`In an embodiment, the adaptive filtering comprises sub(cid:173)
`band adaptive filtering. The adaptive filter comprises an
`analysis filter bank, an adaptive filter matrix, and a synthesis
`filter bank. The analysis filter bank splits the enhanced target
`sound signal from the fixed beamformer and the ambient
`noise signals from the blocking matrix into multiple fre(cid:173)
`quency sub-bands. The adaptive filter matrix adaptively
`filters the ambient noise signals in each of the frequency
`sub-bands in response to detecting the presence or absence
`of the target sound signal in the sound signals received from
`the disparate sound sources. The synthesis filter bank syn(cid:173)
`thesizes a full-band sound signal using the frequency sub(cid:173)
`bands of the enhanced target sound signal. In an embodi(cid:173)
`ment, the adaptive beamforming unit further comprises an
`adaptation control unit for detecting the presence of the
`target sound signal and adjusting a step size for the adaptive
`filtering in response to detecting the presence or the absence
`of the target sound signal in the sound signals received from
`the disparate sound sources.
`The noise reduction unit suppresses the ambient noise
`signals for further enhancing the target sound signal from the 35
`target sound source. The noise reduction unit performs noise
`reduction, for example, by using a Wiener-filter based noise
`reduction algorithm, a spectral subtraction noise reduction
`algorithm, an auditory transform based noise reduction
`algorithm, or a model based noise reduction algorithm. The
`noise reduction unit performs noise reduction in multiple
`frequency sub-bands employed for sub-band adaptive beam(cid:173)
`forming by the analysis filter bank of the adaptive beam(cid:173)
`forming unit.
`The microphone array system disclosed herein compris(cid:173)
`ing the microphone array with an arbitrary number of sound
`sensors positioned in arbitrary configurations can be imple(cid:173)
`mented in handheld devices, for example, the iPad® of
`Apple Inc., the iPhone® of Apple Inc., smart phones, tablet
`computers, laptop computers, etc. The microphone array 50
`system disclosed herein can further be implemented in
`conference phones, video conferencing applications, or any
`device or equipment that needs better speech inputs.
`
`4
`FIG. 3 exemplarily illustrates a microphone array con(cid:173)
`figuration showing a microphone array having N sound
`sensors arbitrarily distributed on a circle.
`FIG. 4 exemplarily illustrates a graphical representation
`5 of a filter-and-sum beamforming algorithm for determining
`output of the microphone array having N sound sensors.
`FIG. 5 exemplarily illustrates distances between an origin
`of the microphone array and sound sensor M 1 and sound
`sensor M3 in the circular microphone array configuration,
`10 when the target sound signal is at an angle 8 from the Y-axis.
`FIG. 6A exemplarily illustrates a table showing the dis(cid:173)
`tance between each sound sensor in a circular microphone
`array configuration from the origin of the microphone array,
`when the target sound source is in the same plane as that of
`15 the microphone array.
`FIG. 6B exemplarily illustrates a table showing the rela(cid:173)
`tionship of the position of each sound sensor in the circular
`microphone array configuration and its distance to the origin
`of the microphone array, when the target sound source is in
`20 the same plane as that of the microphone array.
`FIG. 7 A exemplarily illustrates a graphical representation
`of a microphone array, when the target sound source is in a
`three dimensional plane.
`FIG. 7B exemplarily illustrates a table showing delay
`25 between each sound sensor in a circular microphone array
`configuration and the origin of the microphone array, when
`the target sound source is in a three dimensional plane.
`FIG. 7C exemplarily illustrates a three dimensional work(cid:173)
`ing space of the microphone array, where the target sound
`30 signal is incident at an elevation angle W<Q
`FIG. 8 exemplarily illustrates a method for estimating a
`spatial location of the target sound signal from the target
`sound source by a sound source localization unit using a
`steered response power-phase transform.
`FIG. 9A exemplarily illustrates a graph showing the value
`of the steered response power-phase transform for every 10°.
`FIG. 9B exemplarily illustrates a graph representing the
`estimated target sound signal from the target sound source.
`FIG. 10 exemplarily illustrates a system for performing
`40 adaptive beamforming by an adaptive beamforming unit.
`FIG. 11 exemplarily illustrates a system for sub-band
`adaptive filtering.
`FIG. 12 exemplarily illustrates a graphical representation
`showing the performance of a perfect reconstruction filter
`45 bank.
`FIG. 13 exemplarily illustrates a block diagram of a noise
`reduction unit that performs noise reduction using a Wiener(cid:173)
`filter based noise reduction algorithm.
`FIG. 14 exemplarily illustrates a hardware implementa(cid:173)
`tion of the microphone array system.
`FIGS. 15A-15C exemplarily illustrate a conference phone
`comprising an eight-sensor microphone array.
`FIG. 16A exemplarily illustrates a layout of an eight(cid:173)
`sensor microphone array for a conference phone.
`FIG. 16B exemplarily illustrates a graphical representa(cid:173)
`tion of eight spatial regions to which the eight-sensor
`microphone array of FIG. 16A responds.
`FIGS. 16C-16D exemplarily illustrate computer simula(cid:173)
`tions showing the steering of the directivity patterns of the
`60 eight-sensor microphone array of FIG. 16A in the directions
`of 15° and 60° respectively, in the frequency range 300 Hz
`to 5 kHz.
`FIGS. 16E-16L exemplarily illustrate graphical represen(cid:173)
`tations showing the directivity patterns of the eight-sensor
`65 microphone array of FIG. 16A in each of the eight spatial
`regions, where each directivity pattern
`is an average
`response from 300 Hz to 5000 Hz.
`
`55
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The foregoing summary, as well as the following detailed
`description of the invention, is better understood when read
`in conjunction with the appended drawings. For the purpose
`of illustrating the invention, exemplary constructions of the
`invention are shown in the drawings. However, the invention
`is not limited to the specific methods and instrumentalities
`disclosed herein.
`FIG. 1 illustrates a method for enhancing a target sound
`signal from multiple sound signals.
`FIG. 2 illustrates a system for enhancing a target sound
`signal from multiple sound signals.
`
`Page 38 of 51
`
`SONOS EXHIBIT 1001
`
`

`

`5
`FIG. 17A exemplarily illustrates a graphical representa(cid:173)
`tion of four spatial regions to which a four-sensor micro(cid:173)
`phone array for a wireless handheld device responds.
`FIGS. 17B-17I exemplarily illustrate computer simula(cid:173)
`tions showing the directivity patterns of the four-sensor 5
`microphone array of FIG. 17 A with respect to azimuth and
`frequency.
`FIGS. lSA-18B exemplarily illustrate a microphone array
`configuration for a tablet computer.
`FIG. 18C exemplarily illustrates an acoustic beam formed 10
`using the microphone array configuration of FI GS. lSA-18B
`according to the method and system disclosed herein.
`FIGS. 18D-18G exemplarily illustrate graphs showing
`processing results of the adaptive beamforming unit and the
`noise reduction unit for the microphone array configuration 15
`of FIG. 18B, in both a time domain and a spectral domain
`for the tablet computer.
`FIGS. 19A-19F exemplarily
`illustrate tables showing
`different microphone array configurations and the corre(cid:173)
`sponding values of delay -cm f