Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 1 of 20 PageID #: 74
`
`
`
`
`
`
`
`Exhibit C
`
`U.S. Patent No. 6,744,818
`
`Method and Apparatus for Visual Perception Encoding
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 2 of 20 PageID #: 75
`I 1111111111111111 11111 111111111111111 IIIII IIIII IIIII IIIII 111111111111111111
`US006744818B2
`
`(12) United States Patent
`Sheraizin et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,744,818 B2
`Jun.1,2004
`
`(54) METHOD AND APPARATUS FOR VISUAL
`PERCEPTION ENCODING
`
`(75)
`
`Inventors: Vitaly S. Sheraizin, Mazkeret Batya
`(IL); Semion M. Sheraizin, Mazkeret
`Batya (IL)
`
`(73) Assignee: VLS Com Ltd., Rechovot (IL)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 421 days.
`
`(21) Appl. No.: 09/748,248
`
`(22) Filed:
`
`Dec. 27, 2000
`
`(65)
`
`Prior Publication Data
`
`US 2002/0122494 Al Sep. 5, 2002
`
`(51)
`
`Int. Cl.7 .................................................. H04N 7/12
`
`(52) U.S. Cl. .................................................. 375/240.29
`
`(58) Field of Search ............................ 375/240, 240.01,
`375/240.16, 240.29; 382/264; 704/229,
`500; H04N 7/12
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`U.S. patent application Ser. No. 09/524,618, Sheraizin et al.,
`filed Mar. 14, 2000.
`Raj Talluri, et al., "A Robust, Scalable, Object-Based Video
`Compression Technique for Very Low Bit-Rate Coding",
`IEEE Transaction of Circuit and Systems for Video Tech(cid:173)
`nology, vol. 7, No. 1, Feb. 1997.
`Awad Kh. Al-Asmari," An Adaptive Hybrid Coding Scheme
`for HDTV and Digital Video Sequences," IEEE Transac(cid:173)
`tions on Consumer Electronics, vol. 41, No. 3, pp. 926-936,
`Aug. 1995.
`Kwok-Tung Lo & Jian Feng, "Predictive Mean Search
`Algorithms for Fast VQ Encoding of Images," IEEE Trans(cid:173)
`actions on Consumer Electronics, vol. 41, No. 2, pp.
`327-331, May 1995.
`James Goel, et al., "Pre-processing for MPEG Compression
`Using Adaptive Spatial Filtering", IEEE Transactions on
`Consumer Electronics, vol. 41, No. 3, pp. 687-698, Aug.
`1995.
`Jian Feng, et al., "Motion Adaptive Classified Vector Quan(cid:173)
`tization for ATM Video Coding", IEEE Transactions on
`Consumer Electronics, vol. 41, No. 2, pp. 322-326, May
`1995.
`
`(List continued on next page.)
`
`Primary Examiner-Young Lee
`(74) Attorney, Agent, or Firm---Eitan, Pearl, Latzer &
`Cohen Zedek, LLP
`
`U.S. PATENT DOCUMENTS
`
`(57)
`
`ABSTRACT
`
`8/1994 Barrett
`5,341,442 A
`2/1996 Kim
`5,491,519 A
`5,537,510 A * 7/1996 Kim ........................... 704/229
`5,586,200 A
`12/1996 Devaney et al.
`5,613,035 A * 3/1997 Kim ........................... 704/229
`5,627,937 A * 5/1997 Kim ........................... 375/240
`5,774,593 A
`6/1998 Zick et al.
`5,796,864 A
`8/1998 Callahan
`5,845,012 A
`12/1998 Jung
`5,847,766 A
`12/1998 Peak
`5,870,501 A
`2/1999 Kim
`6,005,626 A * 12/1999 Ding ..................... 375/240.16
`6,466,912 Bl * 10/2002 Johnston ..................... 704/500
`6,473,532 Bl * 10/2002 Sheraizin et al. ........... 382/264
`
`A video encoding system includes a visual perception
`estimator, an encoder, a compression dependent threshold
`estimator and a filter unit. The visual perception estimator
`estimates a perception threshold for a pixel of a current
`frame of a videostream. The encoder encodes the current
`frame. The compression dependent threshold estimator esti(cid:173)
`mates a compression dependent threshold for the pixel at
`least from the perception threshold and information from the
`encoder. The filter unit filters the pixel at least according to
`the compression dependent threshold.
`
`9 Claims, 12 Drawing Sheets
`
`IN: Y/Cr/Cb·
`
`18
`
`TIME
`ALIGNER
`
`Y·
`
`CTHD·
`
`14
`
`16
`
`FILTER
`UNIT
`
`FILTERED STRUCTURAL AND
`DATA
`STATISTICAL
`ENCODER
`
`OUT
`
`y
`PTHD·
`
`VISUAL
`PERCEPTION
`THRESHOLD
`ESTIMATOR
`
`10
`
`COMPRESSION
`DEPENDENT
`THRESHOLD
`DETERMINER
`
`12
`
`ENCODER STATE
`CURRENT PARAMETERS
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 3 of 20 PageID #: 76
`
`US 6,744,818 B2
`Page 2
`
`OIBER PUBLICATIONS
`
`Austin Y. Lan, et al., "Scene-Context-Dependent Refer(cid:173)
`ence-Frame Placement for MPEG Video Coding," IEEE
`Transactions on Circuits and Systems for Video Technology,
`vol. 9, No. 3, pp. 478-489, Apr. 1999.
`Kuo-Chin Fan & Kou-Sou Kan, "An Active Scene Analy(cid:173)
`sis-Based Approach for Pseudoconstant Bit-Rate Video
`Coding", IEEE Transactions on Circuits and Systems for
`Video Technology, vol. 8, No. 2, pp. 159-170, Apr. 1998.
`Takashi Ida & Yoko Sambonsugi, "Image Segmentation and
`Contour Detection Using Fractal Coding", IEEE Transac(cid:173)
`tions on Circuits and Systems for Video Technology, vol. 8,
`No. 8, pp. 968-975, Dec. 1998.
`
`Liang Shen & Rangaraj M. Rangayyan, "A Segmentation(cid:173)
`-Based Lossless Image Coding Method for High-Resolu(cid:173)
`tion Medical Image Compression", IEEE Transactions on
`Medical Imaging, vol. 16, No. 3, pp. 301-316, Jun. 1997.
`Adrian Munteanu, et al., "Wavelet-Based Lossless Com(cid:173)
`pression of Coronary Angiographic Images", IEEE Trans(cid:173)
`actions on Medical Imaging, vol. 18, No. 3, pp. 272-281,
`Mar. 1999.
`Akira Okumura, et al., "Signal Analysis and Compression
`Performance Evaluation of Pathological Microscopic
`Images", IEEE Transactions on Medical Imaging, vol. 16,
`No. 6, pp. 701-710, Dec. 1997.
`* cited by examiner
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 4 of 20 PageID #: 77
`
`N
`~
`~
`i,-
`0'J
`,I;;..
`,I;;..
`-...,l
`_,.a-...
`rJ'J.
`e
`
`'"""' N
`'"""' 0 ....,
`~ ....
`'JJ. =(cid:173)~
`
`,i;;..
`0
`0
`N
`'"""' ~
`
`~ = ?
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`FIG.l
`
`CURRENT PARAMETERS
`
`ENCODER STATE
`
`12
`
`DETERMINER
`THRESHOLD
`DEPENDENT
`COMPRESSION
`
`-
`
`PTHDi
`y
`
`10
`
`ESTIMATOR
`THRESHOLD
`PERCEPTION
`
`VISUAL
`
`STRUCTURAL AND OUT
`
`ENCODER
`STATISTICAL
`
`I
`
`FILTERED
`
`DATA
`
`UNIT
`FILTER
`
`I
`
`CTHDi
`
`Yi
`
`ALIGNER
`
`TIME
`
`18
`
`IN: Y/Cr/Cbi
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 5 of 20 PageID #: 78
`
`N
`~
`~
`i,-
`~
`-..,I;;..
`,I;;..
`-...,l
`-..a-...
`rJ'l
`e
`
`'"""' N
`0 ...,
`~ ....
`rF.J. =(cid:173)~
`
`N
`
`,i;;..
`0
`0
`N
`!'"'
`?
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`0 u
`~
`a.
`0::::
`I~
`cnw
`o- -t;
`z ~
`~ ~ICTHDj
`~a:::
`0 :c
`_J
`FG/BG -o
`. -,
`
`,=r
`
`Gr
`s
`E
`
`....
`
`I
`
`DP
`
`VBF
`T -
`
`Lv
`
`36
`
`I
`
`I
`---"
`I
`I
`1
`
`L-----
`
`.34
`........ 32
`
`PULSE DURATION
`
`ESTIMATOR
`
`H FRAME CENTER
`
`· _ POSITION WINDOW
`'
`
`FIG.2
`GENERATOR
`
`VD
`
`30
`
`PULSE LEVEL
`
`MAXIMUM
`
`QISCRIMINER
`
`SIGNAL
`
`_
`
`DETERMINER
`BACKGROUND
`
`28
`
`20
`
`24
`L--7---.J
`: THRESHOLD l
`1 NOISE
`I
`I
`I
`I
`I
`I
`
`1
`I
`._____. I
`CA1
`I
`·t
`,--
`I
`___.__.___,
`I
`:
`
`,___,r---'
`
`!
`
`I
`--,
`
`I
`r
`
`! 12
`
`FILTER
`
`-------~-PASS ~--"llaoi
`IN l y I HIGH
`22
`
`-----
`
`I
`
`IcoP
`.,..
`---. DETERMINER
`FRAME TYPE
`
`GOP
`
`PTHD·
`
`48
`
`NwFr¢1
`: NwFrCI
`I
`___ -----,
`
`~---------------------c_------;;::i _____ FORE
`,
`I .....____
`l
`
`-------
`
`I
`
`44
`
`42
`
`THRESHOLD
`
`...---·• NwFr
`46
`
`FRAME + INTEGRATOR i----<:___
`
`'-----
`r------···· 1 IF
`
`-
`
`r----,,-i MEMORY
`
`r----
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 6 of 20 PageID #: 79
`
`U.S. Patent
`
`Jun.1,2004
`
`Sheet 3 of 12
`
`US 6,744,818 B2
`
`AFR
`1.0
`
`0.9
`
`0.8
`
`0.7
`
`0.6
`
`0.5
`
`0.4
`
`0.3
`
`0.2
`
`0.1
`
`~ --
`V
`I
`
`V
`
`I
`
`I I
`I
`j I
`I
`/~
`
`0
`
`0.1
`
`0.3
`
`0.4
`
`O.SFs
`
`FIG.3
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 7 of 20 PageID #: 80
`
`'"""' N
`0 ....,
`~ ....
`=-~
`
`'JJ.
`
`,i;;..
`
`,i;;..
`0
`0
`N
`'"""'
`
`~
`
`~
`
`= ?
`
`= ......
`~
`......
`~
`~
`•
`r:JJ.
`d •
`
`Ed
`so
`Gr
`
`Lv
`
`T
`
`N
`~
`~
`i,-
`~
`_,.,I;;..
`,I;;..
`"'--l
`_,.a-...
`rJ'J.
`e
`
`86
`
`COMPARATOR
`
`I BG
`
`DURATION
`
`PULSE
`
`84
`
`SELECTOR
`
`PULSE
`EDGE
`
`FIG.4
`
`Ed
`SD
`Gr
`
`~76
`
`-
`
`74
`~ -
`
`DECODER
`
`DC1
`
`1
`
`..,.
`
`-
`
`I
`
`SIGN
`
`78
`
`OR
`
`80
`
`82
`
`ESTIMATOR
`MAXIMUM
`PULSE
`
`SHAPER
`DURATION
`
`PULSE
`
`r-...i
`
`DCTECTOR
`
`MAXIMUM
`
`LEVEL
`
`MODULE
`
`IMI
`
`_I
`
`-
`
`y
`
`72
`
`70
`
`28~
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 8 of 20 PageID #: 81
`
`U.S. Patent
`
`Jun.1,2004
`
`Sheet 5 of 12
`
`US 6,744,818 B2
`
`_,
`
`_,
`
`......
`
`.......
`
`......
`
`..J
`
`~ Cl
`
`a..
`:::,
`0
`0:
`c.,
`
`,)
`
`~
`<
`C
`ffi-r-<.....
`<
`'-..,
`
`~
`C
`C-
`2
`~
`~
`
`I+
`
`IT
`
`!+
`
`I+
`
`14-
`
`i..;_
`
`-
`IJ -
`I I
`
`I I
`
`1 I
`
`11_
`
`. , ,-=
`
`..--
`
`-
`
`t.O
`C,
`.........i
`~
`
`tO
`_,
`
`...t-
`tO- ....
`
`/
`
`'-'
`
`---...
`l:f
`...
`
`N
`
`tO- " '
`
`I
`
`?
`
`c:§:
`
`" ,
`
`j
`
`(_
`"\
`
`o_ ~
`&
`tO ~
`'
`
`J
`
`_C
`L
`
`r
`
`L
`
`, - -
`
`1+
`
`I..L. -
`
`1+
`
`I±_
`
`I+
`-
`
`L.:....
`
`-
`, I -
`
`I
`
`IJ-=
`
`1j_
`-
`
`IJ-
`
`r
`L
`
`C
`
`z
`
`LL. a..
`:c
`
`z
`
`C)
`(/)
`
`-'
`xw
`<C c;j
`::::E _J
`
`~ w~
`...J _j
`u.J
`~~
`0
`0
`u.J
`
`(/)C)
`
`_j
`
`t-0'
`- a..
`~6
`
`C) 0:
`C!)
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 9 of 20 PageID #: 82
`
`N
`~
`~
`i,-
`0'J
`,I;;;..
`,I;;;..
`-...,l
`-..a-...
`rJ'J.
`e
`
`'"""' N
`0 ....,
`~ ....
`rF.J. =(cid:173)~
`
`O'I
`
`,i;;..
`0
`0
`N
`'"""' ~
`
`~ = ?
`~ SW2
`~ +
`SW
`~ = ......
`swo
`~ ......
`~
`~
`•
`93
`r:JJ.
`d •
`
`q.3
`
`ro,Cbo)
`
`+ ,_
`
`, 93
`
`--
`
`Y(n)
`~ rvc;-=,
`
`0
`C")
`[Tl
`C,
`
`...__ ___
`
`L---
`IL--
`IL_
`I I
`I I
`I I
`1 I
`I I
`
`94
`~J
`I
`
`SW(n-1)
`
`--
`
`SW(n)
`
`11 93'
`
`92111
`
`11 I
`I I
`
`FIG.6
`
`I::
`'-go
`-r--'
`1....-.,.-..--J I
`I-+---'....... I ' I
`~----.,xn
`111
`I
`I It
`..:..--, l : :
`1 CA(n-1) 1-1
`IX(n-1)111
`! ~,J E"::::ic::::::--,7
`
`91
`
`106
`
`(
`
`14
`
`K(n). i--,.;;;;~--+-!-1 CA(n)
`
`Zn
`
`1-
`
`'--?""T----'' I
`
`X2
`
`X1
`
`Zn-1)
`
`K(n-1)
`
`__.___.....,106
`
`22
`
`21
`
`K2
`
`K1
`
`106
`
`-'
`CTHDi
`
`101--1 FILTER
`PASS
`LOW
`
`ALIGNER
`
`-
`
`Y(Cr,Cb)_ I TIME
`
`103
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 10 of 20 PageID #: 83
`
`U.S. Patent
`
`Jun.1,2004
`
`Sheet 7 of 12
`
`US 6,744,818 B2
`
`AFR(f)
`1.0
`
`~ 0.9
`
`"' CT:
`\
`'\ lq)
`---.. r--... t-- r---
`
`0.8
`
`0.7
`
`0.6
`
`0.5
`
`0.4
`
`0.3
`
`0.2
`
`0.1
`
`\
`
`\
`
`\@
`\
`'r---. r---. - ...__
`
`G)
`
`~
`
`0
`
`1.35
`
`2.70
`
`4.05
`
`5.40
`
`6.75 mHz
`
`FIG.7
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 11 of 20 PageID #: 84
`
`N
`~
`~
`i,-
`0'J
`,I;;..
`,I;;..
`-...,l
`-..a-...
`rJ'l
`e
`
`'"""' N
`0 ....,
`00
`~ ....
`rF.J. =(cid:173)~
`
`,i;;..
`0
`0
`N
`'"""' ~
`
`~ = ?
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`I
`I
`I
`I
`I
`kJ99
`
`,
`:
`:
`
`I
`I
`I
`I
`I
`121 1
`
`1 6.YHF
`I I '
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`_________ J
`I
`I
`1
`I
`I
`
`. ------------
`
`-----
`
`L----------------FIG.8
`I
`1
`:
`I ~-~
`
`SL --.
`
`~--..;:_
`~--,
`
`--I
`
`1--
`
`ti.YsL
`
`.--~-------r---
`
`L _______ ,
`----.L---
`
`.---I
`
`L--,--
`
`10
`2
`100
`--, SW1
`
`,____ I DIVIDER L-----:=1 ..... ----,r-
`
`2
`
`30"____
`
`110
`~--
`
`'-----
`
`~-7
`
`--~,
`
`:
`
`l
`
`t
`I
`1
`
`.-----,
`
`2X --'
`
`!-----L__
`
`--~,
`
`30"
`I
`
`I
`I
`
`I
`I
`
`1
`
`-------------7
`
`I
`I
`
`---------
`
`104
`--------
`
`Hrl
`I
`.
`I
`1
`r----------
`
`(
`
`14'
`
`103
`
`y
`
`101
`
`I
`I
`I
`1
`I
`I
`I 1---,-~
`I
`I
`I
`, AYcmo
`:
`I
`I
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 12 of 20 PageID #: 85
`
`U.S. Patent
`
`Jun.1,2004
`
`Sheet 9 of 12
`
`US 6,744,818 B2
`
`144
`140
`r-.....-------·-----..,
`~ ~
`~
`~
`~
`~
`~
`~
`~
`~ ,
`~ i----
`
`146
`l
`
`#
`
`.-
`
`.
`
`,
`
`,
`
`I
`
`,
`
`,,
`.,
`
`I
`30-
`
`142
`V'
`')Fl
`/
`V'
`/ ~
`~<1
`I ~ A /
`11~ I
`18
`:,A l7'
`
`-~
`
`~
`
`/
`
`I
`I
`I
`
`I
`I
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`,
`\L
`,,
`L.____J
`,,
`
`# ,
`
`I , ,
`...._
`1-1_48
`
`I
`,L
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`----1:::.Y
`
`FIG.9
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 13 of 20 PageID #: 86
`
`N
`~
`~
`i,-
`0'J
`,I;;..
`,I;;..
`':...l
`O'I
`rJ'J.
`e
`
`'"""' N
`0 ....,
`'"""' 0
`~ ....
`rF.J. =-
`
`~
`
`,i;;..
`0
`0
`N
`'"""'
`?
`~ =
`
`~
`
`= ......
`~
`......
`~
`~
`•
`r:JJ.
`d •
`
`. --
`
`134
`
`6
`
`MULTIPLIER
`
`'-134
`
`A.
`
`MULTIPLIER
`
`2
`
`. -.
`
`NAFx
`
`I 130"'\
`
`. X
`LPF r
`
`~
`
`ALIGNER
`
`TIME
`
`130
`
`X
`LPF y
`
`FIG.10
`
`...
`Ky
`
`-I Kc~I
`I 'tO"-j SHAPER
`
`1
`r-.y
`
`LPF ~MI
`
`,,,..146
`
`144
`
`ALIGNER
`
`TIME
`
`-
`
`LPF ~
`
`134
`
`MULTIPLIER
`
`5
`
`TIME ~ I I
`
`ALIGNER
`
`132
`
`130-......
`
`y
`LPF b
`
`Cb I I I
`
`I
`
`I 1\1 lf'lt.lLD I
`
`'----134
`
`J
`
`TIME~
`
`ALIGNER
`
`.. IMULTIPLIERI
`
`132
`
`Ky
`
`TIME~
`
`ALIGNER
`
`LPF~ I~
`I
`
`I I
`I I 130, ~ "fHF
`
`Cr
`
`MULTIPU
`
`NAFy
`
`I 130,
`y
`LPF y
`
`y
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 14 of 20 PageID #: 87
`
`N
`~
`~
`i,-
`0'J
`,I;;..
`,I;;..
`""-l
`_,.a-...
`rJ'J.
`e
`
`'"""' N
`'"""' 0 ....,
`'"""'
`~ ....
`'JJ. =(cid:173)~
`
`,i;;..
`0
`0
`N
`'"""' ~
`
`~ = ?
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`OUT
`
`FIG.11
`
`ENCODER STATE CURRENT
`
`PARAMETERS
`
`-
`
`DETERMINER
`
`CTHD
`
`PTHDi
`
`\
`
`10
`
`ESTIMATOR
`THRESHOLD
`· PERCEPTION
`
`VISUAL
`
`I
`
`AND STATISTICAL
`
`ENCODER
`
`STRUCTURAL
`
`r
`
`ENHANCING
`RESOLUTION
`
`FILTER
`
`(
`
`CTHDi
`
`UNIT
`FILTER
`
`14
`
`I
`
`·-
`
`TIME
`/
`
`ALIGNER
`
`IN:Y/Cr/Cb
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 15 of 20 PageID #: 88
`
`N
`~
`~
`i,-
`~
`_,.,I;;..
`,I;;..
`-...,l
`_,.a-...
`rJ'J.
`e
`
`'"""' N
`0 ....,
`'"""' N
`~ ....
`'JJ. =(cid:173)~
`
`r
`?
`~
`
`,i;;..
`0
`0
`N
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`FILTER
`
`1
`
`J
`
`ADDER
`
`/164
`
`158
`
`/150
`
`fKT
`
`SL
`
`/162
`
`FIG.12
`
`L-----------...1
`I
`I PASS FILTER
`:
`FRAME HIGH 1-----1.------'
`FRAME TO
`:
`
`I
`
`1
`I 156
`
`I
`
`r-----------,
`
`TEMPORAL
`
`
`1 1
`
`NONLINEAR
`
`FILTER
`
`2
`
`160
`
`:
`I .-------. 1154
`:
`
`I
`
`I
`
`HIGH PASS
`HORIZONTAL
`
`I
`I
`._ ___________ ...J
`1
`1
`I
`1
`
`FILTER
`
`FILTER
`HIGH PASS
`VERTICAL 1---1-------NONLINEAR
`
`I
`1
`1
`I
`I
`I
`I ...-----1152
`I
`I
`r-----------7
`
`I
`1
`I
`
`.,__ ___ __,
`
`SPATIAL
`
`CTHDj
`
`ALIGNER 18
`FROM TIME
`
`FROM FILTER
`
`UNIT 14
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 16 of 20 PageID #: 89
`
`US 6,744,818 B2
`
`1
`METHOD AND APPARATUS FOR VISUAL
`PERCEPTION ENCODING
`
`FIELD OF THE INVENTION
`
`The present invention relates generally to processing of
`video images,
`
`BACKGROUND OF THE INVENTION
`
`5
`
`There a three types of redundancy in video signals that are
`related to the picture within the video. These are structural,
`statistical and perceptual redundancy. Standard compression
`systems, such as the various forms of MPEG,
`H-compression, etc., mainly reduce structural and statistical 15
`redundancy. U.S. patent application Ser. No. 09/524,618,
`assigned to the common assignees of the present invention
`and incorporated herein by reference, attempts to reduce
`perceptual redundancy independent of whatever other video
`compression might be used afterward.
`
`2
`threshold estimator 10, a compression dependent threshold
`determiner 12, a filter unit 14 and a structural and statistical
`encoder 16.
`Visual perception threshold estimator 10 may receive an
`image having luminance Y and red and blue chrominance Cr
`and Cb signals and may estimate a distinguishing visual
`perception threshold PTHD; for each ith pixel of the image.
`An exemplary estimator 10 is described in U.S. patent
`application Ser. No. 09/524,618, filed Mar. 14, 2000,
`10 assigned to the common assignees of the present invention
`Ad incorporated herein by reference.
`Compression dependent threshold determiner 12 may
`estimate a distinguishing compression dependent threshold
`CTHD; for the ith pixel using the luminance values Y of the
`image, visual perception threshold PTHD; and information
`from encoder 16 about the type of image the current image
`is as will be described in more detail hereinbelow.
`Filter unit 14 filters the ith pixel based on the value of the
`associated compression dependent threshold CTHD;. It can
`20 be a controllable filter set (shown in FIG. 6) or a nonlinear
`filter (shown in FIGS. 8 and 10). Thus, the kind of filtering
`to be performed on a pixel depends on whether the lumi(cid:173)
`nance value Y of that pixel is above or below the specific
`distinguishing threshold for that pixel. Since estimator 10
`25 and determiner 12 typically operate with a time delay, the
`encoding system comprises a time aligner 18 which provides
`the ith pixel of the image to filter unit 14 when filter unit 14
`receives the ith compression dependent threshold CTHD;.
`The filtered data is ten provided to encoder 16 for standard
`encoding. Typically, encoder 16 is a structural and statistical
`encoder such as any of the MPEG types or an H compression
`encoder. As is known in the art, MPEG encoders divide the
`frames of the videostream into "I", "P" and "B" compressed
`35 frames where I frames are compressed in full while, for the
`p and B images, only the differences between the current
`frame and previous predicted frames are encoded. The tpe of
`the frame (i.e. was it an 1, P or B frame?) is provided to
`threshold determiner 12 for use in determining the compres-
`40 sion dependent threshold CTHD;. Thus, the type of encoding
`which encoder 16 performed at least partially affects the type
`of filtering which filter set 14 will ultimately perform.
`Reference is now made to FIG. 2, which generally details
`the elements of compression dependent threshold deter-
`45 miner 12. Determiner 12 comprises a new frame determiner
`20, a high pass filter 22, a noise reducer 24, various
`parameter determiners 26-34 and a compression threshold
`estimator 36. The parameters defining the CTHD value
`comprise at least some of the following parameters:
`whether or not encoder 16 has defined a new frame NwFr
`as an I frame;
`whether the ith pixel is in the foreground FG or the
`background BG of the picture;
`whether die ith pixel forms part of an edge Ed around an
`object in the picture;
`whether or not the ith pixel forms part of a small detail
`SD;
`whether or not the ith pixel is part of a group Gr type of
`details (a set of generally periodic details);
`the contrast level Lv of the detail for the ith pixel;
`the duration -i: (in transmission time) of a detail within a
`picture;
`how full a video buffer of encoder 16 is full (a VBF
`value);
`the distance DP of the ith pixel from the center of the
`frame; and
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present invention will be understood and appreciated
`more fully from the following detailed description taken in
`conjunction with the appended drawings in which:
`FIG. 1 is a block diagram illustration of a system for
`visual perception encoding, for use with standard compres(cid:173)
`sion encoders, constructive and operative in accordance with
`a preferred embodiment of the present invention;
`FIG. 2 is a block diagram illustration of a compression
`dependent threshold determiner, useful in the system of FIG.
`1;
`
`30
`
`FIG. 3 is a graph of the response of a high pass filter,
`useful in the determiner of FIG. 2;
`FIG. 4 is a block diagram illustration of a signal
`discriminator, useful in the determiner of FIG. 2;
`FIG. 5 is a riming diagram illustration, useful in under(cid:173)
`standing the operation of the determiner of FIG. 2;
`FIG. 6 is a block diagram illustration of a filter unit, useful
`in the system of FIG. 1;
`FIG. 7 is a graphical illustration of the frequency response
`of the filter unit of FIG. 6;
`FIG. 8 is a block diagram illustration of an alternative,
`non-linear filter, useful in the system of FIG. 1;
`FIG. 9 is a graphical illustration of the frequency response
`of the filter unit of FIG. 8;
`FIG. 10 is a block diagram illustration of an alternative
`filter unit utilizing the non-linear filter of FIG. 8, useful in 50
`the system of FIG. 1;
`FIG. 11 is a block diagram illustration of a system for
`visual perception encoding having a resolution enhancing
`filter, constructive and operative in accordance with an 55
`alternative preferred embodiment of the present invention;
`and
`FIG. 12 is a block diagram illustration of the resolution
`enhancing filter of FIG. 11.
`
`60
`
`DETAILED DESCRIPTION OF THE PRESENT
`INVENTION
`Reference is now made to FIG. l, which illustrates a video
`encoding system, constructed and operative in accordance
`with a preferred embodiment of the present invention. The 65
`encoding system generally reduces perceptual redundancy
`in video streams and may comprise a visual perception
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 17 of 20 PageID #: 90
`
`US 6,744,818 B2
`
`3
`an initial value C0 for compression dependent threshold
`CTHD.
`The maximum pulse level Lv may be normalized by a
`maximum luminance signal NLv value, the pulse duration
`may be normalized by sampling internal Ni: and the detail 5
`position DP may be defined by the number of lines and the
`pixel position within a line. Estimator 36 may then deter(cid:173)
`mine compression dependent threshold CTHD, from the
`normalized parameters and the visual perception threshold
`PTHD; as follows:
`PTHD; + CE Ed+ CDSD + C0 Gr + C,Nr +)
`
`CTHD; =
`(
`
`CLNLv + CpNwFr+ CsF / B + CpDP
`
`Cv VBF + Co
`
`4
`If comparator 46 indicates that the present frame is a new
`frame NwFr, GOP frame type determiner 48 determines
`whether or not encoder 16 defined the frame as an I frame
`within the current group of pictures and provides this
`information to estimator 36.
`High pass filter 22 filters the pixels of the current frame
`to select only those details of the picture which are of
`generally short duration such as edges, "single details"
`formed of only a few pixels and/or details which have a
`10 group structure,
`An exemplary amplitude-frequency response for high
`pass filter 22 is provided in FIG. 3 to which reference is now
`briefly made. It is noted that the cutoff frequency is about
`0.2Fs where Fs is the sampling frequency of an analog to
`15 digital converter (not shown) used to digitize the input
`signal.
`Returning to FIG. 2, noise reducer 24 takes the output of
`high pass filter 22 and reduces the noise level. Reducer 24
`comprises a comparator 52 and a switch 50. Comparator 52
`20 compares the signal level of the filtered signal produced by
`high pass filter 22 with a noise threshold (typically 3-5 times
`an average noise level). Switch 50 only passes the filtered
`signal if its signal level is high enough, as indicated by
`comparator 52.
`A signal discriminator 28 determines which pixels of the
`filtered and noise reduced signal belong to edges (Ed), single
`detail (SD) and group of details (Gr). FIG. 4 provides one
`embodiment of discriminator 28.
`A foreground/background determiner 26 uses the edge
`30 information to determine if the current pixel is in the
`foreground or background, where a foreground object has
`sharp edges and a background object has blurred edges (i.e.
`ones of long duration).
`A pulse duration estimator 32 measures the length of each
`35 pulse (which may occur over multiple pixels) to generate the
`duration T of a detail and a maximum pulse level determiner
`30 uses the duration to determine the maximum pulse level
`Lv within the pulse duration.
`A detail position generator 34 determines DP, how close
`40 the current pixel is to the center of the frame. To do this,
`generator 34 receives the frame synchronization, i.e. the
`horizontal drive (HD) and vertical drive (VD) signals, and
`the current pixel and uses this information to compare the
`location of the current pixel to that of the center pixel of the
`45 frame
`FIG. 4 is one embodiment of some of the elements of FIG.
`2 showing the operation on the high pass filtered and noise
`reduced signal, FIG. 5, to which reference is also made, is
`a timing diagram indicating how the elements of FIG. 4
`50 operate on different types of input signals.
`The first timing diagram of FIG. 5 shows three types of
`input signals: two edges 60 and 62, two single details 64 and
`66 and a group detail 68. The second timing diagram shows
`the shape of the signals 60-68 after high pass filtering and
`55 noise reduction.
`An absolute value module 70 (FIG. 4) finds the absolute
`value of each pixel and a maximum level detector 72
`converts the current maximum level into sign pulses. The
`output of detector 72 is shown in the fourth dining diagram
`60 of FIG. 5. For edges 60 and 62, there are two points where
`a maximum occurs, as can be seen in the high pass filtered
`signal of the second ting diagram. The single detail 66 has
`three points of maximum while the group detail 68 has many
`of them, relatively regularly spaced.
`A sign indicator 74 (FIG. 4) determines the sign (positive
`or negative) of the high pass filtered and noise reduced
`signal. The output of indicator 74 is shown in the third
`
`25
`
`where CD CD ... CP are weighting coefficients, depen(cid:173)
`dent on the influence of each parameter at CTHD. For
`MPEG encoders, the following empirical values may be
`useful:
`
`CD= 0.8
`
`Cc =0.1
`
`0.7 if NwFr = 1 }
`CF = 0
`{
`otherwise
`
`0.5
`CB = 0
`{
`
`if background}
`if foreground
`
`2
`(tv -0.5V)
`_
`[(tH -0.5H)
`Cp-0.5 - - - + - - -
`0.5H
`0.5V
`
`2
`
`0
`
`5
`·
`
`]
`
`Cv = 1.5
`
`Co= 0.1
`
`where tH and tv are the position, in time, of the pixel
`within a line (tH) and a frame (tv) and H and V are the line
`and frame numbers, respectively, and C0 is the initial CTHD
`value.
`The following other relationships are noted:
`
`Lv
`NLv=(cid:173)
`Lmax
`Tpix
`Nr=-
`r
`
`where Lmax is the maximum value for the luminance
`signal and i:pix is the transmission time of one pixel.
`New frame determiner 20 may determine whether there is
`a new frame NwFr and whether or not it has been defined by
`encoder 16 as an I frame. New frame determiner 20 typically
`comprises a frame memory 40, a summer 42, an integrator
`44, a comparator 46 and a group of pictures (GOP) frame
`type determiner 48.
`Summer 42 finds the differences between the present
`frame and a previous one stored in frame memory 40.
`Integrator 44 sums the differences across the frame to
`produce a change volume I1 indicating the amount change
`between the neighboring frames. If comparator 46 deter(cid:173)
`mines that this change volume I1 is above a certain threshold
`(such as more than 50% of the maximum amount of pixels 65
`in a frame), comparator 46 defines that the present frame is
`a new frame NwFr.
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 18 of 20 PageID #: 91
`
`US 6,744,818 B2
`
`5
`timing diagram of FIG. 5. For edges 60 and 62, the sign
`changes from positive to negative, but after different lengths
`of time. For single details 64 and 66, the sign changes from
`positive, to negative to positive, once again after different
`lengths of time. For group detail 68, the sign continually 5
`changes between negative and positive.
`A decoder 76 uses the output of sign indicator 74 to
`determine whether the current pixel or series of pixels is an
`edge, a single derail or a group detail according to the
`following table:
`
`6
`low pass filtered version of the original frame from the
`original frame and produces thereby a high pass filtered
`frame. The filtered frame is provided to each multiplier 106
`which, in turn, multiplies the signal of the filtered frame,
`This has the effect of changing the shape of the high pass
`filter that operates on the frame. Thus, the output of multi(cid:173)
`pliers 106 is a high pass filtered signal. FIG. 7 shows the
`frequency response of four of the high pass filters, labeled
`1-4.
`Table 2 provides the function of decoder 92, for eight
`multipliers Kl-KS whose weight values are 0.125, 0.25,
`0.375, 0.5, 0.625, 0.75, 0.875 and 1.0, respectively. Their
`outputs are signals zl-z8, respectively, the outputs of their
`respective comparators 91 are signals xl-x8, respectively,
`and the signals to their associated switches are yl-y8,
`15 respectively. The signal y0 instructs a switch sw0 to select
`the high pass filter output of summer 104.
`
`10
`
`TABLE 1
`
`+/-or-/+
`
`+/-/+ or -/+/-
`
`+I-I+!-!+! ...
`
`Edge
`Single Detail
`Group Detail
`
`yes
`no
`no
`
`no
`yes
`no
`
`no
`no
`yes
`
`20 High pass
`filter
`
`TABLE 2
`
`Kl ... KS
`
`signal level, Zi
`
`Xl
`
`X2 X3 X4
`
`XS
`
`X6
`
`X7
`
`XS
`
`ZS< CTHD
`25 ZS"'; CTHD
`Z7 < CTHD
`Z7 "'; CTHD
`Z6 < CTHD
`Z6 "'; CTHD
`ZS< CTHD
`ZS"'; CTHD
`Z4 < CTHD
`Z4 "'; CTHD
`Z3 < CTHD
`Z3 "'; CTHD
`Z2 < CTHD
`Z2 "'; CTHD
`35 Zl < CTHD
`Zl "'; CTHD
`
`30
`
`High pass
`filter
`
`0
`
`0
`0
`
`0
`0
`
`0
`
`0
`0
`
`0
`
`0
`
`0
`0
`
`0
`
`0
`
`0
`
`0
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`KO ... KS
`
`40 signal level, Zi
`
`YO
`
`Yl Y2 Y3 Y4 YS
`
`Y6 Y7 YS
`
`0
`0
`
`0
`0
`
`0
`0
`
`0
`
`0
`
`The output of decoder 76 is shown in the fifth, sixth and
`seventh timing diagrams for edges 60 and 62, single details
`64 and 66, and group detail 68, respectively. It is noted that
`edge 60 is shorter than edge 62 and single detail 64 is shorter
`tan single detail 66.
`A pulse level maximum estimator 80 receives the edge,
`single detail and group detail signals of the fifth, sixth and
`seventh timing diagrams and finds the maximum level Lv of
`the pulse for the signal which currently has a pulse.
`A pulse duration shaper 82 receives the maximum pulse
`level position signal from detector 72 and the edge, single
`detail and group detail signals from OR element 78 after
`decoder 76 and finds the duration X for the signal which
`currently has a pulse. An edge pulse selector 84 uses the
`edge signal from decoder 76 and the signal from shaper 82
`to select an edge duration pulse when an edge is present. The
`edge duration pulse selected by selector 84 is provided to a
`pulse duration comparator 86 which compares the pulse
`duration x for me current edge to a threshold level indicating
`the maximum pulse length which indicates a foreground
`edge. Any pulse length which is longer than the threshold
`indicates a background pixel and any which 1s shorter
`indicates a foreground pixel.
`Reference is now made to FIGS. 6 and 7 which,
`respectively, illustrate the elements of filter unit 14 (FIG. 1)
`and the shapes of the filters which are utilized therein.
`Filter unit 14 is a controllable filter set and typically
`comprises a series of high pass filters ( described in more
`detail hereinbelow), a set of comparators 90, a decoder 92
`and a set of switches 94. Each high pass filter has a different
`frequency response and has a comparator 91 and a switch 93
`associated therewith. The associated comparator 91 com(cid:173)
`pares the level of the filtered data (i.e. filtered pixel) to the
`compression dependent threshold CTHD 1 for the current
`pixel. Decoder 92 decides which filter output to utilize
`(based on which filtered data is above the compression
`dependent threshold CTHD;) and instructs the appropriate ss
`switch 93 to pass that filter output for the current pixel.
`For each pixel, a summer 96 subtracts the high pass
`filtered data output from the appropriate switch 93 from the
`non-filtered data of the frame. Thus, the level of each pixel
`is changed by the selected high pass filter. It will be 60
`appreciated that the operation of controlled filter set 14 is
`equivalent to a low pass filter optimization for every picture
`detail in accordance with the value of the compression
`dependent threshold CTHD;.
`The high pass filters are implemented in the embodiment 65
`of FIG. 6 from a low pass filter 101, a time aligner 103, a
`summer 104 and multipliers 106. Summer 104 subtracts a
`
`ZS< CTHD
`ZS"'; CTHD
`Z7 < CTHD
`Z7 "'; CTHD
`45 Z6 < CTHD
`Z6 "'; CTHD
`ZS< CTHD
`ZS"'; CTHD
`Z4 < CTHD
`Z4 "'; CTHD
`
`50 ~~ ~ ~i:
`
`0
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`Z2 < CTHD
`Z2 "'; CTHD
`Zl < CTHD
`Zl "'; CTHD
`
`Reference is now made to FIGS. 8 and 9, which present
`an alternative embodiment of the filter unit, labeled 14'. In
`this embodiment, the filters of filter unit 14 are non-linear. It
`is expected that this type of filtering is more suitable for
`visual perceptual coding because its picture processing is
`similar to the perceptual process of the human eye which
`uses detected details ( e.g. texture) and distinguished details,
`As in the previous embodiment, filter unit 14' comprises
`low pass filter 101, time aligner 103 and summer 104, where
`summer 104 subtracts a low pass filtered version of the
`original frame from the original frame and produces thereby
`a high pass filtered frame ll Y HF- Typically, the high pass
`
`

`

`Case 1:21-cv-00227-UNA Document 1-3 Filed 02/18/21 Page 19 of 20 PageID #: 92
`
`US 6,744,818 B2
`
`7
`filtered frame comprises the high frequency components that
`correspond to those details of the frame which have small
`dimensions. The high pass filtered frame is then filtered by
`a non-linear filter 99, which