(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2009/0141940 A1
`(43) Pub. Date:
`Jun. 4, 2009
`ZHAO et al.
`
`US 2009014 1940A1
`
`(54)
`
`(75)
`
`(73)
`
`(21)
`(22)
`
`(60)
`
`INTEGRATED SYSTEMIS AND METHODS
`FOR VIDEO-BASED OBJECT MODELING,
`RECOGNITION, AND TRACKING
`Inventors:
`Liang ZHAO, Morrisville, NC
`(US); Matthew G. BERRY,
`Raleigh, NC (US)
`Correspondence Address:
`MORRIS MANNING MARTIN LLP
`3343 PEACHTREE ROAD, NE, 1600 ATLANTA
`FINANCIAL CENTER
`ATLANTA, GA 30326 (US)
`Assignee:
`DIGITALSMITHS
`CORPORATION, Morrisville, NC
`(US)
`12/327,589
`
`Appl. No.:
`
`Filed:
`
`Dec. 3, 2008
`Related U.S. Application Data
`Provisional application No. 60/991,879, filed on Dec.
`3, 2007.
`
`Publication Classification
`
`(51) Int. Cl.
`G06K 9/00
`
`(2006.01)
`
`(52) U.S. Cl. ........................................................ 382/103
`
`ABSTRACT
`(57)
`The present disclosure relates to systems and methods for
`modeling, recognizing, and tracking object images in video
`files. In one embodiment, a video file, which includes a plu
`rality of frames, is received. An image of an object is extracted
`from a particular frame in the video file, and a Subsequent
`image is also extracted from a Subsequent frame. A similarity
`value is then calculated between the extracted images from
`the particular frame and Subsequent frame. If the calculated
`similarity value exceeds a predetermined similarity thresh
`old, the extracted object images are assigned to an object
`group. The object group is used to generate an object model
`associated with images in the group, wherein the model is
`comprised of image features extracted from optimal object
`images in the object group. Optimal images from the group
`are also used for comparison to other object models for pur
`poses of identifying images.
`
`
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 001
`
`

`

`Patent Application Publication
`
`Jun. 4, 2009 Sheet 1 of 15
`
`US 2009/014 1940 A1
`
`
`
`s
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 002
`
`

`

`Patent Application Publication
`
`Jun. 4, 2009 Sheet 2 of 15
`
`US 2009/0141940 A1
`
`205
`RETRIEVEVIDEO
`
`200
`4UV
`
`
`
`SHOT BOUNDARY DETECTION
`
`210
`
`OBJECT DETECTION
`
`220 -
`FEATURE EXTRACTION
`
`
`
`OPTIMAL
`OBJECT2
`
`
`
`
`
`OBJECT RECOGNITION
`
`OBJECT TRACKING
`
`400
`
`245
`
`USER SPECIFIES THEID
`
`300
`
`
`
`OBJECT MODELING
`
`
`
`240
`
`OBJECT
`DENTIFIED?
`
`MODELUPDATING
`
`
`
`VIDEO ANALYSIS
`COMPLETE
`
`YES
`
`END
`
`FIG. 2
`OVERALL PROCESS
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 003
`
`

`

`Patent Application Publication
`
`Jun. 4, 2009 Sheet 3 of 15
`
`US 2009/O141940 A1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`-300
`
`START
`
`RETRIEVE AN OBJECT IMAGE
`
`
`
`ISITAN
`OPTIMALOBJECT?
`
`EXTRACTED OBJECT
`FEATURES
`
`fSTHE MODE
`OF THE OBJECT
`EMPTY
`
`310
`NO
`
`CALCULATE THE SIMILARITYBETWEEN
`THE EXTRACTEDFEATURES AND THOSE
`IN THE MODEL
`
`ADD THE FEATURESTO
`THE MODEL
`
`
`
`
`
`ADD THE NEW FEATURES
`TO THE MODEL AND UPDATE
`THE WEIGHTS OF FEATURES
`
`1 HAVE ALL
`IMAGES BEEN
`PROCESSED?
`
`
`
`FIG. 3
`OBJECT MODELING
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 004
`
`

`

`Patent Application Publication
`
`Jun. 4, 2009 Sheet 4 of 15
`
`US 2009/O141940 A1
`
`
`
`
`
`RETRIEVE EXTRACTED
`OBJECT FEATURES
`
`CALCULATE THE SIMLARITY
`BETWEEN THE EXTRACTED
`FEATURES AND THOSE IN THE
`MODELS
`
`FIND THE MODEL WITH THE
`HIGHEST SMILARITY TO THE
`DETECTED OBJECT
`
`
`
`OBJECT TRACKING
`
`
`
`
`
`
`
`
`
`
`
`FIG. 4
`OBJECT RECOGNITION
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 005
`
`

`

`Patent Application Publication
`
`Jun. 4, 2009 Sheet 5 of 15
`
`US 2009/0141940 A1
`
`(START
`
`-500
`
`505
`
`INITIALIZE THE OBJECT GROUPS (EACHDETECTED OBJECT
`CORRESPONDS TO ASEPARATEGROUP) AND LOCAL
`OBJECT MODELS WITH THE FEATURESEXTRACTED IN THE
`FIRST FRAME OF A SHOT
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`507
`
`RETRIEVE AFRAME
`
`CALCULATE THE SIMLARITYBETWEEN
`THE OBJECT IN THE CURRENT FRAME
`AND THOSE IN THE PREVIOUS FRAME
`
`DOES THE
`SIMILARITY EXCEED
`NHE THRESHOLD2
`
`514
`
`CREATE ANEW GROUP
`FOR THE OBJECT ANDADD
`FEATURESTO TS LOCAL
`OBJECT MODEL
`
`ADD THE OBJECT TO THE CORRESPONDING OBJECT
`GROUPANDADD FEATURESTO THE LOCAL OBJECT
`MODEL
`
`IS THE SHOT COMPLETE?
`
`NO
`
`
`
`520
`
`YES
`CALCULATE THEAVERAGE SIMILARITY
`SCORE BETWEEN THE DETECTED OPTIMAL
`OBJECTS (WITHIN THE OBJECT GROUP)
`AND THE GLOBALOBJECT MODELS
`
`
`
`525
`
`
`
`16OES THE
`SIMILARITY EXCEED
`NTHE THRESHOLD?
`
`535
`
`NO
`
`LABEL THE OBJECT
`GROUP
`AS UNKNOWN
`
`530
`
`YES
`LABEL THE OBJECT GROUP
`WITH THE MODELD
`
`F.G. 5
`OBJECT TRACKING
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 006
`
`

`

`Patent Application Publication
`
`Jun. 4, 2009 Sheet 6 of 15
`
`US 2009/O141940 A1
`
`
`
` SONI LSIT ECJOS|dE CJELSEL 9 ’5)|–}
`
`
`
`
`
`
`
`di opositiºi!!!
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 007
`
`

`

`Patent Application Publication
`
`Jun. 4, 2009 Sheet 7 of 15
`
`US 2009/O141940 A1
`
`
`
`LI IZ606I
`
`
`
`
`
`SdnouÐ 30b). Jo lºquun N | sø3euI jepe u pºqoºyoq Jo laguinN
`
`
`
`
`
`
`
`/ ’5)|–|
`
`
`
`
`
`
`
`SdnOHS) QNW SBSÒ\/WI GELOELEG
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 008
`
`

`

`Patent Application Publication
`
`Jun. 4, 2009 Sheet 8 of 15
`
`US 2009/0141940 A1
`
`
`
`STEGOW BH^d SÐNIST NOILINOOOBH TVIOVH JO BAHmo TTVOER-NOISIOBHA
`
`
`
`
`
`
`
`
`
`
`
`ENOTV EC]OS|dE HOVE WOH-, OE_1\/>?ENES)
`
`uOSoejd
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 009
`
`

`

`Patent Application Publication
`
`Jun. 4, 2009 Sheet 9 of 15
`
`US 2009/O141940 A1
`
`
`
`
`
`(II oposidº
`
`
`
`STEGJOWN BÈHºld (JOH SE SÐ\/_LNEHOXHE, TTVOE}}
`
`
`
`
`
`6 ^<)|–|
`
`‘NOIS!OEYJc]
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 0010
`
`

`

`Patent Application Publication
`
`Jun. 4, 2009 Sheet 10 of 15
`
`US 2009/O141940 A1
`
`
`
`
`
`
`
`
`
`
`(JOH STEGOW B??nd (JOH SETTIVA TTVOEM QNV NOISIOEYJA EÐwyJEAy
`
`
`
`w0dE JO SHE LOVHVHO HOTwW LH9B
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 0011
`
`

`

`Patent Application Publication
`
`US 2009/O141940 A1
`
`r
`
`0 0 1 0 0 0 0 0 0 0 ||
`0 || 0 0 0 0 0 0 Z O º * * = = = = • ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? *-* * * = = • • • •= • !== ) ) = = • – — ± - = = =
`
`E0 0 0 0 0 0 0 0 || 9 50 / 7 0 0 0 0 || 0 | | / ž0 || 0 0 0 0 0 0 9 || 9 §0 0 0 0 0 0 0 0 7 G }0 0 0 0 0 0 0 0 Z #7
`?0 0 0 0 0 1 0 0 8 9 0 7 0 0 0 0 / | O ZE Z
`
`|, }, '5)|–|
`
`| 70dE NI SHE LOV/>HWHO
`
`
`
`
`
`
`
`
`
`
`èJOTWW LHOE HOH STEGOW EYJ/md HO-, XIXJ LWW NOISn-INOO
`
`OO
`
`N
`
`CO
`
`D
`
`V
`
`CY)
`
`CN
`
`w
`
`O
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 0012
`
`

`

`Patent Application Publication
`
`Jun. 4, 2009 Sheet 12 of 15
`
`US 2009/0141940 A1
`
`
`
`|Iepau
`
`
`
`Z|. '5)|–}
`
`CJEÐ HEW SÐNIST NO||LINSOOOEN TVIOV/- -}OBANJOO TTVOE>}
`
`
`
`
`
`
`
`
`
`
`
`SECOSIdE QENIGWOO-ÄTTVILNE WENJONI WOH-, CIELVAJENES) STEGJOW
`
`-NOISIOE}}c} ,
`
`uosoed
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 0013
`
`

`

`Patent Application Publication
`
`Jun. 4, 2009 Sheet 13 of 15
`
`US 2009/O141940 A1
`
`
`
`
`
`
`
`I STECIOWN QE9}{EW JOH SEOVLNE OYJEd TTVOEYH-NOISIOEYJ)
`
`
`
`
`
`
`
`9 | '5)|–|
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 0014
`
`

`

`Patent Application Publication
`
`Jun. 4, 2009 Sheet 14 of 15
`
`US 2009/O141940 A1
`
`
`
`
`
`
`STEGOW GEÐHEW HO + S=nTVA TIVOE, CINw NOISIOBHA EÐwx}=A\}
`
`
`
`
`
`SECIOSleBE CIENIEWOO TTV -JO SMJELOVAJVHO (JOTVIN EATENAL AJO
`
`
`
`uô?spaudCII løp0JN 7.1943 eue?O
`
`
`
`
`
`
`
`
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 0015
`
`

`

`Patent Application Publication
`
`Jun. 4, 2009 Sheet 15 of 15
`
`US 2009/O141940 A1
`
`O O
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 0016
`
`

`

`US 2009/014 1940 A1
`
`Jun. 4, 2009
`
`INTEGRATED SYSTEMIS AND METHODS
`FOR VIDEO-BASED OBJECT MODELING,
`RECOGNITION, AND TRACKING
`
`CROSS REFERENCE TO RELATED
`APPLICATION
`0001. This application claims benefit under 35 U.S.C.
`S119(e) of U.S. Provisional Patent Application No. 60/991,
`879, filed Dec. 3, 2007, and entitled “Integrated Systems for
`Face Recognition, Tracking, and Modeling, which is incor
`porated herein by reference as if set forth herein in its entirety.
`
`TECHNICAL FIELD
`0002 The present systems and methods relate generally to
`Video-based object modeling, recognition, and tracking, and
`more particularly to detection, modeling, recognition, and
`tracking of objects within videos for purposes of indexing and
`retrieving those videos or portions of videos, wherein gener
`ated global object models are insensitive to variations in pose
`and location of the objects within the videos, as well as the
`resolution of the videos and other factors.
`
`BACKGROUND
`0003 Recognition of objects within videos plays an
`important role for many video-related purposes, such as
`indexing and retrieval of videos based on identified objects,
`security and Surveillance, and other similar functions. As
`used herein, the term “object” shall refer to a definable image
`within a video, such as a face, automobile, article of clothing,
`or virtually any other type of object. For example, FIG. 1
`illustrates a sample frame of a video scene. Exemplary
`objects that are capable of being recognized within the illus
`trated video include characters faces, a plantina Vase, a shoe,
`and an automobile tire, each of which is shown within a
`dashed box to indicate its detection and recognition within the
`frame. As will be understood, however, virtually any image
`may be detected and recognized within a given video.
`0004. Many object recognition systems, and particularly
`facial recognition systems, are known in the art, Such as those
`described in R. Gross et. al. Face Recognition Across Pose
`and Illumination, Handbook of Face Recognition, Springer
`Verlag (2004), and W. Zhao et. al. Face Recognition: A Lit
`erature Survey, ACM Computing Surveys (2003), and in
`other similar texts. A typical face recognition system includes
`three general stages: face data collection, facial modeling,
`and facial identification using the learned/generated models.
`Traditional photo-based face recognition technologies, such
`as those described in M. Turk and A. Pentland, Face Recog
`inition Using Eigenfaces, IEEE Conference on Computer
`Vision and Pattern Recognition, pp. 586-91 (1991), utilize a
`single image or a set of images or photos to generate a model
`or models. These systems function properly only when the
`underlying photos, which are used for analysis and generation
`of facial models, are taken in controlled environments, such
`as with uniform or fixed lighting conditions. Further, the faces
`in the photos generally must be frontal poses only, and
`include little or no expression. Because these traditional sys
`tems are constrained in their ability to adapt to variations in
`photos, and because they only provide fixed-face models,
`their applications, especially for videos (as opposed to still
`images), are highly limited.
`0005 Recently, in order to overcome the limitations of
`traditional photo-based technologies, some video-based
`
`facial recognition systems have emerged, such as those
`described in M. Kim et. al., Face Tracking and Recognition
`with Visual Constraints in Real-World Videos, IEEE Confer
`ence on Computer Vision and Pattern Recognition (2008),
`and Krueger and Zhou, Exemplar-Based Face Recognition
`from Video, European Conference on Computer Vision, pp.
`732-46 (2002), and in other similar texts. These proposed
`systems attempt to overcome the recognition and modeling
`problems posed by images with variations in lighting, back
`ground, and character pose, as well as continuous camera
`motion or character movement within a video scene. These
`systems generally function by either treating each frame
`within a video as an independent image (essentially just a
`variation of a traditional photo-based system) and generating
`a plurality of facial models corresponding to each image, or
`they look at all images in the sequence as a whole and weight
`each image in the sequence equally to generate a combination
`model of all equally-weighted images.
`0006 Both types of video-based recognition systems,
`however, are cumbersome and inefficient, and they produce
`facial models that are often inaccurate. Particularly, by ana
`lyzing all images in a video.-the resulting model or models
`are naturally generated using some images that are partially
`occluded, have low resolutions, include non-frontal poses,
`contain poor lighting, and have a host of other issues, result
`ing in poor quality models. Accordingly, recognition systems
`that incorporate models generated by conventional video
`based systems often produce low recognition rates and over
`all poor results.
`0007. The ability to effectively and efficiently index, store,
`and retrieve Videos, or portions of videos, based on objects in
`those videos is important for a variety of fields. For example,
`production companies or advertisement agencies often rely
`on old or previously-created movies, television shows, and
`other video clips for inclusion in new advertisements, promo
`tions, trailers, and the like. Additionally, with the continuing
`advances of technology, online video viewing is becoming
`increasingly popular, and thus the capability to locate,
`retrieve, and present videos or clips based on user-entered
`search criteria is becoming progressively more vital. Further,
`security systems can benefit from accurate and consistent
`identification of perpetrators or victims within surveillance
`Videos. However, existing and conventional object and facial
`recognition systems are neither flexible nor accurate enough
`for these and other commercial applications.
`0008 For these and many other reasons, there is along-felt
`but unresolved need for a system or method that is able to
`generate effective object models for object recognition based
`on video data, and track temporal coherence of videos in
`order to dynamically update and optimize the generated mod
`els.
`
`BRIEF SUMMARY OF THE DISCLOSURE
`0009 Briefly described, and according to one embodi
`ment, the present disclosure is directed to a method for track
`ing object images in video files. The method comprises the
`steps of receiving a video file, wherein the video file com
`prises a plurality of frames; extracting an image of an object
`from a particular frame in the video file; and extracting a
`Subsequent image of an object from a Subsequent frame in the
`video file. Next, a similarity value is calculated between the
`extracted object image from the particular frame and the
`Subsequent extracted object image in the Subsequent frame. If
`the calculated similarity value exceeds a predetermined simi
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 0017
`
`

`

`US 2009/014 1940 A1
`
`Jun. 4, 2009
`
`larity threshold, then the extracted object images from the
`Video file are collected or organized into an object group.
`0010. According to one aspect, the method further com
`prises the steps of identifying one or more optimal object
`images from the images in the object group; extracting a
`plurality of object features from the one or more optimal
`object images, wherein the object features comprise image
`data associated with the one or more optimal object images;
`and generating an object model based on the plurality of
`extracted object features, wherein the object model is associ
`ated with an object-identifier. In one aspect, the object is
`updated with additional object features extracted from one or
`more additional optimal object images extracted from an
`additional video file. In another aspect, the object model is an
`electronic file. In a further aspect, the plurality of object
`features in the object model are weighted based on the relative
`importance of each feature. In one aspect, the relative impor
`tance is determined based on the addition of recurring fea
`tures to an object model.
`0011. According to another aspect, the one or more opti
`mal object images are identified based on properties of the
`image, wherein the image properties comprise one or more of
`resolution, occlusion, brightness, scale, and pose. In one
`aspect, the one or more optimal object images are identified
`by calculating a similarity score between the images in the
`object group and learned examples of optimal object images.
`In another aspect, the one or more optimal object images are
`images with properties that are conducive to modeling.
`0012. According to a further aspect, the method further
`comprises the steps of identifying one or more optimal object
`images from the images in the object group; retrieving one or
`more predefined object models, wherein each object model is
`associated with an object identifier, calculating an average
`similarity value between the one or more identified optimal
`object images and each of the one or more predefined object
`models. If at least one of the calculated average similarity
`values exceeds a predetermined average similarity threshold,
`then the object images in the object group are labeled accord
`ing to the respective object identifier.
`0013. According to yet another aspect, the one or more
`optimal object images are identified based on properties of
`the image, wherein the image properties comprise one or
`more of resolution, occlusion, brightness, Scale, and pose. In
`one aspect, the one or more optimal object images are iden
`tified by calculating a similarity score between the images in
`the object group and learned examples of optimal object
`images. In another aspect, the one or more optimal object
`images are images with properties that are conducive to mod
`eling.
`0014. According to still another aspect, the method further
`comprises the step of if none, of the calculated average simi
`larity values exceeds a predetermined average similarity
`threshold, the object images in the object group are labeled as
`unknown.
`0015. According to yet a further aspect, the average simi
`larity value is calculated based on a predefined algorithm. In
`one aspect, the average similarity value is calculated by com
`paring object features of the optimal object images to object
`features of the one or more predefined object models.
`0016. According to an additional aspect, the similarity
`value is calculated based on a predefined algorithm. In
`another aspect, the similarity value is calculated by compar
`ing object features, spatial features, and contextual features of
`the extracted object image from the particular frame to object
`
`features, spatial features, and contextual features of the Sub
`sequent extracted object image in the Subsequent frame. In
`one aspect, the spatial features comprise data associated with
`physical distances in images, and the contextual features
`comprise data associated with elements surrounding an
`object image in a frame. According to a further aspect, the
`object group is stored in a database. In one aspect, the object
`group comprises a plurality of object images similar to the
`extracted images.
`0017. According to another aspect, the object images
`comprise images of faces.
`0018. According to another embodiment, the present dis
`closure is directed to a method of tracking a specific object
`through a plurality of frames of a video, wherein each frame
`of the video includes a plurality of objects. The method com
`prises the steps of dividing the video into one or more scenes;
`identifying an image of the specific object in a first frame of
`one of the one or more scenes; assigning the image of the
`specific object from the first frame of the respective scene to
`an object group; and analyzing a Subsequent frame of the
`respective scene to identify a potential additional image of the
`specific object. Next, a similarity value is calculated between
`the image of the specific object from the first frame and the
`potential additional image of the specific object from the
`Subsequent frame. If the calculated similarity value exceeds a
`predetermined similarity threshold, the potential additional
`image is assigned to the object group. In one aspect, the steps
`of analyzing and calculating are repeated for additional Sub
`sequent frames of the respective scene whereby the object
`group has assigned thereto all images of the specific object in
`the analyzed frames of the video.
`0019. According to one aspect, the method further com
`prises the step of selecting the specific object from the plu
`rality of objects in the first frame of the one or mote scenes.
`0020. According to another aspect, the step of assigning
`the image of the specific object to the object group includes
`associating in a database the image with the frame from which
`the image was identified.
`0021. According to a further aspect, the subsequent frame
`is a frame immediately following the first frame. In one
`aspect, there is at least one interposing frame between the first
`frame and the Subsequent frame.
`0022. According to yet another aspect, the step of calcu
`lating a similarity value comprises comparing object features
`of the image of the specific object from the first frame and the
`potential additional image of the specific object from the
`Subsequent frame.
`0023. According to another embodiment, the present dis
`closure is directed to a method for generating an object model
`from one or more images in a video. The method comprises
`the steps of receiving a video file, wherein the video file
`comprises a plurality of video frames; and detecting an object
`image from one of the plurality of video frames. If the
`detected object image is an optimal object image, a plurality
`of object features are extracted from the optimal object
`image. Next, it is determined whetheran object model already
`exists for the object represented by the optimal object image.
`If an object model does not exist for the object, an object
`model is generated comprising the plurality of object features
`extracted from the optimal object image.
`0024. According to one aspect, the method further com
`prises the step of if an object model already exists for the
`object, adding the plurality of features extracted from the
`optimal object image to the already-existing object model.
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 0018
`
`

`

`US 2009/014 1940 A1
`
`Jun. 4, 2009
`
`0025. According to another aspect, the plurality of object
`features comprise image data associated with the optimal
`object image. In one aspect, the plurality of object features in
`the object model are weighted based on the relative impor
`tance of each feature.
`0026. According to another aspect, the object model is an
`electronic file.
`0027. According to a further aspect, the method further
`comprises the step of determining that the detected object
`image is an optimal object image based on properties of the
`image, wherein the image properties comprise one or more of
`resolution, occlusion, brightness, scale, and pose. In one
`aspect, the method further comprises the step of determining
`that the detected object image is an optimal object image by
`calculating a similarity value between the detected object
`image and learned examples of optimal object images.
`0028. According to an additional aspect, the object image
`comprises an image of a face.
`0029. According to an additional embodiment, the present
`disclosure is directed to a system for identifying object
`images from videos. The system comprises a processor and a
`computer program product, wherein the computer program
`product includes a computer-readable medium that is usable
`by the processor, the medium having stored thereon a
`sequence of instructions that when executed by the processor
`causes the execution of the steps of retrieving a video,
`wherein the video comprises a plurality of frames; extracting
`a plurality of object images from the plurality of frames; and
`grouping similar object images together into a group based on
`a similarity score calculated between features in the object
`images. Further steps include calculating an average similar
`ity score for one or more object images in the group as
`compared to one or more stored object models, wherein each
`object model is associated with an object identifier; and if the
`calculated average similarity Score exceeds a predefined
`threshold, identifying the object images in the group based on
`the object identifier.
`0030. According to one aspect, the one or more object
`images used to calculate the average similarity score com
`prise optimal object images. In another aspect, the system is
`further configured to execute the step of identifying optimal
`object image based on properties of the image, wherein the
`image properties comprise one or more of resolution, occlu
`Sion, brightness, scale, and pose. In another aspect, the system
`is configured to execute the further step of identifying optimal
`object images by calculating a similarity score between the
`object images in the group and learned examples of optimal
`object images.
`0031. According to another aspect, the system comprises a
`database for storing one or more videos and the one or more
`object models.
`0032. According to a further aspect, the system is config
`ured to execute the step of if the calculated average similarity
`score does not exceed the predetermined threshold, the object
`images in the group are labeled as unknown.
`0033 According to yet another aspect, the average simi
`larity score is calculated based on a predefined algorithm. In
`one aspect, the average similarity score is calculated by com
`paring object features of the one or more object images to
`object features of the one or more stored object models.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`0034. The accompanying drawings illustrate one or more
`embodiments of the disclosure and, together with the written
`
`description, serve to explain the principles of the disclosure.
`Wherever possible, the same reference numbers are used
`throughout the drawings to refer to the same or like elements
`of an embodiment, and wherein:
`0035 FIG. 1 illustrates a sample frame of a video scene
`indicating recognized images from within the frame.
`0036 FIG. 2 illustrates a flowchart describing the steps
`involved, from a high-level, in one embodiment of the present
`system for detecting, modeling, recognizing, and tracking
`object images throughout one or more videos.
`0037 FIG. 3 is an object modeling flowchart showing the
`general steps involved in creating a new object model, as well
`as updating a preexisting model, according to one embodi
`ment of the present system.
`0038 FIG. 4 is a flowchart depicting the steps and pro
`cesses involved in an object recognition procedure according
`to an embodiment of the present system.
`0039 FIG. 5 shows a flowchart of an object tracking pro
`cedure according to one embodiment of the present system.
`0040 FIG. 6 is a listing of episodes of the television pro
`gram Gilmore Girls, including identifiers associated with
`each episode, the title of each episode, and the number of
`frames in each episode, which were used for testing an
`embodiment of the present system.
`0041
`FIG. 7 shows the facial detection and grouping
`results for each tested episode of the television program
`Gilmore Girls.
`0042 FIG. 8 shows a precision-recall curve for facial rec
`ognition using pure models learned from each tested episode
`alone.
`0043 FIG. 9 is a table listing several of the plotted data
`points shown in FIG.8 for 100% and 90% precision values for
`each tested episode.
`0044 FIG. 10 illustrates a table listing the average preci
`sion and recall values for the eight major characters presented
`in ep04 based on testing via a pure episode model.
`0045 FIG. 11 is a confusion matrix for the eight identified
`major characters listed in FIG. 10.
`0046 FIG. 12 shows a precision-recall curve for facial
`recognition using merged models learned by incrementally
`combining models from each tested episode.
`0047 FIG. 13 is a table listing several of the plotted data
`points shown in FIG. 12 for 100% and 90% precision values
`for each merged episode combination.
`0048 FIG. 14 is a table listing the average precision and
`recall values for the twelve major characters presented in
`ep00-ep04 based on testing via merged models.
`0049 FIG. 15 illustrates a confusion matrix for the twelve
`identified major characters listed in FIG. 14.
`
`DETAILED DESCRIPTION
`
`0050 For the purpose of promoting an understanding of
`the principles of the present disclosure, reference will now be
`made to the embodiments illustrated in the drawings and
`specific language will be used to describe the same. It will,
`nevertheless, be understood that no limitation of the scope of
`the disclosure is thereby intended; any alterations and further
`modifications of the described or illustrated embodiments,
`and any further applications of the principles of the disclosure
`
`Google Exhibit 1003 - Google v. CSI
`IPR2025-00877 - Page 0019
`
`

`

`US 2009/014 1940 A1
`
`Jun. 4, 2009
`
`as illustrated therein are contemplated as would normally
`occur to one skilled in the art to which the disclosure relates.
`
`Overview
`0051 Aspects of the present disclosure generally relate to
`systems and methods for detection, modeling, recognition,
`and tracking of objects within video content. Aspects of the
`present disclosure further relate to indexing and retrieval
`systems for videos based on generated object models. Gen
`erally, the system enables computation of a global object
`model for each particular object in one or more videos from
`automatically-selected optimal object images, via an optimal
`object classifier (discussed in greater detail below). These
`objects include people, faces, articles of clothing, plants,
`animals, machinery, electronic equipment, food, and virtually
`any other type of image that can be captured or presented in
`video. Further, the global object models are configured to be
`as broad or granular as desired by a system operator or user,
`such as abroad “automobile” model, or, alternatively, a model
`for a specific make, type, and year of car. The global object
`models are generally comprised of features encoded with an
`affine-invariant descriptor, and thus are insensitive to location
`and positioning (i.e. rotation) of the object image in a video,
`as well as resolution variations, lighting differences, and
`other image characteristics.
`0052 Embodiments of the present system further generate
`local object models for each image recognized in a particular
`Video, and compare those models to the plurality of global
`models to determine if a match has occurred. Generally, a
`weighted graph-matching algorithm is designed and utilized
`for object recognition. Embodiments of the local models
`typically encode both spatial and contextual information
`from the video to enable tracking of the object images under
`going pose or positioning changes within a shot or scene. By
`recognizing optimal object images and tracking them
`throughout a video, both optimal and non-optimal objects are
`identified, thus resulting in an accurate and practical object
`recognition system.
`0053 Embodiments of the present system are useful for a
`wide variety of applications, including video indexing and
`retrieval, video Surveillance and security, unknown person
`identification, advertising, and many other fields. Generally,
`as will be described below, the present system is insensitive to
`variations in scale and focus within videos or images, and
`operates without any human intervention except for unknown
`or initial object labeling. Accordingly, the present system is
`particularly advantageous for any application in which rec
`ognizing an object from a video with camera motion and a
`wide range of camera views, as well as object pose changes,
`is important.
`0054 For purposes of example throughout this document,
`facial recognition examples will often be described. Facial
`recognition is a widely-applicable field, and has many com
`mercial applications. It should be understood, however, that
`the present systems and methods are in no way limited to
`facial images, and the present systems and methods may be
`applied to virtually any object image from within a video or
`still photograph. Accordingly, facial recognition systems will
`often be described for illustrative purposes only, and are in no
`way intended to limit the scope of the present disclosure.
`0055 As will be understood and will become apparent
`below, embodiments of the present system may be operated in
`a computer environment including databases and other Stor
`age apparatuses, servers, processors, terminals and displays,
`
`c