(19) United States
`(12) Patent Application Publication (10) Pub. N0.: US 2012/0051604 A1
`Dudovich et a].
`(43) Pub. Date:
`Mar. 1, 2012
`
`US 20120051604A1
`
`(54) SYSTEM AND METHOD FOR
`VIDEO-ASSISTED IDENTIFICATION OF
`MOBILE PHONE USERS
`
`(76) Inventors;
`
`B031 Dudovicha Rehovot (1L);
`Gideon HaZZani, Rishon Le Zion
`(IL)
`
`(21) Appl. No.:
`
`13/189,514
`
`(22) Filedi
`
`JUL 24, 2011
`
`(30)
`
`Foreign Application Priority Data
`
`Jul. 25, 2010 (IL) ........................................ .. 207176
`_
`_
`_
`_
`Pubhcatlon Classl?catlon
`
`(51) Int. Cl.
`G06K 9/00
`
`(2006.01)
`
`(52) US. Cl. ...................................................... .. 382/115
`
`(57)
`
`ABSTRACT
`
`Methods and systems for identifying and tracking individuals
`in a area-of-interest that may be covered by a video surveil
`lance subsystem and by a communication location sub
`system, and a correlation system correlates the outputs of the
`tWo subsystems. The communication location subsystem
`may monitor communication of mobile phones. The video
`subsystem captures video images of the area-of-interest, and
`processes the video images so as to identify individuals Who
`are present in the area. The correlation system correlates a
`given mobile phone With a given individual Who Was identi
`?ed by the Video Subsystem as being engaged in a phone
`conversation. After correlating the mobile phone With the
`individual using the phone, the correlation system outputs
`correlated information regarding the phone and its user to an
`operator.
`
`37
`
`VIDEO SURVEILLANCE
`SUBSYSTEM
`
`38
`
`/20
`
`40
`8
`CoRRELATIoN SYSTEM
`
`l
`
`i» 56
`
`3
`VIDEO
`‘ INTERFACE ‘L
`CORRELATION
`I» PROCESSOR
`, INTEQFXACE
`8
`52
`
`48
`
`t‘,
`'
`
`I ~
`
`COMMUNICATION
`LoCATIoN SUBSYSTEM
`
`\
`
`3'2
`
`,
`
`)
`K
`44 CORRELATED
`
`PHONE|NFO&
`USER IMAGES V
`
`64
`
`GONDUD
`
`MONITORING
`CENTER
`
`GTL 1006
`PGR of U.S. Patent No. 8,855,280
`
`

`
`Patent Application Publication
`
`Mar. 1, 2012 Sheet 1 0f 2
`
`US 2012/0051604 A1
`
`
`
`2596 292.518
`
`Ow mm
`
`mowwwoomm
`
`wi? 200 M
`
`57g
`
`S N
`
`2“
`
`89>
`
`M
`
`

`
`Patent Application Publication
`
`Mar. 1, 2012 Sheet 2 0f 2
`
`US 2012/0051604 A1
`
`IDENTIFY ACTIVE MOBILE
`PHONE IN AREA-OF-INTEREST
`BY MONITORING
`COMMUNICATION WITH
`WIRELESS NETWORK
`
`OBTA|N \/|DEO "WAGES OF
`AREA_O|I_|NTEREST
`
`82
`8
`
`V
`
`a
`78
`
`3
`70
`
`"
`EXTRACT INFORMATION
`REGARDING MOBILE PHONE
`(E.G., PHONE NUMBER, IMSI)
`FROM MONITORED
`COMMUNICATION
`I
`
`2
`74
`
`PROCESS VIDEO IMAGES So
`AS TO AUTOMATICALLY
`IDENTIFY INDIVIDUAL
`ENGAGED IN PHONE
`CONVERSATION
`
`v
`EXTRACT IMAGES OF
`INDIVIDUAL
`'
`
`8
`86
`
`CORRELATE MOBILE PHONE INFORMATION
`WITH IMAGES OF INDIVIDUAL
`
`PROVIDE CORRELATED PHONE
`INFORMATION AND IMAGES TO OPERATOR
`
`FIG. 2
`
`

`
`US 2012/0051604 A1
`
`Mar. 1, 2012
`
`SYSTEM AND METHOD FOR
`VIDEO-ASSISTED IDENTIFICATION OF
`MOBILE PHONE USERS
`
`FIELD OF THE DISCLOSURE
`
`[0001] The present disclosure relates generally to commu
`nication systems and video surveillance, and particularly to
`methods and systems for combined communication monitor
`ing and video monitoring.
`
`BACKGROUND OF THE DISCLOSURE
`
`[0002] Mobile communication netWorks deploy various
`techniques for measuring the geographical locations of Wire
`less communication terminals. Such techniques are used, for
`example, for providing emergency services (e.g., “911” or
`“112” services) in cellular netWorks. Video surveillance sys
`tems are deployed and operated in various applications, such
`as airport security, crime prevention and access control. In a
`typical video surveillance application, multiple video cam
`eras acquire video footage, Which is vieWed and/or recorded
`at a monitoring center.
`
`SUMMARY OF THE DISCLOSURE
`
`[0003] An embodiment that is described herein provides a
`method, including:
`[0004] identifying a mobile communication terminal that is
`active in an area under surveillance by monitoring commu
`nication conducted betWeen the mobile communication ter
`minal and a Wireless communication network;
`[0005] automatically identifying, in video images of the
`area, that an individual is engaged in a communication ses
`sion; and
`[0006] correlating the identi?ed individual With the identi
`?ed mobile communication terminal.
`[0007] In some embodiments, identifying the individual
`includes applying to the video images an automated image
`recognition process, Which recogniZes a person holding a
`communication terminal. In an embodiment, identifying the
`mobile communication terminal includes detecting a ?rst
`time at Which the mobile communication terminal initiates
`the communication, identifying the individual includes
`detecting a second time at Which the individual initiates the
`communication session, and correlating the individual With
`the mobile communication terminal includes correlating the
`?rst and second times. Detecting the second time may include
`applying to the video images an automated image recognition
`process, Which recogniZes a person initiating the communi
`cation.
`[0008] In a disclosed embodiment, identifying the mobile
`communication terminal includes extracting information
`regarding the mobile communication terminal from the moni
`tored communication, identifying the individual includes
`extracting one or more images of the individual from the
`video images, and correlating the individual With the mobile
`communication terminal includes correlating the information
`regarding the mobile communication terminal With the
`images of the individual. In an embodiment, the method
`includes presenting the information regarding the mobile
`communication terminal and the images of the individual to
`an operator. In another embodiment, extracting the informa
`tion includes extracting at least one information type selected
`from a group of types consisting of a telephone number and an
`International Mobile Station Identity (IMSI) of the mobile
`
`communication terminal. In yet another embodiment, identi
`fying the individual further includes matching the extracted
`images of the individual With at least one image draWn from
`a list of knoWn individuals, and correlating the individual
`With the mobile communication terminal includes correlating
`the mobile communication terminal With the matching image
`draWn from the list.
`[0009] In some embodiments, identifying the mobile com
`munication terminal includes interrogating the mobile com
`munication terminal by an interrogation device that is sepa
`rate from the Wireless communication netWork. In a disclosed
`embodiment, a coverage area of the interrogation device is
`aligned With a ?eld-of-vieW of a video camera that captures
`the video images. In an alternative embodiment, identifying
`the mobile communication terminal includes accepting an
`identi?cation of the mobile communication terminal from the
`Wireless communication netWork. In an embodiment, identi
`fying the mobile communication terminal includes retrieving
`an image of a subscriber of the identi?ed mobile communi
`cation terminal, and the method includes matching a given
`individual Who appears in the video images With the image of
`the subscriber of the identi?ed mobile communication termi
`nal.
`[0010] There is additionally provided, in accordance With
`an embodiment that is described herein, apparatus, including:
`[0011] a communication location subsystem, Which is con
`?gured to identify a mobile communication terminal that is
`active in an area under surveillance by monitoring commu
`nication conducted betWeen the mobile communication ter
`minal and a Wireless communication network;
`[0012] a video subsystem, Which is con?gured to identify,
`in video images of the area, that an individual is engaged in a
`communication session; and
`[0013] a correlation processor, Which is con?gured to cor
`relate the individual With the identi?ed mobile communica
`tion terminal.
`[0014] There is also provided, in accordance With an
`embodiment that is described herein, apparatus, including:
`[0015] a ?rst interface, Which is con?gured to accept an
`identi?cation of a mobile communication terminal that is
`active in an area under surveillance, Wherein the identi?cation
`is obtained by monitoring communication conducted
`betWeen the mobile communication terminal and a Wireless
`communication netWork;
`[0016] a second interface, Which is con?gured to accept
`video images of the area; and
`[0017] a processor, Which is con?gured to identify in the
`video images of the area that an individual is engaged in a
`communication session, and to correlate the individual With
`the identi?ed mobile communication terminal.
`[0018] There is further provided, in accordance With an
`embodiment that is described herein, a computer softWare
`product, the product including a computer-readable medium,
`in Which program instructions are stored, Which instructions,
`When read by one or more processors, cause the processors to
`accept an identi?cation of a mobile communication terminal
`that is active in an area under surveillance, Wherein the iden
`ti?cation is obtained by monitoring communication con
`ducted betWeen the mobile communication terminal and a
`Wireless communication netWork, to accept video images of
`the area, to identify in the video images of the area that an
`individual is engaged in a communication session, and to
`correlate the individual With the identi?ed mobile communi
`cation terminal.
`
`

`
`US 2012/0051604 A1
`
`Mar. 1, 2012
`
`[0019] The present disclosure Will be more fully under
`stood from the following detailed description of the embodi
`ments thereof, taken together With the drawings in Which:
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0020] FIG. 1 is a block diagram that schematically illus
`trates a security monitoring system, in accordance With an
`embodiment of the present disclosure; and
`[0021] FIG. 2 is a How chart that schematically illustrates a
`method for security monitoring, in accordance With an
`embodiment of the present disclosure.
`
`DETAILED DESCRIPTION OF EMBODIMENTS
`OVERVIEW
`
`[0022] Embodiments that are described herein provide
`improved methods and systems for identifying and tracking
`individuals in an area-of-interest, such as in an airport termi
`nal or train station. In some embodiments, the area-of-interest
`is covered by a video surveillance subsystem and by a com
`munication location subsystem, and a correlation system cor
`relates the outputs of the tWo subsystems.
`[0023] The communication location subsystem monitors
`communication of mobile phones (or other kinds of mobile
`communication terminals) in the area-of-interest, and identi
`?es mobile phones that conduct calls in the area at a given
`time. In addition, the communication location subsystem may
`extract information regarding the identi?ed mobile phones,
`such as phone number and/or International Mobile Station
`Identity (IMSI).
`[0024] The video subsystem captures video images of the
`area-of-interest, and processes the video images so as to
`identify individuals Who are present in the area. In particular,
`the video subsystem applies an image recognition process
`that automatically identi?es individuals Who are engaged in a
`phone conversation (or other communication session). For
`example, the image recognition process may recogniZe an
`image of an individual Who holds a mobile phone near his
`head. In some embodiments, the video subsystem extracts
`images of the identi?ed individuals’ faces from the video
`images.
`[0025] The correlation system correlates a given mobile
`phone, Which Was identi?ed by the communication location
`subsystem, With a given individual Who Was identi?ed by the
`video subsystem as being engaged in a phone conversation. In
`some embodiments, the communication location subsystem
`detects the time at Which the given mobile phone initiated a
`call, the video subsystem detects the time at Which the given
`individual placed the mobile phone next to his ear, and the
`correlation system correlates the occurrence times of these
`tWo events.
`[0026] After correlating the mobile phone With the indi
`vidual using the phone, the correlation system outputs corre
`lated information regarding the phone and its user to an opera
`tor. For example, the correlation subsystem may provide the
`operator With the phone number and IMSI of the mobile
`phone (obtained using the communication location sub
`system) together With still or video images of the phone user’s
`face (obtained using the video subsystem). The correlation is
`based on actual recognition that the individual observed in the
`video images is engaged in a phone conversation. The corre
`lation is therefore highly reliable, even in croWded environ
`ments such as airport terminals and train stations.
`
`[0027] Using the disclosed techniques, the correlation sys
`tem provides the operator With valuable information regard
`ing individuals Who are found in the area-of-interest. The
`correlated video-related and communication-related infor
`mation enables the operator to better identify and act upon
`suspicious events With high speed and e?iciency.
`
`System Description
`[0028] FIG. 1 is a block diagram that schematically illus
`trates a security monitoring system 20, in accordance With an
`embodiment of the present disclosure. System 20 monitors a
`certain area-of-interest, and in particular identi?es individu
`als 24 Who use mobile communication terminals 28, using a
`combination of video-based and communication-based
`monitoring. A system of this sort can be operated, for
`example, in an airport terminal or train station in order to
`identify potential terrorists or as part of a crime prevention
`system. Other example applications may cover any other
`suitable environment, such as a street or a football stadium.
`[0029] System 20 comprises a communication location
`subsystem 32, Which identi?es mobile communication termi
`nals that are active in the area-of-interest. Mobile communi
`cation terminals 28 may comprise, for example, cellular
`phones, Wireless-enabled computers or Personal Digital
`Assistants (PDAs), or any other suitable communication or
`computing devices having Wireless communication capabili
`ties. A given terminal 28 communicates With a certain Wire
`less communication netWork (not shoWn in the ?gure). The
`communication terminals may communicate With the net
`Work using any suitable communication standard or protocol,
`such as Global System for Mobile communication (GSM),
`Universal Mobile Telecommunication System (UMTS),
`CDMA2000 or other third generation (3G) cellular standard,
`Long Term Evolution (LTE) or Integrated Digital Enhanced
`NetWork (IDEN). Alternatively, the netWorks and terminals
`may conform to the IEEE 802.16 (WiMAX) standards or
`other Wireless data standard. Although the description that
`folloWs refers to a single netWork, system 20 may identify
`terminals that operate With any desired number of netWorks,
`Which may conform to different standards or protocols.
`[0030] In some embodiments, communication location
`subsystem 32 comprises one or more interrogation devices
`(referred to as interrogators for brevity). A given interrogator
`establishes communication With mobile terminals 28 in a
`given coverage area, in order to extract identity attributes of
`the terminals. Each interrogator typically comprises a direc
`tional antenna, Who se beam pattern (combined With the inter
`rogator’s transmission poWer) determines the coverage area.
`A typical interrogator imitates the operation of a base station,
`and solicits a mobile terminal to start communicating With the
`interrogator. The interrogator typically communicates With
`the terminal for a short period of time, during Which it extracts
`the identity attributes of the terminal. For example, a given
`interrogator may force any terminal that enters its coverage
`area to perform a LOCATION UPDATE process, Which
`reveals its identity.
`[0031] A given interrogation device in subsystem 32 may
`extract various identity attributes of the terminal, such as, for
`example, the terminal’s International Mobile Subscriber
`Identity (IMSI), the terminal’s phone number or any other
`suitable attribute indicating the identity of the terminal. The
`above-described attribute extraction functions can be carried
`out using knoWn Interrogation devices, Which are sometimes
`referred to as “IMSI catchers.” Examples of IMSI catching
`
`

`
`US 2012/0051604 A1
`
`Mar. 1, 2012
`
`techniques are described, for example, by Strobel in “IMSI
`Catcher,” Jul. 13, 2007, Which is incorporated herein by ref
`erence, by Asokan et al., in “Man-in-the-Middle Attacks in
`Tunneled Authentication protocols,” the 2003 Security Pro
`tocols Workshop, Cambridge, UK, Apr. 2-4, 2003, Which is
`incorporated herein by reference, and by Meyer and WetZel in
`“On the Impact of GSM Encryption and Man-in-the-Middle
`Attacks on the Security of Interoperating GSM/UMTS Net
`Works,” proceedings of the 15th IEEE International Sympo
`sium on Personal, Indoor and Mobile Radio Communica
`tions, Barcelona, Spain, Sep. 5-8, 2004, pages 2876-2883,
`Which is incorporated herein by reference. Any of these
`knoWn schemes, as Well as any other suitable type of interro
`gator, can be used to implement the interrogation devices of
`subsystem 32. Typically, the interrogators are detached and
`separate from the communication netWork or netWorks via
`Which terminals 28 communicate.
`[0032] In alternative embodiments, communication loca
`tion subsystem 32 is implemented as part of the communica
`tion netWork With Which terminals 28 communicate. In these
`embodiments, subsystem 32 may apply any suitable location
`tracking technique available in the netWork, or a combination
`of such techniques, in order to measure terminal locations.
`Some location tracking techniques, referred to as netWork
`based techniques, are carried out by base stations and/or other
`netWork-side components of the netWork, Without necessar
`ily using special hardWare or softWare in terminals 28. Other
`location tracking techniques are terminal-based, i.e., use spe
`cial hardWare or softWare in mobile terminals 28. Some
`examples of location tracking techniques that can be used for
`this purpose are described in US. patent application Ser. No.
`12/497,799, ?led Jul. 6, 2009, Whose disclosure is incorpo
`rated herein by reference. The location tracking techniques
`may be passive or active. Passive techniques perform unob
`trusive probing of the signaling information transmitted in the
`netWork, and extract location information from the monitored
`signaling. Active techniques, on the other hand, proactively
`request the netWork or the terminal to provide location infor
`mation. In either case, subsystem 32 is able to detect that a
`given terminal 28 is active in the area-of-interest, and to
`extract information regarding the terminal (e. g., phone num
`ber and/ or IMSI) from the monitored communication.
`[0033] System 20 further comprises a video surveillance
`subsystem 36, Which captures video images of the area-of
`interest. In a typical implementation, video subsystem 36
`comprises one or more video cameras 37 that are connected to
`a video server 38. A given camera may have a ?xed ?eld-of
`vieW, or it may comprise a Pan-Tilt-Zoom (PTZ) camera
`Whose ?eld-of-vieW is adjustable. In some embodiments,
`video server 38 applies an image recognition process that
`automatically identi?es individuals Who are engaged in a
`phone conversation, e.g., an individual Who holds a mobile
`phone next to his head. As Will be explained beloW, this
`identi?cation is used to correlate mobile terminals identi?ed
`by communication location subsystem 32 With individuals
`identi?ed by video subsystem 36.
`[0034] System 20 comprises a correlation system 40, Which
`correlates the information produced by subsystems 32 and 36.
`Correlation system 40 comprises a communication interface
`44, Which is used for receiving location indications from
`communication location subsystem 32. In a typical embodi
`ment, each location indication indicates that a given mobile
`terminal Was identi?ed in the area-of-interest. The location
`indication typically comprises a time stamp indicating the
`
`time at Which the terminal in question Was identi?ed, and
`certain attributes of the terminal such as phone number and/ or
`IMSI.
`[0035] Correlation system 40 further comprises a video
`interface 48, Which is used for receiving video images and
`additional information from video subsystem 36. The infor
`mation obtained from subsystem 36 may comprise, for
`example, an indication that the video sub system has identi?ed
`in the video images an individual operating a mobile phone.
`The indication (referred to herein as “video indication”) may
`also comprise additional information, such as a time stamp
`indicating the time at Which the individual Was observed. In
`some embodiments, the video sub system extracts one or more
`(still or video) images of the individual, or of the individual’s
`face, from the video images. The video subsystem may pro
`vide the extracted images to correlation system 40, as Well.
`[0036] Correlation system 40 comprises a correlation pro
`cessor 52, Which correlates the information obtained from
`subsystems 32 and 36. In particular, processor 52 ?nds cor
`relations mobile phones, Which Were identi?ed by communi
`cation location subsystem 32 as operating in the area-of
`interest, With individuals Who Were identi?ed by video
`subsystem 36 as being engaged in a phone conversation.
`[0037] In a typical implementation, processor 52 receives
`location indications from communication location subsystem
`32, each location indication indicating that a certain mobile
`terminal Was identi?ed as active in the area-of-interest at a
`given time. Processor 52 also receives video indications from
`video subsystem 36, each video indication indicating that a
`certain individual Was identi?ed in the video images as being
`engaged in a phone call at a given time. The video indications
`are typically accompanied With one or more images of the
`individual in question (typically the individual’s face), as
`extracted from the video images.
`[0038] Based on this information, processor 52 correlates a
`mobile terminal identi?ed by subsystem 32 With an indi
`vidual identi?ed by subsystem 36. For example, processor 52
`may correlate the indications of subsystem 32 and subsystem
`36 using their respective time stamps. In other Words, if
`subsystem 32 identi?ed a communication terminal that Was
`active in the area-of-interest at a certain time, and subsystem
`36 identi?ed that a certain individual Was engaged in a phone
`conversation in the area-of-interest at approximately the same
`time, then processor 52 may conclude that the identi?ed
`terminal is associated by the identi?ed individual.
`[0039] In some embodiments, the image recognition pro
`cess applied by subsystem 36 is able to recogniZe the time at
`Which an individual in question initiated the call. For
`example, subsystem 36 may identify the time at Which the
`individual raised his arm and brought the mobile phone in
`proximity to his head. Subsystem 32, on the other hand, may
`be able to detect the time at Which a mobile phone initiates a
`call. By correlating these tWo occurrence times, correlation
`processor 52 can correlate phones With individuals at high
`accuracy and With very loW probability of error. This tech
`nique is especially useful in croWded environments and/or
`large areas-of-interest, Wherein the area-of-interest com
`prises multiple individuals and/ or multiple terminals. In alter
`native embodiments, processor 52 may correlate individuals
`With mobile phones using any other suitable technique.
`[0040] Upon correlating a certain mobile terminal With a
`certain individual, processor 52 produces a correlated infor
`mation set pertaining to the terminal and its user. This set
`typically comprises information obtained from both sub
`
`

`
`US 2012/0051604 A1
`
`Mar. 1, 2012
`
`system 32 and subsystem 36. In an example embodiment, the
`correlated information set comprises a telephone number
`and/or IMSI of the identi?ed terminal (from subsystem 32)
`and an image of the individual’s face (from subsystem 36).
`The correlated information set may also comprise video foot
`age of the individual as captured by subsystem 36.
`[0041] In some embodiments, processor 52 outputs the cor
`related information to a monitoring center 60, Where an
`operator 64 vieWs and acts upon the information. Addition
`ally or alternatively, processor 52 stores the correlated infor
`mation in a database 56, e.g., on a magnetic disk or other
`storage device. Processor 52 may also store video footage
`that, provided by subsystem 36, in database 52.
`[0042] The con?guration of FIG. 1 is an example con?gu
`ration, Which is chosen purely for the sake of conceptual
`clarity. In alternative embodiments, any other suitable system
`con?guration can also be used. For example, the image rec
`ognition process that recogniZes individuals Who are engaged
`in phone conversations can be carried out by correlation pro
`cessor 52 rather than by video subsystem 36. Although FIG.
`1 shoWs a single terminal and a single individual for the sake
`of clarity, the disclosed techniques can also be used When the
`area-of-interest contains multiple mobile terminals and/or
`multiple individuals.
`[0043] In a typical implementation, communication loca
`tion subsystem 32 comprises a single interrogator, and video
`subsystem 36 comprises a single video camera. In this
`embodiment, the camera’s ?eld-of-vieW and the interroga
`tor’s coverage area are designed to cover approximately the
`same geographical area. For example, the camera and inter
`rogator may be collocated and aligned. Alternatively, hoW
`ever, subsystems 32 and 36 may be deployed in any other
`suitable Way. Typically, hoWever, both subsystems are con
`?gured to cover approximately the same area-of-interest.
`[0044] The elements of system 20 may be implemented in
`hardWare, in softWare, or using a combination of hardWare
`and softWare elements. In some embodiments, video server
`38 and correlation processor 52 comprise general-purpose
`computers, Which are programmed in softWare to carry out
`the functions described herein. The softWare may be doWn
`loaded to the computers in electronic form, over a netWork,
`for example, or it may, alternatively or additionally, be pro
`vided and/ or stored on non-transitory tangible media, such as
`magnetic, optical, or electronic memory.
`
`Security Monitoring Method Description
`
`[0045] FIG. 2 is a How chart that schematically illustrates a
`method for security monitoring, in accordance With an
`embodiment of the present disclosure. The method begins
`With communication location subsystem 32 identifying one
`or more mobile communication terminals 28 that are active in
`a certain area-of-interest, at a terminal identi?cation step 70.
`Subsystem 32 identi?es the active terminals by monitoring
`the communication betWeen terminals 28 and the Wireless
`communication netWork. Subsystem 32 extracts information
`regarding the active terminals from the monitored communi
`cation, at a communication extraction step 74. Such informa
`tion may comprise, for example, the phone number and/or
`IMSI of each active terminal.
`[0046] In parallel, video subsystem 36 captures video
`images of the area-of-interest, at a video capturing step 78.
`Subsystem 36 processes the captured video, so as to identify
`one or more individuals that are engaged in a phone conver
`sation, at an image recognition step 82. Example techniques
`
`that can be used for this purpose are described beloW. For each
`identi?ed individual, the video subsystem extracts one or
`more images of the individual (e.g., of the individual’s face)
`from the video images, at a video extraction step 86.
`[0047] Correlation processor 52 correlates the information
`provided by subsystem 32 (information regarding identi?ed
`active terminals) and by subsystem 36 (images of individuals
`identi?ed as conducting phone calls), at a correlation step 90.
`As explained above, the correlation may use time stamps that
`are provided by subsystems 32 and 36. Correlation processor
`52 provides the correlated information to monitoring center
`60 for presenting to operator 64, at an output step 94. Proces
`sor 52 may also store the correlated information in database
`56.
`[0048] Video subsystem 36 (video server 38 in this
`example) may use any suitable image recognition process for
`automatically identifying individuals Who are engaged in
`communication sessions. In an example process, server 38
`uses an image model of an individual holding a phone, and
`another image model of an individual Who does not hold a
`phone. Using these tWo models, server 38 distinguishes
`betWeen an individual holding a phone and an individual Who
`does not hold a phone in the video images.
`[0049] An image models can be constructed, for example,
`by capturing multiple training images of individuals (typi
`cally images of the individuals’ torsos), some holding phones
`and some Who do not. Each training image is divided into
`blocks (e.g., 6-by-6 pixels each). A histogram of gradient
`angles is constructed for each block. (A gradient angle is
`typically de?ned as the rate of change in image color from
`pixel to pixel.) Typically, each gradient angle is Weighted
`according to its magnitude. A set of 6><6><8I288 vectors rep
`resents the visual information of a person holding a phone as
`opposed to a person Who does not hold a phone, and are used
`as input or “Descriptor.” The above-described classi?er is
`trained to distinguish betWeen the tWo classes (betWeen a
`person holding a phone and a person Who does not hold a
`phone). Any suitable classi?er, such as Support Vector
`Machines (SVM) or Neural NetWorks, can be used for this
`purpose. An example classi?cation and training method is
`described by Vedaldi, in “An Implementation of SIFT Detec
`tor and Descriptor,” University of California at Los Angeles,
`2006, Which is incorporated herein by reference. Altema
`tively, any other suitable scheme can also be used.
`[0050] When a neW image is provided to server 38 for
`analysis, the server ?rst determines an image location Where
`an image of an individual is expected to be located (e. g., based
`on motion, face detection, descriptors or any other suitable
`method). The server then applies the above-described classi
`?er to this image location, in order to determine Whether the
`image location contains an image of a person holding a phone
`or a person not holding a phone.
`[0051] Although the embodiments described herein mainly
`address identi?cation of individuals holding mobile phones,
`the principles of the present disclosure can also be used for
`identifying individuals that use other kinds of communication
`terminals and/or for conducting other kinds of communica
`tion sessions. For example, the disclosed techniques can be
`used to automatically identify users of mobile computing
`platforms (e.g. PDAs) Who are in the process of communi
`cating. As another example, the disclosed techniques can be
`used to automatically identify users Who are in the process of
`sending Short Message Service (SMS) messages or e-mails
`
`

`
`US 2012/0051604 A1
`
`Mar. 1, 2012
`
`using a mobile communication terminal, or users Who com
`municate using a handheld radio (“Walkie-talkie”).
`[0052] In some embodiments, additional information
`regarding the identi?ed user, such as the user’s car license
`plate number, can be extracted from the video images and
`combined With the other correlated information (e.g., IMSI
`and image of user). This information can also be presented to
`operator 64 by processor 52.
`[0053] In some embodiments, system 20 may use the image
`of the identi?ed individual ’s face to automate and improve the
`correlation process. In an example embodiment, system 20
`may hold or have access to a blacklist of images of suspect
`individuals (e.g., a picture album of criminals or terrorists).
`Once an image of the individual operating the identi?ed ter
`minal is available, video server 38 can compare this image
`against the blacklist using any suitable face recognition
`method. If a match is found (i.e., if the image of the individual
`in question matches one of the images in the blacklist), cor
`relation system 40 can correlate the blacklist entry (e.g.,
`knoWn criminal or terrorist) With the terminal (e.g., phone
`number or IMSI). Using this method, the operator is provided
`With the phone number or IMSI of a knoWn target. This
`technique is useful, for example, for tracking criminals or
`terrorists that change phones or Subscriber Identity Module
`