(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2006/0187305 A1
`Aug. 24, 2006
`(43) Pub. Date:
`Trivedi et al.
`
`US 2006O1873 05A1
`
`(54) DIGITAL PROCESSING OF VIDEO IMAGES
`(76)
`
`Inventors: Mohan M Trivedi, San Diego, CA
`(US); Kohsia Huang, La Jolla, CA
`(US)
`
`Correspondence Address:
`FISH & RICHARDSON, PC
`P.O. BOX 1022
`MINNEAPOLIS, MN 55440-1022 (US)
`
`(21)
`
`Appl. No.:
`
`10/519,818
`
`(22)
`
`PCT Fed:
`
`Jul. 1, 2003
`
`(86)
`
`PCT No.:
`
`PCT/USO3/20922
`
`100
`
`Related U.S. Application Data
`Provisional application No. 60/393,480, filed on Jul.
`1, 2002.
`
`Publication Classification
`
`Int. C.
`(2006.01)
`H04N 5/225
`(2006.01)
`H04N 700
`(2006.01)
`G06K 9/00
`U.S. Cl. ............................. 348/169; 348/36; 382/118
`
`(60)
`
`(51)
`
`(52)
`
`(57)
`ABSTRACT
`Digital video imaging systems and techniques for efficiently
`transforming warped video images into rectilinear video
`images, real-time tracking of persons and objects, face
`recognition of persons, monitoring and tracking head pose
`of a person and associated perspective view of the person.
`
`110
`
`Video
`Camera
`
`
`
`120
`
`Video processing device having a
`digital processor
`programmed to process warped wide
`angle or panoramic video images
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 001
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 1 of 21
`
`US 2006/0187305 A1
`
`120
`
`Video processing device having a
`digital processor
`programmed to process warped wide
`angle or panoramic video images
`
`
`
`FIG. 2
`
`200 N
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Video
`Transmission
`Mechanism
`
`121
`
`122
`
`
`
`
`
`Video
`Receiver
`(Client 1)
`
`Video
`Receiver
`Client 2
`
`
`
`
`
`
`
`Video
`Receiver
`Client N
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 002
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 2 of 21
`
`US 2006/0187305 A1
`
`FIG. 3
`
`312
`
`331
`
`Video
`Camera
`
`
`
`Scene Of
`a CIe Or
`accident
`
`301
`
`Video
`Camera
`N-
`311
`
`320
`
`Wired or/and
`Wils
`T
`Ideo
`ransmission
`links
`
`300
`
`
`
`
`
`Multiple Simultaneous
`Customized
`Perspectives for each
`client
`
`332
`
`333
`
`334
`
`Video
`Receiver
`Police
`
`Video
`Receiver
`Fire Dept.
`
`Video
`Receiver
`HAZMAT
`
`Video Receiver-Mobile
`Units,
`(e.g., ambulance, fire
`engines)
`
`335
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 003
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 3 of 21
`
`US 2006/0187305 A1
`
`FIG. 4
`
`
`
`400
`
`110
`
`Server
`
`Omni-video
`
`
`
`(Private link)
`410
`
`420 Blocks
`restricted
`
`Tail O
`ed ideo
`(Encrypted internet)
`
`Client in
`
`FIG. 5
`
`
`
`Omnidirectional Camera
`(ODVS)
`
`Omnidirectional Video
`
`New Pan-Tilt-Zoom
`Setting
`
`
`
`Perspective
`Transformafio
`
`Interpolation
`(Rilinear Y
`
`
`
`
`
`High-Pass Filter
`( Jnsharn
`
`Perspective Video
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 004
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 4 of 21
`
`US 2006/0187305 A1
`
`F.G. 6
`
`A
`
`(1-0)
`
`E
`
`O
`
`B
`
`C
`
`(1-0)
`
`F
`
`O
`
`D
`
`FIG. 7
`
`700
`
`Z.
`
`Hyperboloidal
`Mirror
`
`Obiect
`jec
`(Ro, Zo)
`
`
`
`Hyperboloid Equation:
`
`720
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 005
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 5 of 21
`
`US 2006/0187305 A1
`
`FIG. 8
`
`FIG. 9A
`
`FIG. 9B
`
`
`
`Perspective Wiew
`
`Bilinear interpolation + HPF
`
`st
`
`Rai H=
`7T 27 2.
`.
`.
`" ".
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 006
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 6 of 21
`
`US 2006/0187305 A1
`
`FIG. 9C
`
`FIG. 9D
`
`
`
`
`
`
`
`PTZ(1)
`
`PTZ(2)
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 007
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 7 of 21
`
`US 2006/0187305 A1
`
`FIG 11
`
`
`
`Perspective
`Transformation
`on Driver's Seat
`
`r
`
`Head Pattern
`Recognition
`
`Update Kalman
`Filter for Head
`Tracking
`
`Predict Head
`Location in Next
`Frame
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 008
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 8 of 21
`
`US 2006/0187305 A1
`
`F.G. 12A
`Eyebrow, Eye,
`and Mouth
`
`Head Detection -D
`& Tracking
`
`
`
`
`
`Face
`Orientation
`
`yaw,
`pitch, roll
`
`Omnicamera
`
`Parameter Template
`(yaw, pitch, roll) T(ellipse,
`eyebrow, eye, mouth)
`
`Head Tilting
`Compensation
`
`FIG. 12B
`
`
`
`
`
`'Head
`Detection &
`Tracking
`
`Face Orientation
`Likelihood Fns,
`
`0 degree
`
`
`
`FIG. 13
`180 degree
`
`Viewing
`Direction
`S
`
`Omnicamera
`
`360 degree
`
`Direction
`of driver
`
`Direction
`of car
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 009
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 9 of 21
`
`US 2006/0187305 A1
`
`Source omni-video
`
`FIG. 14
`
`
`
`
`
`
`
`FIG. 15
`
`Head
`Detection &
`Tracking
`
`Face
`Orientation
`
`Driver's
`View
`Generation
`
`Head
`Detection &
`
`Face
`Orientation
`Estimation
`
`Driver's
`View
`Generation
`
`Head
`Detection &
`Tracking
`
`;?
`
`-
`
`Face
`Orientation
`Estimation
`
`Driver's
`View
`Generation
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0010
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 10 of 21
`
`US 2006/0187305 A1
`
`
`
`Head Tilting
`Compensation
`
`Head
`Detection &
`8
`Tracking
`
`
`
`FIG. 16
`
`/ YP/YP -----
`VNVN
`
`State
`
`Gaussian Likelihood
`Functions
`
`FIG. 17
`
`
`
`
`
`
`
`
`
`Face Video Stream
`Str. Stream of face images
`Stream Partitioning
`St. i-th segment sequence
`Single-Frame
`Subspace
`Feature Analysis
`
`Sequence of
`Classification Results
`
`Sequence of
`Feature Vectors
`
`
`
`
`
`
`
`
`
`
`
`
`MAJ
`Majority
`Decision
`Rule
`
`
`
`
`
`DMD
`DHMM ML
`Decision
`Rule
`
`CMD
`CDHMMML
`Decision
`Rule
`
`MAJ
`
`DMD
`
`(CMD
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0011
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 11 of 21
`
`US 2006/0187305 A1
`
`Camera
`
`Camera
`
`
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0012
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 12 of 21
`FIG. 20
`
`US 2006/0187305 A1
`
`9 5
`
`
`
`9 O
`
`8 5
`
`8O
`
`75
`O
`
`2O
`15
`1 O
`5
`Number of DHMM States
`
`25
`
`FIG. 21
`
`100
`90
`
`O)
`c
`
`SP
`CD
`
`80
`
`s
`D
`5 70
`O
`is 60
`ce C so
`1OO
`50
`O
`Number of Utilized Dimension in CDHMM
`
`CD
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0013
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 13 of 21
`
`US 2006/0187305 A1
`
`1 O O
`
`
`
`89 5O
`
`8 O
`1
`
`FIG. 22A
`
`4.
`3
`2
`Number of Gaussian Mixtures
`
`5
`
`FIG. 22B
`
`
`
`100
`O
`s
`as g 90
`
`CD
`l
`
`d
`CD
`
`8O
`
`70
`
`CD
`
`ce C 6oL
`O
`
`15
`1O
`5
`Number of CDHMM States
`
`2O
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0014
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 14 of 21
`
`US 2006/0187305 A1
`
`FIG. 23
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 24
`
`
`
`Graphical User Interface
`Person Tracks, Face Tracking, Person IDs, Gestures,
`Events, ... etc.
`Data Archiving/Communication
`
`3D Trackin
`Person Location,
`Bounding Volume,
`Velocity
`
`
`
`
`
`Visual Information Analysis
`Head and Face Tracking,
`Face Recognition,
`Posture/Gesture Recognition,
`Posture/Movement Analysis,
`Event Detection, ... etc.
`
`Active Camera Control
`Array Selection, Camera Selection,
`PTZ Control (Mechanical/Digital)
`
`
`
`Real-Time Visual Information Capture
`
`
`
`Video Arrays
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0015
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 15 of 21
`
`US 2006/0187305 A1
`
`Panoramic Video Camera
`Array
`
`Nr. 1
`
`Sc
`
`FIG. 25
`
`Nc
`
`
`
`Shadow
`Detection &
`Segmentation
`
`
`
`Shadow
`Detection &
`Segmentation
`
`Shadow
`Detection &
`Segmentation
`
`Shadow
`Detection &
`Segmentation
`
`Panorama
`
`Object Profiles
`Azimuth
`Angles
`
`
`
`N-ocular
`Stereo
`x-y Measurements
`Data
`Association
`Associated X-
`y
`Measurements
`
`
`
`
`
`Object Blobs
`Topmost
`Pixels
`
`Height
`Estimation
`Z Measurements
`Weighted
`Averaging
`Averaged Z
`Measurements
`
`Unassociated
`Measurements /
`Empty Tracks
`
`Tracks
`
`Track Candidates
`
`
`
`Track
`Initialization /
`Termination
`Tracks
`
`Moving Track
`Filtering
`
`Output Tracks
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0016
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 16 of 21
`
`US 2006/0187305 A1
`
`FIG. 26
`
`FIG. 27
`
`
`
`In-Car Environment
`in-Car Camera Array
`
`Driver Related
`
`Driver Authorized
`Wiewing Mode
`
`Video Mode
`Selection &
`Streaming
`
`Traffic Related
`C ontext Capture
`& Analysis
`
`Telepresence Interface of Remote Kiewer
`-
`. . Video
`Mode
`
`* .
`
`is
`
`:...
`
`Avatar
`Mode
`
`Cartoon
`Mode
`
`t
`
`...)
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0017
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 17 of 21
`
`US 2006/0187305 A1
`
`FIG. 28
`
`&
`Single-frame face
`analysis & detection
`
`Streaming face
`decision &
`recognition
`
`Edges and
`Contours
`
`Face
`Template
`
`
`
`
`
`
`
`Video
`
`Skin Tone
`Regions &
`Templates
`
`
`
`
`
`
`
`
`
`Multi-Res.
`Window
`Scanning
`
`
`
`ID
`
`Spatial
`Temporal
`Fusion:
`
`HMM /
`Bayesian
`Network
`--
`Classifier
`
`Features relate to:
`eyes, eyebrows,
`mouth, nose,
`cheeks, hairline,
`shoulders
`
`
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0018
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 18 of 21
`
`US 2006/0187305 A1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`6uppe11
`
`p = = = = = = = = = | seleu?eo ;
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0019
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 19 of 21
`
`US 2006/0187305 A1
`
`F.G. 30
`
`Image Compression
`(Motion JPEG)
`
`Stream Video
`to Network
`
`ODWS and Rectilinear
`Video Capture
`
`PTZ camera control
`(one user at a tine)
`
`omni-directional
`camera
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`internet
`
`Multiple Simultaneous Customized
`Perspectives for Each Client
`
`Police emergency Dispatcher
`
`Image Decompressio
`tmage Enhancements
`
`Perspective View
`Transformation
`
`Emergency Medical Services
`Image Decompressor Perspective view
`tmage Enhancements
`Transformation
`
`WIDEO
`
`On Location at Highway
`
`S-N-
`
`At Any Client's Site
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0020
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 20 of 21
`
`US 2006/0187305 A1
`
`FIG. 31
`
`Video
`Initialization
`
`Camera Choice
`Camera Settings
`
`Image Acquisition
`
`Frame
`Decompression
`
`Display Cycle
`Syncronization
`
`Insert live image
`into Visualization
`
`Registration
`Settings
`
`FIG. 32A
`
`FIG. 32B
`
`FIG. 32C
`
`
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0021
`
`

`

`Patent Application Publication Aug. 24, 2006 Sheet 21 of 21
`
`US 2006/0187305 A1
`
`FIG. 33
`
`Real World Image
`Ry)
`
`a camera Transfer
`
`CCDra
`
`Transformed Image
`B*R(x,y)
`
`Median
`Background
`Differencer
`
`vehicle
`Couting
`Module
`
`Noise Filtering
`(Morphological
`Operations)
`
`Forground Mask
`(Binary.Inage).
`F(x,y)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0022
`
`

`

`US 2006/01 87305 A1
`
`Aug. 24, 2006
`
`DIGITAL PROCESSING OF VIDEO MAGES
`0001) This application claims the benefit of U.S. Provi
`sional Application No. 60/393,480 entitled “Digital Tele
`viewer' and filed Jul. 1, 2002.
`
`BACKGROUND
`0002 This application relates to digital imaging and
`machine vision, and in particular, to digital processing of
`digital video images and related applications.
`0003 Video cameras have been used to capture video
`images for various applications such as Surveillance,
`machine vision, security monitoring, inspection, sensing and
`detection. In these and other applications, the captured video
`images may be sent to a nearby or remote image processing
`device to be processed and displayed. Digital image pro
`cessing techniques may be used to process the video images
`to extract information from the video images.
`0004 Certain video cameras are designed with optical
`imaging modules to capture wide-angle or panoramic video
`images. Such video images are distorted due to the designs
`of various optical imaging modules. Digital imaging pro
`cessing may be used to transform the distorted images into
`images that a viewer would normally perceive when directly
`look at the scene being captured.
`
`SUMMARY
`0005. This application includes, among other features,
`implementations of video systems and digital processing
`techniques for delivering wide-angle or panoramic video
`signals to one or more video processing devices and digitally
`processing the panoramic video signals at each video pro
`cessing device for various applications. The video signals
`may be transmitted to one or more receivers through a
`secured server that places certain restrictions on the video
`signals that are transmitted to the receivers. For example,
`certain information in the originally captured video signal
`may be removed to produce a “redacted video signal to a
`selected receiver to provide a limited access. The digital
`processing techniques include, for example, methods to
`efficiently transform warped video images into rectilinear
`Video images, real-time tracking of persons and objects, face
`recognition of persons, monitoring and tracking head pose
`of a person and associated perspective view of the person.
`Systems that include one or more of these and other features
`may be devised for various applications.
`0006.
`In one implementation, the digital processing
`includes a digital tele-viewer module comprising a perspec
`tive transformation part that computes unwarped image
`pixel positions for a set of input pan, tilt, and Zoom param
`eters of the corresponding video camera. A look-up-table is
`included to provide correspondence between image coordi
`nates on the warped image and unwarped image pixel
`positions for a given set of input pan, tilt, and Zoom
`parameters so that the unwarped image may be formed from
`the image pixels taken from input warped video image from
`the video camera. In another implementation, the digital
`processing may include a module for detecting a person’s
`head and determining and tracking the person’s face orien
`tation based on edge detection, ellipse detection, head pat
`tern recognition, and the Kalman filtering for head tracking.
`This module allows for extracting the person’s view when a
`
`panoramic video is taken. Furthermore, the digital process
`ing may include Video-based face recognition to identify
`persons in the captured video against a face image database.
`0007. A technique for 3D real-time tracking of persons is
`also described to use multiple panoramic video camera
`arrays. This technique may be combined with the digital
`tele-viewer module, the face recognition module, and the
`module for detecting a person's head and determining and
`tracking the person’s face orientation in an intelligent room
`system.
`0008. These and other implementation, features, and
`associated applications are described in detail in the follow
`ing drawings, the detailed description, and the claims.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`0009 FIG. 1 shows a general system configuration
`according to one implementation of a digital video system.
`0010 FIGS. 2, 3, and 4 show specific examples of digital
`Video systems.
`0011 FIG. 5 shows one exemplary implementation of
`digital image transformation from a warped video to a
`rectilinear video based on a look-up table.
`0012 FIG. 6 shows one example of image interpolation
`used in FIG. 5.
`0013 FIGS. 7 and 8 show imaging transformation for an
`omnidirectional video camera having a hyperboloidal reflec
`tOr.
`0014 FIGS. 9A and 9B show warped omnidirectional
`video image of the camera in FIG. 7 and an user interface
`for adjusting pan, tilt, and Zoom of a customized view.
`0.015 FIGS. 9C, 9D, 9E, 9F, and 10 show exemplary
`customized images from the digital tele-viewer module for
`the camera in FIG. 7.
`0016 FIG. 11 shows one exemplary implementation of
`detection and tracking of head pose and a person’s view.
`0017 FIGS. 12A and 12B show two different processing
`methods for determining a person’s face orientation and for
`generating the person’s view.
`0018 FIG. 13 illustrates the relative orientation of the
`omnidirectional camera in automobile video system based
`on the system in FIG. 11.
`0.019
`FIGS. 14 and 15 show additional examples of
`estimating face orientation and generating the person's view.
`0020 FIG. 16 illustrates a face orientation estimation by
`the continuous density hidden Markov Model (HMM).
`0021
`FIG. 17 shows one implementation of the video
`based face recognition.
`0022 FIG. 18 shows a warped video image from one
`CaCa.
`0023 FIG. 19A shows six examples of the face images
`in the training and testing video streams which are perspec
`tive views generated from the omni videos.
`0024 FIG. 19B shows face images that were automati
`cally extracted by a testbed system.
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0023
`
`

`

`US 2006/01 87305 A1
`
`Aug. 24, 2006
`
`0025 FIGS. 20, 21, 22, and 23 show results of face
`recognition based on the implementation in FIG. 17.
`FIG. 24 shows one example of an intelligent room
`0026
`system.
`0027 FIG. 25 show one example of a 3D tracking
`system.
`0028 FIG. 26 illustrates the layout and exemplary video
`images of the ODVS array within a test room.
`0029 FIG. 27 shows one example of a driver assistance
`system based on the digital processing in FIG. 11.
`0030 FIG. 28 illustrates a processing module to provide
`streaming face detection and correlation.
`0031
`FIG. 29 shows a system for analyzing multimodal
`human behavior, stress and intent pattern.
`0032 FIG. 30 shows one exemplary system where both
`high-resolution rectilinear video cameras and low-resolution
`omnidirectional video cameras are deployed in a target area
`to further expand the flexibility of customizable viewing of
`the target area by multiple clients.
`0033 FIG. 31 shows one implementation for overlaying
`the live DTV Video over a digital image.
`0034 FIGS. 32A, 32B, and 32C illustrate examples of
`overlaying a live video over a digital map.
`0035 FIG. 33 shows a method for counting vehicles in
`a live video.
`0.036
`FIG. 34 shows cameras deployed over a wide area
`with little or no overlapping between the camera overages
`for monitoring traffic.
`
`DETAILED DESCRIPTION
`0037. A video system may use a video camera and a
`Video processing device that are spatially separate from each
`other so that the video captured by the video camera at one
`location may be processed and viewed through the video
`processing device at a different location. This video system
`allows for remote sensing and may be used in various
`applications.
`0038. Such a video system may be used in situations
`where it is desirable or necessary that an operator or user of
`the video system is absent from the location of the video
`camera. For example, in security and anti-terrorism video
`systems, a network of video cameras may be installed in
`critical locations such as airports, bus and train stations,
`military bases, etc. The video signals can be remotely
`processed and used by various state and federal authorities.
`As another example, such a video system may be installed
`in vehicles to assist pilots, drivers, and security personnel to
`monitor the passenger cabin and luggage cabin. Such video
`systems may also be installed at critical places to help
`security personnel monitor critical sites for any unusual
`situations, including sites where humans are impossible to
`enter Such as nuclear reactors, areas exposed to toxic agents,
`and other hazardous areas. Furthermore, such video systems
`may be deployed on a crisis site to assist police, fire
`department, physicians, and the crisis management com
`mander to monitor the situations of their responsibility, and
`when permissible, to inform relatives of victims of the most
`up-to-date rescue progress in real-time without interfering
`
`the rescue actions. In visual Surveillance. Such video systems
`allows a site of interest to be viewed by remote users like the
`host or the police department through a proper communi
`cation link Such as the Internet or other computer networks
`at any time and simultaneously.
`0039) Other applications may be possible. For example,
`one or more video cameras may be installed in a vehicle to
`monitor the driver's head pose and face orientation as a part
`of a safety alert System, e.g., to warn the driver when the
`driver's direction of view is away from the front direction of
`the vehicle beyond a permissible period during driving. The
`Video processing device may be located at a different loca
`tion in the vehicle, e.g., as part of the on-board computer
`system of the vehicle.
`0040 FIG. 1 illustrates one example of a video system
`100 where a video camera 110 is installed at a location 101
`to capture a video of the location 101. In general, the video
`camera 110 may have an optical imaging system to capture
`either the wide-angle or the panoramic view of the location
`110. As one exemplary implementation of a panoramic
`Video camera, the video camera 110 may be an omni
`direction video camera to capture a full 360-degree view of
`the location 101 surrounding the camera 110 in an inside
`out-coverage configuration. A video processing device 120
`is located at a different location and is linked to the video
`camera 110 by a communication link 130 to receive the
`video signal. The communication link 130 may be a wired
`link, a wireless link, or a combination of both. In some
`applications, the communication link 130 may include one
`or more communication networks Such as the Internet to
`deliver the video signal from the video camera 110 to the
`Video processing device 120. The communication link may
`use, among other links, a wired link for high bandwidth or
`a wireless link such as a wireless 802.11 protocol for high
`mobility.
`0041. The video processing device 120 includes a digital
`processor that is programmed to process the warped wide
`angle or panoramic video images to extract desired infor
`mation. The video processing device 120 may be a desktop
`computer, a portable electronic device such as a PDA or a
`cell phone. The digital processing modules such as the DTV
`module described below may be designed to operate on
`multiple platforms: workstations, desktop computers, laptop
`computers, TabletPCs, PDAs, etc. The DTV module, for
`example, may use the Java implementation which utilizes
`Java Virtual Machine on various platforms and various
`operating systems.
`0042.
`In one implementation, the digital processor may
`be programmed, among other features and functions, to
`transform the warped video images into rectilinear video
`images and allow the user to digitally control the pan, tilt,
`and Zoom of the video to customize the view. This part of the
`digital processor is referred to as the “digital tele-viewer
`(DTV) in part because it enables remote customized viewing
`of the video images. Notably, different users may view the
`same video stream simultaneously with different customized
`viewing settings, such as different pan, tilt, and Zoom
`parameters. Each user may customize the viewing settings
`without interfering customized viewing of the same view
`stream by other users.
`0043 FIG. 2 depicts an exemplary video system 200 that
`includes two or more video processing devices 121 and 122
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0024
`
`

`

`US 2006/01 87305 A1
`
`Aug. 24, 2006
`
`connected to receive the same video signal from the video
`camera 110. A video transmission mechanism 210 is used to
`simultaneously deliver the video signal from the video
`camera 110 to the video processing devices 121, 122, etc.
`The video signal being delivered may be in a digitally
`compressed form. The digital processors in the devices 121
`and 122 uncompress the video signal and allow for different
`customized views by two different users from the same
`Video signal. In operation, the digital tele-viewer takes a
`remote omni-camera video and unwarps it into rectilinear
`Videos of any pan-tilt-Zoom settings. The same video stream
`may be shared among multiple people and each person can
`generate a perspective view of the person's interest. The
`multi-user digital tele-viewing is generally difficult, if not
`possible, by using mechanical pan-tilt-Zoom (PTZ) cameras
`because most mechanical PTZ cameras can be used by only
`one user to look at one point at any time instance. When an
`omni-camera is used to capture the video images with a
`360-degree view, digital tele-viewer may be used to generate
`various different PTZ rectilinear views for all users who
`want to look at different points from the same omnicam.
`0044) Certainly, two or more video cameras may be
`connected to the video transmission mechanism 210 in the
`system 200 to deliver different video signals to the devices
`121, 122, etc. FIG. 3 shows an exemplary video system 300
`for monitoring a scene of a crime or accident 301 by two or
`more video cameras (311, 312, etc.) based on the multi-user
`video system 200 in FIG. 2. The video signals from the
`video cameras 311, 312, etc. are sent to multiple video
`receivers via video transmission links 320 which may be
`wired or wireless channels. The video receiver 331 may be
`located in a dispatch center to process the video signals to
`produce multiple simultaneous customized perspective
`views of the scene 301. Based on the information from the
`Video signals, the dispatch center may send commands to
`dispatch appropriate units to the scene 301, e.g., police 332,
`fire department 333, hazardous material control units
`(HAZMAT) 334, or mobile units 335 such as ambulance or
`fire engines. The police, the fire department, and other units
`may also have video receivers to directly receive live video
`signals from the cameras 311, 312 and to monitor the scene
`310 via their customized views. Therefore, with the aid of
`this video system 300, different units from the dispatch
`center to the rescue agents can obtain valuable live visual
`Video images of the scene 310 and thus cooperate in a highly
`aware manner. For example, with the aid of the real-time
`DTV at each video receiver, appropriate rescue actions can
`be prepared before arriving the crisis site, and lives can be
`saved by shorten delays. Moreover, even relatives of victims
`can know the most up-to-date rescue progress in real-time
`without interfering the rescue actions.
`0045. It is recognized that, however, the scope of access
`to information in the video signal by different users may be
`different in certain applications. In this regard, a video server
`may be connected in the communication links between the
`video camera and the remote client users to filter or edit the
`video signal to produce different “redacted” or “tailored
`versions of the original video signals with different contents.
`For example, for a selected client user, the video server may
`remove video images for certain scenes, e.g., a view within
`a selected arc angle of the 360-degree view, and deliver the
`Video images of the remaining scenes so that the selected
`client user has no access to the removed video images. Such
`filtering or editing may be predetermined or may change
`
`dynamically with time at the video server. This video server
`in general may be a public server or a secured server.
`0046 FIG. 4 illustrates a secured video system 400 that
`includes a secured video server 420 and a secured transmis
`sion link between the video camera 110 and the server 420.
`To ensure the secured delivery of video signals, different
`clients 431, 432, and 433 may need to log in to the server
`420 in order to receive video signals originated from one or
`more video cameras 110. The server 420 may be pro
`grammed to store the client profiles that include client data
`on scope of access. Based on this client data, the server 420
`blocks out restricted areas in the video signals for a particu
`lar client and delivers the tailored video signals. In addition,
`the transmission between the server 420 and the clients may
`use either secured transmission channels or other commu
`nication channels such as the Internet with data encryption
`to secure the transmission.
`0047. In certain implementations, the secured server 420
`in the system 400 may be configured to permit various
`security levels for the clients. For example, a high level
`client may be allowed to have unrestricted video, while a
`lower level client may be restricted to receive some part of
`the video with certain views in the video being blocked by
`the secured server after editing. The scope of the restricted
`part in an edited video may be dynamically adjusted so that
`the blockage of the video for certain clients changes over
`time. This change in blockage may be based on a change of
`a user's level of security or a change in the scene captured
`in the video. As an example of the latter, the video of an
`aircraft carrier parked in a military harbor may be blocked
`to the public but open to navy officers. As the aircraft carrier
`moves, the blockage moves with it. The same scenario may
`apply to airplanes, vehicles, and persons. Techniques to
`implement this time-varying video blockage involve detec
`tion and tracking of motion of an object or person as
`described in this application. For high security, the video
`streams from the server can be encrypted.
`0048. The following sections describe exemplary digital
`processing mechanisms and functions in the digital proces
`sor in each video processing device for a client or user. In
`general, the digital processor may be implemented by using
`a general computer, Such as a computer with a micropro
`cessor. The digital processing mechanisms and functions
`may be implemented with software modules that are stored
`in one or more machine-readable memory devices and can
`be executed by the computer.
`0049. One basic component of the digital processing is
`the digital tele-viewer (DTV) that unwarps the warped
`wide-angle or panoramic video signals received by the
`digital processor into rectilinear videos of any pan, tilt, and
`Zoon settings. As a result, a client may choose any perspec
`tive available in the original video signal and different
`clients may simultaneously choose different perspective
`views, entirely independently from one another without
`affecting another client's viewing, in the same video signal
`from a video camera.
`0050. The video camera may include an optical imaging
`module that captures the wide-angle or panoramic view of a
`scene, and an array of photosensors such as CCDs or other
`Suitable sensors to receive and convert optical images from
`the optical imaging module into electronic signals. Due to
`the nature of the wide-angle or panoramic imaging, the
`
`Google Exhibit 1009 - Google v. CSI
`IPR2025-00877 - Page 0025
`
`

`

`US 2006/01 87305 A1
`
`Aug. 24, 2006
`
`optical imaging module generally warps or distorts the video
`images. The array of photosensors may be a planar 2D array
`and thus the optical images projected on the array are
`warped. For a given optical configuration of the optical
`imaging module, a warped image at the 2D array may be
`mathematically transformed into a rectilinear perspective
`image. This perspective transformation may be implemented
`in the DTV Software.
`0051 FIG. 5 shows one implementation of the DTV
`perspective view generation. Upon initialization and each
`request of new pan-tilt-Zoom (PTZ) settings for the perspec
`tive view, the PTZ values are sent to the perspective trans
`formation module to unwarp a portion of the warped wide
`angle or panoramic image into a perspective view by
`updating a look-up-table. The look-up-table includes data
`that directs to the corresponding image coordinates on the
`warped image for each pixel of the unwarped perspective
`image. This use of the look-up-table speeds up the process
`because the values of the look-up-table need to be computed
`only once if the PTZ value is not changed. With the values
`of the look-up-table for a given set of PTZ values, the
`unwarped perspective image can be formed by filling the
`pixels with the corresponding pixels in the warped image. In
`general, the corresponding warped image coordinates may
`not be integers, i.e., they may be located between the
`adjacent warped image pixels, an image interpolation may
`be used to compute the unwarped image pixels.
`0.052
`In one implementation, the interpolation may be a
`bilinear interpolation. FIG. 6 illustrates the operation of this
`method. For given pixel values A, B, C, and D at integer
`coordinates of adjacent pixels, the non-integer coordinate
`pixel G to be interpolated can be calculated by using the
`following equations:
`
`wh