(12) United States Patent
`Garcia
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9,696,897 B2
`Jul. 4, 2017
`
`US0096.96897B2
`
`(58) Field of Classification Search
`CPC ..................................................... GO6F 3/0487
`USPC ...
`.
`.
`. . . . .
`. . . . .
`348777
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`(54) IMAGE-BASED MEASUREMENT TOOLS
`(71) Appli
`The R
`ts of the Uni
`ity of
`pplicant: The Regents of the University o
`California, Oakland, CA (US)
`
`(72) Inventor: Maurice Garcia, San Francisco, CA
`US (US)
`(73) Assignee: The Regents of the University of
`California, Oakland, CA (US)
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 353 days.
`
`(21) Appl. No.:
`
`14/352,943
`
`(22) PCT Filed:
`
`Oct. 19, 2012
`
`(86). PCT No.:
`S 371 (c)(1),
`(2) Date:
`
`PCT/US2O12/061032
`
`Apr. 18, 2014
`
`(87) PCT Pub. No.: WO2013/059599
`PCT Pub. Date: Apr. 25, 2013
`
`(65)
`
`Prior Publication Data
`US 2014/O3OO722 A1
`Oct. 9, 2014
`Related U.S. Application Data
`(60) Provisional application No. 61/548,780, filed on Oct.
`19, 2011.
`
`(2006.01)
`(2013.01)
`(2006.01)
`(2013.01)
`(2017.01)
`
`(51) Int. Cl.
`H04N 7/8
`G06F 3/0487
`G0IB II/02
`G06F 3/0482
`G06T 7/62
`(52) U.S. Cl.
`CPC ............ G06F 3/0487 (2013.01); G0IB II/02
`(2013.01); G0IB II/022 (2013.01); G06F
`3/0482 (2013.01); G06T 7/62 (2017.01); G06T
`2207/10004 (2013.01)
`
`412
`
`
`
`U.S. PATENT DOCUMENTS
`2004/00 1687.0 A1
`1/2004 Pawlicki et al.
`2010.0020221 A1* 1/2010 RON e .
`.
`.
`.
`.
`.
`.
`
`.
`
`.
`
`. G06F 3/04883
`348,333.01
`2011/0164128 A1* 7/2011 Burgett ................ A61B 5, 1076
`348,77
`
`.
`
`FOREIGN PATENT DOCUMENTS
`
`JP
`KR
`KR
`
`2008-211409 A
`10-2010-0033634 A
`10-1032466 B1
`
`9, 2008
`3, 2010
`5, 2011
`
`OTHER PUBLICATIONS
`
`RulerPhone, retrieved from the internet, www.trytap.com, printed
`on Apr. 9, 2014.
`
`(Continued)
`
`Primary Examiner — Richard Torrente
`
`(57)
`
`ABSTRACT
`
`Provided are methods, systems, devices, and computer pro
`grams enabling the measurement of various objects using
`imaging. The inventions are adaptable to mobile devices
`Such as Smartphones, allowing a range of unique measure
`ments to be conveniently made, including length, area,
`distance, Velocity, and height. The methods, systems,
`devices, and computer programs include the novel applica
`tion of image-based measurement in medical, research, and
`other contexts.
`
`15 Claims, 9 Drawing Sheets
`
`Petitioner's Exhibit 1012
`Page 1 of 35
`
`

`

`US 9,696.897 B2
`Page 2
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Baker, et al., “Shape-from-silhouette of articulated objects and its
`use for human body kinematics estimation and motion capture.”
`Computer Vision and Pattern Recognition, Proceedings 2003 IEEE
`Computer Society Conference, Jun. 2003, vol. 1, pp. 77-84. Whole
`document.
`
`* cited by examiner
`
`Petitioner's Exhibit 1012
`Page 2 of 35
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 1 of 9
`Sheet 1 of 9
`
`US 9,696,897 B2
`US 9,696,897 B2
`
`FG. A
`FIG. 1A
`
`FIG. 1B
`FIG. 1B
`
`Petitioner's Exhibit 1012
`Page 3 of 35
`
`Petitioner's Exhibit 1012
`Page 3 of 35
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 2 of 9
`
`US 9,696,897 B2
`
`2O6
`
`{ - -D
`
`205
`
`FIG. 2A
`
`
`
`21 O
`
`2O1
`
`
`
`2O2
`
`Petitioner's Exhibit 1012
`Page 4 of 35
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 3 of 9
`Sheet 3 of 9
`
`US 9,696,897 B2
`US 9,696,897 B2
`
`.
`FIG.3C
`
`
`
`FIG.3B
`
`S
`503
`
`as
`-e
`
`S
`302
`
`FIG.3A
`
`Petitioner's Exhibit 1012
`Page 5 of 35
`
`Petitioner's Exhibit 1012
`Page 5 of 35
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 4 of 9
`Sheet 4 of 9
`
`US 9,696,897 B2
`US 9,696,897 B2
`
`
`
`3.
`
`FIG.AC
`
`FIG.4B
`
`s
`
`
`
`
`
`Petitioner's Exhibit 1012
`Page 6 of 35
`
`Petitioner's Exhibit 1012
`Page 6 of 35
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 5 of 9
`Sheet 5 Of 9
`
`US 9,696,897 B2
`US 9,696,897 B2
`
`
`
`FIG,4EFIG.4F
`
`FIG.AD
`
`:
`
`Petitioner's Exhibit 1012
`Page 7 of 35
`
`Petitioner's Exhibit 1012
`Page 7 of 35
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 6 of 9
`
`US 9,696,897 B2
`
`5O1
`
`TURN ON LEVEL
`HARDWARE AND
`SOFTWARE
`
`
`
`
`
`INPUT
`DES RED
`ORIENTATION
`
`OVERLAY GUIDES
`
`502
`
`
`
`
`
`DISPLAY
`CAMERA
`PREVIEW
`
`505
`
`
`
`
`
`
`
`
`
`S CAMERA N
`PROPER
`ORIENTATION?
`
`
`
`
`
`
`
`504
`
`503
`
`DISPLAY
`ORIENTATION
`STATUS/
`CORRECTION
`FACTORS
`
`
`
`USER
`CORRECTS
`ORIENTATION
`
`509
`
`510
`
`508
`
`FIRE CAMERA
`
`
`
`513
`
`USER
`COMMANDS
`CAMERATO
`FIRE
`
`
`
`
`
`
`
`512
`
`
`
`
`
`
`
`
`
`DISPLAY
`MAGE ON
`SCREEN
`
`514
`
`500
`
`515
`FIG. 5
`
`DISPLAY
`OK TO FIRE
`
`511
`
`coin ND
`INPUT BY
`USER
`
`STORE
`MAGE
`FILE N
`MEMORY
`
`517
`
`516
`
`END
`
`Petitioner's Exhibit 1012
`Page 8 of 35
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 7 Of 9
`
`US 9,696,897 B2
`
`601
`
`STAR
`
`REREVE MAGE
`
`
`
`DISPLAY
`MAGE
`REME
`SELECT
`SANDARD
`
`603
`
`64
`
`
`
`DSPAY
`MENU OF
`STANDARDS
`
`sis
`ESRED
`SANDARD
`
`SO5
`
`SSANDARD
`NCDED EN
`DAABASEP
`
`606
`
`
`
`
`
`RETREVE
`MENSONS OF
`STANDARD FROM
`DAAEASE
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`DSPLAY SANDAR
`CON PROPMPT SER
`TO POSION AND SIZE
`CON OVER STANDARD
`DEPECTED N MAGE
`
`
`
`
`
`
`
`
`
`USER
`POSIONS
`AND SES
`CON
`
`MEASURE
`SANDARO'S
`SZEN PXES
`
`CACUAE
`SEZE PER
`PXEN MAGE
`
`67
`
`ROPMPT
`SERO
`DELNEAE
`
`MEASRED
`
`
`
`
`
`
`
`68
`
`69
`
`SER
`OENEAES
`EM
`
`MEASRE
`OENEAED
`EMS SIZEN
`PXES
`
`32O
`
`CA CAE SIZE OF
`TEM BASED ON
`PREVIOUSLY
`CALCUATED SIZE
`OF PEXE
`
`FIG. 6
`
`600 /
`
`6
`
`PROMP
`USERO
`DELNEATE
`STANDARD
`
`SER
`DELNEATES
`SANDARD
`
`WEASRE
`OEN EATED
`STANDARDS
`SZEN PXELS
`
`PROPM USERO
`NAN/E SANDARD
`AND ENP
`STANDARDS
`DVENSONS
`
`612
`
`S13
`
`64
`
`615
`
`SERNS
`STANARS
`MENSONS
`
`
`
`66
`
`NAVE AND
`DVSENSIONS
`N MEMORY
`
`621
`
`622
`
`
`
`
`
`
`
`
`
`DSPAY
`CACAED
`SZES
`
`
`
`STORE
`CACAE
`S2E IN
`MEMORY
`
`Petitioner's Exhibit 1012
`Page 9 of 35
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 8 of 9
`
`US 9,696,897 B2
`
`
`
`O.4
`
`O.35
`
`O.3
`
`O.25
`
`O.2
`
`O.5
`
`O.
`
`O
`
`2O
`
`AO
`
`6O
`
`8O
`
`OO
`
`DISTANCE FROM REFERENCE FIGURE (fi)
`
`FIG. 7
`
`O
`
`C
`
`O
`
`3O
`
`4O
`
`5O
`
`60
`
`7O
`
`DISTANCE FROM CAMERA (ft)
`
`FIG. 8
`
`Petitioner's Exhibit 1012
`Page 10 of 35
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 9 Of 9
`
`US 9,696,897 B2
`
`Es-CS
`
`N92
`
`c
`s
`
`s
`
`N
`M
`V
`M
`V
`V
`V
`V
`N
`V
`V
`V
`
`-
`is
`
`s
`ce 1N
`
`?
`/
`d W
`A
`A.
`A.
`f
`A.
`
`M
`/
`
`V
`VA
`
`is
`
`se
`o
`
`V
`N
`Y-
`V
`cs
`V
`V old
`N
`V
`V
`V
`V
`
`S
`
`W
`W
`/
`A.
`W
`A.
`?
`A
`A.
`W
`A.
`M
`
`s
`
`c
`
`On
`CD
`L
`
`I
`
`Petitioner's Exhibit 1012
`Page 11 of 35
`
`

`

`1.
`IMAGE-BASED MEASUREMENT TOOLS
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application is based on and claims the benefit of
`priority of U.S. Provisional Patent Application Ser. No.
`61/548,750, filed on Oct. 19, 2011, the contents of which are
`incorporated by reference.
`
`10
`
`STATEMENT REGARDING FEDERALLY
`SPONSORED RESEARCH OR DEVELOPMENT
`
`This invention was made with United States government
`support under Grant Number NIH NICHD K08 HD069462
`01 awarded by the National Institutes of Health. The United
`States government has certain rights in the invention.
`
`15
`
`REFERENCE TO SEQUENCE LISTING, A
`TABLE, OR A COMPUTER PROGRAM LISTING
`COMPACT DISKAPPENDIX
`
`US 9,696,897 B2
`
`2
`venient measurement of items, for the measurement of items
`that are remote, and for the fast measurement of multiple
`items.
`The methods disclosed herein are readily implemented, in
`whole or in part, on mobile devices, for example, Smart
`phones. The use of image-based measurement systems
`implemented on mobile devices to determine the dimensions
`of an object is known in the art. For example, the Ruler
`Phone application, by Ben Kamens (2008), provides a
`mobile-phone based tool for the measurement of one-di
`mensional lengths using images acquired by the user. The
`invention disclosed herein provides novel features and meth
`ods not included in the prior art measurement tools which
`improve upon the accuracy, convenience, versatility, and
`usability of the prior art methods.
`The invention disclosed herein provides methods,
`devices, and systems that address various measurement
`needs and which provide advantageous Solutions in a wide
`variety of specific contexts. Such as medicine, health and
`fitness, Scientific research, engineering, and even novelty
`SC.
`
`Not Applicable
`
`FIELD OF THE INVENTION
`
`The invention relates to the field of measurement, spe
`cifically, the use of novel image-based measurement tools to
`accurately determine the dimensions of objects that have
`been imaged, i.e., photographed, in a variety of contexts.
`The invention further includes methods for the determina
`tion of using image analysis to determine the distance of
`objects from a camera, as well the Velocity of a moving
`object. Certain embodiments of the invention are directed to
`the use of the disclosed methods on mobile devices, for
`example, Smartphones.
`
`25
`
`30
`
`35
`
`BACKGROUND OF THE INVENTION
`
`In medicine, Science, engineering, and just about every
`other aspect of human endeavor, there is the need to measure
`the size of items. The object of the invention disclosed
`herein is to provide new methods and devices for the facile
`and accurate measurement of length, width, height, area,
`Volume, curvature, and other dimensional features of various
`items, ranging from microscopic to enormous. The inven
`tions disclosed herein are further directed to the measure
`ment of distance, Velocity, and very tall heights. The meth
`ods, systems, software, and devices provided herein provide
`the art with novel solutions for a variety of measurement
`problems.
`In many situations, direct measurement of an object is not
`easily achieved. For example, the scale of an item can make
`its measurement difficult, as in the case of microscopic cells
`or tall trees. In other cases, accurate measurement is difficult
`to achieve, as in the case of irregularly shaped objects. Such
`as skin lesions or plant leaves. In some situations, such as
`remote medicine, the item to be assessed and the persons
`capable of making the assessments are separated by great
`distance. In yet other cases, there is the need for a large
`number of measurements to be taken and it would be
`advantageous to have a means of quickly collecting massive
`amounts of measurement data, for examples, as in agricul
`ture. Thus, there is a need in the art for solutions that solve
`these many problems. There is a need in the art for systems
`that allow for accurate measurement of items, for the con
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`SUMMARY OF THE INVENTION
`
`The basic function of the invention is to conveniently and
`accurately measure the size attributes of various items using
`imaging. The length, area, and angular features of an item
`will often have particular relevance in certain contexts. In
`various fields of biological research, determining the size
`(e.g. area, Volume) of cells, organs, or whole organisms is of
`great importance in assessing the health, growth rate, and
`other features of living things. In medicine, the size or other
`geometrical attributes of a feature are of importance in
`diagnosing various conditions. In other medical contexts,
`serial measurements of a wound, a lesion, or other feature
`can be used to assess progression, prognosis and treatment
`efficacy of a condition.
`The methods of the invention can be utilized to determine
`the size of an object in one dimension, i.e. a linear mea
`Surement. Typical linear measurements are length, width,
`and height. For many items, the concepts of length, width,
`and height are commonly understood to refer to the one
`dimensional measure of a certain axis with specific end
`points. These commonly accepted measures rely on easily
`delineated endpoints and they allow for size comparisons
`between individual items. Even complex three-dimensional
`items can be meaningfully compared using measures of a
`single axis common to all objects of the same type, using
`accepted and easily identified endpoints. For example, the
`height of a person is commonly understood to refer to the
`one-dimensional distance from the bottom of the person's
`feet to the top of their head.
`The methods of the invention may also be used to measure
`the area of an item which is substantially flat or which has
`a planar aspect. In the case of objects having known pro
`portional relationships, the methods of the invention can
`also be utilized to derive one or more indicative measure
`ments which can then be used to estimate other parameters
`with known mathematical relationships to the measured
`aspect(s). For example, in the case of objects which are
`substantially cylindrical or spherical, the methods of the
`invention can be used to measure the diameter of the object
`and then calculate the circumference and volume of the
`object. The methods of the invention may also be used to
`measure angular aspects of an item, such as curvature, angle
`of taper, and other geometrical features of the item that have
`particular relevance. In some embodiments, multiple mea
`
`Petitioner's Exhibit 1012
`Page 12 of 35
`
`

`

`3
`Surements of the item from different angles can be analyzed
`in order to determine the desired parameter.
`The description contained herein, for convenience,
`describes various embodiments directed to measuring the
`size of objects. However, it is understood that the methods,
`systems, software, and devices systems of the invention may
`also be applied to the measurement of any spatial area of
`interest, for example, the distance between objects. For
`example, the linear measurement techniques described
`herein could be utilized to measure the distance between a
`person’s wrist and the floor, which is a medically useful
`parameter for fitting patients with walkers or other mobility
`assist devices.
`The basic function of the invention will be referred to as
`“image-based measurement.’ Image based measurement is
`carried out in five basic steps. First, a reference standard, i.e.,
`an object of known dimensions, is placed on, near, or
`otherwise in plane with the item to be measured. Secondly,
`an image capture means, such as a digital camera, is used to
`capture an image of the item and the reference standard in
`the same image. Third, a processing means, guided by user
`delineations, analyzes the image and compares the image
`size of the reference standard to its actual size, deriving the
`relationship between image size and the actual size of the
`objects depicted in the image. Fourth, the image size of the
`item of interest is measured. Lastly, using the previously
`calculated relationship between image size and actual size,
`and the image size of the item of interest, the actual size of
`the item of interest is determined. Reduced to its most basic
`sense, image-based measurement is akin to placing a ruler
`next to an item and photographing them together; then using
`the size of the ruler in the image to determine the actual size
`of the item.
`Image-based measurement is known in the art. For
`example, the RulerPhone (by Ben Kamens, 2008) mobile
`phone application utilizes the basic image-based measure
`ment system described above, in which a credit card is used
`as the reference standard. The RulerPhone application does
`not allow the use of objects other than credit cards to be
`utilized as a reference standard.
`The novel inventions disclosed herein include improve
`ments to the prior art, which Such improvement increase the
`accuracy, convenience, and versatility of the prior art meth
`ods. These further includes unique methods, devices, sys
`tems and computer programs to carry out image-based
`measurement, and the use of these methods, devices, sys
`tems and computer programs in novel and useful contexts.
`Disclosed herein is a method for the accurate measure
`ment of distance. This method employs the basic linear
`measurement techniques disclosed herein, taking advantage
`of certain optical properties of cameras, to yield facile and
`accurate measurements of distance. The distance measure
`ment method is employed in further embodiments for the
`measurement of Velocity and determining the height of very
`tall objects.
`The methods of the invention are carried out using a
`combination of hardware elements and non-transitory com
`puter-readable storage medium having computer-readable
`program instructions stored therein. The computer-readable
`program instructions may include a plurality of program
`instructions which carry out the methods of the invention on
`one or more devices. It will be understood by one of skill in
`the art that the steps of the various processes described
`herein may be carried out using a wide variety of program
`ming Solutions and computer controlled hardware to effect
`the processes described in these exemplary embodiments.
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 9,696,897 B2
`
`10
`
`15
`
`4
`The methods, systems, and devices of the invention can
`be configured in various ways. In some embodiments, all the
`elements of the invention are housed in a single device,
`allowing the desired measurements to be calculated and
`analyzed on-site. In other embodiments, the image capture
`steps and the image analysis steps are performed on separate
`devices. In some embodiments, multiple devices are utilized
`at various sites. Additionally, off-site databases may be
`accessed by the devices to provide inputs for the analysis
`process, or other data.
`The methods, systems, devices, and computer program
`products of the invention may be carried out on a wide
`variety of devices, including mobile telephones, tablet PC's,
`laptop computers, desktop computers, and cloud computers.
`Hardware for image acquisition may include, for example,
`stand-alone digital cameras, digital camera modules embed
`ded in mobile phones, tablets, or laptop computers, and
`plug-in webcams. User input devices may include, for
`example, touchscreens, mouse interfaces, pen tool inter
`faces, keyboard interfaces, and Voice activated controllers. It
`is recognized by one of skill in the art that there are multiple
`programming options for practicing the invention, including
`a variety of operating Systems and programming languages.
`One of skill in the art, guided by this disclosure, can readily
`choose the properhardware drivers and control modules and
`implement the image acquisition and image analysis func
`tions described herein.
`In many embodiments of the invention, all or some
`portion of the processes claimed herein are carried out on a
`Smartphone. In recent years, there has been an incredible
`proliferation of smartphones. In 2010, it was estimated that
`over 45 million Smartphones were in use in the United States
`alone. Exemplary smartphones include the Apple iPhoneTM
`and the Motorola DroidTM. Smartphones are wireless, com
`pact, hand-held devices that, in addition to basic cellular
`telephone functions, include a range of compact hardware.
`Typical Smartphones have embedded digital cameras, large
`touchscreen displays, and broadband or Wi-Fi capabilities
`allowing for the receipt and transmission of large amounts of
`data to and from the Internet.
`Smartphones not only include a range of hardware, they
`are also configured to download and run a wide variety of
`Software applications, commonly called 'apps. Although
`the proliferation of Smartphones is a recent phenomenon,
`there already exists a mature and well developed system for
`the delivery of apps to Smartphone users.
`The proliferation of Smartphones, with their combination
`of portable hardware and readily loaded software applica
`tions, creates a platform upon which many embodiments of
`the invention may be practiced. While the methods, devices,
`and systems of the invention are not limited to the use of
`Smartphones, the invention most advantageously utilizes
`basic features of Smartphones and extends the capabilities of
`these devices to include accurate and convenient measure
`ment of many items. In essence, the inventions described
`herein may conveniently convert a common Smartphone into
`a medical device, research tool, or measurement system, and
`a means of compiling, communicating and comparing mea
`Surements, all without requiring the user to obtain any
`additional specialized equipment.
`
`DESCRIPTION OF THE DRAWINGS
`
`FIG.1. This drawing depicts the effect of foreshortening.
`When the finger (101) is oriented perfectly vertically (FIG.
`1A), its length (102) measures 62 mm in the image. When
`the same finger is oriented away from the camera, at an angle
`
`Petitioner's Exhibit 1012
`Page 13 of 35
`
`

`

`5
`that is not perpendicular with the plane of the image (FIG.
`1B), the length of the finger (103) is 51 mm in the image.
`FIG. 2 depicts the apparent change in size of a dynamic
`reference standard that pivots as it is pressed against objects
`of varying thickness. FIG. 2 depicts a dynamic reference
`standard comprising a bar (202) of known length attached to
`a rod (204) extending from the camera, the bar attached to
`the rod by a pivoting means (203). In FIG. 2A, the dynamic
`reference standard is pushed against an object (205) that is
`substantially flat, and the bar of the dynamic reference
`standard (202) is not deflected, i.e. to an observer or camera
`(201), the bar appears in its maximum size (206). In FIG.
`2B, the dynamic reference standard is pressed against an
`object having some depth (207), resulting in the bar (202)
`pivoting around the pivot means (203). The reference stan
`dard bar appears smaller (208) to an observer or camera
`(201) because it is not in-plane with the image and is
`foreshortened. In FIG. 2C, the dynamic reference standard is
`pressed against a larger object (209) and is deflected such
`that its apparent size (210) is even smaller to an observer or
`camera (201).
`FIG. 3. This figure depicts an exemplary implementation
`of a visual interface displaying inclinometer data on the
`screen of a device during the camera aiming step in image
`acquisition. A virtual “bubble’ (301) moves in response to
`25
`the forward, backward, and lateral tilt of the device. When
`the camera is not level, the bubble appears outside the
`encircling guide (302), as in FIG. 3A, the direction of the
`deviance from the center corresponding to the orientation of
`the camera, and the degree of deviance from the center being
`proportional to the angle of tilt. When the camera is near
`level, the bubble (301) is still outside the encircling guide
`but is closer to being centered (302), as in FIG. 3B. When
`the camera is perfectly level, the bubble (301) appears
`centered within the guide (302), as in FIG. 3C. A color
`indicator, depicted as an indicator light or shading over some
`portion of the screen (303) also aids in orienting the camera,
`this feature changing color as the camera is leveled. For
`example, the color indicator may appear red when the
`camera is not level, yellow when the camera is near level.
`and green when the camera is substantially level, within an
`acceptable range of tilt.
`FIG. 4. This figure depicts a series of simulated screen
`shots which encompass the delineation steps wherein the
`reference standard object and the target object are delineated
`in the image so that their image sizes may be measured, as
`set forth in detail in Example 1.
`FIG. 5. This figure depicts a software diagram illustrating
`a potential architecture for a software application designed
`to implement the image acquisition step of the invention.
`FIG. 6. This figure depicts a software diagram illustrating
`a potential architecture for a software application designed
`to implement the image analysis step of the invention.
`FIG. 7 depicts the relationship between minification and
`distance. In this example, a person standing 68 inches in
`height was photographed at distances ranging from 10 to 95
`feet from the camera. The camera used was the standard
`camera module of an Apple iPhone 4. Image size of the
`person in the resulting images was measured by displaying
`each image on the iPhone 4 screen and measuring the
`person’s height in the displayed image using a ruler. The
`person’s image height in each picture was plotted against the
`person's distance from the camera at the time each image
`was acquired.
`FIG. 8 This figure depicts a minification Distance Func
`tion that was derived by photographing three objects of
`varying height (a US $20 bill (length=6.14 inches), a plastic
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 9,696,897 B2
`
`10
`
`15
`
`6
`bin (height 15.5 inches), and a car (height from ground to
`roof rack=71.5 inches)) at various distances from a digital
`camera, ranging from 1 to 60 feet. The digital camera
`employed was the standard camera module of an Apple
`iPhone 3G. The height of the items in the resulting images
`was measured using Photoshop CS3. The actual height of
`each item was divided by its size in each image, and this
`value was plotted against distance from the camera to
`generate a linear minification vs. distance function.
`FIG. 9. Exemplary illustration of the height measurement
`method of the invention using a camera- and inclinometer
`equipped mobile device.
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`
`The invention comprises various methods of carrying out
`image-based measurement. In some aspects, the invention
`comprises novel systems for measuring parameters of inter
`est. In other aspects, the invention comprises novel improve
`ments to known methods of image-based measurement. In
`other aspects, the invention comprises novel implementa
`tions of image-based measurement on mobile devices. In yet
`other aspects, the invention comprises the application of
`image-based measurement to novel contexts. In still other
`aspects, the invention comprises novel image-based techni
`cal Solutions to measurement problems. In still other aspects,
`the invention comprises Software programs, i.e. computer
`instructions stored on a nontransitory computer readable
`storage medium that when executed by a machine result in
`the desired operations occurring. The methods of the inven
`tion are enabled and executed by the use of various software
`programs, comprising nontransitory computer readable Stor
`age medium carrying computer programs that will cause a
`Suitable general purpose computer to carry out the methods
`described herein. It will be appreciated by one of skill in the
`art that these computer operations enabled by the software
`encompass interactions with a variety of standard processing
`means, data storage and retrieval means, hardware elements,
`user interfaces, and user inputs to effect the methods
`described herein.
`Image-Based Measurement.
`Image-based measurement may be used to determine a
`dimension of interest, for example length along a particular
`axis, diameter, or area, such as the height of a person, the
`diameter of a coin, or the area of a rash. Image-based
`measurement encompasses the imaging of a “target object.”
`which is an object of unknown dimensions, and a “reference
`standard object.” (or “reference standard”) which is an
`object, at least one dimension of which (for example length,
`diameter, etc.) is known. Subsequently, the image sizes of
`the target object and reference standard object are measured.
`As used herein, “image size refers to a dimension of
`interest of an object pictured in an image, i.e. the actual
`physical size of the object in the image. For example, in a
`photograph of an upright person, the person’s height in the
`image would be a measure of image size. Generally, as used
`herein, image size will refer to the physical size of an object
`represented in an image which is stored as a computer image
`file, i.e. a digital photograph. Physical size of features within
`the image may be measured in pixels. The actual physical
`size of pixels within a digital image will vary between
`images, depending on the pixel resolution and pixel density
`(e.g. pixels per inch) of the image, however, within an image
`these properties are fixed and physical features of objects
`within the image may be compared by determining linear or
`areal pixel dimensions. In some contexts, image size will
`
`Petitioner's Exhibit 1012
`Page 14 of 35
`
`

`

`US 9,696,897 B2
`
`7
`refer to the dimension of interest of a feature pictured within
`a displayed or printed image. The actual physical size of
`such features will vary depending on the resolution of the
`original image and the resolution of the display or print, but
`in any case, these parameters are typically fixed within a
`displayed or printed image and meaningful comparisons of
`sizes within Such displayed or printed image may be made.
`As used herein, the proportion between an objects actual
`size to its image size will be called minification, as described
`in Equation 1.
`
`Actual Size of Object
`-w
`Minification=
`Object's Size in Image
`
`Equation 1
`
`Although this proportion is referred to as minification herein
`for convenience, it is understood that objects are not nec
`essarily de-magnified in an image. Microscopic objects and
`even Small objects (e.g. of a few inches in length) may have
`an image size many times their actual size when pictured in
`images several inches wide by several inches long, such as
`are attained by standard digital camera modules included in
`Smartphones. The degree of minification increases as an
`object is moved further from the camera, for example, as
`depicted in FIG. 7. Therefore, at any given distance, the
`degree of minification of all objects imaged at that distance
`will be substantially equal. The degree of minification of a
`reference standard object in an image can be determined by
`dividing the reference object's size by its image size.
`Assuming proper alignment, as discussed later herein, if the
`target object has been photographed at substantially the
`same distance from the camera as the reference object, its
`degree of minification will be the same as that of the
`reference standard object, for example as expressed in
`Equation 2.
`
`5
`
`10
`
`15
`
`25
`
`30
`
`35
`
`8
`size might be employed as reference standards. Depending
`on the resolution of the images being captured, reference
`standards of 1% to 10,000% of the size of the item being
`measured may be used. Preferred reference standards are in
`the range of 20% to 500% of the size of the target object.
`Especially preferred reference standards are in the range of
`50% to 200% of the size of the target object. In general, the
`use of a reference standard having a size that is close to that
`of the target object (e.g. 50% to 200% of the size of target
`object) is advantageous. If a reference standard and target
`object are of widely varying scales, the resolution of the
`Smaller item in the image may be compromised due to the
`limitations of the image acquisition means (e.g. low reso
`lution) or defects in the image (e.g., low light, blur from
`camera motion, etc). This may result in either an erroneous
`delineation of the reference standard or the target object,
`compromising the accuracy of the resulting measurement
`calculation. Additionally, in the event that the image is not
`taken in-plane, the measurement errors introduced by fore
`shortening, as discussed herein, will be amplified if the
`reference standard and item being measured are not of
`similar sizes.
`In the prior art image-based measurement method
`employed by RulerPhone, only a single object, a credit card,
`may be utilized as a reference standard. During the reference
`standard delineation step of the process, RulerPhone super
`imposes a box said to conform to the proportions of a
`standard credit card over the acquired image. RulerPhone's
`limitation to a single reference standard and calibration icon
`presents many disadvantages for the user. First, the propor
`tions of various bank cards, credit cards, ID's, and like items
`may vary considerably and all such cards do not fit the
`fixed-size calibration icon box utilized in RulerPhone. Sec
`ondly, the user may not have a credit card available to use
`in the image acquisition step. Third, as discussed above,
`accuracy may be compromised when the reference standard
`and the target object are of widely varying scale, and a credit
`card will not be the optimal reference standard for measure
`ments of items that are Substantially smaller (e.g. less than
`one inch) or larger (e.g. greater than a foot) than a credit
`card.
`The invention advantageously overcomes the limitations
`of the prior art by allowing the user to utilize any object, or
`any dimension defined within an image, as a reference
`standard. Preferred reference standards include