www.uspto.gov
`
`UNITED STATES DEPARTMENT OF COMMERCE
`United States Patent and Trademark Office
`Address: COMMISSIONER FOR PATENTS
`P.O. Box 1450
`Alexandria, Virginia 22313-1450
`
`17/405,790
`
`08/18/2021
`
`Tobias Anderberg
`
`113748-0279UTO1
`
`4111
`
`Procopio - SPE
`525 B Street
`Suite 2200
`San Diego, CA 92101
`
`GADOMSKI, STEFAN J
`
`2485
`
`PAPER NUMBER
`
`NOTIFICATION DATE
`
`DELIVERY MODE
`
`12/04/2024
`
`ELECTRONIC
`
`Please find below and/or attached an Office communication concerning this application or proceeding.
`
`The time period for reply, if any, is set in the attached communication.
`
`Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
`following e-mail address(es):
`
`docketing @procopio.com
`
`PTOL-90A (Rev. 04/07)
`
`

`

`
`
`Disposition of Claims*
`1,4-11 and 15-20 is/are pending in the application.
`)
`Claim(s)
`5a) Of the above claim(s) _ is/are withdrawn from consideration.
`C} Claim(s)__ is/are allowed.
`Claim(s) 1,4-11 and 15-20 is/are rejected.
`(] Claim(s)__ is/are objectedto.
`C] Claim(s
`are subjectto restriction and/or election requirement
`)
`* If any claims have been determined allowable, you maybeeligible to benefit from the Patent Prosecution Highway program at a
`participating intellectual property office for the corresponding application. For more information, please see
`http://www.uspto.gov/patents/init_events/pph/index.jsp or send an inquiry to PPHfeedback@uspto.gov.
`
`) ) ) )
`
`Application Papers
`10) The specification is objected to by the Examiner.
`11)0) The drawing(s) filedon__ is/are: a)(J accepted or b)( objected to by the Examiner.
`Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a).
`Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d).
`
`Priority under 35 U.S.C. § 119
`12)7) Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d)or (f).
`Certified copies:
`c)Z None ofthe:
`b)() Some**
`a)C All
`1.1.) Certified copies of the priority documents have been received.
`2.2) Certified copies of the priority documents have been received in Application No.
`3.1.) Copies of the certified copies of the priority documents have been receivedin this National Stage
`application from the International Bureau (PCT Rule 17.2(a)).
`*“ See the attached detailed Office action for a list of the certified copies not received.
`
`Attachment(s)
`
`1) [[] Notice of References Cited (PTO-892)
`
`Information Disclosure Statement(s) (PTO/SB/08a and/or PTO/SB/08b)
`2)
`Paper No(s)/Mail Date
`U.S. Patent and Trademark Office
`
`3)
`
`4)
`
`(LJ Interview Summary (PTO-413)
`Paper No(s)/Mail Date
`(Qj Other:
`
`PTOL-326 (Rev. 11-13)
`
`Office Action Summary
`
`Part of Paper No./Mail Date 20241120
`
`Application No.
`Applicant(s)
`17/405, 790
`Anderbergetal.
`
`Office Action Summary Art Unit|AIA (FITF)StatusExaminer
`STEFAN GADOMSKI
`2485
`Yes
`
`
`
`-- The MAILING DATEof this communication appears on the cover sheet with the correspondence address --
`Period for Reply
`
`A SHORTENED STATUTORYPERIOD FOR REPLYIS SET TO EXPIRE 3 MONTHS FROM THE MAILING
`DATE OF THIS COMMUNICATION.
`Extensionsof time may be available underthe provisions of 37 CFR 1.136(a). In no event, however, may a reply betimely filed after SIX (6) MONTHSfrom the mailing
`date of this communication.
`If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHSfrom the mailing date of this communication.
`-
`- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133).
`Any reply received by the Office later than three months after the mailing date of this communication, evenif timely filed, may reduce any earned patent term
`adjustment. See 37 CFR 1.704(b).
`
`Status
`
`1) Responsive to communication(s) filed on 09/27/2024.
`C} A declaration(s)/affidavit(s) under 37 CFR 1.130(b) was/werefiled on
`
`2a)() This action is FINAL. 2b)¥)This action is non-final.
`3) An election was madeby the applicant in responseto a restriction requirement set forth during the interview
`on
`; the restriction requirement and election have been incorporated into this action.
`4)(2) Since this application is in condition for allowance except for formal matters, prosecution as to the merits is
`closed in accordance with the practice under Exparte Quayle, 1935 C.D. 11, 453 O.G. 213.
`
`

`

`Application/Control Number: 17/405,790
`Art Unit: 2485
`
`Page 2
`
`DETAILED ACTION
`
`Notice ofPre-AIA or AIA Status
`
`1.
`
`The presentapplication, filed on or after March 16, 2013, is being examined underthe
`
`first inventorto file provisions of the AIA.
`
`Continued Examination Under 37 CFR 1.114
`
`2.
`
`A request for continued examination under 37 CFR 1.114, includingthe fee set forth in
`
`37 CFR 1.17(e), was filed in this application after final rejection. Since this applicationis
`
`eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e)
`
`has been timely paid, the finality of the previous Office action has been withdrawn pursuantto
`
`37 CFR 1.114. Applicant's submission filed on 09/27/2024 has been entered.
`
`Response to Amendment
`
`3.
`
`The Amendmentfiled 09/27/2024 has been entered. No claims have been added. Claims
`
`2,3,and 12-14 have been cancelled. Claims 1,9, and 18 have been amended. Claims 1, 4-11,
`
`and 15-20 remain pending in the application.
`
`Response to Arguments
`
`4.
`
`Applicant’s arguments, see pages 6-10, filed 09/27/2024, with respect to the 103
`
`rejections have been fully considered andare persuasive. Therefore, the rejection has been
`
`withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view
`
`of a previously cited reference, specifically Rowell etal. US 2019/0158813 Al.
`
`

`

`Application/Control Number: 17/405,790
`Art Unit: 2485
`
`Page 3
`
`Examiner concurs the previously cited references fail to disclose or suggest the amended
`
`claim language. However, an updated search foundthat the Rowell reference discloses or
`
`suggest the amendedclaim language. Rowell discloses real time automatic calibration of
`
`multiple cameras. The combination of Islam, Liu, and Rowell discloses or suggest the claim
`
`language of independent claims 1, 9 and 18.
`
`Information Disclosure Statement
`
`5.
`
`The information disclosure statement (IDS) submitted on 1 1/02/2024 was considered by
`
`the examiner.
`
`Claim Rejections - 35 USC § 103
`
`6.
`
`In the eventthe determination ofthe status of the application as subject to AIA 35 U.S.C.
`
`102 and 103 (or as subject to pre-AJA 35 U.S.C. 102 and 103) is incorrect, any correction of the
`
`statutory basis (1.e., changing from AIA to pre-AIA)for the rejection will not be considered a
`
`new groundof rejection if the prior art relied upon, and the rationale supporting the rejection,
`
`would be the same undereitherstatus.
`
`7,
`
`The followingis a quotation of 35 U.S.C. 103 which formsthe basis for all obviousness
`
`rejections set forth in this Office action:
`
`A patent fora claimed invention may not be obtained, notwithstanding that the claimed invention is not
`identically disclosed as set forth in section 102, if the differences between the claimed invention and the
`prior art are such that the claimed invention as a whole would have been obviousbefore the effective
`filing date of the claimed invention to a person having ordinary skill in the art to which the claimed
`invention pertains. Patentability shall not be negated by the manner in which the invention was made.
`
`

`

`Application/Control Number: 17/405,790
`Art Unit: 2485
`
`Page 4
`
`8.
`
`The factual inquiries for establishing a background for determining obviousness under 35
`
`U.S.C. 103 are summarized as follows:
`
`1. Determining the scope and contents of the priorart.
`
`2. Ascertaining the differences betweentheprior art and the claimsat issue.
`
`3. Resolving the level of ordinary skill in the pertinentart.
`
`4. Considering objective evidence present in the application indicating obviousness or
`
`nonobviousness.
`
`9.
`
`Claims 1, 4, 8-11, 15, and 18-20 rejected under 35 U.S.C. 103 as being unpatentable
`
`over Islam et al. US 2021/0012534 A1, hereafter Islam, in view of Liu et al. US
`
`2019/0308326 Al, hereafter Liu, further in view of Rowell et al. US 2019/0158813 Al,
`
`hereafter Rowell.
`
`Regarding claim 1, Islam discloses a machine vision and control system to capture
`
`images using cameraandto calibrate the cameras using the images (method and systemfor
`
`performing automatic camera calibrationfor a scanning system) [title], the machine vision and
`
`control system comprising:
`
`a machine vision system including:
`
`a plurality of imaging devices to capture or detect one of (1) calibration target or (2)
`
`feature within a scene or capture volume (control circuit receives a first set ofcalibration images
`
`via the communication interface from the first camera, wherein the first set ofcalibration images
`
`capturethe first set offaces of the polyhedron and capturethefirst set of2D calibration pattems
`
`disposed respectively on the first set offaces 201; control circuit receives a second set ofone or
`
`

`

`Application/Control Number: 17/405,790
`Art Unit: 2485
`
`Page 5
`
`more calibration images via the communication interface from the second camera, wherein the
`
`second set ofone or more calibration images capture the additionalface ofthe polyhedron 203)
`
`[FIG.2];
`
`a processor (control circuit may include one or more processors) [0035] to determine a
`
`configuration of the plurality of imaging devices using the captured or detected calibration target
`
`or feature (control circuit determines, based on first set ofcoordinates and the second set of
`
`coordinates, a transformation function for describing a spatial relationship between thefirst
`
`camera and the second camera 209) [FIG. 2],
`
`wherein the determined configuration includes arranging the plurality of imaging devices
`
`in an array of different types and groupings of imaging devices including at least one of groups
`
`of a certain numberof imaging devices, groupsof infra-red and color imaging devices, and
`
`arrays of imaging devices in a spherical structure (control circuit determines, based on first set of
`
`coordinates and the second set ofcoordinates, a transformation function for describing a spatial
`
`relationship betweenthe first camera and the second camera 209) [FIG. 2],
`
`the processor to automatically calibrate the plurality of imaging devices using the
`
`determined configuration of the plurality of imaging devices (camera calibration according to
`
`steps 201-209) [0045]; and
`
`a control system (robot operation system 101) including motorized device (robot) [0023],
`
`wherein the processor automatically adjusts, positions, aligns, and calibrates the
`
`motorized device of the control system using the determined configuration of the plurality of
`
`imaging devices (after camera calibration has been performed and when an objectother than
`
`the 3D pattern is disposed on a first surface ofthe platform, control circuit generates a 3D
`
`model for representing the object, wherein the 3D model is generated based on the
`
`

`

`Application/Control Number: 17/405,790
`Art Unit: 2485
`
`Page 6
`
`transformation function, based on imagesof the object 211; 3D model ofmayfacilitate an ability
`
`ofa robot to interact with the object; picking up the object) [FIG. 2; 0023; 0024].
`
`However, while Islam discloses using a pattern to determine a camera configuration to
`
`generate a 3D object for a robot to interact with the corresponding real world object, Islam fails
`
`to explicitly disclose automatically adjust, position, align, and calibrate lens parameters,
`
`including focus and aperture; motorized device mounts on whichthe plurality of imaging devices
`
`is placed, wherein the processor automatically adjusts, positions, aligns, and calibrates the
`
`motorized device mounts of the control system using the determined configuration of the
`
`plurality of imaging devices.
`
`Liu, in an analogous environment, discloses motorized device mounts on whichthe
`
`plurality of imaging devices is placed (robot 105 with a camera 110 mounted thereon; in some
`
`embodiments, more than one camera can be used) [0038; 0084 ], wherein the processor
`
`automatically adjusts, positions, aligns, and calibrates the motorized device mounts of the contol
`
`system using the determined configuration of the plurality of imaging devices (the motion
`
`controller can be the machine vision master...the motion controller can control the pose and
`
`movement ofthe robot and to cause the machine vision processor to acquire images...cause the
`
`machine vision processorto perform the calibration described herein) [0039].
`
`Islam and Liu are analogous becausethey are both related to calibration of cameras for
`
`machinevision. Therefore, it would have been obviousto one of ordinary skill in the art before
`
`the effective filing date of the claimed invention to use the mounted cameras, as disclosed by
`
`Liu, with the invention disclosed by Islam, the motivation being accuracy [0003].
`
`Further, Rowell, in an analogous environment, discloses the processor automatically
`
`adjusts, positions, aligns, and calibrates lens parameters needed including focus and aperture
`
`

`

`Application/Control Number: 17/405,790
`Art Unit: 2485
`
`Page 7
`
`(real time camera setting 902... video sequences captured using camera configurations
`
`comprising one or more camera setting (e.g., baseline, zoom, focus aperture; slight changesto
`
`the position ofone or more lenses...an auto re-calibration process that modifies 3D calibration
`
`metadata in real time to correctfor changesin the position ofone or more camera module
`
`components) [0119; 0135].
`
`Islam and Liu and Rowell are analogous becausetheyare related to calibration of a
`
`plurality of cameras. Therefore, it would have been obvious to one of ordinary skill in the art
`
`before the effective filing date of the claimed invention to use the lens auto adjustment for
`
`calibration, as disclosed by Rowell, with the invention disclosed by Islam andLiu, the
`
`motivation being compensating for intrinsic lens parameters [0004].
`
`Regarding claim 4, Islam, Liu, and Rowell address all of the features with respectto
`
`claim | as outlined above.
`
`Liu further discloses the control system manually activates or initiates a control process
`
`of the processor, wherein the control process includes adjusting, positioning, aligning, and
`
`calibrating (control system 115 can manipulate the pose of robot 105, e.g., based on analysis of
`
`image data from camera 110) [0039].
`
`Therefore, it would have been obvious to one of ordinaryskill in the art before the
`
`effective filing date of the claimed invention to use the control process, as disclosed by Liu, with
`
`the invention disclosed by Islam, the motivation being accuracy [0003].
`
`Regarding claim 8, Islam, Liu, and Rowell address all of the features with respectto
`
`claim | as outlined above.
`
`

`

`Application/Control Number: 17/405,790
`Art Unit: 2485
`
`Page 8
`
`Islam further discloses lenses electronically connectedto the plurality of imaging devices
`
`(lens) [0030].
`
`Claims 9 are drawn to a method implementedby the control system of claims 1, and are
`
`therefore rejected in the same manneras above.
`
`Regarding claim 10, Islam, Liu, and Rowell address all of the features with respect to
`
`claim 9 as outlined above.
`
`Islam further discloses the detected images include a calibration target (3D calibration
`
`target) [abstract].
`
`Regarding claim 11, Islam, Liu, and Rowell address all of the features with respect to
`
`claim 9 as outlined above.
`
`Islam further discloses the detected images include a feature within a scene or capture
`
`volume (3D calibration target) [abstract].
`
`Regarding claim 15, Islam, Liu, and Rowell address all of the features with respect to
`
`claim 9 as outlined above.
`
`Liu further discloses wherein adjusting, positioning, aligning, and calibrating the plurality
`
`of imaging devices comprises manually activating or initiating adjusting, positioning,aligning,
`
`andcalibrating of the plurality of imaging devices (control system 115 can manipulate the pose
`
`ofrobot 105, e.g., based on analysis ofimage data from camera 110) [0039].
`
`

`

`Application/Control Number: 17/405,790
`Art Unit: 2485
`
`Page 9
`
`Therefore, it would have been obvious to one of ordinaryskill in the art before the
`
`effective filing date of the claimed invention to use the control process, as disclosed by Liu, with
`
`the invention disclosed by Islam, the motivation being accuracy [0003].
`
`Regarding claim 18-20, non-transitory computer readable storage medium claims 18-20
`
`are drawnto the instructions corresponding to the method of claims 9, 10 and 16. Therefore,
`
`non-transitory computer readable storage medium claims 18-20 correspond to method claims9,
`
`10, and 16 andare rejected for the same reasons of unpatentability as used above.
`
`10.
`
`Claims 5, 6, 16, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over
`
`Islam and Liu in view of Yi-Ping Hung “A Simple Real-Time Method for Calibrating a
`
`Camera Mounted on A Robot For Three Dimensional Machine Vision”, hereafter Hung.
`
`Regarding claim 5, Islam, Liu, and Rowell address all of the features with respectto
`
`claim | as outlined above.
`
`However, the combination fails to disclose the processor continuously aligns and
`
`calibrates the control system according to defined conditions.
`
`Hung,in an analogous environment, discloses the processor continuously aligns and
`
`calibrates the control system accordingto defined conditions (off-line stage ...using observations
`
`ofthe calibration objects directly as described in section IIIA ...calibr ate the robot as accurately
`
`as possible) [section IV].
`
`Islam and Liu and Hungare analogous because they are both related to calibrating a
`
`camera mounted onarobot. Therefore, it would have been obviousto one of ordinary skill in
`
`

`

`Application/Control Number: 17/405,790
`Art Unit: 2485
`
`Page 10
`
`the art before the effective filing date of the claimed invention to use offline calibration, as
`
`disclosed by Hung,with the invention disclosed by Islam and Liu, the motivation being
`
`improving calibration accuracy [section I].
`
`Regarding claim 6, Islam and Liu and Hung address all of the features with respectto
`
`claim 5 as outlined above.
`
`Liu further discloses the defined conditions include whenthe system is not recording any
`
`images (off-line stage ...using observations ofthe calibration objects directly as described in
`
`section IILA...calibrate the robot as accurately as possible) [section IV].
`
`Claims 16 and 17 are drawn to a method implemented by the control system of claims 5
`
`and 6, and are therefore rejected in the same manneras above.
`
`
`
`11. Claim7is rejected under 35 U.S.C. 103 as being unpatentable over Islam and Liu in
`
`view of De Villiers et al. US 9,330,463 B2, hereafter De Villiers.
`
`Regarding claim 7, Islam, Liu, and Rowell address all of the features with respectto
`
`claim 1 as outlined above.
`
`However, the combination fails to explicitly disclose wherein the motorized device
`
`mounts include one of (1) Brushless DC Motor (BLDC)or (2) Synchronous Servo Motor
`
`(SSVM).
`
`

`

`Application/Control Number: 17/405,790
`Art Unit: 2485
`
`Page 11
`
`De Villiers, in an analogous environment, discloses wherein the motorized device mounts
`
`include one of (1) Brushless DC Motor (BLDC)or (2) Synchronous Servo Motor (SSVM)(the
`
`mechanical actuator 18is movable by servo motors) [column3, lines 36-38].
`
`Islam and Liu and De Villiers are analogous becausethey are both related to calibrating
`
`cameras on robotarms. Therefore, it would have been obviousto one of ordinary skill in the art
`
`before the effective filing date of the claimed invention to use the servo motor, as disclosed by
`
`De Villiers, with the invention disclosed by Islam andLiu, to obtain the predictable result of
`
`using various types of motorsin the robot arms.
`
`Conclusion
`
`Any inquiry concerning this communication or earlier communications from the
`
`examiner should be directed to STEFAN GADOMSKIwhosetelephone numberis (571)270-
`
`5701. The examiner can normally be reached Monday- Friday, 12-8PM EST.
`
`Examinerinterviews are available via telephone, in-person, and video conferencing using
`
`a USPTO supplied web-based collaboration tool. To schedule an interview, applicantis
`
`encouragedto use the USPTO Automated Interview Request (AIR)at
`
`http://www.uspto.gov/interviewpractice.
`
`If attempts to reach the examinerby telephone are unsuccessful, the examiner’s
`
`supervisor, Jay Patel can be reached on 571-272-2988. The fax phone numberfor the
`
`organization where this application or proceedingis assigned is 571-273-8300.
`
`Information regarding the status of published or unpublished applications may be
`
`obtained from Patent Center. Unpublished application information in Patent Centeris available
`
`to registered users. To file and managepatent submissions in Patent Center, visit:
`
`

`

`Application/Control Number: 17/405,790
`Art Unit: 2485
`
`Page 12
`
`https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more
`
`information about Patent Center and https://www.uspto.gov/patents/docx for information about
`
`filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC)
`
`at 866-217-9197 (toll-free). If you would like assistance from a USPTO CustomerService
`
`Representative, call 800-786-9199 (IN USA OR CANADA)or 571-272-1000.
`
`STEFAN GADOMSKI
`Primary Examiner
`Art Unit 2485
`
`/STEFAN GADOMSKI/
`Primary Examiner, Art Unit 2485
`
`