US0094.06.177B2
`
`(12) United States Patent
`Attard et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9.406,177 B2
`Aug. 2, 2016
`
`(54)
`
`(71)
`
`(72)
`
`(73)
`
`(*)
`
`(21)
`(22)
`(65)
`
`(63)
`
`(51)
`
`(52)
`
`FAULTHANDLING IN AN AUTONOMOUS
`VEHICLE
`
`Applicant: Ford Global Technologies, LLC,
`Dearborn, MI (US)
`Inventors: Christopher Attard, Ann Arbor, MI
`(US); Shane Elwart, Ypsilanti, MI (US);
`Jeff Allen Greenberg, Ann Arbor, MI
`(US); Rajit Johri, Ann Arbor, MI (US);
`John P. Joyce, West Bloomfield, MI
`(US); Devinder Singh Kochhar, Ann
`Arbor, MI (US); Douglas Scott Rhode,
`Farmington Hills, MI (US); John
`Shutko, Ann Arbor, MI (US); Hongtei
`Eric Tseng, Canton, MI (US)
`Assignee: Ford Global Technologies, LLC,
`Dearborn, MI (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 3 days.
`Appl. No.: 14/184,860
`
`Notice:
`
`Filed:
`
`Feb. 20, 2014
`
`Prior Publication Data
`US 2015/O178998 A1
`Jun. 25, 2015
`
`Related U.S. Application Data
`Continuation-in-part of application No. 14/136,495,
`filed on Dec. 20, 2013, now Pat. No. 9,346,400.
`
`Int. C.
`GOIC 22/00
`G05D I/00
`G07C5/00
`G07C5/08
`U.S. C.
`CPC .............. G07C5/008 (2013.01); G07C5/0808
`(2013.01)
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`
`(58) Field of Classification Search
`CPC ........ G05B 13/026; G05B 13/02; G08G 1/16;
`G05D 1/0257; G05D 2201/0213; G05D
`1/0261; G05D 1/0289; G05D 1/02
`USPC ............................................ 701/2, 23, 24, 25
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,331,561 A * 7/1994 Barrett et al. ................... TO1/23
`5,572,449 A * 1 1/1996 Tang et al. .....
`... 700/304
`5,887,268 A * 3/1999 Furukawa ....................... TO1/23
`6,128,559 A * 10/2000 Saitou ..................... B61L23/34
`340,436
`6,236,915 B1* 5/2001 Furukawa et al. .............. 7O1/25
`6,313,758 B1 * 1 1/2001 Kobayashi ............. G08G 1,162
`340,436
`
`
`
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`CN
`DE
`EP
`
`102867393 A
`102O08052322 A1
`1862985 A3
`
`1, 2013
`4/2010
`6, 2008
`
`OTHER PUBLICATIONS
`
`Non-Final Office Action dated Sep. 21, 2015; U.S. Appl. No.
`14136,495.
`
`Primary Examiner — Jaime Figueroa
`(74) Attorney, Agent, or Firm — Frank MacKenzie; Beijin
`Bieneman PLC
`
`(57)
`ABSTRACT
`Data is collected during operation of a vehicle. A determina
`tion is made that a confidence assessment of at least one of the
`data indicates at least one fault condition. A first autonomous
`operation affected by the fault condition is discontinued,
`where a second autonomous operation that is unaffected by
`the fault condition is continued.
`
`20 Claims, 3 Drawing Sheets
`
`
`
`
`
`corrunication
`
`w
`
`IPR2025-00943
`Tesla EX1004 Page 1
`
`

`

`US 9.406,177 B2
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`6,553,288 B2 * 4/2003 Taguchi ................. G08G 1, 164
`18Of 167
`6,882,923 B2 * 4/2005 Miller et al. .................... TO1.96
`6,985,089 B2 *
`1/2006 Liu ........................ G08G 1, 161
`340,436
`
`3/2009 Allard et al.
`7,499,776 B2
`7,664,589 B2 * 2/2010 Etori et al. ...................... TO1.96
`7,689.230 B2 * 3/2010 Spadafora et al. ......... 455,456.1
`7,831,345 B2 * 1 1/2010 Heino ...................... B62D 1.28
`299/19
`8, 116,921 B2 * 2/2012 Ferrin et al. ...................... TO1f1
`8, 135,507 B2
`3/2012 Okabe et al.
`8,504.233 B1* 8/2013 Ferguson et al. ............... TO1/23
`8,510,029 B2 * 8/2013 Curtis et al. ...
`... 701,301
`8,718,861 B1* 5/2014 Montemerlo ......... B6OW 3O/OO
`TO1/26
`9,076.341 B2* 7/2015 Funabashi
`... GO8G 1/22
`2005/0134440 A1* 6/2005 Breed ........................... 340/.435
`
`2008/0055068 A1* 3/2008 Van Wageningen
`et al. .......................... 340,539.3
`2008/O140278 A1* 6/2008 Breed ............................. 7O1/29
`2009/0076673 A1
`3/2009 Brabec
`2010/0256852 A1* 10/2010 Mudalige ........................ TO1/24
`2012/O126997 A1* 5, 2012 Bensoussan .
`340,905
`2012/0314070 A1* 12/2012 Zhang et al. .
`... 348/148
`2012/0323474 A1* 12/2012 Breed et al. ................... 701 117
`2013/0024084 A1
`1/2013 Yamashiro .......... B6OW 50,029
`TO1.96
`2013/0030606 A1* 1/2013 Mudalige et al. ................. 7O1/2
`2013, O154853 A1* 6, 2013 Chen ................ G08G 1,096716
`340,905
`2013/0279491 A1* 10/2013 Rubin et al. .................. 370,347
`2013,0297.195 A1* 11, 2013 Das et al. ........
`701 117
`2013/0325241 A1* 12/2013 Lombrozo et al. .............. TO1/23
`2014/0186052 A1* 7, 2014 Oshima et al. ................ 398,130
`2014/03.02774 A1* 10/2014 Burke .................... HO4H2O,57
`455,305
`
`
`
`2014/0303827 A1 10/2014 Dolgov et al.
`* cited by examiner
`
`IPR2025-00943
`Tesla EX1004 Page 2
`
`

`

`U.S. Patent
`
`Aug. 2, 2016
`
`Sheet 1 of 3
`
`US 9.406,177 B2
`
`OOT
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`IPR2025-00943
`Tesla EX1004 Page 3
`
`

`

`U.S. Patent
`
`Aug. 2, 2016
`
`Sheet 2 of 3
`
`US 9.406,177 B2
`
`
`
`2OO
`
`Y
`
`YES
`
`Autonomous driving
`205
`
`Collect data
`210
`
`Compute confidence estimates
`215
`
`Compare scalar estimate to
`stored parameter
`220
`
`Identify vector values for alert
`230
`
`Provide alert
`235
`
`
`
`Continue?
`240
`
`NO
`
`END
`
`FIG. 2
`
`IPR2025-00943
`Tesla EX1004 Page 4
`
`

`

`U.S. Patent
`
`Aug. 2, 2016
`
`Sheet 3 of 3
`
`US 9.406,177 B2
`
`Autonomous driving
`305
`
`Collect data
`310
`
`Compute confidence estimates
`315
`
`Compare scalar estimate to
`stored parameter
`32O
`
`Send w2w Communication
`330
`
`Receive response(s)
`335
`
`Determine action(s)
`340
`
`FIG. 3
`
`YES
`
`
`
`Continue?
`345
`
`NO
`
`END
`
`IPR2025-00943
`Tesla EX1004 Page 5
`
`

`

`1.
`FAULTHANDLING IN AN AUTONOMOUS
`VEHICLE
`
`RELATED APPLICATION
`
`This application is a continuation-in-part of, and as such,
`claims priority to, U.S. application Ser. No. 14/136,495,
`entitled AFFECTIVE USER INTERFACE IN AN
`AUTONOMOUS VEHICLE, filed Dec. 20, 2013, the con
`tents of which are hereby incorporated herein by reference in
`their entirety.
`
`10
`
`BACKGROUND
`
`15
`
`A vehicle, e.g., a car, truck, bus, etc., may be operated
`wholly or partly without human intervention, i.e., may be
`semi-autonomous or autonomous. For example, the vehicle
`may include sensors and the like that convey information to a
`central computer in the vehicle. The central computer may
`use received information to operate the vehicle, e.g., to make
`decisions concerning vehicle speed, course, etc. However,
`mechanisms are needed for evaluating a computer's ability to
`autonomously operate the vehicle, and for determining an
`action or actions to take when one or more faults are detected.
`
`DRAWINGS
`
`FIG. 1 is a block diagram of an exemplary vehicle system
`for autonomous vehicle operation, including mechanisms for
`detecting and handling faults.
`FIG. 2 is a diagram of an exemplary process for assessing,
`and providing alerts based on confidence levels relating to
`autonomous vehicle operations.
`FIG. 3 is a diagram of an exemplary process for assessing,
`and taking action based on, confidence levels relating to
`autonomous vehicle operations.
`
`DESCRIPTION
`
`Introduction
`FIG. 1 is a block diagram of an exemplary vehicle system
`100 for operation of an autonomous vehicle 101, i.e., a
`vehicle 101 completely or partly operated according to con
`trol directives determined in a vehicle 101 computer 105. The
`computer 105 may include instructions for determining that
`an autonomous driving module 106, e.g., included in the
`vehicle computer 105, may not be able to operate the vehicle
`101 autonomously or semi-autonomously with acceptable
`confidence, e.g., confidence expressed numerically that is
`lower than a predetermined threshold. For example a fault or
`faults could be detected with respect to one or more data
`collectors 110, e.g., sensors or the like, in a first vehicle 101.
`Further, once a fault is detected, the first vehicle 101 may send
`a vehicle-to-vehicle communication 112 to one or more sec
`ond vehicles 101 and/or may send data via a network 120 to a
`remote server 125. Moreover, further operation of the first
`vehicle 101 may use data 115 from collectors 110 in the first
`vehicle 101 to the extent such data 115 is not subject to a fault,
`and may further use data 115 from one or more second
`vehicles 101 that may be received in a vehicle-to-vehicle
`communication 112.
`Alternatively or additionally, when a fault is detected in a
`vehicle 101, the vehicle 101 could cease and/or disable one or
`more particular autonomous operations dependent on a data
`collector 110 in which the fault was detected. For example,
`the vehicle 101 computer 105 could depend on radar or lidar
`data 115 to detect and/or to maintain a distance from other
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 9,406,177 B2
`
`2
`vehicles 101. Accordingly, if radar and/or lidar data collectors
`110 needed for such distance detection and/or maintenance
`were associated with a fault condition, the vehicle 101 could
`cease and/or disable an adaptive cruise control or like mecha
`nism for detecting and maintaining a distance from other
`vehicles 101. However, if other data collectors 110 were
`available for other autonomous operations, e.g., detecting and
`maintaining a lane, clearing vehicle 101 windows, etc., the
`vehicle 101 could continue to conduct such operations.
`Reasons for lower confidence could include degradation of
`data collection devices 110 Such as sensors, e.g., caused by
`weather conditions, blockage or other noise factors. Lower
`confidence in autonomous operations could also occur if
`design parameters of the autonomous vehicle 101 operation
`are exceeded. For example, confidence assessments 118 may
`arise from data 115 provided by data collectors 110 included
`in a perceptual layer (PL) of the autonomous vehicle 101, or
`from data collectors 110 in an actuation layer (AL). For the
`PL, these confidence estimates, or probabilities, may be inter
`preted as a likelihood that perceptual information is sufficient
`for normal, safe operation of the vehicle 101. For the AL, the
`probabilities, i.e., confidence estimates, express a likelihood
`that a vehicle 101 actuation system can execute commanded
`vehicle 101 operations within one or more design tolerances.
`Accordingly, the system 100 provides mechanisms for detect
`ing and addressing lower than acceptable confidence(s) in one
`or more aspects of vehicle 101 operations.
`Autonomous operations of the vehicle 101, including gen
`eration and evaluation of confidence assessments 118, may be
`performed in an autonomous driving module 106, e.g., as a set
`of instructions stored in a memory of, and executable by a
`processor of a computing device 105 in the vehicle 101. The
`computing device 105 generally receives collected data 115
`from one or more data collectors, e.g., sensors, 110. The
`collected data 115, as explained above, may be used to gen
`erate one or more confidence assessments 118 relating to
`autonomous operation of the vehicle 101. By comparing the
`one or more confidence assessments to one or more stored
`parameters 117, the computer 105 can determine whether to
`provide an alert or the like to a vehicle 101 occupant, e.g., via
`an interface 119. Further additionally or alternatively, based
`on the one or more confidence assessments 118, message 116.
`e.g., an alert, can convey a level of urgency or importance to
`a vehicle 101 operator, e.g., by using prosody techniques to
`include emotional content in a voice alert, a visual avatar
`having an appearance tailored to a level of urgency, etc. Yet
`further additionally or alternatively based on the one or more
`confidence assessments 118, i.e., an indication of a detected
`fault or faults, the computer 105 can determine an action to
`take regarding autonomous operation of the vehicle 101, e.g.,
`to disable one or more autonomous functions or operations, to
`limit or cease operation of the vehicle 101, e.g., implement a
`“slow to a stop’ or “pull over and stop' operation, implement
`a "limp home' operation, etc.
`Concerning messages 116, one example from many pos
`sible, an example, an alert may inform the vehicle 101 occu
`pant of a need to resume partial or complete manual control of
`the vehicle 101. Further, as mentioned above, a form of a
`message 116 may be tailored to its urgency. For example, an
`audio alert can be generated with prosody techniques used to
`convey a level of urgency associated with the alert. Alterna
`tively or additionally, a graphical user interface included in a
`human machine interface of the computer 105 may be con
`figured to display particular colors, fonts, fontsizes, an avatar
`or the like representing a human being, etc., to indicate a level
`of urgency, e.g., immediate manual control is recommended,
`manual control may be recommended within the next minute,
`
`IPR2025-00943
`Tesla EX1004 Page 6
`
`

`

`US 9,406,177 B2
`
`10
`
`15
`
`3
`within the next five minutes, manual control is recommended
`for mechanical reasons, manual control is recommended for
`environmental or weather conditions, manual control is rec
`ommended because of traffic conditions, etc.
`Relating to an action or actions in response to one or more
`detected faults, examples include a first vehicle 101 receiving
`a communication 112 from one or more second vehicles 101
`for operation, e.g., navigation, of the first vehicle 101.
`Examples relating to action or actions in response to one or
`more detected faults alternatively or additionally include the
`first vehicle 101 disabling and/or ceasing one or more autono
`mous operations, e.g., steering control, speed control, adap
`tive cruise control, lane maintenance, etc.
`Exemplary System Elements
`A vehicle 101 may be a land vehicle such as a motorcycle,
`car, truck, bus, etc., but could also be a watercraft, aircraft, etc.
`In any case, the vehicle 101 generally includes a vehicle
`computer 105 that includes a processor and a memory, the
`memory including one or more forms of computer-readable
`media, and storing instructions executable by the processor
`for performing various operations, including as disclosed
`herein. For example, the computer 105 generally includes,
`and is capable of executing, instructions such as may be
`included in the autonomous driving module 106 to autono
`mously or semi-autonomously operate the vehicle 101, i.e., to
`operate the vehicle 101 without operator control, or with only
`partial operator control.
`Further, the computer 105 may include more than one
`computing device, e.g., controllers or the like included in the
`vehicle 101 for monitoring and/or controlling various vehicle
`components, e.g., an engine control unit (ECU), transmission
`control unit (TCU), etc. The computer 105 is generally con
`figured for communications on a controller area network
`(CAN) bus or the like. The computer 105 may also have a
`connection to an onboard diagnostics connector (OBD-II).
`Via the CAN bus, OBD-II, and/or other wired or wireless
`mechanisms, the computer 105 may transmit messages to
`various devices in a vehicle and/or receive messages from the
`various devices, e.g., controllers, actuators, sensors, etc.,
`including data collectors 110. Alternatively or additionally, in
`cases where the computer 105 actually comprises multiple
`devices, the CAN bus or the like may be used for communi
`cations between devices represented as the computer 105 in
`this disclosure.
`In addition, the computer 105 may be configured for com
`45
`municating with the network 120, which, as described below,
`may include various wired and/or wireless networking tech
`nologies, e.g., cellular, Bluetooth, wired and/or wireless
`packet networks, etc. Further, the computer 105, e.g., in the
`module 106, generally includes instructions for receiving
`data, e.g., collected data 115 from one or more data collectors
`110 and/or data from an affective user interface 119 that
`generally includes a human machine interface (HMI), such as
`an interactive Voice response (IVR) system, a graphical user
`interface (GUI) including a touchscreen or the like, etc.
`As mentioned above, generally included in instructions
`stored in and executed by the computer 105 is an autonomous
`driving module 106 or, in the case of a non-land-based or road
`vehicle, the module 106 may more generically be referred to
`as an autonomous operations module 106. Using data
`received in the computer 105, e.g., from data collectors 110.
`data included as stored parameters 117, confidence assess
`ments 118, etc., the module 106 may control various vehicle
`101 components and/or operations without a driver to operate
`the vehicle 101. For example, the module 106 may be used to
`regulate vehicle 101 speed, acceleration, deceleration, Steer
`ing, braking, etc.
`
`55
`
`4
`Data collectors 110 may include a variety of devices. For
`example, various controllers in a vehicle may operate as data
`collectors 110 to provide data 115 via the CAN bus, e.g., data
`115 relating to vehicle speed, acceleration, etc. Further, sen
`sors or the like, global positioning system (GPS) equipment,
`etc., could be included in a vehicle and configured as data
`collectors 110 to provide data directly to the computer 105,
`e.g., via a wired or wireless connection. Data collectors 110
`could also include sensors or the like for detecting conditions
`outside the vehicle 101, e.g., medium-range and long-range
`sensors. For example, sensor data collectors 110 could
`include mechanisms such as RADAR, LIDAR, Sonar, cam
`eras or other image capture devices, that could be deployed to
`measure a distance between the vehicle 101 and other
`vehicles or objects, to detect other vehicles or objects, and/or
`to detect road attributes, such as curves, potholes, dips,
`bumps, changes in grade, lane boundaries, etc.
`A data collector 110 may further include biometric sensors
`110 and/or other devices that may be used for identifying an
`operator of a vehicle 101. For example, a data collector 110
`may be a fingerprint sensor, a retina Scanner, or other sensor
`110 providing biometric data 105 that may be used to identify
`a vehicle 101 operator and/or characteristics of a vehicle 101
`operator, e.g., gender, age, health conditions, etc. Alterna
`tively or additionally, a data collector 110 may include a
`portable hardware device, e.g., including a processor and a
`memory storing firmware executable by the processor, for
`identifying a vehicle 101 operator. For example, such por
`table hardware device could include an ability to wirelessly
`communicate, e.g., using Bluetooth or the like, with the com
`puter 105 to identify a vehicle 101 operator.
`A memory of the computer 105 generally stores collected
`data 115. Collected data 115 may include a variety of data
`collected in a vehicle 101 from data collectors 110. Examples
`of collected data 115 are provided above, and moreover, data
`115 may additionally include data calculated therefrom in the
`computer 105. In general, collected data 115 may include any
`data that may be gathered by a collection device 110 and/or
`derived from such data. Accordingly, collected data 115 could
`include a variety of data related to vehicle 101 operations
`and/or performance, as well as data related to motion, navi
`gation, etc. of the vehicle 101. For example, collected data
`115 could include data 115 concerning a vehicle 101 speed,
`acceleration, braking, detection of road attributes Such as
`those mentioned above, weather conditions, etc.
`As mentioned above, a vehicle 101 may send and receive
`one or more vehicle-to-vehicle (v2v) communications 112.
`Various technologies, including hardware, communication
`protocols, etc., may be used for vehicle-to-vehicle communi
`cations. For example, V2V communications 112 as described
`herein are generally packet communications and could be
`sent and received at least partly according to Dedicated Short
`Range Communications (DSRC) or the like. As is known,
`DSRC are relatively low-power operating over a short to
`medium range in a spectrum specially allocated by the United
`States government in the 5.9 GHz band.
`A V2V communication 112 may include a variety of data
`concerning operations of a vehicle 101. For example, a cur
`rent specification for DSRC, promulgated by the Society of
`Automotive Engineers, provides for including a wide variety
`of vehicle 101 data in a V2v communication 112, including
`vehicle 101 position (e.g., latitude and longitude), speed,
`heading, acceleration status, brake system status, transmis
`sion status, steering wheel position, etc.
`Further, V2v communications 112 are not limited to data
`elements included in the DSRC standard, or any other stan
`dard. For example, a V2v communication 112 can include a
`
`25
`
`30
`
`35
`
`40
`
`50
`
`60
`
`65
`
`IPR2025-00943
`Tesla EX1004 Page 7
`
`

`

`5
`wide variety of collected data 115 obtained from a vehicle
`101 data collectors 110, such as camera images, radar or lidar
`data, data from infrared sensors, etc. Accordingly, a first
`vehicle 101 could receive collected data 115 from a second
`vehicle 101, whereby the first vehicle 101 computer 105
`could use the collected data 115 from the second vehicle 101
`as input to the autonomous module 106 in the first vehicle
`101, i.e., to determine autonomous or semi-autonomous
`operations of the first vehicle 101, such as how to execute a
`“limp home” operation or the like and/or how to continue
`operations even though there is an indicated fault or faults in
`one or more data collectors 110 in the first vehicle 101.
`A V2v communication 112 could include mechanisms
`other than RF communications, e.g., a first vehicle 101 could
`provide visual indications to a second vehicle 101 to make a
`v2v communication 112. For example, the first vehicle 101
`could move or flash lights in a predetermined pattern to be
`detected by camera data collectors or the like in a second
`vehicle 101.
`A memory of the computer 105 may further store one or
`more parameters 117 for comparison to confidence assess
`ments 118. Accordingly, a parameter 117 may define a set of
`confidence intervals; when a confidence assessment 118 indi
`cates that a confidence value falls within a confidence interval
`at or passed a predetermined threshold, such threshold also
`specified by a parameter 117, then the computer 105 may
`include instructions for providing an alert or the like to a
`vehicle 101 operator.
`In general, a parameter 117 may be stored in association
`with an identifier for a particular user or operator of the
`vehicle 101, and/or a parameter 117 may be generic for all
`operators of the vehicle 101. Appropriate parameters 117 to
`be associated with a particular vehicle 101 operator, e.g.,
`according to an identifier for the operator, may be determined
`in a variety of ways, e.g., according to operator age, level of
`driving experience, etc. As mentioned above, the computer
`101 may use mechanisms, such as a signal from a hardware
`device identifying a vehicle 101 operator, user input to the
`computer 105 and/or via a device 150, biometric collected
`data 115, etc., to identify a particular vehicle 101 operator
`whose parameters 117 should be used.
`Various mathematical, statistical and/or predictive model
`ing techniques could be used to generate and/or adjust param
`eters 117. For example, a vehicle 101 could be operated
`autonomously while monitored by an operator. The operator
`could provide input to the computer 105 concerning when
`autonomous operations appeared safe, and when unsafe. Vari
`ous known techniques could then be used to determine func
`tions based on collected data 115 to generate parameters 117
`and assessments 118 to which parameters 118 could be com
`pared.
`Confidence assessments 118 are numbers that may be gen
`erated according to instructions stored in a memory of the
`computer 105 in a vehicle 101 using collected data 115 from
`the vehicle 101. Confidence assessments 118 are generally
`provided in two forms. First, an overall confidence assess
`ment 118, herein denoted as did, may be a continuously or
`nearly continuously varying value that indicates an overall
`confidence that the vehicle 101 can and/or should be operated
`autonomously. That is, the overall confidence assessment 118
`may be continuously or nearly continuously compared to a
`parameter 117 to determine whether the overall confidence
`meets or exceed a threshold provided by the parameter 117.
`Accordingly, the overall confidence assessment 118 may
`serve as an indicia of whether, based on current collected data
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 9,406,177 B2
`
`10
`
`15
`
`6
`115, a vehicle 101 should be operated autonomously, may be
`provided as a scalar value, e.g., as a number having a value in
`the range of 0 to 1.
`Second, one or more vector of autonomous attribute
`assessments 118 may be provided, where each value in the
`vector relates to an attribute and/or of the vehicle 101 and/or
`a Surrounding environment related to autonomous operation
`of the vehicle 101, e.g., attributes such as vehicle speed,
`braking performance, acceleration, steering, navigation (e.g.,
`whether a map provided for a vehicle 101 route deviates from
`an actual arrangement of roads, whether unexpected con
`struction is encountered, whether unexpected traffic is
`encountered, etc.), weather conditions, road conditions, etc.
`In general, various ways of estimating confidences and/or
`assigning values to confidence intervals are known and may
`be used to generate the confidence assessments 118. For
`example, various vehicle 101 data collectors 110 and/or sub
`systems may provide collected data 115, e.g., relating to
`vehicle speed, acceleration, braking, etc. For example, a data
`collector 110 evaluation of likely accuracy, e.g., sensor accu
`racy, could be determined from collected data 115 using
`known techniques. Further, collected data 115 may include
`information about an external environment in which the
`vehicle 101 is traveling, e.g., road attributes such as those
`mentioned above, data 115 indicating a degree of accuracy of
`map data being used for vehicle 101 navigation, data 115
`relating to unexpected road construction, traffic conditions,
`etc. By assessing Such collected data 115, and possibly
`weighting various determinations, e.g., a determination of a
`sensor data collector 110 accuracy and one or more determi
`nations relating to external and/or environmental conditions,
`e.g., presence or absence of precipitation, road conditions,
`etc., one or more confidence assessments 118 may be gener
`ated providing one or more indicia of the ability of the vehicle
`101 to operate autonomously.
`An example of a vector of confidence estimates 118
`include a vector (p =({p, q}, ..., (b,f), relating to the
`vehicle 101 perceptual layer (PL), where n is a number of
`perceptual Sub-systems, e.g., groups of one or more sensor
`data collectors 110, in the PL. Another example of a vector of
`confidence estimates 118 includes a vector (p'=(p",
`(p', ..., p.'), relating to the vehicle 101 actuation layer
`(AL), e.g., groups of one or more actuator data collectors 110.
`in the AL.
`In general, the vector (p may be generated using one or
`more known techniques, including, without limitation, Input
`Reconstruction Reliability Estimate (IRRE) for a neural net
`work, reconstruction error of displacement vectors in an opti
`cal flow field, global contrast estimates from an imaging
`system, return signal to noise ratio estimates in a radar sys
`tem, internal consistency checks, etc. For example, a Neural
`Network road classifier may provide conflicting activation
`levels for various road classifications (e.g., single lane, two
`lane, divided highway, intersection, etc.). These conflicting
`activations levels will result in PL data collectors 110 report
`ing a decreased confidence estimate from a road classifier
`module in the PL. In another example, radar return signals
`may be attenuated due to atmospheric moisture Such that
`radar module reports low confidence in estimating the range,
`range-rate or azimuth of neighboring vehicles.
`Confidence estimates may also be modified by the PL
`based on knowledge obtained about future events. For
`example, the PL may be in real-time communication with a
`data service, e.g., via the server 125, that can report weather
`along a planned or projected vehicle 101 route. Information
`about a likelihood of weather that might adversely affect the
`PL (e.g., heavy rain or Snow) can be factored into the confi
`
`IPR2025-00943
`Tesla EX1004 Page 8
`
`

`

`7
`dence assessments 118 in the vector (p' in advance of actual
`degradation of sensor data collector 110 signals. In this way
`the confidence assessments 118 may be adjusted to reflect not
`only the immediate sensor state but also a likelihood that the
`sensor State may degrade in the near future.
`Further, in general the vector (p" may be generated by
`generally known techniques that include comparing a com
`manded actuation to resulting vehicle 101 performance. For
`example, a measured change in lateral acceleration for a given
`commanded steering input (steering gain) could be compared
`to an internal model. If the measured value of the steering gain
`varies more than a threshold amount from the model value,
`then a lower confidence will be reported for that subsystem.
`Note that lower confidence assessments 118 may or may not
`reflect a hardware fault; for example, environmental condi
`tions (e.g., wet or icy roads) may lower a related confidence
`assessment 118 even though no hardware failure is implied.
`When an overall confidence assessment 118 for a specified
`value or range of values, e.g., a confidence interval, meets or
`exceeds a predetermined threshold within a predetermined
`margin of error, e.g., 95 percent plus or minus three percent,
`then the computer 105 may include instructions for providing
`a message 116, e.g., an alert, via the affective interface 119.
`That is, the affective interface 119 may be triggered when the
`overall confidence assessment 118 (d) drops below a speci
`fied predetermined threshold dd. When this occurs, the
`affective interface 119 formulates a message 116 (M) to be
`delivered to a vehicle 101 operator. The message 116 M
`generally includes two components, a semantic content com
`ponent S and an urgency modifier U. Accordingly, the inter
`30
`face 119 may include a speech generation module, and inter
`active voice response (IVR) system, or the like, such as are
`known for generating audio speech. Likewise, the interface
`119 may include a graphical user interface (GUI) or the like
`that may display alerts, messages, etc., in a manner to convey
`a degree of urgency, e.g., according to a fontsize, color, use of
`icons or symbols, expressions, size, etc., of an avatar or the
`like, etc.
`Further, confidence attribute Sub-assessments 118, e.g.,
`one or more values in a vector p or p", may relate to
`particular collected data 115, and may be used to provide
`specific content for one or more messages 116 via the inter
`face 119 related to particular attributes and/or conditions
`related to the vehicle 101, e.g., a warning for a vehicle 101
`occupant to take over steering, to institute manual braking, to
`take complete control of the vehicle 101, etc. That is, an
`overall confidence assessment 118 may be used to determine
`that an alert or the like should be provided via the affective
`interface 119 in a message 116, and it is also possible that, in
`addition, specific content of the message 116 alert may be
`based on attribute assessments 118. For example, message
`116 could be based at least in part on one or more attribute
`assessments 118 and could be provided indicating that
`autonomous operation of a vehicle 101 should cease, and
`alternatively or additionally, the message 116 could indicate
`as content a warning Such as "caution: slick roads, or “cau
`tion: unexpected lane closure ahead. Moreover, as men
`tioned above and explained further below, emotional prosody
`may be used in the message 116 to indicate a level of urgency,
`concern, or alarm related to one or more confidence assess
`ments 118.
`In general, a message 116 may be provided by the com
`puter 105 when d-d (note that appropriate hysteresis may
`be accounted for in this evaluation to prevent rapid switch
`ing). Further, when it is determined that d(dB
`components
`of each of the vectors (p and p' may be evaluated to deter
`mine whether a value of the vector component falls below a
`
`50
`
`8
`predetermined threshold for the vector component. For each
`vector component that falls below the threshold, the computer
`105 may formulate a message 116 to be provi