I 1111111111111111 1111111111 lllll 111111111111111 11111 111111111111111 IIII IIII
`
`
`
`
`
`
`
`US010377303B2
`
`c12) United States Patent
`McNew et al.
`
`US 10,377,303 B2
`(IO) Patent No.:
`(45) Date of Patent:
`Aug. 13, 2019
`
`(54) MANAGEMENT OF DRIVER AND VEHICLE
`MODES FOR SEMI-AUTONOMOUS
`DRIVING SYSTEMS
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`(71) Applicant: Toyota Motor Engineering &
`Manufacturing North America, Inc.,
`Erlanger, KY (US)
`
`(72)
`
`Inventors: John Michael McNew, Ann Arbor, MI
`(US); Vladimeros Vladimerou,
`Ypsilanti, MI (US)
`
`(73) Assignee: Toyota Motor Engineering &
`Manufacturing North America, Inc.,
`Erlanger, KY (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 23 days.
`
`(21) Appl. No.: 14/477,445
`
`(22) Filed:
`
`Sep. 4, 2014
`
`(65)
`
`Prior Publication Data
`
`US 2016/0068103 Al
`
`Mar. 10, 2016
`
`(51)
`
`Int. Cl.
`B60Q 9/00
`B60W50/14
`B60W 30/16
`B60W 30112
`(52) U.S. Cl.
`CPC ............... B60Q 9/00 (2013.01); B60W 30112
`(2013.01); B60W 30/16 (2013.01); B60W
`50/14 (2013.01)
`
`(2006.01)
`(2012.01)
`(2012.01)
`(2006.01)
`
`( 58) Field of Classification Search
`CPC ...................................................... G08B 21/02
`See application file for complete search history.
`
`Semi-aLl\unomous
`rnode1nc1:,::.2Pment
`
`7,479,892 B2
`8,103,398 B2
`8,428,843 B2
`8,577,550 B2
`8,655,537 B2
`8,670,903 B2
`8,823,530 B2 *
`
`2008/0055114 Al*
`
`2008/0091318 Al
`2010/0082195 Al*
`
`1/2009
`1/2012
`4/2013
`11/2013
`2/2014
`3/2014
`9/2014
`
`Ling et al.
`Duggan et al.
`Lee et al.
`Lu et al.
`Ferguson et al.
`Lee et al.
`Green
`
`4/2008 Deng et al.
`4/2010 Lee .
`
`B60K28/06
`340/576
`3/2008 Kim .......................... B60R 1/00
`340/937
`
`B62D 15/025
`701/25
`B60W 40/08
`180/272
`
`G08G 1/16
`701/41
`B60W 30/0953
`701/301
`
`2011/0284304 Al * 11/2011 Van Schoiack .
`
`2012/0206252 Al
`2012/0283913 Al
`2013/0131906 Al
`2013/0253767 Al
`2014/0195120 Al*
`
`8/2012 Sherony et al.
`11/2012 Lee et al.
`5/2013 Green et al.
`9/2013 Lee et al.
`7/2014 McClain
`
`.
`
`2015/0314783 Al*
`
`11/2015 Nespolo .
`
`* cited by examiner
`
`Jelani A Smith
`Primary Examiner -
`Assistant Examiner - Kelly D Williams
`(74) Attorney, Agent, or Firm - Obion, McClelland,
`Maier & Neustadt, L.L.P.
`ABSTRACT
`(57)
`A least restrictive allowable driving state of a semi-autono(cid:173)
`mous driving system is determined based on one or more
`threats and sensor performance. A current driving state and
`a future driving state are determined based on an attention
`state and a steering state of a driver. Warnings are provided
`to the driver in order to match the current driving state to the
`future driving state. Driver interaction and attention are
`enforced when the driver does not respond to the warnings.
`20 Claims, 15 Drawing Sheets
`
`SiC14
`
`Cst1maternrrei"!l
`driving states
`
`3106 Ge~eratewarnmgs \
`!
`
`I
`S108!
`
`~~-~
`
`IPR2025-00943
`Tesla EX1006 Page 1
`
`

`

`U.S. Patent
`
`Aug. 13, 2019
`
`Sheet 1 of 15
`
`US 10,377,303 B2
`
`Semi-autonomous
`mode management
`(SAMM) process
`
`100~
`
`Start
`
`1
`
`S102
`
`Estimate allowable
`driving states
`
`,,
`
`S104
`
`Estimate current
`driving states
`
`"
`
`S106
`
`Generate warnings
`
`•
`Offer appropriate
`level of automation
`
`S108
`
`,,
`
`S110
`
`Warning
`enforcement
`
`',
`( End
`
`Fig. 1
`
`IPR2025-00943
`Tesla EX1006 Page 2
`
`

`

`U.S. Patent
`
`Aug. 13, 2019
`
`Sheet 2 of 15
`
`US 10,377,303 B2
`
`Sen1i-aut,onom()i.JS
`mmle manal)er
`(&\MM)
`........_,
`
`"
`
`( Start j
`
`.... "~:
`,C:,C i MapfGPS
`
`21,q
`
`Rat!ars i----
`2161 Cameras t
`
`Tr.rem !O and
`TC Calcuia!ion
`
`Catu!ata
`Alk>waMe
`Dri'ling Stales
`swz
`
`Driving State
`D;,,-erminaii,.m
`
`D;lver Monitoring .
`Gam,mi ___
`
`224
`
`____,
`
`226[
`!_ ___
`
`HMl
`., .. ~•
`
`Cabi,aie
`Er<fwcerrret:t or •-··
`E~errrent
`
`S210
`
`Fig. 2
`
`Wilick:
`(;(r('.l;Oi
`Determination
`SW8
`
`IPR2025-00943
`Tesla EX1006 Page 3
`
`

`

`U.S. Patent
`
`Aug. 13, 2019
`
`Sheet 3 of 15
`
`US 10,377,303 B2
`
`$314
`HASIS" '
`DFWERJS
`_STEERING
`
`~---S3~'~~1cc,B
`(l Warnings i
`HAO "
`
`
`'£.YES __ ON_J;:OAD .. W/'l;:Nj
`
`S32G
`
`S316
`~N_o _ ____,..,i HASiS ~ DRIVER __ MiJST _STEER 1---------------l
`HAD O NONE
`
`S322
`HAD :-:NONE
`
`i
`DALC c MANU/\L,
`Ne
`~ HASIS0 LTC.8LOCKED_BY __ STEERING.
`HAO 0 NONE
`i
`/ Does CSS >--.......__
`Yes
`•• 7 -to-~RIVER.JS_STEERING)
`'-.!i>ANU/.l.?.
`'-..__ .NOW? _.,..,,-
`/_....'-........_
`~ ..
`,
`,
`"-...
`'-..__
`//
`, .. ..._.,_ S32i3
`_a,,H/.>t) • ~
`.?c,c·>-'-'-3t'-, __._ _ __,
`
`./
`'-,
`{
`i,:
`.WARt,....___ . i
`<,.. Or LTC_BLOCKEG __ BY. ;;:-,o[• warrmir· I
`Future
`_,../(EYES __ ON_ROAD
`Ye
`l
`i''0
`'
`Vl/aming
`'°oo)
`-._____/
`
`Yes
`
`$324
`
`3/~
`
`S304
`Aloes cur~
`
`< L'.C Cor.!rol O
`
`I,,,
`
`""-
`
`determination
`i..
`
`S3Q8
`
`Fig. 3
`
`lt1ATTENTl0N' &&
`DRIVER EYES
`~•-lOT __ oN:_ROAD).,../
`//
`.
`Yes
`'-------.i
`
`Wammgs m:ght occur
`
`i
`~
`derermmation
`i
`
`S3:)2 /
`END ;
`DALC ~ LC ACTlVE :~
`HAS:8 ° HANDS FR.ff OK : \.... ........ ,,
`HAD O NON£
`
`$328
`
`•
`~-------~----,
`DALC C MANU/\L
`
`HA.SIS~ DRIVER.JS .. STEERING
`HAD~ l TC __ BLOCKEDJW
`.. INATTENTION
`
`IPR2025-00943
`Tesla EX1006 Page 4
`
`

`

`U.S. Patent
`
`Aug. 13, 2019
`
`Sheet 4 of 15
`
`US 10,377,303 B2
`
`\,'\faming
`determination
`S308
`
`Start
`
`S400
`
`Order SAFETY
`Threats By lowest
`TC
`
`S402
`
`i,
`
`No HMI
`
`Process Threat -
`Ust
`
`S406 ,~
`Order COMFORT
`Threats By lowest
`TC
`
`S408 ,,
`
`N oHM!
`
`Process Threat
`Ust
`
`I--
`
`HM! Warning
`Generated
`
`S410
`
`1
`
`DALC = LTC
`HASIS ::-: HANDS_FREE_OK
`HAD= NONE
`
`"
`
`~
`
`ENO
`
`Fig. 4A
`
`IPR2025-00943
`Tesla EX1006 Page 5
`
`

`

`U.S. Patent
`
`Aug. 13, 2019
`
`Sheet 5 of 15
`
`US 10,377,303 B2
`
`Process Threat List
`8402. S408
`
`,,.---
`~
`Start
`
`DALC = LTCj,CTlVE S420
`
`Fig. 4B
`
`EYES __ ON
`ROAD
`
`3434
`
`[
`
`,, ;::, "
`.... HANDS FR.EE OK
`
`8436
`
`Yes
`
`8438
`HASlS=
`0:,s,red Action
`
`0
`
`D,
`
`IPR2025-00943
`Tesla EX1006 Page 6
`
`

`

`U.S. Patent
`
`Aug. 13, 2019
`
`Sheet 6 of 15
`
`US 10,377,303 B2
`
`C
`
`D
`
`Fig. 4B Cont
`
`Yes
`
`Yes
`
`No
`
`S458
`
`HASIS = HANDS..FREE .. OK
`HAD" EYES .. ON_ROAD_WARN
`
`$454
`
`HAD=
`
`EYES_ ON_ROAD jJARN
`
`HANDS_ON
`HAD,,,NONE
`
`S452
`HAD
`
`=EYES_ON .. ROAD
`_WARN
`
`$462---
`NoHMI
`
`IPR2025-00943
`Tesla EX1006 Page 7
`
`

`

`U.S. Patent
`
`Aug. 13, 2019
`
`Sheet 7 of 15
`
`US 10,377,303 B2
`
`Eyes on road
`warning
`determination
`8318 ~
`
`Start
`
`S500
`
`Order SAFETY
`Threats By lowest
`TC
`
`S502
`
`'"
`Process Threat
`List
`
`-
`
`No HM!
`
`S504 'I'
`Order COMFORT
`Threats By lowest
`TC
`
`8506 H
`
`No HM!
`
`Process Threat
`......
`list
`
`HM! Warning
`Generated
`
`S508 ,
`
`HAD= NONE
`
`....
`
`,.
`
`1,
`
`END )
`
`Fig. 5A
`
`IPR2025-00943
`Tesla EX1006 Page 8
`
`

`

`U.S. Patent
`
`Aug. 13, 2019
`
`Sheet 8 of 15
`
`US 10,377,303 B2
`
`Process Threat list
`S502,S506
`
`Start
`
`Yes
`
`No
`
`S516
`
`No
`
`HAD=
`EYES_ON_ROAD_WARN
`
`S518
`HM! Warning Generated
`
`S520
`
`NoHMI
`
`Fig. 5B
`
`IPR2025-00943
`Tesla EX1006 Page 9
`
`

`

`U.S. Patent
`
`Aug. 13, 2019
`
`Sheet 9 of 15
`
`US 10,377,303 B2
`
`Future warning
`determination ~
`S330 ~
`
`( Start
`
`Fig . 6A
`
`arning might
`HM!w
`beg enerated
`
`S600 "
`
`Order SAFETY
`Threats By lowest TC
`
`8602
`
`w
`
`N oHMI
`
`Process Threat -
`List
`
`I
`Ustdone
`!
`8604
`
`Order Cotv1FORT
`Threats By lowest TC
`
`S606
`
`,
`
`Process Threat -
`List
`
`N oHMI
`
`S608
`
`I
`List Done
`-1-
`OALC = LTC
`HASIS = HANDS_FREE_OK
`HAD= NONE
`
`S610
`'
`DALC = MANUAL
`HASIS=
`DR!VER_IS_8TEERING
`
`~
`
`'
`
`END
`
`IPR2025-00943
`Tesla EX1006 Page 10
`
`

`

`U.S. Patent
`
`Aug. 13, 2019
`
`Sheet 10 of 15
`
`US 10,377,303 B2
`
`Precess Threat Ust
`S602,S606
`
`-----....
`
`Fig. 6B
`
`
`
`~-"D-""H'-"-lV=ER'-'-,··=SE:;.;...N=SE=D~. How is threat sensed? SYSTEM-SENSED
`
`HANDS_ON
`
`HANDS_ON
`
`8618
`
`s TC -tff .! WC_DAT +
`WC_HOWT?
`/
`
`No
`
`A.
`
`B
`
`C
`
`IPR2025-00943
`Tesla EX1006 Page 11
`
`

`

`U.S. Patent
`
`Aug. 13, 2019
`
`Sheet 11 of 15
`
`US 10,377,303 B2
`
`$622
`
`is TG - t;T 5"-..>Yes
`/
`<_ WC_DA/
`
`No
`
`(E\
`
`No
`
`S628
`HADz NONE
`
`No
`
`~--'!'----Joi:
`
`HAD=
`1 LTC __ Bl.OCKED __ BY__INA TTENTION
`
`S632
`HMI warning might be ---~
`\._
`generated _,.
`
`~---
`NoHMl
`
`Fig. 6B Cont
`
`IPR2025-00943
`Tesla EX1006 Page 12
`
`

`

`U.S. Patent
`
`Aug. 13, 2019
`
`Sheet 12 of 15
`
`US 10,377,303 B2
`
`Enforcement and
`engagement
`calculation ~
`
`Fig. 7A
`
`Start
`
`S702 Update Timers
`
`Yes
`
`S706
`Is scenario
`safe for
`
`Yes
`
`Is CLC = Manual,
`DALC = LTC_ACTIVE,
`and
`are HANDS_OFF?
`
`No
`
`8712
`Engagement =
`None
`
`S714
`
`Enforcement
`intervention
`
`No
`
`8720
`Engagement =
`None
`
`S710
`
`Disengage
`
`S718
`
`Engage
`
`END
`
`IPR2025-00943
`Tesla EX1006 Page 13
`
`

`

`U.S. Patent
`
`Aug. 13, 2019
`
`Sheet 13 of 15
`
`US 10,377,303 B2
`
`Steeering Timer
`
`Process I
`
`Start
`
`S726
`Steering Timer =1---------------------~
`OFF
`S730
`Add new enforcenmenl
`event to enforcement history
`
`No
`
`S?22 Does HASIS :::
`DRIVER_MUST _STEER or
`DRIVER_MUST _
`HANDS_ON?
`
`S728
`Steering Timer=
`ON
`
`No
`
`Yes
`
`END
`
`S734
`Steering Timer "
`EXPIRED
`
`Fig. 7B
`
`IPR2025-00943
`Tesla EX1006 Page 14
`
`

`

`U.S. Patent
`
`Aug. 13, 2019
`
`Sheet 14 of 15
`
`US 10,377,303 B2
`
`Fig. 8
`800
`/
`
`I ______________________ / -----------------------1
`
`812
`
`809
`
`I
`
`804
`
`I
`!
`I
`I
`I
`I 810
`I
`!
`I
`
`:
`~~o
`! __ I __ .?____1"~ _r_ __ .? __ \~ _ _:
`
`1
`1
`
`I
`
`I
`I
`
`808~
`
`,,.~,
`
`:
`
`~~.,,L---..
`
`__
`
`~ l_J
`
`,__ _
`
`___,
`
`IPR2025-00943
`Tesla EX1006 Page 15
`
`

`

`U.S. Patent
`
`Aug. 13, 2019
`
`Sheet 15 of 15
`
`US 10,377,303 B2
`
`Processing
`System
`
`~
`
`Monitor
`
`Display
`Controller
`
`Memory
`
`Disk
`Controller
`
`Sensor 1
`
`Sensor 2
`
`Sensor N
`
`I
`i
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`BUS
`
`Network
`Interface
`
`CPU
`
`~---------------·
`
`e
`
`1/0
`Interface :======.=;--Actuator(s)
`
`----:
`
`GPS
`
`Peripheral(s)
`
`Fig. 9
`
`IPR2025-00943
`Tesla EX1006 Page 16
`
`

`

`US 10,377,303 B2
`
`1
`MANAGEMENT OF DRIVER AND VEHICLE
`MODES FOR SEMI-AUTONOMOUS
`DRIVING SYSTEMS
`
`BACKGROUND
`
`A semi-autonomous driving system can manage lane
`centering and automatic cruise control based on current and
`future driving states. U.S. Patent Application Publication
`No. 2013/0253767 Al entitled "System and Method for
`Vehicle Lateral Control" to Lee et al. describes a method for
`lane correction and torque-assisted steering based on a
`planned path to reduce driver input.
`
`SUMMARY
`
`The present disclosure is directed to managing vehicle
`control states of a semi-autonomous vehicle while simulta(cid:173)
`neously managing a driving state of a driver, which can
`include driver steering interaction as well as driver attention
`to the road. In an exemplary implementation, a least restric(cid:173)
`tive allowable driving state of a semi-autonomous driving
`system is determined based on one or more threats and
`sensor performance. A current driving state and a future
`driving state are determined based on an attention state and 25
`a steering state of a driver. Warnings are provided to the
`driver in order to match the current driving state to the future
`allowable driving state. Furthermore, driver interaction and
`attention are enforced when the driver does not respond to
`the warnings.
`The threats can be determined to be a safety threat or a
`comfort threat and can be driver-sensed or system-sensed.
`Each of the threats can have a desired action by the driver.
`The priority of the threats can be determined based on an
`associated time of occurrence of the threats. A driver 35
`response time to the threats can be based on a driver
`attention time, an eyes on the road time, and a hands-on(cid:173)
`the-wheel time. The least restrictive allowable driving state
`can be based on a comparison of the time of occurrence of
`the threats to the driver response time.
`The future driving state can be set to the least restrictive
`allowable driving state if it is determined that the driver has
`complied with one or more previous requests by the semi(cid:173)
`autonomous driving system and if the future driving state
`will not generate one or more warnings within a predeter(cid:173)
`mined time period.
`The warnings can include attention warnings and steering
`warnings and can be provided when the future driving state
`is more restrictive than at least one of the current driving
`state and a current lane trace control state. The current lane 50
`trace control state can be determined based on a level of
`automation of a lane trace control system.
`An amount of time the driver takes to comply with the
`warnings can be determined, and driving interaction can be
`enforced when the amount of time the driver takes to comply 55
`with the warnings is greater than a first predetermined
`threshold. The driving interaction can be enforced by reduc-
`ing a speed of an automatic cruise control system.
`The number of enforcement events in an enforcement
`history can be compared to a second predetermined thresh-
`old, and a safe scenario for disengaging the semi-autono(cid:173)
`mous driving system can be determined. The semi-autono(cid:173)
`mous driving system can be disengaged when the number of
`the enforcement events in the enforcement history is greater
`than the second predetermined threshold.
`A process can include: determining a least restrictive
`allowable driving state of a semi-autonomous driving sys-
`
`2
`tern based on one or more threats and sensor performance;
`determining a current driving state and a future driving state
`based on an attention state and a steering state of a driver;
`providing warnings to the driver to match the current driving
`5 state to the future driving state; and enforcing driving
`interaction or attention when the driver is not responding to
`the warnings.
`storage medium
`A non-transitory computer-readable
`including executable instructions, which when executed by
`10 circuitry, can cause circuitry to perform the process.
`A system can include: one or more sensors to obtain threat
`data and driver monitoring data; and a controller including
`a processor. The processor can: determine a least restrictive
`15 allowable driving state of a semi-autonomous driving sys(cid:173)
`tem based on one or more threats and sensor performance;
`determine a current driving state and a future driving state
`based on an attention state and a steering state of a driver;
`provide warnings to the driver to match the current driving
`20 state to the future driving states; and enforce driving inter(cid:173)
`action or attention when the driver is not responding to the
`warnings.
`The system can be a part of vehicle system or a vehicle
`sub-system, and can be removable from the vehicle as a
`detachable module.
`The foregoing general description of the illustrative
`implementations and the following detailed description
`thereof are merely exemplary aspects of the teachings of this
`disclosure, and are not restrictive.
`
`30
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`40
`
`A more complete appreciation of this disclosure and many
`of the attendant advantages thereof will be readily obtained
`as the same becomes better understood by reference to the
`following detailed description when considered in connec(cid:173)
`tion with the accompanying drawings, wherein:
`FIG. 1 is an exemplary illustration of a semi-autonomous
`mode management process;
`FIG. 2 is an exemplary illustration of a semi-autonomous
`mode manager (SAMM);
`FIG. 3 is an exemplary flowchart of a calculation of
`allowable driving states;
`FIGS. 4A and 4B are exemplary flowcharts of a warning
`45 determination;
`FIGS. SA and SB are exemplary flowcharts of an eyes on
`the road warning determination;
`FIGS. 6A and 6B are exemplary flowcharts of a future
`warning determination;
`FIG. 7A illustrates a flowchart of an enforcement and
`engagement calculation;
`FIG. 7B is an exemplary flowchart of a steering timer
`process;
`FIG. 8 is an exemplary illustration of an interior com(cid:173)
`partment of a motor vehicle; and
`FIG. 9 schematically illustrates a processing system for a
`controller and/or a computer system.
`
`DETAILED DESCRIPTION
`
`60
`
`In the drawings, like reference numerals designate iden(cid:173)
`tical or corresponding parts throughout the several views.
`Further, as used herein, the words "a," "an" and the like
`generally carry a meaning of "one or more," unless stated
`65 otherwise. The drawings are generally drawn to scale unless
`specified otherwise or illustrating schematic structures or
`flowcharts.
`
`IPR2025-00943
`Tesla EX1006 Page 17
`
`

`

`US 10,377,303 B2
`
`10
`
`3
`Furthermore, the terms "approximately," "proximate,"
`and similar terms generally refer to ranges that include the
`identified value within a margin of 20%, 10%, or preferably
`5%, and any values therebetween.
`In semi-autonomous driving systems, steering control,
`speed control, and sensing systems can be controlled by
`processing circuitry within a vehicle. In some aspects,
`activation of the semi-autonomous driving systems can be
`limited to one or more operating regions, such as when
`driving on highways or in low traffic areas. In some imple-
`mentations, a semi-autonomous mode manager (SAMM)
`can interface between vehicle control systems and a human
`machine interface (HMI) to manage allowable driving states
`based on current and future driving scenarios as well as an
`attention state of a driver. The SAMM can manage when 15
`lane trace control (LTC) and/or automatic cruise control
`(ACC) is offered to the driver, generate warnings and/or
`requests to prompt the driver to interact with the steering
`wheel and/or pay attention to the road, and enforce driver
`attention or steering interaction when the driver is not 20
`responding to the warnings. The attention state of the driver
`can be based on the driver having eyes on the road as
`determined by head and gaze angle and eye closure mea(cid:173)
`surements, interacting with a steering wheel, brake pedal, or
`gas pedal, or explicitly setting the cruise control.
`FIG. 1 is an exemplary illustration of a semi-autonomous
`mode management process 100. At step S102, an estimation
`of a least restrictive allowable driving state is made by the
`processing circuitry of the SAMM. The SAMM can receive
`input from sensors such as radar, lidar (light detection and 30
`ranging), vehicle-mounted cameras, GPS, stored maps, and
`the like in order to determine current and upcoming threats.
`The sensor data that is used to determine the threats can be
`referred to as threat data according to an implementation.
`The threats can include any type of scenarios through which
`a consequence could occur, such as lane splits or merges,
`lane marker degradation, and the like. In some implemen(cid:173)
`tations, the threats have associated criteria for identifying the
`threats that are stored in a database and are accessed by the
`processing circuitry. For example, in the case of a lane split,
`the processing circuitry can retrieve lane split information
`from map information stored in the database. In addition, the
`vehicle-mounted camera can obtain images of the road that
`indicate that the vehicle is approaching a lane split. Each
`threat can also have an associated consequence level ( e.g.,
`safety or comfort) and a desired driver state (e.g., driver
`must steer, driver must have hands on steering wheel, or
`driver must look at road). Each threat can also have a time
`to consequence (TC), which is the amount of time the driver
`has to achieve the desired state in order to avoid the
`consequence. The TC can be based on location of the threat,
`speed of the vehicle, and the like.
`The least restrictive allowable driving state can also be
`affected by sensor performance. For example, if the quality
`of images obtained from the vehicle-mounted camera is
`degraded due to weather conditions, confidence in the accu(cid:173)
`racy of the LTC system may be decreased, and the allowable
`driving state may be set to a level that requires a greater
`amount of driver intervention. Examples of allowable driv(cid:173)
`ing
`states can
`include LTC_ACTIVE, CO-STEER,
`MANUAL, or OFF. In some implementations, LTC_AC(cid:173)
`TIVE means that the LTC system is centering the vehicle in
`a lane without driver input. The CO-STEER driving state
`can mean that the LTC system is active but operating at a
`reduced capability, and the driver can be required to place
`hands on the steering wheel to provide additional steering
`torque and/or supervision. The MANUAL driving state can
`
`4
`indicate that the driver is actively steering the vehicle and
`the LTC system on and sensing but not providing steering
`output. The OFF driving state can indicate that the driver is
`actively steering the vehicle and the LTC system is turned
`5 off.
`At step S104, an estimation of a current driving state is
`made. In an implementation, current driver steering control
`states (CSS) can include steering states such as DRIVER_
`IS_STEERING, HANDS_ON, and HANDS_FREE. DRIV(cid:173)
`ER_IS_STEERING can mean that the driver is actively
`steering the vehicle. In some aspects, HANDS_ON can
`mean that the driver's hands are resting on the steering
`wheel but are not actively steering the vehicle. HANDS_
`FREE can mean that the driver's hands are resting some(cid:173)
`where other than the steering wheel. A steering wheel grip
`sensor can detect whether the driver's hands are in contact
`with the steering wheel and can include one or more tactile,
`pressure, or capacitive sensors. A database can store posi(cid:173)
`tions of proper or acceptable hand placements on the steer(cid:173)
`ing wheel or patterns in time or space of hand pressures, and
`detected positions or contact pressures can be compared
`with the stored positions and/or time and space patterns to
`determine whether the driver's hands are properly grasping
`the steering wheel. An optical sensor, such as a camera, can
`25 also detect hand position, either collaboratively with the grip
`sensor or alone. In some aspects, a steering wheel torque
`sensor can detect whether the driver is actively steering the
`vehicle by measuring the speed, direction, and/or force of
`rotation of the steering wheel.
`Another aspect of the current driving state can include an
`attention state of the driver such as whether or not the driver
`has eyes on the road. In some implementations, the deter(cid:173)
`mination of whether or not the driver has eyes on the road
`is made based on processing driver monitoring data from a
`35 driver monitoring camera. The driver monitoring camera
`can detect information related to the driver's current viewing
`angle (i.e., indicia of the viewing angle of the driver). In an
`implementation, the driver monitoring camera can be an
`in-vehicle device such as a video camera, charge coupled
`40 device (CCD) camera, or complementary metal-oxide-semi(cid:173)
`conductor (CMOS) camera that captures images of the
`driver. Other cameras, such as infrared cameras, can also be
`used. The cameras can also be utilized with or replaced by
`time-of-flight sensors, such as a lidar device, that recognize
`45 objects or features. The driver images can be processed to
`obtain information related to the driver's viewing angle.
`In addition, estimations regarding a driver response time
`are also made. In some implementations, Driver Attention
`Time (DAT), is an estimate of the time that may elapse
`50 before the driver looks at (and sees) the road again without
`being warned to do so. Driver Eyes on Road to Warning
`Time (EORW) is an estimate of the time that may elapse
`before the driver looks at (and sees) the road again after
`receiving a warning. Driver Hands-on-the-Wheel Time
`55 (HOWT) is an estimate of the time that may elapse before
`the driver moves his or her hands to the steering wheel. The
`DAT, EORW, and HOWT can be constant values that are
`determined from pre-existing data measured from a popu(cid:173)
`lation of drivers. For example, if the average DAT for a
`60 population of 1000 drivers is approximately 1.5 seconds, the
`DAT can be set to 1.5 seconds.
`The DAT, EORW, and HOWT can also be determined
`based on observations of the driver over time. For example,
`the DAT, EORW, and HOWT can be set to initial values that
`65 can be modified based on observing the driver's reaction
`time over time since different drivers can have reaction times
`that are widely varied. For example, machine learning
`
`IPR2025-00943
`Tesla EX1006 Page 18
`
`

`

`US 10,377,303 B2
`
`5
`
`15
`
`5
`techniques, such as artificial neural networks, clustering, or
`the like, can be implemented to determine the most accurate
`DAT, EORW, and HOWT for a driver.
`At step S106, warnings or requests are generated to
`correct the current driving state if the allowable driving
`states are more restrictive than the current driving states. For
`example, there may be an upcoming threat of a curve in the
`road that is too sharp for the LTC system to navigate and has
`a desired driving state of DRIVER_MUST_STEER. If the
`driver currently has hands off the steering wheel, the SAMM
`can issue a warning to the driver via a human-machine
`interface (HMI) to place his or her hands back on the
`steering wheel. In addition, if the TC for a threat is less than
`the DAT, the SAMM can issue a warning to the driver via the
`HMI to look straight ahead with eyes on the road.
`The HMI can include one or more visual warning devices
`within the vehicle, such as a heads up display or an instru(cid:173)
`ment cluster. In some implementations,
`the HMI visual
`warning devices can be visible to the driver when the driver
`has eyes on the road (i.e., the one or more visual displays are 20
`within the driver's view when the driver is looking straight(cid:173)
`ahead along the forward trajectory of the vehicle). The HMI
`visual warning devices can also be outside the straight-ahead
`view of the driver but within the driver's peripheral vision,
`such as a navigational screen, instrument cluster, rear-view 25
`or side-view mirrors displays, and the like.
`The HMI can also issue auditory warnings that can be
`transmitted by a standard vehicle audio system, an audio
`loudspeaker, a piezoelectric
`transducer, magnetostrictive
`transducer, electrostatic transducer, or other sound trans- 30
`ducer. The auditory warnings can be output as spoken words,
`chimes, alarms, music, or other sound configured to com(cid:173)
`municate
`information or stimulate driver emotion or
`response. In some implementations, the HMI can also issue
`warnings via haptic devices that can stimulate the driver's
`tactile sense. The haptic warning devices can include a
`mechanical transducer connected to a seat, a brake pedal, a
`seat belt, a steering wheel, or other part of the vehicle that
`is in physical contact with the driver. The haptic warning
`devices can also include air blown from a ventilation system 40
`in the vehicle and other forms of tactile communication
`configured to alert or otherwise communicate information or
`stimulate an emotion to the driver.
`At step S108, the SAMM offers a level of automation to
`the driver that is higher than the current level of automation
`if the allowable driving state is less restrictive than the
`current driving state and predetermined criteria are met. In
`order for the higher level of automation to be offered to the
`driver, the driver can be required to comply with previous
`requests from the SAMM, according to some implementa(cid:173)
`tions. In one example, a current vehicle control state is
`MANUAL, and the next allowable vehicle control state is
`LTC_ACTIVE. If the previous warning to the driver was
`"DRIVER_MUST_STEER," and the current driver steering
`control state is "HANDS_FREE,"
`the driver will not be
`offered a higher level of automation until the driver places
`hands on the steering wheel and demonstrates steering
`control of the vehicle. The HMI can
`issue a LTC_
`BLOCKED_BY_STEERING warning
`to the driver
`to
`prompt the driver to demonstrate positive steering control
`before implementing a higher level of automation.
`At step Sll0,
`the SAMM
`implements enforcement
`actions, if necessary, in order to force the driver to comply
`with previously issued warnings. When a warning is issued
`by the HMI, a timer can be set. When the driver acknowl-
`edges the warning by taking the proper action ( e.g., placing
`eyes on the road or steering), the warning stops and the timer
`
`6
`is reset. In some implementations, when the timer reaches a
`threshold before being acknowledged by the driver the
`processing circuitry of the SAMM can send control signals
`to enforce the desired driving state by altering among other
`things the vehicle control. According to some aspects, speed
`enforcement can include lowering the ACC speed to a level
`that may be uncomfortable to the driver or pumping the
`brakes or by inducing uncomfortable lateral motion in order
`to encourage the driver to acknowledge the warning. In
`10 some implementations, enforcement actions can include
`other types of actions that can induce driver compliance with
`the warnings such as using an audio signal that may cause
`discomfort to a driver who is ignoring the warnings from the
`SAMM.
`In addition, a determination can be made that the driver
`may be purposely ignoring the semi-automatic driving sys(cid:173)
`tem if the driver does not comply with the issued warnings
`in a predetermined number of seconds. In some implemen(cid:173)
`tations, the warnings may have been issued repeatedly
`without response by the driver. If the driver has not complied
`with the predetermined number of warnings within a pre-
`determined window of time, the SAMM can issue control
`signals to shut off the semi-automatic driving system in a
`controlled marmer, which can be referred to as disengaging
`the system. In an implementation, the window of time can be
`a predetermined period of time that a driver that has not
`complied with warnings that may indicate that the driver is
`purposely ignoring the semi-autonomous driving system.
`In an example where the driver has been offered a higher
`level of automation, the SAMM can issue a control signal to
`engage the semi-automatic driving system if the driver
`properly acknowledges the offer. For example, if the SAMM
`has offered LTC to the driver, a control signal can be sent to
`engage the LTC system after the driver has lifted his or her
`35 hands from the steering wheel to acknowledge the offer. In
`other implementations acknowledgement of the offer of a
`transition to LTC can occur via tactile interaction with
`buttons or stalks, voice command, etc.
`Details regarding the semi-autonomous mode manage(cid:173)
`ment process 100 are discussed further herein. The present
`disclosure is focused toward describing a semi-autonomous
`mode management system of a vehicle with respect to
`managing a steering state of a semi-autonomous driving
`system in order to provide a more concise description.
`45 Extending the scope of the semi-autonomous mode man(cid:173)
`agement system to include managing speed control states
`and other types of driving states can be seen as straightfor(cid:173)
`ward to one of ordinary skill in the art.
`FIG. 2 is an exemplary illustration of a configuration of a
`50 SAMM, according to an implementation. A calculation of an
`allowable driving state at step S202 is made based on inputs
`received from a threat identification (ID) and TC calculation
`step S204, driving state determination step S206, and a
`vehicle control determination step S208. The processing
`55 circuitry can determine the current lane centering control
`(CLC) state and a driver available LTC control (DALC). In
`certain aspects, the CLC can indicate the current state of the
`LTC
`system,
`such
`as LTC_ACTIVE, CO-STEER,
`MANUAL, or OFF. The DALC can be the driving state that
`60 the driver can transition to in the next time step based on the
`least restrictive allowable driving state and the driver's
`response to previous warnings.
`In addition to determining the allowable driving state at
`step S202, the processing circuitry can send control signals
`to a HMI 226 to display requests or warnings to the driver
`based on current and future driving states. The allowable
`driving states are also utilized in the calculation of enforce-
`
`65
`
`IPR2025-00943
`Tesla EX1006 Page 19
`
`

`

`US 10,377,303 B2
`
`7
`mentor engagement step S210 in order to determine if the
`driver has complied with warnings within a predetermined
`amount of time. Details regarding the calculation of the
`allowable driving state at step S202 are discussed further
`with respect to FIG. 3.
`At step S204, threat identification and TC calculation is
`performed and provided as an input to the calculation of the
`allowable driving state at step S202. Data from maps/GPS
`212, radars 214, cameras 216, vehicle measurements 218,
`and the like can be used to determine the location of threats
`and