`
`
`
`1111111111111111 IIIIII IIIII 111111111111111 lllll 111111111111111 1111111111 1111111111 11111111
`
`
`
`
`US 20160267335Al
`
`c19) United States
`c12) Patent Application Publication
`Hampiholi
`
`c10) Pub. No.: US 2016/0267335 Al
`Sep. 15, 2016
`(43) Pub. Date:
`
`(54) DRIVER DISTRACTION DETECTION
`SYSTEM
`
`(71) Applicant: Harman International Industries,
`Incorporated, Stamford, CT (US)
`
`(72)
`
`Inventor: Vallabha Vasant Hampiholi, Bangalore
`(IN)
`
`(21) Appl. No.: 14/657,070
`
`(22) Filed:
`
`Mar. 13, 2015
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`G06K9/00
`H04N7/18
`B60R 161037
`G06T 7100
`G06T 7120
`G02B 27101
`G06F 3101
`(52) U.S. Cl.
`CPC ........ G06K 9/00845 (2013.01); G02B 2710172
`(2013.01); H04N 71181 (2013.01); H04N 71185
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`
`(2013.01); G06K9/00771 (2013.01); G06K
`9/00597 (2013.01); G06F 3/013 (2013.01);
`G06T 71004 (2013.01); G06T 7120 (2013.01);
`B60R 161037 (2013.01); G02B 2027/0138
`(2013.01); G02B 2027/014 (2013.01); G06T
`2207/30241 (2013.01); G06T 2207/30268
`(2013.01); G06T 2207/30252 (2013.01)
`
`ABSTRACT
`(57)
`Embodiments are described for determining and responding
`to driver distractions. An example in-vehicle computing sys(cid:173)
`tem of a vehicle includes an external device interface com(cid:173)
`municatively connecting the in-vehicle computing system to
`a mobile device, an inter-vehicle system communication
`module communicatively connecting the in-vehicle comput(cid:173)
`ing system to one or more vehicle systems of the vehicle, a
`processor, and a storage device storing instructions execut(cid:173)
`able by the processor to receive image data from the mobile
`device, and determine a driver state based on the received
`image data. The instructions are further executable to receive
`vehicle data from one or more of the vehicle systems, deter(cid:173)
`mine a vehicle state based on the vehicle data, determine a
`distraction severity level based on the driver state and the
`vehicle state, and control one or more devices of the vehicle to
`perform a selected action based on the distraction severity
`level.
`
`102
`
`150
`
`IPR2025-00943
`Tesla EX1007 Page 1
`
`

`

`Patent Application Publication
`
`Sep. 15, 2016 Sheet 1 of 10
`
`US 2016/0267335 Al
`
`...---
`(9
`LL
`
`N
`0
`
`IPR2025-00943
`Tesla EX1007 Page 2
`
`

`

`....
`0 ....
`N
`('D .....
`rJJ =(cid:173)
`0 ....
`'? ....
`
`('D
`
`O'I
`
`N
`~Ul
`
`('D
`rJJ
`
`O'I --- 0
`0 ....
`N
`rJJ
`c
`
`Ul
`~
`~
`--....J
`O'I
`N
`
`> ....
`
`0
`
`0 =
`....
`~ .....
`
`(')
`
`....
`~ .....
`
`(')
`
`0 = ""O = O" -....
`('D = ..... t "e -....
`
`~ .....
`""O
`
`FIG. 2
`
`~
`
`262
`
`CONTROL
`STEERING
`
`'
`!
`
`CONTROLS 261
`
`VEHICLE
`
`I __________________
`l
`
`! TELECOMM. SYS. 236
`
`! CLIMATE CONTROL SYS. 234
`
`i
`i
`
`! AUDIO SYSTEM 232
`
`VEHICLE SYSTEM 231
`
`VEHICLE CONTROL SYSTEM 230
`
`EXTERNAL STORAGE DEVICES 254
`
`BLUETOOTH DEVICES 252
`
`APP. 248 ]°]
`
`.1.1
`
`l.t; EXTERNAL SERVICES 246
`
`I APP. 244 11
`
`l..~OBILE DEVICE 242
`
`EXTERNAL DEVICES 240
`
`'\.
`',r~~~
`.. --"',
`
`~
`
`__ .J--(A)
`
`----:'- ~:;:~
`-~
`/
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`r201
`I
`
`1
`
`TOUCH SCREEN
`
`218
`
`USER INTERFACE
`
`MODULE 222
`
`COMMUNICATION
`
`SYSTEM
`
`INTER-VEHICLE
`
`INTERFACE 212
`
`EXTERNAL DEVICE
`
`ANTENNA 2Q.6_
`
`INTERFACE PROCESSOR 22Q
`
`SUBSYSTEM 211.
`NAVIGATION
`
`204
`
`PROCESSING UNIT
`
`SPEECH
`
`APP(S)
`
`208
`
`STORAGE DEVICE
`
`MEMORY 216
`
`PROCESSOR 214
`
`! ........ CAMERA(S)······___i
`
`SYSTEM
`
`OPERATING
`
`SUBSYSTEM 210
`
`SENSOR
`
`MICROPHONE 202
`
`IN-VEHICLE COMPUTING SYSTEM 200
`
`,---
`
`IPR2025-00943
`Tesla EX1007 Page 3
`
`

`

`O'I --- 0
`0 ....
`N
`rJ'1
`c
`
`Ul
`(,H
`(,H
`-....J
`O'I
`N
`
`> ....
`
`....
`0 ....
`('D .....
`rJ'1 =(cid:173)
`0 ....
`'? ....
`
`0
`
`(,H
`
`('D
`
`O'I
`
`N
`~Ul
`
`('D
`rJ'1
`
`0 =
`....
`~ .....
`
`(')
`
`....
`~ .....
`
`(')
`
`0 = ""O = O" -....
`('D = ..... t "e -....
`
`~ .....
`""O
`
`I ~ r········1
`
`I
`
`,
`
`CAN INTERFACE
`
`AUDIO SUBSYSTEM
`
`318
`
`SUBSYSTEM 316
`
`DISPLAY
`
`DECOMPRESSOR 314
`
`VIDEO
`
`VISUAL ALERT I I AUDIO ALERT I I ii~~~~L
`
`.................. =--~--
`
`'
`
`~--~---··············-.
`
`i
`
`!
`
`! _____________
`
`!
`
`MOTION ANALYSIS
`
`IMAGE CORRECTION
`
`SEVERITY RANKING
`
`ENHANCEMENT
`
`VIDEO
`
`DISTRACTION SEVERITY ANALYSIS 308
`
`DISTRACTION ANALYSIS 306
`
`HEAD UNIT 304a
`
`,--------------------------------------
`
`,300a
`
`CONTROL
`ENGINE
`
`~
`
`~
`
`AUDIOIWARNING
`
`FIG. 3A
`
`SENSOR(S)
`
`OTHER
`
`FAC ING CAMERAS
`FRONT /OUTWARD(cid:173)
`
`WEARABLE DEVICE 302
`
`_,,f-:SI
`
`.... ,-- .. --
`
`~
`
`BLUETOOTH 312
`
`--
`
`,:,/-~:•'
`
`~
`
`WI-Fl 310
`
`-
`
`IPR2025-00943
`Tesla EX1007 Page 4
`
`

`

`....
`0 ....
`('D .....
`rJJ =(cid:173)
`0 ....
`'? ....
`
`0
`
`.i;...
`
`('D
`
`O'I
`
`N
`~Ul
`
`('D
`rJJ
`
`0 =
`....
`~ .....
`
`(')
`
`....
`~ .....
`
`(')
`
`0 = ""O = O" -....
`('D = ..... t "e -....
`
`~ .....
`""O
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-1-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`HEAD UNIT INTERFACE 328 +--------~'
`
`'-----t
`
`-
`
`-
`
`-
`
`-
`
`DECOMPRESSOR 314
`
`...:..._
`! BLUETOOTH 312 !
`. ..... ,
`.
`
`-
`
`-
`
`-
`
`-
`
`VIDEO
`
`i.
`
`,300b
`
`L. .............................................
`
`:
`.i
`l l VEHICLE CONTROL l
`I
`
`: ...............................................
`.!
`i
`i AUDIO ALERT
`l VISUAL ALERT i
`:
`
`t ~
`
`1/ !
`!
`i
`! SEVERITY RANKING !
`:
`i
`
`DISTRACTION SEVERITY ANALYSIS 308
`
`CLOUD COMPUTING DEVICE -3.22_
`
`:
`l MOTION ANALYSIS
`·,
`! ENHANCEMENT
`!
`VIDEO
`~--,------_,
`DISTRACTION ANALYSIS .3Q.6.
`
`VIDEO SCENE
`
`ANALYSIS
`
`IMAGE CORRECTION l
`
`I
`I
`I.--__
`I
`I -----'
`I
`I ~--____,:
`I
`I
`I
`.----------------~
`,---------------------------------------
`
`O'I --- 0
`0 ....
`N
`rJJ
`c
`
`Ul
`(,H
`(,H
`-....J
`O'I
`N
`
`> ....
`
`CONTROL
`ENGINE
`
`VEH ICLEI CONTROL
`
`AUDIO WARNING
`
`~
`•
`
`~
`
`I,
`
`VISUAUWARNING
`
`FIG. 38
`
`FACING CAMERAs ... ..J SENSOR(S)
`FRONT/OUTWARD-l OTHER
`
`• ~ I +
`
`! DISPLAY SUBSYSTEM 316 ! l AUDIO SUBSYSTEM~ ! ! CAN INTERFACE~ i
`BLU;~~~~:ill:
`
`HEAD uN1T 304b
`
`_,..-:
`
`I v101::oco1v1PR1::ssORm ~ cLouD INTERFACE J2§ i
`
`~EARABLE DE~ICE ~ ~--·;:::~~:::
`f
`l,i::;
`• " " " "
`-J-
`,
`WI-Fl 310
`.,___---,
`
`I
`
`-
`
`1
`
`IPR2025-00943
`Tesla EX1007 Page 5
`
`

`

`Patent Application Publication
`
`Sep. 15, 2016 Sheet 5 of 10
`
`US 2016/0267335 Al
`
`---~ 404
`
`PROCESS RECEIVED DATA
`
`r400
`
`START
`
`RECEIVE IMAGE DATA
`FROM MOBILE DEVICE 402
`,-------- --- --- ----,
`:
`IMAGE DATA FROM
`:
`: FRONT-FACING CAMERA :
`,----- ----- --- --- --,
`, _________________
`J
`:
`IMAGE DATA FROM
`:
`: DRIVER-FACING CAMERA :
`, _________________
`J
`
`,-------------------
`'
`I
`: SEND DATA TO CLOUD :
`'-------- ,---------]
`,--------- ----------
`: RECEIVE DATA FROM
`:
`CLOUD
`:
`:
`_________
`J
`----------
`
`DETERMINE IF DRIVER IS
`DISTRACTED (FIG. 5) 406
`
`CALCULATE SEVERITY
`RANK, R (FIG. 6) 410
`
`NO
`
`YES PRESENT VISUAL WARNING
`414
`>----1~
`
`YES PRESENT AUDIO WARNING
`418
`>----1~
`
`YES
`
`PERFORM ENGINE
`CONTROL422
`
`RETURN
`
`FIG. 4
`
`IPR2025-00943
`Tesla EX1007 Page 6
`
`

`

`Patent Application Publication
`
`Sep. 15, 2016 Sheet 6 of 10
`
`US 2016/0267335 Al
`
`START
`
`r500
`
`SEND DRIVER-FACING
`CAMERA DATA 502
`r--------,------,
`: HEAD UNIT : : CLOUD :
`, ________
`JI
`-----1
`
`SEND DATA FROM
`OTHER SENSORS 504
`,--------,------,
`: HEAD UNIT : : CLOUD :
`-----1
`, ________
`JI
`
`SEND FRONT-FACING
`CAMERA DATA 506
`,--------,------,
`: HEAD UNIT:: CLOUD:
`, ________
`JI
`-----1
`
`I ..........
`
`••••••••••••••••••••••••••••••:
`
`DATA PROCESSING 508
`,------------------------------------
`!
`!
`PERFORM DAT A
`i
`i
`CORRECTION
`.
`.
`·-----------------------------------·
`j
`PERFORM DATA
`j
`
`
`.__ _____ L-----~~~~~-?.~-~-~-~-: _______ j ____ ___.
`! PERFORM SATURATION !
`_ __j
`! ___ LEVEL_CORRECTION
`,------------------------------------
`.
`.
`i
`:
`PERFORM DATA
`. ___________________________________
`i
`i
`SMOOTHING
`:
`
`DETERMINE DRIVER
`DISTRACTION 514
`
`ANALYZE DATA TO DETERMINE
`ANALYZE PROCESSED DAT A FOR
`DRIVER, VEHICLE, AND
`
`DISTRACTION INDICATORS 510
`ENVIRONMENTAL STATES 512
`i PERFORM COMPARISON WITH i
`-------------------------------------------·
`!
`i
`! PERFORM VIDEO SCENE !
`HISTORICAL DATA
`·----------------------------------------------·
`ANALYSIS
`:
`:
`·------------------------------------------·
`.
`.
`·------------------------------------------·
`! _ PERFORM _MOTION ANALYSIS _j
`.
`.
`.
`.
`·-------------------------------------------
`~~-:~~~-~~-~~~:-~ _________
`L_ _______
`j
`i DETERMINE OBJECT AND !
`
`! ____ VEHICLE_TRAJECTORIES ____ J
`.
`·------------------------------------------·
`! COMPARE OBJECT AND
`'
`j
`L_ __ ~-~~'.?.~-~-!-~-~~-?_:~-~!~~-----j
`! PERFORM COMPARISON WITH l
`!
`i
`HISTORICAL DATA
`..:
`:_ _________________________________________
`
`FIG. 5
`
`IPR2025-00943
`Tesla EX1007 Page 7
`
`

`

`Patent Application Publication
`
`Sep. 15, 2016 Sheet 7 of 10
`
`US 2016/0267335 Al
`
`r600
`
`RETRIEVE ANALYZED DATA 602
`
`DETERMINE TYPES OF DRIVER
`DISTRACTION 604
`
`ANALYZE/PREDICT ENVIRONMENTAL
`DANGERS/RISKS 606
`
`DETERMINE NUMBER OF DRIVER
`DISTRACTIONS 608
`
`DETERMINE NUMBER OF
`ENVIRONMENTAL DANGERS/RISKS 610
`
`PRIORITIZE AND RANK SEVERITY
`BASED ON 604-610 612
`
`PERFORM ACTION BASED ON
`SEVERITY RANK (FIG.4) 614
`
`RETURN
`
`FIG. 6
`
`IPR2025-00943
`Tesla EX1007 Page 8
`
`

`

`O'I --- 0
`0 ....
`N
`rJJ
`c
`
`Ul
`~
`~
`-....J
`O'I
`N
`
`> ....
`
`....
`0 ....
`QO
`('D .....
`rJJ =(cid:173)
`0 ....
`'? ....
`
`O'I
`
`N
`~Ul
`
`('D
`rJJ
`
`0
`
`('D
`
`0 =
`....
`~ .....
`
`(')
`
`....
`~ .....
`
`(')
`
`0 = ""O = O" -....
`('D = ..... t "e -....
`
`~ .....
`""O
`
`FIG. 7
`
`8
`
`8
`
`5
`
`4
`
`1
`
`1
`
`turns within city limits, high speed
`Driver talking on the phone Not using hands free, upcoming
`
`city limits
`vehicle speed, upcoming turn within
`Vehicle not in reverese gear, high
`
`Driver not looking ahead
`
`Vehicle parked on side of freeeway
`
`Parking light off
`
`upcoming turns, medium speed
`
`Driver talking on the phone Using hands free, freeway, no
`
`Vehicle parked in garage
`
`Parking light off
`
`Vehicle stopped at traffic signal
`
`Driver not looking ahead
`
`Severity Rank ( R )
`
`Road/Vehicle Condition
`
`Distraction
`
`r700
`
`IPR2025-00943
`Tesla EX1007 Page 9
`
`

`

`Patent Application Publication
`
`Sep. 15, 2016 Sheet 9 of 10
`
`US 2016/0267335 Al
`
`START
`
`,800
`
`RECEIVE IMAGES FROM
`CAMERAS/SENSORS 802
`
`NO
`
`MONITOR DRIVER DATA.
`VEHICLE DATA, CAN DATA
`806
`
`DETERMINE VEHICLE
`LOCATION AND SPEED 808
`
`NO
`
`NO
`
`NO
`
`INDICATE SEVERITY RANK
`R IN 1sr RANGE, PRESENT
`VISUAL WARNING fila
`
`INDICATE SEVERITY RANK
`IN 2ND RANGE, PRESENT
`AUDIO WARNING ll2Q
`
`INDICATE SEVERITY RANK
`IN 3RD RANGE, PERFORM
`ENGINE CONTROL822
`
`RETURN
`
`FIG. 8
`
`IPR2025-00943
`Tesla EX1007 Page 10
`
`

`

`Patent Application Publication
`
`Sep. 15, 2016 Sheet 10 of 10
`
`US 2016/0267335 Al
`
`START
`
`RECEIVE DATA FROM
`MOBILE DEVICE 902
`,------------------,
`: RECEIVE COMPRESSED :
`: DATA FROM HEAD UNIT :
`'-----------------~
`
`PROCESS RECEIVED DATA,
`DETERMINE VEHICLE DATA,
`OBJECT DATA, DRIVER
`DATA 904
`
`NO
`
`NO
`
`NO
`
`YES
`
`YES
`
`CALCULATE
`TRAJECTORIES OF
`VEHICLE, OBJECT 912
`
`R IN 2ND RANGE, SEND
`AUDIO ALERT FROM CLOUD
`916
`: SEND DATA TO HEAD :
`:
`: UNIT, SEND AUDIO
`'-----------------~
`ALERT
`I
`I
`
`NO
`
`R IN JRD RANGE, PERFORM
`ENGINE CONTROL FROM
`:--SENDC6~~2 {68
`HEAD--:
`: UNIT, PERFORM ENGINE :
`
`: ______ C_QN_T_R.9.!-______ :
`
`YES
`
`FIG. 9
`
`IPR2025-00943
`Tesla EX1007 Page 11
`
`

`

`US 2016/0267335 Al
`
`Sep. 15,2016
`
`1
`
`DRIVER DISTRACTION DETECTION
`SYSTEM
`
`FIELD
`
`[0001] The disclosure relates to assessing driver distraction
`based on the output of a wearable device and other sensors.
`
`BACKGROUND
`
`[0002] Distracted driving may include any activity that
`could divert a person's attention away from the primary task
`of driving. All distractions endanger driver, passenger, and
`bystander safety and could increase the chance of a motor
`vehicle crash. Some types of distraction include visual dis(cid:173)
`traction, where the driver takes his/her eyes off the road,
`manual distraction, where the driver takes his/her hands off
`the wheel, and cognitive distraction, where the driver takes
`his/her mind off of driving. The severity of the distraction
`could depend on both the level and duration of these distrac(cid:173)
`tions and may be compounded by external factors such as
`speed and location of vehicle and objects in the path of the
`vehicle, for example.
`
`SUMMARY
`
`[0003] Embodiments are described for determining and
`responding to driver distractions. An example in-vehicle
`computing system of a vehicle includes an external device
`interface communicatively connecting the in-vehicle com(cid:173)
`puting system to a mobile device, an inter-vehicle system
`communication module communicatively connecting the in(cid:173)
`vehicle computing system to one or more vehicle systems of
`the vehicle, a processor, and a storage device storing instruc(cid:173)
`tions executable by the processor to receive image data from
`the mobile device via the external device interface, the image
`data imaging a driver and a driver environment, and deter(cid:173)
`mine a driver state based on the received image data. The
`instructions are further executable to, responsive to determin(cid:173)
`ing that the driver state indicates that the driver is distracted,
`receive vehicle data from one or more of the vehicle systems
`via the inter-vehicle system communication module, deter(cid:173)
`mine a vehicle state based on the vehicle data, determine a
`distraction severity level based on the driver state and the
`vehicle state, and control one or more devices of the vehicle to
`perform a selected action based on the distraction severity
`level.
`[0004] An example method of determining driver distrac(cid:173)
`tion includes receiving driver data from a wearable device, the
`driver data including image data from a driver-facing camera,
`receiving object data from one or more imaging devices of at
`least one of the wearable device and the vehicle, the object
`data including image data of a vehicle environment, and
`receiving vehicle data from one or more vehicle systems, the
`vehicle data including an indication of an operating condition
`of the vehicle. The example method further includes deter(cid:173)
`mining whether a driver is distracted by correlating the driver
`data with the object data, responsive to determining that the
`driver is distracted, selecting an action based on correlating
`the driver data with the object data and the vehicle data and
`performing the selected action, and responsive to determining
`that the driver is not distracted, maintaining current operating
`parameters.
`[0005] An example distraction monitoring system includes
`a wearable device including a driver-facing camera, an out(cid:173)
`ward-facing camera, and one or more sensors, an in-vehicle
`
`computing system communicatively connected to the wear(cid:173)
`able device and one or more vehicle systems, the in-vehicle
`computing system comprising a first processor and a first
`storage device, and a cloud computing device remote from the
`in-vehicle computing system and communicatively con(cid:173)
`nected to the in-vehicle computing system via a network, the
`cloud computing device comprising a second processor and a
`second storage device. One or more of the first storage device
`and the second storage device storing first instructions
`executable by a respective one or more of the first processor
`and the second processor to receive image data from the
`driver-facing camera and sensor data from the one or more
`sensors of the wearable device indicating a driver state,
`receive image data from the outward-facing camera of the
`wearable device indicating object states of one or more
`objects, receive vehicle data from one or more vehicle sys(cid:173)
`tems to indicate vehicle state, and select an action to be
`performed based on the indicated driver state, object states,
`and vehicle state. The first storage device may store second
`instructions executable by the first processor to transmit a
`command to one or more of a display device of the in-vehicle
`computing system, an audio device of the vehicle, and an
`engine control unit of the vehicle to perform the selected
`action.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0006] The disclosure may be better understood from read(cid:173)
`ing the following description of non-limiting embodiments,
`with reference to the attached drawings, wherein below:
`[0007] FIG. 1 shows an example partial view of a vehicle
`cabin in accordance with one or more embodiments of the
`present disclosure;
`[0008] FIG. 2 shows an example in-vehicle computing sys(cid:173)
`tem in accordance with one or more embodiments of the
`present disclosure;
`[0009] FIG. 3A shows a block diagram of an example dis(cid:173)
`traction determination system in accordance with one or more
`embodiments of the present disclosure;
`[0010] FIG. 3B shows a block diagram of an example dis(cid:173)
`traction determination system in accordance with one or more
`embodiments of the present disclosure;
`[0011] FIG. 4 is a flow chart of an example method for
`generating warnings based on a calculated severity rank range
`from the perspective of a head unit in accordance with one or
`more embodiments of the present disclosure;
`[0012] FIG. 5 is a flow chart for an example method of
`determining a distraction level of a driver in accordance with
`one or more embodiments of the present disclosure;
`[0013] FIG. 6 is a flow chart for an example method of
`determining a distraction severity rank in accordance with
`one or more embodiments of the present disclosure;
`[0014] FIG. 7 shows an example table mapping severity
`rank R to driver states and vehicle conditions in accordance
`with one or more embodiments of the present disclosure;
`[0015] FIG. 8 is a flow chart for an example method of
`determining an action to be performed responsive to example
`driver states and vehicle conditions in accordance with one or
`more embodiments of the present disclosure; and
`[0016] FIG. 9 is a flow chart for an example method of
`determining an action to be performed responsive to example
`driver states, object states, and vehicle conditions in accor(cid:173)
`dance with one or more embodiments of the present disclo(cid:173)
`sure.
`
`IPR2025-00943
`Tesla EX1007 Page 12
`
`

`

`US 2016/0267335 Al
`
`Sep. 15,2016
`
`2
`
`DETAILED DESCRIPTION
`
`[0017] As described above, driver distraction may be dan(cid:173)
`gerous to occupants of a vehicle, as well as people in a vicinity
`of the vehicle. However, by only monitoring a driver state, or
`by monitoring only a small number of distraction-related
`behaviors of a driver, an inappropriate response to a distracted
`driver may be provided. For example, if a driver is nodding off
`to sleep, a visual warning may be insufficient to correct the
`distracted behavior. As another example, if a driver briefly
`looks away from the road, but there are no objects in the path
`of the vehicle, a loud, audible warning of driver distraction
`may be unnecessary and instead startle the driver, causing the
`driver to lose control of the vehicle. A distraction monitoring
`system that not only monitors the level of driver distraction,
`but also effectively responds to such distraction in a timely
`manner may address the issue of distracted driving and the
`major traffic safety issue that such driving poses.
`[0018] Accordingly, the disclosure provides a distraction
`monitoring system including one or more of an in-vehicle
`computing system and a cloud computing device that receives
`sensor data from a wearable device and/or other sensor
`devices to determine a driver state and a state of objects in a
`vehicle environment. The in-vehicle computing system and/
`or cloud computing device may also receive data from vehicle
`systems in order to determine a vehicle state. By correlating
`the driver, object(s), and vehicle state, the distraction moni(cid:173)
`toring system may first determine whether the driver is dis(cid:173)
`tracted, and then determine a severity of that distraction. The
`distraction monitoring system may perform a different action
`(e.g., provide a visual alert, provide an audible alert, and/or
`perform a vehicle control) responsive to different levels of
`distraction severity, as different types of distractions may
`benefit from different types of warnings/responses. In this
`way, a driver may be alerted to his/her distraction in an appro(cid:173)
`priate manner based on an intelligent combination of the
`different types of data.
`[0019] FIG. 1 shows an example partial view of one type of
`environment for a communication system: an interior of a
`cabin 100 of a vehicle 102, in which a driver and/or one or
`more passengers may be seated. Vehicle 102 of FIG. 1 may be
`a motor vehicle including drive wheels (not shown) and an
`internal combustion engine 104. Internal combustion engine
`104 may include one or more combustion chambers, which
`may receive intake air via an intake passage and exhaust
`combustion gases via an exhaust passage. Vehicle 102 may be
`a road automobile, among other types of vehicles. In some
`examples, vehicle 102 may include a hybrid propulsion sys(cid:173)
`tem including an energy conversion device operable to absorb
`energy from vehicle motion and/ or the engine and convert the
`absorbed energy to an energy form suitable for storage by an
`energy storage device. Vehicle 102 may include a fully elec(cid:173)
`tric vehicle, incorporating fuel cells, solar energy capturing
`elements, and/or other energy storage systems for powering
`the vehicle.
`[0020] As shown, an instrument panel 106 may include
`various displays and controls accessible to a driver (also
`referred to as the user) of vehicle 102. For example, instru(cid:173)
`ment panel 106 may include a touch screen 108 of an in(cid:173)
`vehicle computing system 109 ( e.g., an infotainment system),
`an audio system control panel, and an instrument cluster 110.
`While the example system shown in FIG. 1 includes audio
`system controls that may be performed via a user interface of
`in-vehicle computing system 109, such as touch screen 108
`without a separate audio system control panel, in other
`
`embodiments, the vehicle may include an audio system con(cid:173)
`trol panel, which may include controls for a conventional
`vehicle audio system such as a radio, compact disc player,
`MP3 player, etc. The audio system controls may include
`features for controlling one or more aspects of audio output
`via speakers 112 of a vehicle speaker system. For example,
`the in-vehicle computing system or the audio system controls
`may control a volume of audio output, a distribution of sound
`among the individual speakers of the vehicle speaker system,
`an equalization of audio signals, and/or any other aspect of
`the audio output. In further examples, in-vehicle computing
`system 109 may adjust a radio station selection, a playlist
`selection, a source of audio input ( e.g., from radio or CD or
`MP3), etc., based on user input received directly via touch
`screen 108, or based on data regarding the user (such as a
`physical state and/or environment of the user) received via
`external devices 150 and/or mobile device 128.
`[0021]
`In some embodiments, one or more hardware ele(cid:173)
`ments of in-vehicle computing system 109, such as touch
`screen 108, a display screen, various control dials, knobs and
`buttons, memory, processor( s ), and any interface elements
`(e.g., connectors or ports) may form an integrated head unit
`that is installed in instrument panel 106 of the vehicle. The
`head unit may be fixedly or removably attached in instrument
`panel 106. In additional or alternative embodiments, one or
`more hardware elements of the in-vehicle computing system
`may be modular and may be installed in multiple locations of
`the vehicle.
`[0022] The cabin 100 may include one or more sensors for
`monitoring the vehicle, the user, and/or the environment. For
`example, the cabin 100 may include one or more seat(cid:173)
`mounted pressure sensors configured to measure the pressure
`applied to the seat to determine the presence of a user, door
`sensors configured to monitor door activity, humidity sensors
`to measure the humidity content of the cabin, microphones to
`receive user input in the form of voice commands, to enable a
`user to conduct telephone calls, and/or to measure ambient
`noise in the cabin 100, etc. It is to be understood that the
`above-described sensors and/or one or more additional or
`alternative sensors may be positioned in any suitable location
`of the vehicle. For example, sensors may be positioned in an
`engine compartment, on an external surface of the vehicle,
`and/or in other suitable locations for providing information
`regarding the operation of the vehicle, ambient conditions of
`the vehicle, a user of the vehicle, etc. Information regarding
`ambient conditions of the vehicle, vehicle status, or vehicle
`driver may also be received from sensors external to/separate
`from the vehicle (that is, not part of the vehicle system), such
`as from sensors coupled to external devices 150 and/or
`mobile device 128.
`[0023] Cabin 100 may also include one or more user
`objects, such as mobile device 128, that are stored in the
`vehicle before, during, and/or after travelling. The mobile
`device may include a smart phone, a tablet, a laptop computer,
`a portable media player, and/or any suitable mobile comput(cid:173)
`ing device. The mobile device 128 may be connected to the
`in-vehicle computing system via communication link 130.
`The communication link 130 may be wired ( e.g., via Univer(cid:173)
`sal Serial Bus [USB], Mobile High-Definition Link [MHL],
`High-Definition Multimedia Interface [HDMI], Ethernet,
`etc.) or wireless (e.g., via BLUETOOTH, WI-FI, Near-Field
`Communication [NFC], cellular connectivity, etc.) and con(cid:173)
`figured to provide two-way communication between the
`mobile device and the in-vehicle computing system. For
`
`IPR2025-00943
`Tesla EX1007 Page 13
`
`

`

`US 2016/0267335 Al
`
`Sep. 15,2016
`
`3
`
`example, the communication link 130 may provide sensor
`and/or control signals from various vehicle systems (such as
`vehicle audio system, climate control system, etc.) and the
`touch screen 108 to the mobile device 128 and may provide
`control and/or display signals from the mobile device 128 to
`the in-vehicle systems and the touch screen 108. The com(cid:173)
`munication link 130 may also provide power to the mobile
`device 128 from an in-vehicle power source in order to charge
`an internal battery of the mobile device.
`[0024]
`In-vehicle computing system 109 may also be com(cid:173)
`municatively coupled to additional devices operated and/or
`accessed by the user but located external to vehicle 102, such
`as one or more external devices 150. In the depicted embodi(cid:173)
`ment, external devices 150 are located outside of vehicle 102
`though it will be appreciated that in alternate embodiments,
`external devices may be located inside cabin 100. The exter(cid:173)
`nal devices may include a server computing system, personal
`computing system, portable electronic device, electronic
`wrist band, electronic head band, portable music player, elec(cid:173)
`tronic activity tracking device, pedometer, smart-watch, GPS
`system, etc. External devices 150 may be connected to the
`in-vehicle computing system via communication link 136,
`which may be wired or wireless, as discussed with reference
`to communication link 130, and configured to provide two(cid:173)
`way communication between the external devices and the
`in-vehicle computing system. For example, external devices
`150 may include one or more sensors and communication
`link 136 may transmit sensor output from external devices
`150 to in-vehicle computing system 109 and touch screen
`108. External devices 150 may also store and/or receive infor(cid:173)
`mation regarding contextual data, user behavior/preferences,
`operating rules, etc. and may transmit such information from
`the external devices 150 to in-vehicle computing system 109
`and touch screen 108.
`[0025]
`In-vehicle computing system 109 may analyze the
`input received from external devices 150, mobile device 128,
`and/or other input sources and select settings for various
`in-vehicle systems (such as climate control system or audio
`system), provide output via touch screen 108 and/or speakers
`112, communicate with mobile device 128 and/or external
`devices 150, and/or perform other actions based on the
`assessment. In some embodiments, all or a portion of the
`assessment may be performed by the mobile device 128 and/
`or the external devices 150.
`[0026]
`In some embodiments, one or more of the external
`devices 150 may be communicatively coupled to in-vehicle
`computing system 109 indirectly, via mobile device 128 and/
`or another of the external devices 150. For example, commu(cid:173)
`nication link 136 may communicatively couple external
`devices 150 to mobile device 128 such that output from exter(cid:173)
`nal devices 150 is relayed to mobile device 128. Data received
`from external devices 150 may then be aggregated at mobile
`device 128 with data collected by mobile device 128, the
`aggregated data then transmitted to in-vehicle computing sys(cid:173)
`tem 109 and touch screen 108 via communication link 130.
`Similar data aggregation may occur at a server system and
`then transmitted to in-vehicle computing system 109 and
`touch screen 108 via communication link 136/130.
`[0027]
`In the example environment illustrated in FIG. 1, the
`in-vehicle computing system 109 may be connected to one or
`more vehicle systems, such as speakers 112, display 108,
`vehicle sensors, and/or other suitable vehicle systems via any
`suitable network. In some examples, the in-vehicle comput(cid:173)
`ing system 109 includes a talker device configured to transmit
`
`audio/video data to listener devices, such as speakers 112 and
`display 108 via a network. The network may be configured in
`accordance with Layer 2 of the Open Systems Interconnec(cid:173)
`tion (OSI) model, in which routing and forwarding decisions
`or determinations in the network may be performed on a
`media access control (MAC) addressing basis. An example
`Layer 2 network may be an Ethernet AudioNideo Bridging
`(AVB) network. For Layer 2 networks configured as AVB
`networks, the talkers and the listeners may be configured to
`communicate over the AVB network using various AVB stan(cid:173)
`dards and protocols, including the Institute of Electrical and
`Electronics Engineers (IEEE) 802.lAS-2011 (gPTP) for net(cid:173)
`work timing and synchronization, IEEE 802 .1Q-2011 clause
`34 for queuing and forwarding streaming data, IEEE 802 .1 Q-
`2011 clause 35 (Stream Reservation Protocol (SRP)) for
`reserving a network connection or path and/or resources such
`as bandwidth for communication over the network connec(cid:173)
`tion, and/or IEEE 1722-2011 related to a possible data
`streaming format. Other AVE-related standards and proto(cid:173)
`cols, and/or other versions of the AVB standards and proto(cid:173)
`cols, previously, currently, or later developed, may addition(cid:173)
`ally or alternatively be used.
`[0028]
`It is to be understood that FIG. 1 depicts one
`example environment, however the communication systems
`and methods described herein may be utilized in any suitable
`environment. Any suitable devices that transmit and/or
`receive information, sense data, and/or otherwise contribute
`to a driver distraction detection and/or alert system may be
`utilized as the systems and/or to perform the methods
`described herein.
`[0029] FIG. 2 shows a block diagram of an in-vehicle com(cid:173)
`puting system 200 configured and/or integrated inside vehicle
`201. In-vehicle computing system 200 may be an example of
`in-vehicle computing system 109 of FIG. 1 and/or may per(cid:173)
`form one or more of the methods described herein in some
`embodiments. In some examples, the in-vehicle computing
`system may be a vehicle infotainment system configured to
`provide information-based media content (audio and/or
`visual media content, including entertainment content, navi(cid:173)
`gational services, etc.) to a vehicle user to enhance the opera(cid:173)
`tor's in-vehicle experience. The vehicle infotainment system
`may include, or be coupled to, various vehicle systems, sub(cid:173)
`systems, hardware components, as well as software applica(cid:173)
`tions and systems that are integrated in, or integratable into,
`vehicle 201 in order to enhance an in-vehicle experience for a
`driver and/or a passenger.
`[0030]
`In-vehicle computing system 200 may include one
`or more processors including an operating sy