USOO8825258B2
`
`(12) United States Patent
`Cullinane et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,825.258 B2
`Sep. 2, 2014
`
`(54)
`
`(71)
`(72)
`
`(73)
`(*)
`
`(21)
`(22)
`(65)
`
`(60)
`
`(51)
`
`(52)
`
`(58)
`
`ENGAGING AND DISENGAGING FOR
`AUTONOMOUS DRIVING
`
`Applicant: Google Inc., Mountain View, CA (US)
`
`Inventors: Brian Cullinane, San Jose, CA (US);
`Philip Nemec, San Jose, CA (US);
`Manuel Christian Clement, Felton, CA
`(US); Robertus Christianus Elisabeth
`Mariet, Sunnyvale, CA (US); Lilli
`Ing-Marie Jonsson, Mountain View, CA
`(US)
`Assignee: Google Inc., Mountain View, CA (US)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`Appl. No.: 13/792,774
`
`Filed:
`
`Mar 11, 2013
`
`Prior Publication Data
`US 2014/O156133 A1
`Jun. 5, 2014
`
`Related U.S. Application Data
`Provisional application No. 61/731,717, filed on Nov.
`30, 2012.
`
`(2006.01)
`(2006.01)
`
`Int. C.
`G05D I/00
`G05D I/02
`U.S. C.
`CPC ............ G05D I/0212 (2013.01); G05D I/0223
`(2013.01)
`USPC ............... 701/23: 701/96; 701/117: 701/300;
`701/301
`
`Field of Classification Search
`CPC .............. G05D 1/0212: G05D 1/0223; G05D
`1/02019; G05D 1/0231; G05D 1/0234:
`G05D 1/0236; G05D 1/02; G05D 1/021;
`
`1 FO968
`USPC ........... 701/23, 27, 41, 96, 117, 118, 119, 36,
`701/300, 301; 340/900-935, 438,937,
`34O74255
`See application file for complete search history.
`References Cited
`
`(56)
`
`U.S. PATENT DOCUMENTS
`
`5,521,579 A
`5,644,386 A
`
`5/1996 Bernhard
`7/1997 Jenkins et al.
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`JP
`
`3, 2010
`2168835 A1
`9, 2005
`2005.250564. A
`OTHER PUBLICATIONS
`
`Bakambu et al., “Autonomous system for exploration and navigation
`in drift networks', 2004 IEEE Intelligent Vehicles Symposium, Uni
`versity of Parma, Parma, Italy, Jun. 14-17, pp. 212-217, 2004.
`(Continued)
`Primary Examiner — Richard Camby
`(74) Attorney, Agent, or Firm — Lerner, David, Littenberg,
`Krumholz & Mentlik, LLP
`(57)
`ABSTRACT
`Aspects of the present disclosure relate Switching between
`autonomous and manual driving modes. In order to do so, the
`vehicle's computer may conduct a series of environmental,
`system, and driver checks to identify certain conditions. The
`computer may correct some of these conditions and also
`provide a driver with a checklist of tasks for completion. Once
`the tasks have been completed and the conditions are
`changed, the computer may allow the driver to Switch from
`the manual to the autonomous driving mode. The computer
`may also make a determination, under certain conditions, that
`it would be detrimental to the driver's safety or comfort to
`make a Switch from the autonomous driving mode to the
`manual driving mode.
`26 Claims, 10 Drawing Sheets
`
`
`
`
`
`
`
`
`
`
`
`-issile Cenaccessor
`
`
`
`- 8.
`
`82
`
`8:
`
`88
`
`IPR2025-00943
`Tesla EX1019 Page 1
`
`

`

`US 8,825.258 B2
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,684,697 A 11, 1997 Mullen
`5,774,069 A
`6, 1998 Tanaka et al.
`5,835,870 A 11/1998 Kagawa
`5,838,562 A 11/1998 Gudat et al.
`5,874,904 A
`2/1999 Hirabayashi et al.
`5,906,645 A
`5/1999 Kagawa et al.
`5,983,161. A 1 1/1999 Lemelson et al.
`6,044,312 A
`3, 2000 Sudo et al.
`6,185,499 B1
`2/2001 Kinoshita et al.
`6,275,773 B1
`8/2001 Lemelson et al.
`6,324.463 B1 1 1/2001 Patel
`6,337,638 B1
`1/2002 Bates et al.
`6,396,417 B2
`5, 2002 Lee
`6,487,500 B2 11/2002 Lemelson et al.
`6,516.262 B2
`2/2003 Takenaga et al.
`6,590,499 B1
`7/2003 D'Agosto
`6,768,962 B2
`7/2004 Bullinger et al.
`6,810,330 B2 10/2004 Matsuura
`6,868,934 B2
`3/2005 Dirrig
`7,209,221 B2
`4/2007 Breed et al.
`7,295,904 B2 11/2007 Kanevsky et al.
`7,319,777 B2
`1/2008 Morcom
`7,321,311 B2
`1/2008 Rieth et al.
`7,327,238 B2
`2/2008 Bhogal et al.
`7,430,473 B2
`9, 2008 Foo et al.
`7,446,649 B2 11/2008 Bhogaletal.
`7,486, 176 B2
`2/2009 Bhogal et al.
`7,486, 177 B2
`2/2009 Wilbrink et al.
`7,499,774 B2
`3/2009 Barrett et al.
`7,630,806 B2 12/2009 Breed
`7,769.498 B2
`8/2010 Isaji et al.
`7,783.403 B2
`8, 2010 Breed
`7,865,310 B2
`1/2011 Nakano et al.
`7,894,951 B2
`2/2011 Norris et al.
`7,925.438 B2
`4/2011 Lo
`7,979, 172 B2
`7, 2011 Breed
`7.979,173 B2
`7/2011 Breed
`8,050,863 B2 11/2011 Trepagnier et al.
`8,126,642 B2
`2/2012 Trepagnier et al.
`8,204,670 B2
`6, 2012 Watanabe
`8,269,652 B2
`9, 2012 Seder et al.
`
`8,320,628 B2 11/2012 Cheng et al.
`8,352,110 B1
`1/2013 Szybalski et al.
`8,384,532 B2
`2/2013 Szczerba et al.
`8,395,529 B2
`3/2013 Seder et al.
`8.433,470 B1
`4/2013 Szybalski et al.
`8,482.431 B2
`7/2013 Kushi et al.
`8,527,199 B1* 9/2013 Burnette et al. .............. TO1/450
`2005/0234612 A1 10/2005 Bottomley et al.
`2006/0089765 A1
`4/2006 Packet al.
`2008.OO71177 A1
`3/2008 Yanagidaira et al.
`2008/01 19994 A1
`5/2008 Kameyama
`2008/0306668 Al 12/2008 Wang et al.
`2009/0147996 A1
`6/2009 Peng
`2009/0322547 A1 12/2009 Wen
`2010 OO63663 A1
`3/2010 Tolstedt et al.
`2010.0085173 A1
`4/2010 Yang et al.
`2010, 0168998 A1
`7/2010 Matsunaga
`2010/01797 15 A1
`7/2010 Puddy
`2010/0222976 A1
`9/2010 Haug
`2010/0228417 A1
`9/2010 Lee et al.
`2010/0332 127 A1 12/2010 Imai et al.
`2011/O140919 A1
`6, 2011 Hara et al.
`2011/O193722 A1
`8/2011 Johnson
`2012.0035788 A1
`2/2012 Trepagnier et al.
`2012/0109418 A1
`5, 2012 Lorber
`2012fO269391 A1 10, 2012 Saito
`2012,0283942 A1 11, 2012 TSiobbel et al.
`2012fO288156 A1 11, 2012 Kido
`2013/012915.0 A1
`5, 2013 Saito
`2013,013 1905 A1
`5, 2013 Green et al.
`2013,013 1906 A1
`5, 2013 Green et al.
`2013,013 1907 A1
`5, 2013 Green et al.
`
`OTHER PUBLICATIONS
`
`Sebastian Thrun, Toward Robotic Cars, Apr. 2010, pp. 99-106, No. 4.
`Communications of the ACM.
`Sebastian, et al., Stanley: The Robot That Won the DARPA Grand
`Challenge, Journal of Field Robotics 23(9), 661-692 (2006), 2006
`Wiley Periodicals, Inc.
`International Search Report and Written Opinion for Application No.
`PCT/US2013/07 1846 dated Mar. 7, 2014.
`
`* cited by examiner
`
`IPR2025-00943
`Tesla EX1019 Page 2
`
`

`

`U.S. Patent
`U.S. Patent
`
`Sep. 2, 2014
`Sep. 2, 2014
`
`Sheet 1 of 10
`Sheet 1 of 10
`
`US 8,825.258 B2
`US 8,825,258 B2
`
`
`
`LUSISASUOHEOIABNSOILOA
`
`
`
`
`
`
`
`LWUAISASJayndwiegGulaugsnowouoiny
`
`LOLa1lusa
`
`wiaishsuoneIele00yee
`JOSSASIOI(BAUSDssaOOFSoUSUB!-deypaleisd
`
`
`
`
`
`jusuodiuosuotsodsudeuboagheo-—pel
`
`suonongsal
`
`ASOLLIOHY
`
`JOSSSIO0Idy”
`
`
`
`
`
`fsadA}}suopoasia}ui-
`
`
`
`UORBULOLUYSSINIeS}1YIO-
`
`Siupoads-
`
`
`
`BIEGjOSO}OId
`
`
`
`SWAeQUONEIajeIoy¥PrSFL
`
`
`
`Apidsipuonoayo}-
`
`
`
`wiaishsuonIejagl-
`
`OLbSYNOlS
`
`IPR2025-00943
`Tesla EX1019 Page 3
`
`IPR2025-00943
`Tesla EX1019 Page 3
`
`

`

`U.S. Patent
`
`Sep. 2, 2014
`
`Sheet 2 of 10
`
`US 8,825.258 B2
`
`FGURE 2
`
`
`
`s
`
`25
`
`28
`
`IPR2025-00943
`Tesla EX1019 Page 4
`
`

`

`
`
`HVSy
`
`NnwemEE
`
`eo
`
`Sep. 2, 2014
`Sep. 2, 2014
`
`car LNOdSs
`
`U.S. Patent
`U.S. Patent
`
`Sheet 3 of 10
`Sheet 3 of 10
`
`US 8,825.258 B2
`US 8,825,258 B2
`
`£agNois
`
`IPR2025-00943
`Tesla EX1019 Page 5
`
`IPR2025-00943
`Tesla EX1019 Page 5
`
`

`

`U.S. Patent
`
`Sep. 2, 2014
`
`Sheet 4 of 10
`
`US 8,825.258 B2
`
`451
`
`i
`
`422 421 420
`-
`^
`
`412 411
`
`4
`
`450
`
`r-
`
`s
`Y.
`
`461443 433
`
`Y
`432 442
`
`44
`
`FIGURE 4
`
`IPR2025-00943
`Tesla EX1019 Page 6
`
`

`

`U.S. Patent
`
`Sep. 2, 2014
`
`Sheet 5 of 10
`
`US 8,825.258 B2
`
`512 511
`
`510 550
`
`FIGURE 5
`
`IPR2025-00943
`Tesla EX1019 Page 7
`
`

`

`U.S. Patent
`
`Sep. 2, 2014
`
`Sheet 6 of 10
`
`US 8,825.258 B2
`
`
`
`• • • • • • • • • • • • • • • • • • • • • • • • • • •\
`
`FGURE 6
`
`IPR2025-00943
`Tesla EX1019 Page 8
`
`

`

`U.S. Patent
`U.S. Patent
`
`Sep. 2, 2014
`Sep. 2, 2014
`
`Sheet 7 of 10
`Sheet 7 of 10
`
`US 8,825.258 B2
`US 8,825,258 B2
`
` IOSSa30id
`
`
`
`V¥2iANDAs
`
`
`
`0wL-—~
`
`
`
`
`
`wigsASJeirmiuo>Biusaugsnawouony
`
`
`
`
`
`d/anos
`
`umOEE
`
`IPR2025-00943
`Tesla EX1019 Page 9
`
`IPR2025-00943
`Tesla EX1019 Page 9
`
`

`

`U.S. Patent
`U.S. Patent
`
`Sep. 2, 2014
`Sep. 2, 2014
`
`Sheet 8 of 10
`Sheet 8 of 10
`
`US 8,825.258 B2
`US 8,825,258 B2
`
`
`
`B40 ~
`
`
`<A x SS
`
`
`
`
`
`
`
`NS
`
`
`SS
`
`x
`
`\
`
`FIGURE 8
`
`IPR2025-00943
`Tesla EX1019 Page 10
`
`IPR2025-00943
`Tesla EX1019 Page 10
`
`

`

`U.S. Patent
`
`Sep. 2, 2014
`
`Sheet 9 of 10
`
`US 8,825.258 B2
`
`
`
`930 -
`
`
`
`
`
`
`
`Š - Y X Šy
`Š i.
`
`s
`
`S
`
`
`
`
`
`FGURE 9
`
`IPR2025-00943
`Tesla EX1019 Page 11
`
`

`

`
`
`adessauiainjiesAedsiqau}SuisnjuawuoKAUSueOo}apowBuiAup
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`JOALB20><snnuenaidfuEIEpjodojoidssacoy__-HSfuionspuosIO_-—aisjuompuoeayySuisnjuswwuoKAUS¥OOL<JO@100OARS
`SPBIGUaDNLoonSAALUNe&nsantoOaF-
`
`
`—a“Eisjuoqipucs=ONBEDchat
`“SNBaquaaaddMuy"poBo}O:dou]BushJeAuDapzaaU]JOSNE]s
`
`
`
`
`degndusiooAq(s}uoyipuceainGuisnstuaysAs
`aanuaAeidAUBjoa0SPOMOUOINEOFFENUEMSDLIGAPUSQIUSA
`
`
`
`™_aanduiosueg2SAAaNSKvSUNNYS.SPSIOA
`
`SAAONa“endaawByepjosajoidFOO!
`
`|apour2004USShOboNPHOLau]JOsnjeysssesoyiON
`
`
`
`
`
`BUSA39°LUO,LIOUAASUBMpoaoGldaun1sereeeasey
`ssassy
`
`
`
`
`OcteyepjooojaidYOOLBulaupsnotuoucine
`
`
`zcolapouoFabgyepjooojoidOFOL
`BY]JOSNjeysssassya}isanbs/8acisoey
`
`
`WUBINS.SfIYSAPeNnUBLBLUOYOUAS
`
`
`
`U.S. Patent
`U.S. Patent
`
`Sep. 2, 2014
`Sep. 2, 2014
`
`Sheet 10 of 10
`Sheet 10 of 10
`
`US 8,825.258 B2
`US 8,825,258 B2
`
`
`
`
`pestle,eidoyAvidsicaépapeauucaela>LUPUOdSSLIOD{S}{SE}Bunuesousiy+—e
`
`
`
`yerf<paaidSYDOOFS
`erepyeetveezOLaa—YroO
`
`9e0LaoObaHNDISSo
`{onNI
`
`
`
`
`
`
`
`
`
`SS
`ran
`CONoS—-
`
`
`
`
`
`
`
`
`
`IPR2025-00943
`Tesla EX1019 Page 12
`
`
`
`
`
`

`

`US 8,825,258 B2
`
`1.
`ENGAGING AND DISENGAGING FOR
`AUTONOMOUS DRIVING
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`The present application claims the benefit of the filing date
`of U.S. Provisional Patent Application No. 61/731,717 filed
`Nov.30, 2012, the entire disclosure of which is hereby incor
`porated herein by reference.
`
`BACKGROUND
`
`Autonomous vehicles use various computing systems to
`aid in the transport of passengers from one location to
`another. Some autonomous vehicles may require Some initial
`input or continuous input from an operator, Such as a pilot,
`driver, or passenger. Other systems, for example autopilot
`systems, may be used only when the system has been
`engaged, which permits the operator to Switch from a manual
`driving mode (where the operator exercises a high degree of
`control over the movement of the vehicle) to an autonomous
`driving mode (where the vehicle essentially drives itself) to
`modes that lie somewhere in between.
`
`10
`
`15
`
`25
`
`BRIEF SUMMARY
`
`2
`vehicle's lane proximate to the vehicle, whether the vehicle is
`at least Some pre-determined minimum distance from other
`vehicles or moving objects in the roadway, whether the
`vehicle must make any Small maneuvers to avoid non-moving
`or slow moving objects in the roadway, whether the vehicle is
`traveling too fast or too slow, whether any upcoming maneu
`vers would prevent a Switch from manual to autonomous
`driving mode, whether the vehicle can be driving in the
`autonomous mode for a minimum period of time before a
`Switch to manual mode is required, whether the various fea
`tures of the autonomous driving system of vehicle are work
`ing properly, whether the vehicle has met minimum mainte
`nance standards (for example, oil changes are up to date),
`whether the vehicle's tires are properly inflated, whether the
`vehicle's doors are closed, whether the vehicle is currently in
`“drive' (as opposed to “neutral.” “reverse,” or “park').
`whether the vehicle is in two wheel drive (for example, on a
`vehicle which allows for the driver to manually switch into
`four wheel or all wheel drive), whether the vehicle's auto
`matic wipers are currently on, whether the vehicles head
`lights or fog lights are on, and whether no other warning
`indicators (such as check engine lights) are currently active,
`and whether the driver's seatbelt is properly buckled.
`Other conditions that prevent a switch from manual to
`autonomous mode may include that the vehicle is not in
`“drive,” that the automatic wipers are not on, that the auto
`matic lights are not on, and that the vehicle is not in a lane
`pre-approved for initiating autonomous driving. In Such
`cases, the generated set of tasks may include putting the
`vehicle in drive, turning on automatic wipers, turning on
`automatic lights, and moving the vehicle to a lane pre-ap
`proved for initiating autonomous driving. In this example, the
`sequence of displayed tasks includes a first displayed task for
`putting the vehicle in drive, a second displayed task for turn
`ing on automatic wipers, a third displayed task turning on
`automatic lights, and a fourth displayed task moving the
`vehicle a lane pre-approved for initiating autonomous driv
`ing. The first displayed task is displayed until confirmed
`completed. When the first displayed task is completed, the
`second displayed task is displayed until the second task is
`completed. When the second displayed task is completed, the
`third displayed task is displayed until the third task is com
`pleted. When the third displayed task is completed, the fourth
`displayed task is displayed until the fourth task is completed.
`In another example, the preventive conditions may include
`that the vehicle is not in “drive,” that the automatic wipers are
`not on, that the automatic lights are not on, that the vehicle is
`not in a lane pre-approved for initiating autonomous driving,
`that the vehicle is not centered in a lane, that the vehicle is then
`moving too fast, and that the vehicle is too close to another
`object in the roadway. In this example, the generated set of
`tasks may include putting the vehicle in drive, turning on
`automatic wipers, turning on automatic lights, moving the
`vehicle a lane pre-approved for initiating autonomous driv
`ing, centering the vehicle in the lane, slowing the vehicle
`down, and increasing the distance between the vehicle and the
`another object in the roadway.
`In another example, the method also includes, before using
`the task data to generate the set of tasks, correcting by the
`processor any of the identified one or more conditions pre
`approved for correction by the processor. In another example,
`assessing the status is performed continuously while the tasks
`are being completed. In this regard, additional conditions
`may be identified and used to add additional tasks to the set of
`tasks.
`In another example, the method may also include, after
`Switching to the autonomous driving mode, maneuvering the
`
`30
`
`35
`
`40
`
`Aspects of the disclosure provide a method. In Such
`aspects, the method includes receiving a request to Switch a
`vehicle from a manual driving mode to an autonomous driv
`ing mode. In response, protocol data is accessed. This proto
`col data is used by a processor to assess the status of the
`vehicle's environment, the vehicle and systems of the vehicle,
`and a driver and identify one or more conditions. The method
`also includes accessing task data which associates conditions
`with tasks that may be performed by a driver to change the
`condition. The task data is used to generate a set of tasks. Each
`task of the set of tasks is displayed in an ordered sequence.
`Once the tasks are complete, the switch from the manual
`driving mode to the autonomous driving mode may proceed.
`In one example, assessing the status of the vehicle's envi
`ronment, the vehicle and systems of the vehicle, and a driver
`includes accessing sensor data, accessing detailed map infor
`mation, communicating with various systems of the vehicle,
`and communicating with a remote computer.
`45
`The assessments may include, for example, one or more of
`determining whether the current or future weather would
`make autonomous driving unsafe, uncomfortable for the
`vehicle's passengers or damage the vehicle, whether the vehi
`cle's actual location agrees with the vehicle's location rela
`tive to the detailed map information, whether data received
`from the sensors of the detection system agrees with the
`corresponding detailed map information, whether the vehicle
`is currently driving in an area pre-approved for initiating the
`autonomous mode, whether the vehicle is currently driving in
`a lane pre-approved for initiating the autonomous mode,
`whether the roadway is paved (as opposed to dirt), whether
`the road is wide enough (as opposed to being too narrow for
`two vehicles to pass one another), whether the vehicle is on a
`straight roadway (as opposed to driving around a curve, down
`or up a hill, etc.), whether the vehicle is sufficiently centered
`in the lane, whether the vehicle is surrounded by or boxed in
`by other vehicles (for example, in dense traffic conditions),
`whether the vehicle is currently in a school Zone, whether the
`vehicle is facing oncoming traffic (for example, driving
`northbound in a Southbound lane according to the detailed
`map information), whether another vehicle is pulling into the
`
`50
`
`55
`
`60
`
`65
`
`IPR2025-00943
`Tesla EX1019 Page 13
`
`

`

`US 8,825,258 B2
`
`3
`vehicle in the autonomous driving mode and receiving a
`request to Switch from the autonomous driving mode to the
`manual driving mode. This request may be not recommended
`ordenied when the vehicle is performing an action that cannot
`be safely or easily transferred to the driver before the action is
`complete. In this example and in response to an indication by
`the user that the user wants to return to manual mode, a
`warning may be displayed to the driver when Switching to
`manual driving mode is not recommended.
`Other aspects of the disclosure provide systems including
`a processor which performs some or all of the various features
`of the methods described above. Further aspects of the dis
`closure provide a non-transitory, tangible computer-readable
`storage medium on which computer readable instructions of
`a program are stored. The instructions, when executed by a
`processor, cause the processor to perform some or all of the
`various features of the methods described above.
`
`10
`
`15
`
`4
`If, as a result of the assessments, the computer identifies
`one or more preventive conditions, the computer may deter
`mine whether the computer or driver can change those iden
`tified conditions immediately or within a certain period of
`time. If the preventive conditions cannot be so changed, the
`computer may display a failure message to the driver. In some
`examples, this failure message may indicate the particular
`issue for the failure.
`If the computer can change the identified preventive con
`ditions, the computer may take the actions necessary to do so.
`Even if the computer is otherwise capable of changing the
`preventive condition, the vehicle may be designed to require
`the driver to change the condition. For example, the driver
`may be in a better position to assess if and how the preventive
`condition can be changed. If no identified preventive condi
`tions remain, the computer may proceed with the Switch from
`the manual to the autonomous driving mode.
`If after the computer corrects any of the preventive condi
`tions other preventive conditions remain, the computer may
`generate a corresponding task or task for each of the remain
`ing problem conditions. The computer may then display the
`tasks in order of priority to the driver. For example, the com
`puter may be programmed to always display the tasks one at
`a time in a predetermined order, or dynamically determine the
`order in which the remaining tasks should be displayed. Once
`the task is completed, the computer may display the next task
`for completion. The compute may continue to display the
`tasks until each is corrected, and the computer may then
`proceed with the Switch from manual to autonomous driving
`mode. In addition, this list may be recomputed periodically
`such that if one of the steps is completed while another step is
`being shown, then the completed task may be removed.
`While the driver is performing the tasks, the computer may
`continue to make the aforementioned assessments to identify
`any additional preventive conditions. These conditions may
`then cause the computer to display the error message, correct
`the additional preventive conditions, or generate and display
`additional tasks for completion by the driver. Again, if all of
`the preventive conditions have been corrected, the computer
`may proceed with the Switch from the manual to the autono
`mous driving mode.
`The computer may similarly make assessments to deter
`mine whether switching the vehicle from the autonomous
`driving mode to the manual driving mode is not currently
`recommended.
`As shown in FIG. 1, an autonomous driving system 100 in
`may include a vehicle 101 with various components. While
`certain aspects of the disclosure are particularly useful in
`connection with specific types of vehicles, the vehicle may be
`any type of vehicle including, but not limited to, cars, trucks,
`motorcycles, busses, boats, airplanes, helicopters, lawnmow
`ers, recreational vehicles, amusement park vehicles, farm
`equipment, construction equipment, trams, golf carts, trains,
`and trolleys. The vehicle may have one or more computers,
`Such as computer 110 containing a processor 120, memory
`130 and other components typically present in general pur
`pose computers.
`The memory 130 stores information accessible by proces
`sor 120, including instructions 132 and data 134 that may be
`executed or otherwise used by the processor 120. The
`memory 130 may be of any type capable of storing informa
`tion accessible by the processor, including a computer-read
`able medium, or other medium that stores data that may be
`read with the aid of an electronic device, such as a hard-drive,
`memory card, ROM, RAM, DVD or other optical disks, as
`well as other write-capable and read-only memories. Systems
`and methods may include different combinations of the fore
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a functional diagram of a system in accordance
`with aspects of the disclosure.
`FIG. 2 is an interior of an autonomous vehicle in accor
`dance with aspects of the disclosure.
`FIG. 3 is an exterior of an autonomous vehicle in accor
`dance with aspects of the disclosure.
`FIG. 4 is an illustration of a highway used by way of
`example in accordance with aspects of the disclosure.
`FIG. 5 is an example of map information in accordance
`with aspects of the disclosure.
`FIG. 6 is another example of map information in accor
`dance with aspects of the disclosure.
`FIG. 7A is a pictorial diagram of a system in accordance
`with aspects of the disclosure.
`FIG. 7B is a functional diagram of a system in accordance
`with aspects of the disclosure.
`FIG. 8 is a series of Screen images in accordance with
`aspects of the disclosure.
`FIG. 9 is a series of screen images in accordance with
`aspects of the disclosure.
`FIG. 10 is a flow diagram in accordance with aspects of the
`disclosure.
`
`25
`
`30
`
`35
`
`40
`
`DETAILED DESCRIPTION
`
`A vehicle may make a variety of determinations before
`allowing a driver to operate a vehicle in an autonomous driv
`ing mode, Such as handing off control of the vehicle to the
`vehicle's computer, or allowing the driver to switch from the
`autonomous driving mode to the manual driving mode. This
`may include conducting a series of environmental, system,
`and driver checks and also providing a driver with a checklist
`of tasks before allowing the driver to hand off control of the
`vehicle to the vehicle's computer.
`In one example, a computer associated with a vehicle hav
`ing an autonomous driving mode and a manual driving mode
`may receive a request to Switch from a manual driving mode
`to an autonomous driving mode. The computer may then
`respond by accessing protocol data and using it to assess
`status of the vehicle's current environment, the vehicle's
`likely future environment, the vehicle, and the driver. The
`computer may then determine if these assessments have iden
`tified any conditions that the system uses to prevent vehicle
`from being placed into an autonomous driving mode (here
`after, “preventive conditions'). If not, the computer may pro
`ceed with the switch from the manual to the autonomous
`driving mode.
`
`45
`
`50
`
`55
`
`60
`
`65
`
`IPR2025-00943
`Tesla EX1019 Page 14
`
`

`

`5
`going, whereby different portions of the instructions and data
`are stored on different types of media.
`The instructions 132 may be any set of instructions to be
`executed directly (such as machine code) or indirectly (Such
`as Scripts) by the processor. For example, the instructions may
`be stored as computer code on the computer-readable
`medium. In that regard, the terms “instructions” and “pro
`grams' may be used interchangeably herein. The instructions
`may be stored in object code format for direct processing by
`the processor, or in any other computer language including
`Scripts or collections of independent source code modules
`that are interpreted on demand or compiled in advance. Func
`tions, methods and routines of the instructions are explained
`in more detail below.
`The data 134 may be retrieved, stored or modified by
`processor 120 in accordance with the instructions 132. For
`instance, although the claimed Subject matter is not limited by
`any particular data structure, the data may be stored in com
`puter registers, in a relational database as a table having a
`plurality of different fields and records, XML documents or
`flat files. The data may also be formatted in any computer
`readable format. By further way of example only, image data
`may be stored as bitmaps comprised of grids of pixels that are
`stored in accordance with formats that are compressed or
`uncompressed, lossless (e.g., BMP) or lossy (e.g., JPEG), and
`bitmap or vector-based (e.g., SVG), as well as computer
`instructions for drawing graphics. The data may comprise any
`information sufficient to identify the relevant information,
`Such as numbers, descriptive text, proprietary codes, refer
`ences to data stored in other areas of the same memory or
`different memories (including other network locations) or
`information that is used by a function to calculate the relevant
`data.
`The processor 120 may be any conventional processor,
`such as commercially available CPUs. Alternatively, the pro
`cessor may be a dedicated device such as an ASIC or other
`hardware-based processor. Although FIG. 1 functionally
`illustrates the processor, memory, and other elements of com
`puter 110 as being within the same block, it will be under
`stood by those of ordinary skill in the art that the processor,
`computer, or memory may actually comprise multiple pro
`cessors, computers, or memories that may or may not be
`stored within the same physical housing. For example,
`memory may be a hard drive or other storage media located in
`a housing different from that of computer 110. Accordingly,
`references to a processor or computer will be understood to
`include references to a collection of processors or computers
`or memories that may or may not operate in parallel. Rather
`than using a single processor to perform the steps described
`herein, some of the components, such as steering components
`and deceleration components, may each have their own pro
`cessor that only performs calculations related to the compo
`nent's specific function.
`In various aspects described herein, the processor may be
`located remote from the vehicle and communicate with the
`vehicle wirelessly. In other aspects, some of the processes
`described herein are executed on a processor disposed within
`the vehicle and others by a remote processor, including taking
`the steps necessary to execute a single maneuver.
`Computer 110 may include all of the components normally
`used in connection with a computer Such as a central process
`ing unit (CPU), memory (e.g., RAM and internal hard drives)
`storing data 134 and instructions such as a web browser, an
`electronic display 152 (e.g., a monitor having a screen, a
`small LCD touch-screen or any other electrical device that is
`operable to display information), user input 150 (e.g., a
`mouse, keyboard, touch screen and/or microphone), as well
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 8,825,258 B2
`
`10
`
`15
`
`6
`as various sensors (e.g., a video camera) for gathering explicit
`(e.g., a gesture) or implicit (e.g., “the person is asleep') infor
`mation about the states and desires of a person.
`In one example, computer 110 may be an autonomous
`driving computing system incorporated into vehicle 101.
`FIG. 2 depicts an exemplary design of the interior of an
`autonomous vehicle. The autonomous vehicle may include
`all of the features of a non-autonomous vehicle, for example:
`a steering apparatus, such as steering wheel 210; a navigation
`display apparatus, Such as navigation display 215 (which may
`be a part of electronic display 152); and a gear selector appa
`ratus, such as gear shifter 220. The vehicle may also have
`various user input devices 140 in addition to the foregoing,
`such as touch screen 217 (which may be a part of electronic
`display 152), or button inputs 219, for activating or deacti
`Vating one or more autonomous driving modes and for
`enabling a driver or passenger 290 to provide information,
`Such as a navigation destination, to the autonomous driving
`computer 110.
`The autonomous driving computing system may be
`capable of communicating with various components of the
`vehicle. For example, returning to FIG. 1, computer 110 may
`be in communication with the vehicle's central processor 160
`and may send and receive information from the various sys
`tems of vehicle 101, for example the braking system 180,
`acceleration system 182, signaling system 184, and naviga
`tion system 186 in order to control the movement, speed, etc.
`of vehicle 101. In one example, the vehicle's central proces
`sor 160 may performall of the functions of a central processor
`in a non-autonomous computer. In another example, proces
`Sor 120 and 160 may comprise a single processing device or
`multiple processing devices operating in parallel.
`In addition, when engaged, computer 110 may control
`some or all of these functions of vehicle 101 and thus be fully
`or partially autonomous. It will be understood that although
`various systems and computer 110 are shown within vehicle
`101, these elements may be external to vehicle 101 or physi
`cally separated by large distances.
`The vehicle may also include a geographic position com
`ponent 144 in communication with computer 110 for deter
`mining the geographic location of the device. For example,
`the position component may include a GPS receiver to deter
`mine the device's latitude, longitude and/or altitude position.
`Other location systems such as laser-based localization sys
`tems, inertial-aided GPS, or camera-based localization may
`also be used to identify the location of the vehicle. The loca
`tion of the vehicle may include an absolute geographical
`location, such as latitude, longitude, and altitude as well as
`relative location information, such as location relative to
`other cars immediately around it, which can often be deter
`mined with better accuracy than absolute geographical loca
`tion.
`The vehicle may also include other devices in communi
`cation with computer 110. Such as an accelerometer, gyro
`scope or another direction/speed detection device 146 to
`determine the direction and speed of the vehicle or changes
`thereto. By way of example only, acceleration device 146
`may determine its pitch, yaw or roll (or changes thereto)
`relative to the direction of gravity or a plane perpendicular
`thereto. The device may also track increases or decreases in
`speed and the direction of Such changes. The device's provi
`sion of location and orientation data as set forth herein may be
`provided automatically to the user, computer 110, other com
`puters and combinations of the foregoing.
`The computer 110 may control the direction and speed of
`the vehicle by controlling various components. By way of
`example, if the vehicle is operating in a completely autono
`
`IPR2025-00943
`Tesla EX1019 Page 15
`
`

`

`US 8,825,258 B2
`
`10
`
`15
`
`25
`
`35
`
`7
`mous driving mode, computer 110 may cause the vehicle to
`accelerate (e.g., by increasing fuel or other energy provided to
`the engine), decelerate (e.g., by decreasing the fuel Supplied
`to the engine or by applying brakes) and change direction
`(e.g., by turning the front two wheels).
`The vehicle may also include components for detecting
`objects external to the vehicle such as other vehicles,
`obstacles in the roadway, traffic signals, signs, trees, etc. The
`detection system 154 may include lasers, Sonar, radar, cam
`eras or any other detection devices which record data which
`may be processed by computer 110. For example, if the
`vehicle is a small passenger vehicle, the car may include a
`laser mounted on the roof or other convenient location.
`