US009304654B2
`
`US 9,304,654 B2
`(10) Patent No.:
`az) United States Patent
`Bakhash
`(45) Date of Patent:
`*Apr. 5, 2016
`
`
`(54) SYSTEM AND METHOD FOR DISPLAYING A
`TIMELINE ASSOCIATED WITH A
`PLURALITY OF APPLICATIONS
`
`(71)
`
`(72)
`
`Applicant: Ezra Eddie Bakhash, New York, NY
`(US)
`
`Inventor: Ezra Eddie Bakhash, New York, NY
`(US)
`
`Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`US.C. 154(b) by 84 days.
`
`3/04845 (2013.01); GO6F 1730572 (2013.01):
`G06F 17/30864 (2013.01); GO6F 17/308 73
`(2013.01); GO6T 15/20 (2013.01); G1IB 27/34
`(2013.01)
`
`(58) Field of Classification Search
`CPC wee GO6F 3/0481; GO6F 3/04815; GO6F
`3/04817; GO6F 3/0482; GO6F 3/0483; GO6F
`3/04845; GO6F 17/30864; GO6F 17/30873;
`GO6F 17/30572; GO6T 15/04; GO6T 15/20;
`G11B 27/34
`USPC oes 715/782, 781, 764, 788, 766, 767
`See application file for complete search history.
`
`This patent is subject to a terminal dis-
`claimer.
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`(21)
`
`Appl. No.: 14/503,142
`
`(22)
`
`Filed:
`
`Sep. 30, 2014
`
`(65)
`
`(63)
`
`(60)
`
`(51)
`
`(52)
`
`Prior Publication Data
`
`US 2015/0128059 Al
`
`May7, 2015
`
`Related U.S. Application Data
`
`Continuation of application No. 12/751,879, filed on
`Mar. 31, 2010, now Pat. No. 8,881,048, which is a
`continuation of application No. 11/531,676, filed on
`Sep. 13, 2006, now Pat. No. 7,735,018.
`
`Provisional application No. 60/717,019, filed on Sep.
`13, 2005.
`
`Int. Cl.
`
`G06F 3/0482
`G06F 3/0481
`GO6F 3/048
`G06F 1730
`G06T 15/20
`GI1B 2734
`GO06F 3/0484
`US. Cl.
`
`(2013.01)
`(2013.01)
`(2013.01)
`(2006.01)
`(2011.01)
`(2006.01)
`(2013.01)
`
`6/1994 Yoshikawa.........c6005 345/179
`5,319,387 A *
`
`6/1995 Kahletal. o.. 715/839
`5,428,735 A *
`
`9/2000 Jainetal. wo. 715/850
`6,121,969 A *
`6,499,029 BIL* 12/2002 Kurapatietal.
`......... 707/750
`
`6,577,330 BL*
`6/2003 Tsuda etal... 715/782
`
`(Continued)
`
`Primary Examiner — Ting Lee
`(74) Attorney, Agent, or Firm — Fitzsimmons IP Law
`
`(57)
`
`ABSTRACT
`
`Methods and systemsare provided for providing an improved
`three-dimensional graphical user interface. In one embodi-
`ment, the method generally comprises: receiving an input
`from an end user, and capturing computing output from at
`least one computer source in response to the received end-
`user input. The computing output can be presented as two or
`more objects within a three-dimensional virtual space dis-
`played to the end user. Inone embodiment, the method further
`comprises generating a timeline that includes an icon for each
`object presented within the virtual space. In another embodi-
`ment, the method further comprises providing a database for
`storing and categorizing data regarding each object presented
`within the virtual space.
`
`CPC we. GO6F 3/04815 (2013.01); GO6F 3/0482
`(2013.01); GO6F 3/04817 (2013.01); GO6F
`
`19 Claims, 30 Drawing Sheets
`
`
`
`
`
`
`
`
`
`
`
`
`
`APPLE 1001
`
`APPLE 1001
`
`1
`
`

`

`US 9,304,654 B2
`
` Page 2
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`6,725,427 B2*
`4/2004 Freeman etal.
`.............. 715/273
`
`6,768,999 B2*
`7/2004 Prageretal. ....
`707/999 .002
`2001/0050687 Al* 12/2001 Tidaetal. wo. 345/581
`
`2002/0033848 Al*
`3/2002 Sciammarellaet al.
`...... 345/838
`2003/0164827 Al*
`........... 345/419
`9/2003 Gottesman etal.
`
`2003/0179231 A1*
`........... 345/757
`9/2003 Kamiwada etal.
`
`2004/0268451 Al* 12/2004 Robbin etal. ......... 999/999.999
`
`* cited by examiner
`
`2
`
`

`

`U.S. Patent
`
`Apr.5, 2016
`
`Sheet 1 of 30
`
`US 9,304,654 B2
`
`10~
`
`32
`
`
`
`
`Load Custom Script andits Icon to Alter
`Application Logic of 30 Cartesian Space
`
`
`through Programmatic Access
`
`
`(External Proto Node) Represented by
`
`
`interactive Icon on Interface Panel
`
`@QOL
`
`©
`
`(F)
`
`©
`
`@
`
`oO
`
`@
`
`QOOO®@
`
`Visit Viewpoint
`
`Record Viewpoint or
`x,y,z Coordinate
`
`Add Desktop
`Window Command
`
`Add Browser
`
`Add Documentor
`Application Command
`
`Add Text
`Command
`-
`Add Picture
`ommand
`Add Music Command
`
`Add Video Command
`
`Record Sound
`Command
`Add Map Command
`
`Add 3D VRML or
`X3D File Command
`
`Save Space/Email
`Space Command
`
`Add Advertising to
`3D Scene
`
`42
`
`50
`
`62
`
`75
`74
`
`38
`
`92
`
`94
`
`104
`
`30
`SeanEvent Handler
`
`
`or
`End
`UserInput
`
`on Interface Panel to
`Execute Appropriate
` Helper Application
`
`
`
`
`
`
`
`Recalculate
`Geometry in
`Scene Basedo
`New Viewpoint
`or Perspective
`
`FIG. 1A
`
`3
`
`

`

`U.S. Patent
`
`Apr.5, 2016
`
`Sheet 2 of 30
`
`US 9,304,654 B2
`
`36
`©—4 aaa|PO©
`
`Program Prompts End
`End SorotHelperoad User to Locate Script Fite
`(D) App
`from Local Computer,
`lication) Icon from
`Network, Internet
`Input Interface Panel
`or other Location
`
`
`
`Prompt User to Choose Viewpoint Name
`Load x,y,z Position or
`from CompassorClick on ican of Previously4Viewpoint from Compass
`Recorded Viewpoint in 3D Cartesian Space
`File and Bindto it
`©
`
`Prompt User to
`
`
`Input Viewpoint
`
`
`Name(s) to be Added
`
`
`
`to Compass File
`
`&
`
`
`
`(Q)
`
`
`
`
`
`
`
`
`
`Draw Interactive 3D
`Store x,y,z Position
`and Viewpoint Name
`Icon/Geometry in Scene
`
`
`
`within 3D Cartesian
`to Represent Viewpoint
`
`
`
`Space to CompassFile
`with Viewpoint Name
`
`
`
`
`
`Initiate Open File Class or other Web
`Service, API, or Method for Loading
`
`Data from Local Computer,
`
`Networkor the World Wide Web
`
` Allow Drag and Dropof File, Document
`or Application into 3D Cartesian Space
`
`fram 2D Desktop
`
`
` Draw Interactive 3D Icon/Geometry in
`
`
`Draw Text, Map, Picture, Video,
`VRML, X3D or other Content
`Scene to Represent Sound or Music
`
`that if Clicked with Play Sound/Music
`into 3D Cartesian Space
`
`
`
`
`
`Prompt User to Input Name of 3D
`Save Space on Local Computer
`
`
`Cartesian Space to Save or Email.
`and/or Our Company Server (FTP).
`
`
`
`Add Nameto Compass File
`Server Files can be Public or Private
`
`
`Determine Subject of 3D Cartesian Space by Scanning all Words
`Associated with or within the File Names, Files, htm! Pages,
`
`
`Viewpoint Names, Hyperiinked Files or any other Available
`Information Associated with Items within the 3D Cartesian Space
`
`
`or Words Entered into the Input Interface
`$$
`©-
`FIG. 1B
`Oe)
`
`4
`
`

`

`U.S. Patent
`
`Apr.5, 2016
`
`Sheet 3 of 30
`
`US 9,304,654 B2
`
`
`
`
`
`Interactive Icon that Triggers Script is | End User is Prompted to Click
`
`Dynamically Added to Interface Panel to
`|-Interactive Icon on Interface Panel
`
`Initiate Command Using XML Encoding||to Initiate Program Altering Script
`
`
`
`
`
`
`
`There is no Control
`Determine Proper OS Controlto
`
`for this Information
`Retreive the Kind of information Requested
`
`
`
`
`
`
`Periodically Capture the
`
`
`on Screen Output of
`Execute Proper OS Control Specific to the
`Window for
`Said
`Kind of information Requested which
`
`will Act as a Mirror to its 3D Representation
`Program/Information as
`
`
`
`a Bit Map Image
`
`Map Visual Output of OS Control, Bit Map
`or API onto Arbitrary 3D Geometry
`
`Store Bit Mapin
`a Frame buffer
`
`
`
`Pass MouseClicks, Cursor Position,
`
`
`
`Create Device
`Keyboard Input and Movements from
`
`
`Input Event
`Control Map on 3D Geometry to the Mirror
`
`Handler in 3D
`Controf Running on 2D Environment
`
`
`
`Generate Table of Contents Page in the Form of an htmi Pagethat is Stored on
`
`
`Company Server Next to Saved Space File that Stores Space Name, Hyperlinks
`
`
`in 3D Space, Description, Picture(s) of 3D Space with the Purposeof being
`
`
`an Index for the 3D Spaceto be Crawied by the Major Search Engines as Part
`of their Web Crawling/Indexing. Place Subject (See Item "A") of 3D Spaceinto
`
`
`Meta Tags of html Page. Embed Option in html Page to Download PROGRAM
`Active X Control within html Page to Auto Install Saved Space from
`
`
`CompanyServerfor which T.O.C. htm! Page was Generated and
`Execute on their Computer with the Click of One Button on htmi T.0.C. Page
`
`
`Email Table of Contents html Page to Recipient(s) and Embed
`
`
`Option for Recipient to Download PROGRAMActive X Control within
`
`html Page to Auto Install and Load Saved Space from Company
`
`
`Serverfor which T.O.C. html Page was Generated and Display
`on their Computerwith the Click of One Button off html Page
`
`
`
`
`
`Return Appropriate text,
`Sort Words by Frequency to
`
`
`Picture(s), Video(s), Sound(s),
`Determine Subject of 3D
`
`
`Music, Hyperiinks or other
`Cartesian Space by Identifying
`
`
`
`Advertisement Content from
`Mast Frequently
`
`Advertising Server, Web
`
`
`Appearing Words
`
`Service or other Location
`
`
`Send Subject of 3D Cartesian
`Draw text, Picture(s), Video(s),
`
`Space via xmi Fite of Most
`Sounds(s), Music, Hyperlinks or
`
`
`other Ad Content from
`Frequently Appearing Words to
`
`Advertising Server or other
`Advenising Server through
`
`internet Request
`
`
`108
`
`Location into 3D Cartesian Space|
`
`FIG. 1C
`
`
`
`
`
`
`5
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 4 of 30
`
`US 9,304,654 B2
`
`20-~\
`
` User Input
`
`
`
`
`
`Recalculate Geometry
`in Scene Based on
`New Viewpoint
`or Perspective
`
`FIG. 2
`
`6
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 5 of 30
`
`US 9,304,654 B2
`
`jONUODOUSIQeU;
`
`pPrl
`
`8cl
`
`ysanbaysasayeqiu;puesaysuesy
`
`
`sanbayJasnaimdey
`
`ssaooyajewwel6olgyBnosu}
`
`
`doyseqdz0)soedg-qewoy
`
`uoHeziWOjsnDWweibolgUOposed)
`
`(
`
`soedsCEULM
`
`9cl
`
`
`
`uolesyddyyouney
`
`
`
`GESanoe1ayyyayea9
`
`
`
`aoedsueiseye@5
`
`eC
`
`01
`
`vel
`
`
`
`paysenbayUOEWJOJU]JOPUYSY}SAISOY0}[0.09SOJado1gsuIUIA}Ag
`
`
`
`UOHJEWJOJU]SIU}40}
`
`
`
`
`
`
`
`UOHRUWWO;U[/WeEIBOIgPIES40}
`
`
`MODUIAAJOINdINOu9aI0SUO
`au)aamdegAyjeapousd
`
`
`
`ysonboyMODULA,
`
`
`
`JOMNIOa]ealD
`
`yusWND0q
`
`abey
`
`fallgvedo
`genvado
`
`O€l
`
`Ort
`
`JoyNgeweeuldetqal01S
`
`8rl
`
`
`AysuioesqeAeutaiy01U0{diyJO
`
`
`dew1g‘104UODSO40NdynCjensiAdey
`
`Zvh
`
`a6eudewiqe@se
`
`
`
`UOEJUSSAIdaYGESH!0}JOLY@SBJOYIMYoU
`
`
`
`pajsanbayvoneuuojuljopulyayy0}aytoadgjoNUODSQsadolg91NVaxy
`
`€‘Sid
`
`‘UOHISOdJOSIND‘syoIIDSsnoWsseq
`
`JOUIWYBuy0}Asjauioes)GEUOdeyfoujU04
`WO]SJUBLdAOWYpuejndu}pueoahay
`
`
`
`yUaWUOJIAUAGzUOBuluUNyjomUOD
`
`
`CEUlJa;puRHJuaAg
`
`
`
`Induso1neqa}ea75
`
`
`
`
`
`OSL
`
`7
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 6 of 30
`
`US 9,304,654 B2
`
`
`
`
`
`00zabeyj0AuendaurwatagjOjuCZJasmolgdeyajealdSYUIEdALAyuapy
`
`
`
`
`v6ulS}INSaySAlgoaykgyaueagjonpuo5
`
`VP‘Sls@)861)S6t
`
`peuno|Lsay}-—feoesGM}—|weiborgvoneonddy[|[5484]
`
`
`
`platyyxays,o6edqay,s,ulGugYyoueasouloBedJesmoigGoAy
`
`
`
`
`
`
`
`
`
`
`
`
`
`SyNSayYDeISJoUGEWoysynsayYoreagBulsiegJO}ajqisuodsoyjdiogWoysn>peo7
`Bunsyndujyenbeyyoeaswo}UB8ISHOU!abegGay,ebednordGEM
`
`
`
`ysonbayyoseagpainjdesJeys,ulBUCQyatesJo}FUN18]UySIDSHOMei
`
`
`
`
`
`
`
`Pa}SUUDTHNYde3JO}18SpueabegUosyuIyadAyaioigpueabeuo
`
`O8lBZ}OLLpLcLOLb
`[Jeu|ereeeererroSNOOURA
`abeggamasledabe1asmoigMONabe29Buren
`
`
`
`4UILEdAHYORO}ALUNVADHOWLSSes1KqAbuUeUI
`
`899,49919991S991Yoo,991“991taal
`
`
`
`JasnwosGuysyoswasaalaoayynduyjsonbey
`
`
`
`
`synsayyoeagjoulBuCj9ajag0}JeSAJdwoig
`88h981ysl
`ZBLO61
`
`BurnjdesJoypoyjsyouleiag
`
`weJbolgoulByegyess8a
`
`aoepayu
`
`col
`
`
`
`
`
`POL
`
`c9L
`
`Ne09t
`
`8
`
`
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 7 of 30
`
`US 9,304,654 B2
`
`
`
`
`yeu)UONSaBedGaAAIKON10}Aujauu0asayEynojeosy
`
` aoedsq-¢UlUOISO,5}Buersuer,pueumogiBuyeogAgyoeysutAemy
`
`Jayynysieaddyi!
`
`
`
`
`OY}10}abeJOMUODJESMOIGGaAV}xaNJojNdjnojensi,MEIC
`
`
`siAeuyjopugjunaoedgueisayedq-¢au)ulaBegsynsay
`
`YyoleasS,uIBUQYoIRagWoyyUTEdAHBu}O}pajoauuOyFN
`
`
`
`poyoray
`
`902v0?
`
`
`
`jeUegaoepayy]S,wWesbo1dUOUOHNGYBnoU}soedsUe\seYEDGEUlUMEIG9q0}sabegayjoyORIS
`
`
`
`
`
`
`Jensijojnoke7Ajpoyw)0}wesboig0}ayejdwiaywoysngpeoojses)pugmory
`
`
`
`
`
`UeUlPayoidabeGay}S27]OU}0}YUITJeu)SaBEYGan,OU}|[epuesdedgUeIsaeDGEau)UlPanoig
`
`a6eGanjse]94}UOpuNoaeyeLySyUIPOdALSy)josabeGon,94)IIeJO}g0}JOSHpuyMOI
`
`
`
`
`
`weJbeiqye}Se9INESyUIT/U|SYUI]Bag-BBegjojUODJesmolgGayJUeLIND0}soedSjudDElpy
`
`
`
`
`
`seabedJaSmolgGafAJOYOR}SJENSIASU}Jo}xepUyseyoy0}adedsUEISAaLED
`
`
`
`
`
`
`
`
`
`QeulUMBIGSq0}YORISJeENSIAeUleBegJsuI4seaBeYsynseayMei
`
`c0e
`
`
`
`yoR}Sjensi,ulSebeyjoAeuyposwesBeiqaa-soRpa}u]WoysNn95
`
`
`
`
`
`
`yBnory)yoR}S[eNsiAUlSeabedJaSMOJGGaLOS0}JaspuyMoly
`
`jduiolg
`yORISJENSIAMONBumelSAqaGedsynsayYeasJO9gIXONJO}SSa001qJeaday0}Jasq
`
`
`
`
`BABS Bbc
`JasnAgpaydwosdUONEDO7JaUIOJOJaAJAaSqanA‘OMANVAINdWOD(e907©}aIL4
`
`
`
`
`
`
`av‘Sls
`
`9
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Apr.5, 2016
`
`Sheet 8 of 30
`
`US 9,304,654 B2
`
`FIG. 5A
`
`
`
`
`FIG. 5B
`
`10
`
`10
`
`

`

`U.S. Patent
`
`Apr.5, 2016
`
`Sheet 9 of 30
`
`US 9,304,654 B2
`
`718
`
`720,760
`
`=
`
`aLI
`
`FIG. 5C
`
`11
`
`11
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 10 of 30
`
`US 9,304,654 B2
`
`
`
`YORISIXBNulSHEYS14JOJUIOdMaIAdnaso|DojPUIG
`Ovevee
`
`yOB}ISSNOINaulBBEjsutJoyuIOdmalAdnesojya}pug
`
`
`
`
`yoerlsjensiAulabeyse7jojujodmaiadnasojy0}pug
`
`
`
`yorysjensi,ula6eqIXxenjoJulodmaindnasoly0}puig
`
`
`rensinensabeabegabegabed
`essnoinaig||S21XNSNOIAD|Isu4
`0SzYORISJeNsiUjaedSilJOJUIOdMa!Adneso}y0)pulg
`
`
`BES9cve?C&S0&282292
`
`
`
`adedsuRiseye)GeUla}eUIPI00DZ'A'xploday
`
`
`
`YOR}SJensiAulabeysnoiraldjowiodmel,dnaso}y0}puig
`
`
`
`
`
`Ja|pueyyueagueag
`
`
`
`
`
`jouegsoepayyyuojnduyJes)pu40}
`
`206
`
`0c2
`
`ove
`
`vyc
`
`OZ
`
`82
`
`rASTA
`
`9‘O14
`
`SZyohe
`
`
`
`ynduysasuo
`
`
`
`JepicpayioadsulyesAeuypos
`
`yoeis|}de}
`
`12
`
`
`
`
`
`
`
`peseguonoyoyeudoddyajnoax3
`
`12
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Apr.5, 2016
`
`Sheet11 of 30
`
`US 9,304,654 B2
`
`260
`
`End UserClicks Close Button on any Texture Map
`of Web Browser Control Representedin Visual Stack
`
`262
`
`Remove Web Browser Page
`from Visual Stack
`
`264
`
`266
`
`Position of Removed Pagein Stack
`Animated Page Before it was Moved
`
`Animate Forward Page after
`Removed Page fo Replace
`
`Animate Page after Previously
`Animated Page to Original
`Position of Previously
`
`268
`
`270
`
`of Previously Animated Page
`
`Animate Forward Next Page in
`3D Visual Stack to Replace Position
`of Previously Animated Page
`
`Animate Next Page in
`Stack to Position
`
`272
`
`274
`
`Exit Loapif the End of Visual
`Stack or Last Elementin
`
`Array is Reached
`
`FIG. 7
`
`13
`
`13
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 12 of 30
`
`US 9,304,654 B2
`
`
`
`
`
`
`
`
`IENLIAPapeo|uMmog||/ENUIAPapeo|uMod||feNHIAPapeojumag||jENUIApepeojuMmog||jenuIApapeojumog||jenw1Apapeojumog
`
`
`
`
`
`
`
`
`
`
`Bursnyeynduo9BuisnJaynduo9BuispJayndwioeyBuisnsayndwo5GulsnsayndworyBuisnsayndwos
`
`
`
`
`
`JUBIDJST)BIGHINW|[JUS2asAedAINYY||AIDJesnajdyiNyy|JUaIIDJesqaidyiny||yUaHDJeshaidninw
`
`
`AueyjoauAuewJo8uQAuewJo9u9AuewJoaudAueywjo3u9
`
`eoedse0eds
`
`
`0}sabueydpazieusnorJoJejndwogUsd0}aItyfeUIBUOUsNg
`
`dlUME$3Ulyoe3woysabueu5oziyjeuinor
`
`
`
`
`payeusbuCseBueyDaayJayndwo5Buipnjox3ayyeuibuc
`
`
`
`MONYdeWso}aBueuDpazipewnor
`
`
`
`JO}NdWODJOAIBSay}O}
`
`JUalDAangWO,JUBAZ[eNsiA,,
`
`
`
`yoesJO}oilyYesYstid
`
`J9AIBSUOSO]4
`puUeoIySaAlaIayJaAlag
`
`
`
`Osadedsjenul,panes
`SIU}0}TUNseysiigng
`}1SSa00yUPDaUOAUY
`
`
`
`
`S$!adedgjenpi,jeulsu¢d
`
`Paysign,puepajyeaip
`
`pugWolyJaniag0}
`
`Jayndwo9ssasn
`
`$82c8C
`
`UaHDJasyaidqinyy
`AueyyjoauG
`
`
`
`‘062
`
`14
`
`
`
`JU9AZ[eNsIA,
`
`8“Sid
`
`8,]Ua}19u)aBBueYD
`yBnosuypayeoipu|
`
`siasedsjenviA
`
`wJUSAZIENSIA,,
`
`SJUa}IOueaBueyD
`
`yBnoiuy)payeoipuy
`
`$1asedsjenut,
`
`JUBAZ[eNSIA,,
`SJUaI|DulaBueYyD
`
`yBnoiy)payesipuy
`
`$]e08dS[ENUIA
`
`JUSAZ|ENSIA,,
`$jU3IIDul!sbueyD
`
`yBnoJy)payesipuS|90edg
`Jeni,
`
`,UBAZJENSIA,
`
`$1U9I]DUleBBueyD
`
`yBnouuypayeoipuy
`
`stagedsjenuia
`
`JWUAAZfensip,,
`SJUA!DulaBueyD
`
`yBnou)payeaipuy
`
`s|a0edgjenuiA
`
`UBAZ[ENSIA,
`
`bz6z
`
`14
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 13 of 30
`
`US 9,304,654 B2
`
`6‘Did
`
`ve
`
`Itave<b
`
`15
`
`15
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 14 of 30
`
`US 9,304,654 B2
`
`©
`
`80b
`
`Ob‘Sid
`Lp(zo)
`
`elyOlLP¢9)
`blyLp
`BO8¢
`
`o.
`
`16
`
`16
`
`
`
`
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 15 of 30
`
`US 9,304,654 B2
`
`bb‘Sid
`
`
`
`mmnrdny
`
`
`a16009-Wos'sihoosmauchiyFB
`
`premioKelelpuMor
`seyuoney=Melh,
`
`
`
`WOONSW-WoorusurmMayediy(--
`
` t}pest2Mon-Ange-wooxegommyrccny(BFRe?
`
`Ory
`
`17
`
`J|
`
`17
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 16 of 30
`
`US 9,304,654 B2
`
`eb‘Sls
`
`La
`
`sowosey=maix
`
`
`
`18
`
`18
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 17 of 30
`
`US 9,304,654 B2
`
`b-56ESLE
`
`‘susuraepse
`
`
`
`is
`
`ith
`
`awe!OltTEVEL“SIACucrmonmooneymangerSL|wiccieO—Gey
`
`
`seyoneyaah
`
`
` hootterrithori
`Oz8smousAxiafensasoumswarstsGunesadeunmBindwucaAquaunarloay
`
`
`
`
`
`
`Auge20seerspenneuJs9yYRWOPEALpureedetyJOsoryoyUEeeddeOF
`
`SIROFvoRDeAGGJOUseyTRVREYenp3vumoiybutdeoousvoqaosd
`
`
`
`
`aayse“sreuopeuspoyera,Aue30juounoopjuatedReunestanudsin3Uonanposdat
`
`
`
`
`TuSuideo0pelgnsjeusTewSuNAKDWewNOCpJuatedjeuotsHax!stut
`
`
`
`
`
`
`TWNOISNAMID33YH1ONIGIAOUdHOdCOHANONWW3LSAS
`
`
`
`“wonoosyeipnyuupesexeseuoserueigoInq‘Aaurico2010
`
`
`
`SUAGHODALLNIEUNST1d03dMOH
`
`
`SOVIUBLNIBISTWOdveo
`SOUONTRSRAOD$
`eens
`
`
`
`
`Unu’suoneniesoy~yenury
`
`
`
`Ind"4-sBoregyeodBurmeuryE71
`
`
`
`byuouner-sueoedsGdepZeumuared
`
`
`
`soeg660S1“ecionuFl
`
`
`
`wang(GP
`
`
`
`Ipuruojensitasegopy((}
`
`SoptuatedjauorsaoxdmauLApogLL”(4oapis(2)
`
`
`
`seedsponesAW[)
`
`syoogeAW(3
`
`somnytedAWforsAWied
`
`19
`
`19
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 18 of 30
`
`US 9,304,654 B2
`
`
`
`gel‘Sls
`
`.webdaNodatagnaaMleycn,heyyread
`
`ssce“OIG,
`:tanepee)
`
`
`
`.aSeSE."aeae
`
`00
`
`
`
`
`
`
`
`*Gonyeang-usrrsMopegery-toosasustomanydoy(6)9.
`
`bp
`
`OLS
`
`Ovy
`
`O2e
`
`
`[_Witwosison|sawoisacu[1J|romenerscoe-saay-worcoptneanran(Bo.
`
`
`smannoymneycay(ED4CORETTOTOTSHILATHONHTT,»yyfNOH:worurmmrolTwae!
`snauyday(BD£31'TEER»'“HSN|SHAHSuryeaig-wea:
`
`
`
`
`
`
`{29:1}suondoyysowosoped‘|~ABojoulgay~wooayBRO-worunnsay -.
`
`
`
`
`
`$5¢/-4$24Suey0a¥dno2BUD.
`
`
`
`
`
`etWEUET[COU9,0PU0DJp9Uy0]BIEL
`
`
`
`“9SAAN«WOOAVOOLYSN-woorkeporesemmayrdin(2by:
`
`
`
`
`
`saanaBoog-weaabosnouicny(BDoj
`
`
`
`
`
`UATTyUaUsFOqUIaRw0D*
`
`
`
`(C__feoouormmmndaue)(6)(5)
`
`
`
`Premio}Ael4PuIMey
`
`seoreyMak
`
`
`
`SMAN18311=aH
`
`So‘shvirdery10gwezpgauvadn
`
`
`
`Tovoseumpeayseudoyt-worroouel
`
`“NSHWORSansAepo)-wooUSWoqusUrMMMyrct[EPBa:orgwong0H|SMBHOBEHOOT
`
`SMOKYNYED1-3
`
`SHOUUNDSHONDEYwenOBsMaLAMEIdN[EDG1
`
`20
`
`20
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 19 of 30
`
`US 9,304,654 B2
`
`PLSls
`
`oogedsoueuyirdny
`345|voqeonpa[oven{seinsnpu[svondy{spuodJ313[spuneramy|St00(g|Brigmeen|mOnUEAGOe|anee
`7
`ATWYNOKueg
`
`
`rWersett]camaags)12WidOre
`meenOEWeHETSueseatapes|
`
`
`O0'OLLSe189
`JoaBeByso)
`EUOLUINGS
`
`
`yuregooueig|doomsG9}[seen
`
`wulg-goas4empTpNOOUbesosambaoepeomrequdasSavesDIRSTRATOAOFSHUTDYSTON[|_NS3GreAvy]
`
`
`ppsaeend|AendsigoOuen
`onube
`
`+wor
`
`21
`
`21
`
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 20 of 30
`
`US 9,304,654 B2
`
`
`
`SbOld[Cerne
`
`
`
`
`
`BatewoyyayssaAgudyngyon"
`
`«Met
`
`
`
`piemo4heldpuumay
`
`S3aUOAB
`
`
`
`Ove
`
`22
`
`22
`
`

`

`U.S. Patent
`
`Apr.5, 2016
`
`Sheet 21 of 30
`
`US 9,304,654 B2
`
`4)FIG.16A
`
`YiewFavontes
`
`Bi
`
`Aa
`
`l
`
`23
`
`23
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 22 of 30
`
`US 9,304,654 B2
`
`€9b“Old
`
`ify SaLOAR
`
`—-MOIX
`
`24
`
`24
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 23 of 30
`
`US 9,304,654 B2
`
`
`
`VAL‘Old
`
`
`
`seysoaey=maiK
`
`
`
`25
`
`25
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 24 of 30
`
`US 9,304,654 B2
`
`ALL“Oldfa SOWOARTMOA
`
`26
`
`26
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 25 of 30
`
`US 9,304,654 B2
`
`DLL
`Sid
`
`SewOAR]Mal
`
`27
`
`27
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 26 of 30
`
`US 9,304,654 B2
`
`sewioAey=maik,
`
`Ove
`
`28
`
`28
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 27 of 30
`
`US 9,304,654 B2
`
`‘smau
`
`29
`
`bya
`
`6Old
`
`
`
`
`“W,N0K‘KepAlanaayesJoysway!OILGJ8A0UAA“SedldMO}
`paeuNoAsauaulWe185)
`
`SOULiO,MONSUL
`
`
`yeSw}!SMeupul4
`
`/“AUAHIOTGSOL,90)MONOU
`
`
`
`““HAOK‘KapArendates10)sw0y
`
` ‘spqsUoArpyueutuesPeaunOAmeuOU1B10¢)HOUALWOTMONRUTwoskeqowan ouReVEPORSequinaywoo
`
`
`UOT!GS@A0INIA“BeOudFO}
`ibaasmaupuesyooysMaa
`KegsudSmay
`
`
`woo'AeqaMMM
`
`““SJBU][OPBUEayyPURSMoU
`
`AFMPOLLIOIH-TOMAB}SOF
`
` suqndWPSTUIESTOYOUTTNTWoo
`
`
`MO){SB=SIPPYOWpUE
`
` {GUIS}WIGSOUGUT]1Sq)JeonERyTRENthoosesoLunAUAA,
`
`
`““SUOHEISOfPRLSNOUBA
`
`
`Cugytgary‘opyy
`smenpirestoos
`
`“gory‘smoukreyeas
`]ORIKOROU
`
`78SLUG}SOUPuls
`RESUSTHA
`
`29
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 28 of 30
`
`US 9,304,654 B2
`
`02‘Sid
`
`LequvoAesa
`“puaMio4feldpulmey
`
`AO
`soyuoney=mai,a
`
`csp
`
`30
`
`30
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 29 of 30
`
`US 9,304,654 B2
`
`be‘Old
`
`sayuoney=Maik
`[Rp9-5eaeWdES:s-goOZ‘BisnBnyOHeg-:eeSaWwdaG'¢-GO02‘p1enBeyRaSSwaz5's-902‘Bientiny
`
`
`
`
`099Sit‘guntey@a!
`
`
`
`iy siG fi
`
`6
`
`vor
`
`31
`
`31
`
`

`

`U.S. Patent
`
`Apr. 5, 2016
`
`Sheet 30 of 30
`
`US 9,304,654 B2
`
`eT)|e|
`
`saoreymaiz
`“sveppy
`
`SIETESacesoRNOTTSa
`
`ardosdBuyoOJaMoqffoO
`
`
`
`Angsnojeartf
`
`
`
`DIBAIOSAbidpLIMEN
`
`32
`
`32
`
`

`

`US 9,304,654 B2
`
`1
`SYSTEM AND METHOD FORDISPLAYING A
`TIMELINE ASSOCIATED WITH A
`PLURALITY OF APPLICATIONS
`
`CROSS-REFERENCE TO RELATED
`APPLICATION
`
`This application is a continuation of U.S. patent applica-
`tion Ser. No. 12/751,879, filed Mar. 31, 2010, which is a
`continuation of U.S. patent application Ser. No. 11/531,676,
`filed Sep. 13, 2006, now U.S. Pat. No. 7,735,018, which
`claimspriority pursuant to 35 U.S.C. §119(e) to U.S. Provi-
`sional Application No. 60/717,019, filed Sep. 13, 2005, which
`application is specifically incorporated herein, in its entirety
`by reference.
`
`COPYRIGHT NOTICE
`
`This patent document contains material that is subject to
`copyright protection. The copyright ownerhas no objection to
`the reproduction of this patent documentor related materials
`as they appearin the files of the patent offices of the United
`States or other countries, but otherwise reserves all copyright
`rights whatsoever.
`
`BACKGROUNDOF THE INVENTION
`
`1. Field of the Invention
`
`The present invention is directed toward graphical user
`interfaces for operating and accessing information on acom-
`puter, and moreparticularly, to a three-dimensional (“3D”)
`interactive computing interface and sorting interface com-
`prising information from real-time andstatic sources, includ-
`ing, but not limited to, meta search results from the Web;
`information from APIs, webservices, search engines, appli-
`cation programs, and networks; andfiles on the end user’s
`desktop.
`2. Description of Related Art
`Currently, people use computers by inputting information
`into the computerto achieve a given output. Often this can be
`aseries oftedious steps (mouse clicks and keyboard inputs) to
`run applications and documents or navigate to information.
`To get to new computing experiences, people often have to
`close their current applications and documents, hide them or
`overlap them on a finite desktop by drawing them ontop of
`each other, and then mine through folders within folders to
`find them againat a later date. The user’s desktopis finite, and
`one mustredo the sametasks over and over again. This wastes
`time by (1) requiring many mouse clicks to open and close
`documents, (i1) requiring one to rememberall the combina-
`tions of programs and documents one might need for a given
`purposeand(iii) requiring one to create elaborate hierarchical
`folder systems to aid in the process of storing and recalling
`applications and documents. This is primarily due to the
`limited space the end user has on their desktop.
`People currently compute within operating systems that
`present computer output, such as documents, applications,
`and operating system’s interface in a 2D (two-dimensional)
`visual display. After initially being loaded into the computer
`by the boot program, the operating system controls all the
`other programs in a computer. Typically, the component of
`the operating system that summonsthe style in which this
`output is displayed is called the GUIor graphicaluserinter-
`face. A successful GUI will use screen presentations includ-
`ing metaphorsthat utilize graphic elements such as icons to
`make an operating system’s input and output easier to man-
`age. Most computer operating systems incorporate a GUI that
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`utilizes two-dimensional graphics to capture, process, and
`output all input from an end user in a 2D form—havingheight
`and width only.
`This output is usually confined within a window thatis
`drawn on a finite-sized desktop, i.e., the working area of a
`computer, that has a given length and width. When the com-
`puter’s output exceeds this finite working graphical area,
`elements ofthe GUI(the windows)are typically drawn on top
`of each other such that the GUI components overlap one
`anotherother. In some operating systems, a shadow is drawn
`beneath these overlapping windows on the desktop to make
`them appearas if they have depth. This technique allows an
`end user to identify the overlapping windows moreeasily.
`Welive in a 3D (three-dimensional) world where we see
`that objects not only have a horizontal position (x) and verti-
`cal position (y) but also have depth (z) that is also known as
`time, according to the three-dimensional coordinate system
`of mathematics. This notion of expressing depth or time ina
`visual computer metaphor is important for the creation of a
`visual history of the end user’s computing sessions. By plot-
`ting new output of the computer (instead of replacing) in a
`virtual space that does not overlap or substitute whatexists on
`the finite desktop, a new virtual space through depth and time
`is created. For example, ifone wereto pullup the webpagefor
`the URLhttp://www.yahoo.com, and then click on a hyper-
`link (e.g., finance), the current webpagein its window would
`be replaced by the webpage for Yahoo! finance.
`3D has shownitself in computing primarily in the follow-
`ing areas: (1) games, (2) CAD/medical visualization, and (3)
`virtual worlds. A virtual world is a computer-simulated envi-
`ronmentthatits users can inhabit andinteract with via avatars.
`
`This habitation usually is represented in the form of two- or
`three-dimensional graphical representations of humanoids
`(or other graphical or text-based avatars).
`The navigation window of many desktop operating sys-
`temsuse controls and buttonsto allow endusersto navigate to
`other folders and windowsin the hierarchical structure of the
`file system. Often, in navigating to new windows, the new
`windowsreplace the display of the current window. Accord-
`ingly,
`it would be very desirable to provide an improved
`graphical user interface that allows the user to efficiently
`navigate though a virtual space wherein groups of windows
`can be easily organized, stored, and retrieved.
`
`SUMMARYOF THE INVENTION
`
`The present invention addresses the shortcomings of the
`prior-art systems and methods. In particular,
`the present
`invention is directed to a system and methodfor providing an
`improved 3D graphicaluser interface.
`In accordance with one aspect of the embodiments
`described herein, there is provided a graphical user interface
`that uses the two-dimensional (“2D”) display of a user’s
`computer to display three-dimensional (“3D”) objects in a
`simulated real-time 3D immersive Cartesian space.
`In one embodiment, there is provided a system whereby
`new computing output occupies new virtual space near the
`original output, without losing the original output. When an
`end user clicks on a hyperlink on the webpage,there appears
`in the virtual space a new webpagethatis linked to but does
`not replace the current webpage in its window; rather, the new
`webpageis drawn in a new virtual space. This way, the end
`user can Visit past visual computing momentsin time.
`In accordance with another aspect of the embodiments
`described herein, there is provided a methodfor providing a
`three-dimensional graphical user
`interface,
`comprising
`receiving an input from an end user, capturing computing
`
`33
`
`33
`
`

`

`US 9,304,654 B2
`
`3
`output from at least one computer source in response to the
`received end-user input, and presenting the computing output
`as at least two objects within a three-dimensional virtual
`space displayed to the end user.
`In one embodiment, the method further comprises gener-
`ating a timeline that includes an icon for each object presented
`within the virtual space, wherein the icons are organized in
`linear chronological order according to whenthe objects were
`presented withinthe virtual space anddisplayingthe timeline
`within the virtual space. In another embodiment, the method
`further comprises providing a database module for storing
`and categorizing data regarding each object presented within
`the virtual space, providing a hyperlink within the database
`module to respective viewpoint of each object presented
`withinthe virtual space, and displayingthe data regarding one
`or more of the objects within the database module presented
`along with virtual space.
`In accordance with another aspect of the embodiments
`described herein, there is provided a system for providing a
`three-dimensional graphicaluserinterface, comprising a dis-
`play screen, an inputdevice for receiving an input from an end
`user, a processor module operatively coupled to the display
`screen and the user input device, and a memory module
`operatively coupled to the processor module. The memory
`module preferably comprises executable code for the proces-
`sor to capture computing output from at least one computer
`source in responseto the received end-user input and present
`the computing output as at least two objects within a three-
`dimensional virtual space displayed on the display screen.
`Tn one embodiment, the memory module further comprises
`executable code for the processor to generate a timeline that
`includes an icon for each object presented within the virtual
`space, wherein the icons are organized in linear chronological
`order according to whenthe objects were presented within the
`Cartesian space, and display the timeline within the virtual
`space. In another embodiment, the memory module further
`comprises executable code for the processor to provide a
`database module for storing and categorizing data regarding
`each object presented within the virtual space, provide a
`hyperlink within the database module to respective viewpoint
`of each object presented within the virtual space, and display
`the data regarding one or more of the objects within the
`database module presented along with virtual space.
`In accordance with another aspect of the embodiments
`described herein, there is provided a system for providing a
`three-dimensional graphicaluser interface in a computernet-
`work, comprising a server connected to the computer network
`and a user-interface application executing in association with
`the server to provide the functions of receiving an input from
`an end user, capturing computing output from at least one
`computer source in response to the received end-user input,
`and presenting the computing output as at least two objects
`within a three-dimensionalvirtual space displayed to the end
`user.
`
`In one embodiment, the user-interface application execut-
`ing in association with the server further provides the func-
`tions of generating a timeline that includes an icon for each
`object presented within the virtual space, wherein the icons
`are organized in linear chronological order according to when
`the objects were presented within the Cartesian space, and
`displaying the timeline within the virtual space. In one
`embodiment,the user-interface application executing in asso-
`ciation with the server further providesthe functionsof: pro-
`viding a database module for storing and categorizing data
`regarding each object presented withinthe virtual space, pro-
`viding a hyperlink within the database module to respective
`viewpoint of each object presented within the virtual space,
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`65
`
`4
`and displaying the data regarding one or moreof the objects
`within the database module presented along with virtual
`space.
`In accordance with another aspect of the embodiments
`described herein, there is provided a network system for
`providing a three-dimensional graphical user interface, com-
`prising: a computer-server network comprising a plurality of
`servers in communication with each other;at least one display
`screen operatively coupled to the computer-server network;
`at least one input device for receiving an input from an end
`user, the input device being operatively coupled to the com-
`puter server network; and a software module for providing a
`series of screen displays to the end user, the software module
`being accessible by one or more ofthe servers of the com-
`puter-server network. The software module preferably com-
`prises instructions for directing the servers to capture com-
`puting output from at least one network source in response to
`the received end-user input and to present the computing
`outputas at least two objects within a simulated three-dimen-
`sional Cartesian space displayed on the display screen.
`In one embodiment, the software module further com-
`prises instructionsfor directing the servers to generate a time-
`line that includes an icon for each object presented within the
`Cartesian space, wherein the icons are organized in linear
`chronological order according to when the objects were pre-
`sented within the Cartesian space andto display the timeline
`within the Cartesian space. In another embodiment, the soft-
`ware module further comprises instructions for directing the
`servers to provide a database module for storing and catego-
`rizing data regarding each object presented within the virtual
`space, to provide a hyperlink within the database module to
`respective viewpoint of each object presented within the vir-
`tual space, and to display the data regarding one or moreofthe
`objects within the database module presented along with
`virtual space.
`In accordance with another aspect of the embodiments
`described herein,
`there is provided a computer-readable
`recording mediumfor storing a computer program that makes
`a computer execute: receiving an input from an end user;
`capturing computing output from at at least one computer
`source in response to the received end-user input; a