`Case 4:20-cv-01509—DMR Document 1-1 Filed 02/28/20 Page 1 of 24
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`EXHIBIT A
`
`EXHIBIT A
`
`
`
`
`
`USO095651 49B2
`
`(12) United States Patent
`Coatta et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9,565,149 B2
`Feb. 7, 2017
`
`(54) MEDIA MESSAGING METHODS, SYSTEMS,
`AND DEVICES
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`(71) Applicant: Brilliant Mobile, L.L.C., Chanhassen,
`MN (US
`(US)
`
`(72) Inventors: Jay David Coatta, Chanhassen, MN
`(US); John Mikkelsen, Wayzata, MN
`(US); Sergey Tolkachev, Minneapolis,
`MN (US); Robert Freidson, Saint
`Petersburg (RU); Max Freidson, Saint
`Petersburg (RU
`g (RU)
`(73) Assignee: Phoji, LLC. Minneapolis, MN (US)
`(*) Notice:
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 285 days.
`(21) Appl. No.: 14/072,418
`(22) Filed:
`Nov. 5, 2013
`
`(65)
`
`Prior Publication Data
`US 2014/0129.650 A1
`May 8, 2014
`Related U.S. Application Data
`(60) Provisional application No. 61/722,560, filed on Nov.
`5, 2012, provisional application No. 61/835,975, filed
`on Jun. 17, 2013.
`(51) Int. Cl.
`G06F 5/16
`H04L 2/58
`(52) U.S. Cl.
`CPC ..................................... H04L 51/10 (2013.01)
`(58) Field of Classification Search
`CPC .......... H04L 51/10; H04L 12/58; H04L 29/06;
`G06Q 10/107
`USPC ................ 709/204, 205, 206, 217, 218, 219
`See application file for complete search history.
`
`(2006.01)
`(2006.01)
`
`7,035,803 B1 * 4/2006 Ostermann et al. .......... TO4,260
`2002.0137507 A1* 9, 2002 Winkler ................ HO4L 12.583
`455,412.1
`2006.0053386 A1* 3, 2006 Kuhl ..................... G06F 3,0236
`71.5/773
`2006/0129633 A1* 6/2006 Potluri et al. ................. TO9,203
`2007/0203995 A1* 8/2007 Wang et al. ...
`TO9,206
`(Continued)
`FOREIGN PATENT DOCUMENTS
`
`KR
`KR
`
`4/2005
`10-2005-0037070
`2, 2007
`10-0680O34
`(Continued)
`
`OTHER PUBLICATIONS
`
`“Notification of Transmittal of the International Search Report and
`the Written Opinion of the International Searching Authority, or the
`Declaration” for PCT/US2013/035209 filing date Nov. 5, 2013, 4
`pageS.
`
`(Continued)
`Primary Examiner — Liangche A Wang
`Assistant Examiner Kaylee Huang
`(74) Attorney, Agent, or Firm — Patterson, Thuente,
`Pedersen, P.A.
`ABSTRACT
`(57)
`A wireless communications system that allows a mobile
`phone, tablet or personal computer user the ability to initiate
`the sending of a text message or email whereby the sender
`is able to include photographs, graphs, pie charts and the like
`within the flow of the actual word by word texting or email
`writing process, without depending on the traditional nec
`essary step to “attach' the photograph.
`13 Claims, 14 Drawing Sheets
`
`
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 2 of 24
`
`
`
`US 9,565,149 B2
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2007/0266090 A1* 11/2007 Len ....................... GO6F 17,211
`TO9.204
`
`2008/0285071 A1 11, 2008 Mori et al.
`2009/0055484 A1* 2/2009 Vuong et al. ................. TO9,206
`2009.0156170 A1* 6/2009 Rossano et al. ........... 455,412.1
`2013/001224.5 A1
`1/2013 Choi ................. HO4M 3/42.382
`455,466
`2014/0101553 A1* 4/2014 Nagel ................... G06F 3.0482
`71.5/728
`2014/0143330 A1* 5, 2014 Shaw ..................... G06Q 50/01
`TO9.204
`
`
`
`FOREIGN PATENT DOCUMENTS
`
`KR
`WO
`
`10-2009-0057655
`WO 2004/017596
`
`6, 2009
`2, 2004
`
`OTHER PUBLICATIONS
`
`International Preliminary Report on Patentability from PCT Appli
`cation PCT/US2013/068509, dated May 14, 2015, 8 pgs.
`* cited by examiner
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 3 of 24
`
`
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 4 of 24
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 4 of 24
`
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 1 of 14
`
`US 9,565,149 B2
`
`1&3!
`
`3m .
`
`
`gygfigm
`fiemmaj
`
`
` @mwans-Commun (5555
`
`fierswaé Library
`
`
`
`
`
`
`
`
`
`Wei: zswgfiicafii‘ime Mime; agispzsiii‘miarzsa
`
`
`Fig. l
`
`
`
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 2 of 14
`
`US 9,565,149 B2
`
`a
`
`28
`
`roo.
`8
`:
`8:38:
`83;xicatios:
`
`
`
`
`
`'-'-'-'-'-'-'-'-'.
`
`i:
`::
`$8:338
`:grw:“
`
`
`
`8:8:
`
`
`
`
`
`exti&essaging
`$8:::::8
`
`ixiefs
`88:
`i:8:y 3:
`88:338
`
`
`
`
`
`o
`it
`i.
`i-Mar Protocar Proteca.
`-MMP exting wet socket básed Protocol
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 5 of 24
`
`Fig. 2
`
`
`
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 3 of 14
`
`US 9,565,149 B2
`
`
`
`
`
`
`
`E.
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 6 of 24
`
`
`
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 4 of 14
`
`US 9,565,149 B2
`
`&
`
`Cloud
`wet Service
`
`
`
`8
`
`&
`&
`&
`&
`
`MMP organizer
`--
`
`
`
`
`
`Pisotocon's Es
`
`yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 7 of 24
`
`Fig. 4
`
`MMP exting
`
`
`
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet S of 14
`
`US 9,565,149 B2
`
`
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 8 of 24
`
`Fig. 5
`
`
`
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 6 of 14
`
`US 9,565,149 B2
`
`
`
`Exx: 8:888:
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 9 of 24
`
`Fig. 6
`
`
`
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 7 of 14
`
`US 9,565,149 B2
`
`
`
`Scrossing
`
`is extreis
`
`Fig. 7
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 10 of 24
`
`
`
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 8 of 14
`
`US 9,565,149 B2
`
`
`
`Organizer
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 11 of 24
`
`
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 12 of 24
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 12 of 24
`
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 9 of 14
`
`US 9,565,149 B2
`
`
`
`........................§
`QM? CLZEfiT ~ SERVER Wrofiac $3
`
`
`
`1 33“,.
`
`“= 3’?
`
`
` W3? mem’fw PmmmE
`
`
`
`
`WIMP ”filming? Megsagéng memi
`
`
`
`£358! 2
`
`Kisser "Z
`
`
`
`Texiing 2’
`{mapmder}
`Messaging
`{imfimer‘}
`
`
`Sewfie
`
`
`
`
`
`ding tr
`3
`‘.
`"- Til)”,
`.
`..
`.
`,
`.M"
`
`'= #Wfig‘m “33° m 3
`21. ire-{mm Sfimz‘é if:
`g
`_
`.
`x
`«i
`_
`_Q f5? .m
` :1
`
`, me an in ”'3‘
`3mm: 3,;er 22m {Jim
`scsaérsn smegma c;
`"
`
`
`
`22" “v 322 “1",x {
`x...
`
`Web Service
`
`233%?
`F‘hmewns
`
`
`
`{923mm}
`
`Library Same .
`
`3
`1
`3
`
`3
`:
`-'
`
` ;
`
`fihcfiocon refiners;
`
`
`mm fies: US$355;
`
`
`
`
`i? Mama? mats:
`
`
`
`
`
`
`éi Magma» deiiraemzs a";
`User 2 Er: £12m: {mt-ifs”: car
`E
`5
`if555E$5§§SF§21§ 1'0- Lésefl
`Memsagge send in
`
`Naiifimfinn farm?
`
`
`3 Mwasga pmgsarm
`2 mfléfig 35's,?em
`{m- sgmfiefizzgg in 5,5322?
`
`weperw,
`
`.'
`
`:
`
`_:
`
`if}. Emma
`
`
`
`
`
`Fig. 9
`
`
`
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 10 of 14
`
`US 9,565,149 B2
`
`
`
`
`
`33.38:8:
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 13 of 24
`
`8.33:38:8:3
`
`Fig. 10
`
`
`
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 11 of 14
`
`US 9,565,149 B2
`
`input
`
`*ressess
`
`stipat
`
`
`
`i. Pictestion ibrary Service
`
`
`
`
`
`
`
`-
`
`... ii:38 388: 8xxo:
`13. Assign proper it as -
`3.3, 88ogarize .ixaty
`3.8, sewirie views act existing
`toois 3 cc 3: 88.28
`if 888
`8, 88t: 838 &; gi:is
`18quiest
`
`2. exting Service
`
`
`
`3, .. 3i: 88f 3:{x}:8
`8.2. Esiasis gi:8::::::: xii:
`wis-à-vis
`2.3, 388v8 essages 388xi
`{:} :8388 888
`3.3. 388 88ssages 38
`{ei}: 8xxxxxis for itse
`*88si:ing
`
`3:38.8
`
`aw
`
`*88:3:38
`
`Photocons
`
`8:8° exting
`&ia:3888
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 14 of 24
`
`;
`
`3.3. iter-re: 88i set:328i
`3.3. *:::::::::::::::38
`
`
`
`:
`
`Fig. 11
`
`
`
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 12 of 14
`
`US 9,565,149 B2
`
`
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 15 of 24
`
`
`
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 13 of 14
`
`US 9,565,149 B2
`
`--------------------------------------
`
`3:33
`
`save picturer
`
`a---
`
`
`
`i&is
`*:::::::g::8
`p:rsora itsary
`
`ise
`
`Ste: i.
`xiak88 3: $888; it: 88 (3
`saw: it *ote:{x}; i.ittay
`
`
`
`Sir
`&pgicatio:
`
`Step: 2.
`8: 83:38::::::
`i:it:388
`with 8:8 cite register:
`&
`S:8:
`
`-
`
`Y
`
`&is
`{x}8:88:38:3:38
`Syster
`
`1.&
`&
`
`-
`323
`
`
`
`x8
`Step 3.
`s& X:
`yes 3r. 888 essages
`x8
`with MMP tags within 1
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 16 of 24
`
`323
`
`Fig. 13
`
`
`
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 14 of 14
`
`US 9,565,149 B2
`
`Pair text MMP message
`
`Parsed and rendered essage
`
`ir
`1 hi 8c:38: working 838 of
`i&isió:38::::::.
`...
`
`be
`
`8: 8888t: viorking 8:38
`ieyek;88::: p33}.
`sixxix."
`X
`
`sight {xx' --> wic::ier where air fight
`3:::::::::::::::::::.
`
`(location)
`
`Food is great:
`
`So, tikirt; if sic:3 stay is: {xug;&
`ge cays (weeks (3: 38y 38 years.
`
`&c, 88king : : sixxix: stay
`if {x}{3:33:38: Riays
`weeks 33 83y 88 years.
`
`$8is is y ::::::::stats.3 tie at:
`:
`Sinai: isote: (cal 1)
`
`is is try caret state citi
`art smartphore:
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 17 of 24
`
`
`
`
`
`Your frienci - Yo: Si:{x:
`
`Fig. 14
`
`
`
`1.
`MEDIA MESSAGING METHODS, SYSTEMS,
`AND DEVICES
`
`US 9,565,149 B2
`
`CLAIM TO PRIORITY
`
`This application claims the benefit of U.S. Provisional
`Patent Application 61/772,560 entitled “RMS Communica
`tion Service' filed Nov. 5, 2012, and U.S. Provisional Patent
`Application 61/835,975 entitled “Reach Messaging Meth
`ods, System, and Service' filed Jun. 17, 2013, each of which
`is incorporated herein by reference.
`
`10
`
`TECHNICAL FIELD
`
`This application relates generally to electronic device
`communication, and more particularly, to systems, methods,
`and devices for the insertion of media data within electronic
`device communication.
`
`15
`
`BACKGROUND
`
`In a traditional texting or messaging environment, in
`order to insert media data other than text, a user must go
`through a cumbersome process of attaching the media data,
`select the media data through a series of menus, or otherwise
`go through similar cumbersome processes of attaching the
`media data. Further, such inserted data is typically generic
`and impersonal to the users.
`Therefore, there is a need for a messaging system that
`optimizes the information power of the communication
`process between users and further adds a personal feeling
`and experience, which in turn, Substantially increases the
`amount of information per message.
`
`SUMMARY
`
`Embodiments of the present application substantially
`meet the aforementioned needs of the industry. In an
`embodiment, a Media Messaging Platform (MMP) is a
`cloud-based communication platform, allowing mobile and
`tablet users to seamlessly integrate texts, images, photo
`graphs audio and video (Rich Media Data) within texting
`and messaging interactions with correspondents without
`having to interrupt the mobile user's texting flow skipping
`the laborious “step” of “attaching such rich media data into
`a mobile user's message.
`In embodiments, MMP provides users with a semantically
`organized multi-dimensional communication space for per
`Sonalizing rich media content (for example, images, photo
`graphs, audio, video, etc.) and integrating the content into a
`more emotionally enriched communication experience
`between MMP users.
`In embodiments, MMP can include a set of tools for
`content organization and integration of Such personalized
`rich media content within mobile related texts, chats and
`email communications.
`In embodiments, MMP’s “Content Organizer visually
`presents Rich Media Data Objects in the form of unique
`icons called “Photocons' or “Phoji’s.’ Photocons and Phojis
`are real photos of faces, brands, logos, animals and any other
`true photographic images (although the MMP shortcuts and
`“Content Organizer” can also be used with traditional non
`photographic emojis or emoticons, as discussed below).
`According to embodiments, a wireless communications
`platform allows a mobile phone, tablet or personal computer
`user the ability to initiate the sending of a text message or
`email whereby the sender is able to include photographs,
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 18 of 24
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`graphs, pie charts and the like within the flow of the actual
`word by word texting or email writing process, without
`depending on the traditional necessary step to “attach” said
`photograph (graph, pie chart and the like) as is currently
`experienced with all existing texting and email platforms.
`As the user types email or text message, user is able to
`incorporate photographs, graphs, pie charts and the like by
`tapping on, clicking on or dragging existing photographs
`stored on the user's device. User also having the option to
`use “qwerty symbol short-cuts' like the exclamation point,
`colon, semi colon and the like which are pre-assigned by
`user to user-identified photographs, pie charts, graphs and
`the like, giving user the ability to instantly “shortcut the
`inclusion of said photographs, images, pie charts graphs and
`the like within the user's texting flow experience.
`For example, user is writing an email explaining with
`words a particular “financial ownership break down. Rather
`than the user attaching an image of a "pie chart graph' to the
`bottom of the email (which must then be opened by receiver
`separately taking reader away from text), the user (sender)
`can simply type “pc' within the flow of their email typing
`and the image of said "pie chart graph will automatically
`“place itself within the paragraph of the words being typed,
`the symbol “pc' having been pre-assigned to said image of
`"pie chart graphic' within user's device or photograph
`library.
`A similar example can be made within a text messaging
`experience between users in a chat or texting experience. For
`example, the platform user can type a text message to a
`friend explaining, “We won the game!' Traditional texting
`platforms allow a user to add Small, animated "Smiley
`faces.” “sad faces.” “heart symbols and the like. These
`“non-rich media icons' have come to be known worldwide
`as ubiquitous yellow “smileys, otherwise known as “emoti
`cons” or “emojis.” These small animated icons are not real
`photographs or real images, yet add to the expression of
`texted words within a given message's context.
`According to embodiments, the platform allows the user
`to send the same expressive text message, “We won the
`game adding a real photograph of say the team cheering,
`or the individual smiling. Thus the emotional experience by
`the receiver of the text message is far more personal as it is
`a photo of the actual person who just “won the game' rather
`than a “small yellow smiley face' which may or may not
`represent the truest nature of the text message.
`A text message might be sent reading, "Let's have a beer
`. . .
`. The sender typing the symbol “beer allowing the
`pre-assigned photograph of a tall cold mug of beer to follow
`the text message without the extra step of “attaching” said
`photograph. User can insert or otherwise couple an unlim
`ited number of photographs within the texting cell, which
`cannot be accomplished by “attaching photos' within the
`traditional texting experience.
`In an embodiment, a communications system comprises a
`first user device presenting a first user interface; a second
`user device presenting a second user interface; a messaging
`cloud including: at least one database of rich media data, a
`messaging service configured to: receive a message, the
`message including a rich media data request from the first
`user, retrieve the requested rich media data from the at least
`one database of rich media data, insert the retrieved rich
`media data into the message, and present the message to the
`second user device via the second user interface; and a
`network operably coupling the first user device, the second
`user device, and the messaging cloud, wherein the message
`
`
`
`3
`is originally formatted on the first user device via the first
`user interface Such that the rich media data request is within
`the message.
`In an embodiment, a method of streamlined media inser
`tion on an electronic device comprises initiating, by the
`electronic device, a messaging session with a receiving user;
`receiving a message on the electronic device, the message
`including at least one media insertion request; evaluating the
`at least one media insertion request; retrieving media from
`at least one database of media based on the evaluated at least
`one media insertion request; and inserting the retrieved
`media into the message.
`In an embodiment, a wireless device in electronic com
`munication with a messaging service along a wireless net
`15
`work, the messaging service configured to evaluate at least
`one media insertion request, retrieve media from at least one
`database of media based on the evaluated at least one media
`insertion request, insert the retrieved media into the mes
`sage, and present the message to a receiving user, the
`wireless device comprises a memory; and a processor in
`electronic communication with the memory, the processor
`configured to: initiate a messaging session with a receiving
`user, receive a message including at least one media inser
`tion request, and transmit the message to the messaging
`service.
`The above summary of the invention is not intended to
`describe each illustrated embodiment or every implementa
`tion of the present invention. The figures and the detailed
`description that follow more particularly exemplify these
`embodiments.
`
`10
`
`25
`
`30
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The invention may be more completely understood in
`consideration of the following detailed description of vari
`ous embodiments of the invention in connection with the
`accompanying figures, in which:
`FIG. 1 is a block diagram of an MMP Architecture,
`according to an embodiment of the invention.
`FIG. 2 is a block diagram of an MMP Cloud Architecture,
`according to an embodiment of the invention.
`FIG. 3 is an illustration of an MMP Semantic Model,
`according to an embodiment of the invention.
`45
`FIG. 4 is a block diagram of components of an MMP
`architecture, according to an embodiment of the invention.
`FIG. 5 is an illustration of raw pictures in a picture library,
`according to an embodiment of the invention.
`FIG. 6 is an illustration of an MMP Organizer Layout
`page, according to an embodiment of the invention.
`FIG. 7 is a block diagram of the design structure of a
`mobile MMP application, according to an embodiment of
`the invention.
`FIG. 8 is a block diagram of generic Smartphone views
`and controls, according to an embodiment of the invention.
`FIG. 9 is a flow diagram of MMP information and
`communication, according to an embodiment of the inven
`tion.
`FIG. 10 is an example illustration a photocon library
`having images and the respective emotional tags, according
`to an embodiment of the invention.
`FIG. 11 is a diagram of interaction between MMP users in
`relation to one or more MMP databases, according to an
`embodiment of the invention.
`FIG. 12 is an MMP database class diagram, according to
`an embodiment of the invention.
`
`35
`
`40
`
`50
`
`55
`
`60
`
`65
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 19 of 24
`
`US 9,565,149 B2
`
`4
`FIG. 13 is a block diagram of interaction between MMP
`users in relation to Personal Environments and Services,
`according to an embodiment of the invention.
`FIG. 14 is a diagram of an example inserting Photocons/
`Phojis, according to an embodiment of the invention.
`While embodiments are amenable to various modifica
`tions and alternative forms, specifics thereof have been
`shown by way of example in the drawings and will be
`described in detail. It should be understood, however, that
`the intention is not to limit the invention to the particular
`embodiments described. On the contrary, the intention is to
`cover all modifications, equivalents, and alternatives falling
`within the spirit and scope of the invention as defined by the
`appended claims.
`
`DETAILED DESCRIPTION OF THE DRAWINGS
`
`According to embodiments of the invention, MEDIA
`MESSAGING PLATFORM (MMP) is an alternative Mobile
`Messaging Communication Service to Short Messaging
`Service (SMS) and Multimedia Messaging Service (MMS)
`implemented in such applications like Skype, Microsoft IM,
`ICO, Google Chat, etc.
`MMP improves Social messaging and texting services by
`increasing information power of communication process
`between participants by adding “personal feeling and expe
`rience, which should optimize an amount of information per
`message.
`In an embodiment, MMP uses two form of MMP of
`exchange objects: Messages and Texts. Message is HTML
`string, which constructed by the Sender using internal
`HTML format. Receiver will see them in traditional rich web
`format. Such Message when arrived at Receiver's environ
`ment will be properly rendered by Web browser or by MMP
`Application. Text is a plain string where tags presented as a
`combination of characters on the Sender's side. On the
`Receiver's side, these tags would be interpreted and substi
`tuted with rich HTML objects (pictures, audio, etc.).
`In an embodiment, MMP can operate in two service
`modes: Interactive Chat and Notification Service. MMP
`Interactive Chat allows participants to exchange with Mes
`sages and Texts while both participants are being online in
`real time. MMP Notification Service allows sending Texts
`and Messages without all participants having to be online
`simultaneously.
`MMP can comprise a family of protocols and applica
`tions, which allows Socially engaged users to include real
`photographs or another "rich objects within a text messages
`without having to “attach' the photo to the text or interrupt
`user’s “texting flow.”
`Similar to the Emoticon's libraries inserted inside the
`texting field, the MMP real photos icons (Photocons) are
`similarly inserted from the personal "emotional library
`within the user's texting field. The user can add his/her
`personalized photos and replace standard Emoticons in
`MMPUser's Photocon Library or alternatively the user can
`assign additional text symbols, numbers and letters to asso
`ciated photographs for future “easy insert” in texting field.
`During the chat session or preparing SMS message in the
`framework of Notification Service the user can open the
`MMP Photocon Library and click on a MMP Photocon or,
`alternatively the user can assign text symbols, numbers and
`letters to associated photographs.
`The assigned symbols, letters or numbers can be typed
`while texting, resulting in immediate and automatic place
`ment of the associated Photocon where the symbol, number
`or letter was typed. An example of the uninterrupted “Pho
`
`
`
`6
`fication includes ping as one of the variety of notifications,
`when one of MMP users enters MMP and his/her correspon
`dent is offline (hibernate).
`MMP Photocon Library System and Database/MMP Seman
`tic Model
`In embodiments, a Photocon Library 222 is a Cloud
`Service and Database System based on a MMP Semantic
`Model illustrated in FIG. 3. See also FIG. 10. This model
`allows categorizing of Personal Image Library and signifi
`cantly optimizes the image search during chatting and
`texting. It provides personal rich media content organization
`based on multi-dimensional Social and emotional content
`classification, preparing it for seamless integration into
`mobile messaging, comprising: semantically organized
`multi-dimensional communication space and multilayer rich
`content structure; content representation, which associated
`with every particular contact person in user's contact list,
`including types of interpersonal relationship (friendship,
`family, professional, romantic, etc.) and user's current emo
`tional state 810 (Happy 811, Excited 812, Angry 813, etc.).
`In an example, consider the following scenario:
`Step 1. A user selects the contact for MMP chat session. This
`selection will automatically select the corresponding layer
`with a unique set of Photocons.
`Step 2. The user is able to define his/her “Emotional State'
`or “Temperature' for this particular chat by using slider 810
`or sequential cyclic color setter for MMP Mobile applica
`tion. This selection will pick out one particular column
`(color) on the selected layer: green (Happy 811), yellow
`(Excited 812) or red (Angry 813).
`Step 3. During the dialogue, the user is able to dynamically
`change the current emotional state 810 of the dialogue and
`shortcut will select the Photocon from the correspondent
`column automatically.
`In embodiments, a particular Substitution of emoticon's
`symbol instead of the shortcut will depend on the current
`square of the Semantic Model 800, namely on contacts,
`dialogue “temperature' and current emotion 810.
`MMP Web Service
`In embodiments, MMP Cloud Web Service 221 connects
`users to MMP Photocons Library System 203 and Database
`204, and at the same time provides support for the Social
`Networks content share and other traditional Web services.
`FIG. 4 illustrates the components of MMP architecture at a
`system level.
`MMP Cloud Web Service 221 integrates different System
`components such as MMP Organizer 220, MMP Parser 230,
`MMP Application 240, the Photocon's IDs component 231
`and Main UI 250 to organize rich media personal content
`and to create individual layer for each person from user's
`contact list.
`MMP Parser 230 analyzes text string and substitute MMP
`shortcuts with appropriate Photocons. MMP Application
`240 is a platform dependent implementation of MMP client
`side functions such as photo transfer, asynchronous push
`notification and other. Photocons' IDs 231 is a part of MMP
`Database storing and accessing relations between Photocons
`and Rich Media Objects. Main UI 250 is client based
`integration of MMP Web based components and platform
`dependent applications 240.
`MMP Organizer 220 provides an ability to upload rich
`content items from the Web, desktop or from a Picture
`Library, such as that illustrated in FIG. 5 on smartphones or
`tablets. An example MMP Organizer layout page is illus
`trated in FIG. 6.
`MMP Organizer allows user to collect, classify and rear
`range Photocons, as well as to initiate chats and invite others
`
`Case 4:20-cv-01509-DMR Document 1-1 Filed 02/28/20 Page 20 of 24
`
`40
`
`45
`
`65
`
`US 9,565,149 B2
`
`5
`
`10
`
`15
`
`5
`tocon Placement within the user texting experience would
`be the ubiquitous “smiley face' emoticon. For example, the
`universal “colon symbol combined with a “right parentheti
`cal symbol' automatically places a “smiley face' emoticon
`within the user's text message where the combo of these two
`symbols is typed.
`MMP has the universal QWERTY keyboard symbols
`associated with all existing emoticons, e.g. Smiley, Sad,
`Angry, Nerd, Heart, Crying, Yawn and so forth. Upon a
`user's MMP application download to their mobile device,
`the MMP application can include a library of stock Photo
`COS.
`System Architecture
`According to an embodiment, the MMP general architec
`ture 100 incorporates multiple components, applications and
`environments, as illustrated in FIG. 1. Architecture 100 can
`include MMP Cloud 201, MMP Communication System
`202, and MMP Photocons' Personal Library 203 and further
`includes MMP Personal Environment 300, which in turn
`contains Mobile Applications 301 and Web applications 302.
`An embodiment of the System’s Cloud architecture is
`depicted in FIG. 2. It includes a representation of main MMP
`components 100 and a representation of the communication
`protocols 110, 111, 112 between structural elements of MMP
`25
`System.
`MMP Communication System
`According to an embodiment, MMP Communication Sys
`tem 202 comprises different modules performing system
`critical tasks and provides connection and command inter
`pretation received from users’ devices 300. In embodiments,
`system 202 can include MMP Web Service 221, MMP
`Photocons Library and Database 222 and MMP Text/Mes
`saging Service 223.
`35
`MMP Text/Messaging Service
`In embodiments, MMP Text/Messaging Service 223 com
`prises a module configured to handle the Texting/Messaging
`modality. The interaction can be handled both ways using
`duplex channels based on MMP Photocon Protocol 111 and
`HTML5 Web socket texting protocol 112, in an embodi
`ment. The System provides interface and command proces
`Sor to interpret tags and Support interaction with Photocon
`Library and Database System 222.
`MMP Notification System
`In embodiments, MMP Notification System 210 provides
`delivery and notification mechanism to users currently
`offline using MMP Notification Protocol 110. Notification
`System 210 checks if a user has subscribed for SMS
`notification and, if yes, will send SMS and/or e-mail with
`50
`URL to MMP formatted message. If not, Notification Sys
`tem 210 will periodically check user status and will deliver
`the message when it is possible. Notification System 210
`includes custom alert Sounds notification when it is received
`by a MMP user. A MMP user is able to transmit a photocon
`55
`to a non-MMP user's mobile phone number using MMP
`Demo account invitation, in embodiments.
`MMP Push Notifications System
`In embodiments, MMP Push Notification System com
`bines sound and vibration alerts with specific pictures and
`60
`automatic actions such as open chat with calling person,
`show a picture or play an audio or video clip. MMP Mobile
`Applications 310 installed on mobile device is capable to
`receive notifications even in a “hibernate” mode. MMP Push
`Notification System includes protocols, which are capable to
`send “push notifications” from MMP Text/Messaging Ser
`vice 223 to MMP Mobile Application 310. The push noti
`
`30
`
`
`
`8
`MMP Protocols can define the set of control commands
`and its flow that manages the users' texting/messaging
`session. FIG. 9 depicts interaction between users and Pho
`tocon Library 222 and the information flow between the
`modules of the System based on BMP Protocols. It also
`depicts the architecture of communication and connections
`between System components. The three major Services are
`the backbone of MMP system: Web Service 221, Photocons
`Library (for example, that shown in FIG. 10) and Database
`Service 222 and Texting/Messaging Service 223.
`An interaction between the Users through these major
`modules 221, 222 and 223 is established and controls by
`Photocons Protocol 111 and Texting/Messaging Protocol
`112, which shown in FIG. 9 and can be illustrated by the
`following steps and control commands according to the
`session scenarios:
`1. User 1 (initiator) initiates texting/messaging session with
`User 2 (responder) via module 223.
`2. Module 223 sends invitation to User 2 and Chat session
`is established or request for Notification sent to User 1
`3. User 1 sends Text or Message to User 2 via module 223.
`4. Module 223 sends text to the Parser of the module 221
`with the request: “Photocon included (yes/no'?)
`5. Module 221 sends Photocon request to the Library 222.
`6. Module 222 checks if Photocon URL exists in the Library
`(yes/no?) and sends respond to module 223.
`7. Module 223 wrapped Photocon into internal format and
`prepared Photocon for transferring to User 2 (responder).
`8. Message delivered to User 2 in Chat mode or Message
`sent to User 2 in Notification format.
`9. Message prepared for rendering in User 2 texting system
`and sent to module 222.
`10. Module 222 provides Image to User 2.
`11. User 1 sends request to module 222 to saves Image into
`Photocon Library.
`12. Image saved into Photocon Library by module 222.
`13. User 2 requests the Photocon Library 222 to show the list
`of available Photocons.
`14. Module 222 provides User 1 with the List of Photocons
`available for selection.
`MMP Integrated Environment
`In embodiments, MMP Integrated Environment top-level
`structure consists of three main services: Photocon Library
`Service, Texting Service and Presentation Service. FIG. 11
`depicts a diagram of interaction between MMP users in
`conjunction with MMP Database 204, including data struc
`tures for personalization database 204-1 and structure for
`texting and messaging database 204-2. As depicted, the
`interaction can comprise a number of Subsections.
`In an embodiment, the photocon library service can
`interface to MMP users 103 via Design Structure 320 and
`personalization database 204-1. In an embodiment, the pho
`tocon library service is configured to, among other pro
`cesses, initiate a user account, upload raw images, assign
`proper ID emotional tags to images, organize and reorganize
`the library, provide views and editing tools to crop and resize
`images, and return image(s) on the user's request.
`In an embodiment, the texting service can interface to
`MMP