Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 1 of 34
`
`
`
`6:20cv693
`
`EXHIBIT A
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 2 of 34
`111111
`1111111111111111111111111111111111111111111111111111111111111
`US009691429B2
`
`(12) United States Patent
`Leiberman et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9,691,429 B2
`Jun. 27,2017
`
`(54) SYSTEMS AND METHODS FOR CREATING
`MUSIC VIDEOS SYNCHRONIZED WITH AN
`AUDIO TRACK
`
`(71) Applicant: Mibblio, Inc., Brooklyn, NY (US)
`
`(72)
`
`Inventors: David Leiberman, Brooklyn, NY (US);
`Samuel Rubin, Brooklyn, NY (US)
`
`(73) Assignee: MIBBLIO, INC., Brooklyn, NY (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.c. 154(b) by 43 days.
`
`(21) Appl. No.: 141708,805
`
`(22) Filed:
`
`May 11, 2015
`
`(65)
`
`(51)
`
`(52)
`
`(58)
`
`Prior Publication Data
`
`US 2016/0336039 Al
`
`Nov. 17,2016
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2011.01 )
`(2011.01 )
`
`Int. Cl.
`H04N 5/93
`G11B 27/031
`H04N9/82
`G11B 27/34
`H04N 21/43
`H04N 21/8547
`U.S. Cl.
`CPC ............ G11B 27/031 (2013.01); G11B 27/34
`(2013.01); H04N 9/8211 (2013.01); H04N
`21/4307 (2013.01); H04N 21/8547 (2013.01)
`Field of Classification Search
`CPC ... GliB 2220/90; GliB 27/034; GliB 27/34;
`GliB 27/024; GliB 27/036
`USPC .......................................................... 386/285
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,265,248 A
`7,026,536 B2
`
`1111993 Moulios et al.
`4/2006 Lu et al.
`
`7,027,124 B2
`7,301,092 Bl
`7,500,176 B2
`8,046,688 B2
`8,244,103 Bl *
`
`4/2006 Foote et al.
`1112007 McNally et al.
`3/2009 Thomson et al.
`1012011 Adams et al.
`8/2012 Shore.
`
`912012 Salvucci
`8,271,872 B2
`8,704,069 B2 * 4/2014 Naik
`
`8,896,609 B2
`
`1112014 Xu et al.
`(Continued)
`
`OTHER PUBLICATIONS
`
`G11B 27/34
`386/278
`
`GI0H 1140
`84/612
`
`Musicstory: A personalized Music Video Creator; David A.
`Shamma, Bryan Paredo, Kristian J. Hammond; Proceedings of the
`13th Annual ACEM International Conference on Multimedia;
`ACM,2005.
`
`(Continued)
`
`Primary Examiner - William Tran
`(74) Attorney, Agent, or Firm - Robert W. Morris;
`Eckert Seamans Cherin & Mellott, LLC
`
`(57)
`
`ABSTRACT
`
`Systems and methods for creating music videos synchro(cid:173)
`nized with an audio track are provided. In some embodi(cid:173)
`ments' an audio track may be selected and one or more video
`takes may be captured while the selected audio track plays.
`The video takes may be analyzed while they are captured to
`determine, for example, a video intensity level and/or a
`number of faces recognized within each take. By capturing
`the video takes with the audio track, the video takes may be
`synchronized to the audio tracks so that they are in time with
`one another. Portions or subsets of the video takes may be
`paired or matched with certain sections of the audio track
`based on, for example, the audio characteristics for a par(cid:173)
`ticular section and video characteristics of a particular take.
`
`19 Claims, 17 Drawing Sheets
`
`TAKE: 1 TAKE: 2 TAKE: 1
`TAKE: 3
`TIME: 00:00 TIME:
`TIME:
`TIME: 01:00
`·00:30
`01:30
`00:30
`00:50
`00:50
`01:00
`722
`I32
`
`742
`
`712
`
`/1/700
`
`TAKE: 1 TAKE: 3
`TAKE: 2 TAKE: 3
`TIME:
`TIME: 02:00 TIME:
`TIME: 03:00
`02:40
`03:36
`02:40
`01:30
`02:00
`03:00
`~ m
`Z6Z
`
`Z6Z
`
`760
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 3 of 34
`
`US 9,691,429 B2
`Page 2
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`U.S. PATENT DOCUMENTS
`
`Creating Music Videos Using Automatic Media Analysis; Jonathan
`
`200210035475 Al * 3/2002 Yoda ....................... GI0L 15/24
`704/270
`
`3/2004 Mizushima
`2004/0060070 Al
`2005/0143915 Al * 6/2005 Odagawa.
`
`912005 Brown et al.
`2005/0190199 Al
`2006/0288849 Al * 1212006 Peeters
`
`2008/0037953 Al * 212008 Kawamura
`
`2008/0055469 Al * 3/2008 Miyasaka
`
`6/2009 Cohen et al.
`200910164034 Al
`1112010 Xu et al.
`201010290538 Al
`1212012 Luo et al.
`201210316660 Al
`2013/0330062 Al * 1212013 Meikle.
`
`2014/0160250 Al * 6/2014 Pomerantz.
`
`1012014 Chau et al.
`2014/0317480 Al
`1012014 Lanuners et al.
`2014/0320697 Al
`2015/0050009 Al * 212015 Svendsen.
`
`G08G 110962
`7011443
`
`GI0H 110008
`84/616
`H04N 5/783
`386/343
`GlOH 11368
`348/521
`
`H04N9/87
`386/285
`H04N 5/23229
`348/47
`
`Foote, Matthew Cooper, and Andreas Girgensohn; Proceedsings of
`
`the 10th International Conference on Multimedia; ACM, 2002.
`MuViSync: Realtime Music Video Alignment; R. Macrae, X.
`Anguera, N. Oliver; 2010 IEEE International Conference on Mul(cid:173)
`timedia and Expo (ICME), 534-9, 2010; ISBN-13: 978-1-4244-
`7491-2; DOl: 10.1109/ICME.2010.5583863; Conference: 2010
`IEEE International Conference on Multimedia and Expo (ICME),
`Jul. 19-23, 2010, Suntec City, Singapore; Publisher:
`IEEE,
`Piscataway, NJ, US.
`Cati Dance: Self-Edited, Self-Synchronized Music Video; Tristan
`Jehan, Michael Lew, and Cati Vaucelle; ACM SIGGRAPH 2003
`Sketches & Applications. ACM, 2003.
`Dubsmash by Mobile Motion GmbH; https:llitunes.apple.comiapp/
`dubsmashlid918820076 retrieved on May 11, 2015.
`Video Star by Frontier Design Group; https:llitunes.apple.comius/
`app/video-starlid438596432?mt~8 retrieved on May 11, 2015.
`
`G11B 27/036
`386/280
`
`* cited by examiner
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 4 of 34
`
`u.s. Patent
`
`Juo.27,2017
`
`Sheet 1 of 17
`
`US 9,691,429 B2
`
`PROCESSOR(S)
`
`MEMORY
`
`STORAGE
`
`[
`
`I
`
`I
`
`r~102
`
`t--104
`
`~106
`
`COMMUNICATIONS
`CIRCUITRY
`
`------108
`
`OUTPUT INTERFACE
`I
`I
`
`DISPLAY
`
`SPEAKERS
`
`110 ........... ---l
`112 -
`- ""-
`
`114
`
`-
`
`INPUT INTERFACE
`
`CAMERA(S)
`
`I
`
`MICROPHONE(S)
`
`EXTERNAL DEVICE
`INPUT
`
`116
`
`FIG. 1
`
`r-r--120
`~ r- ~ 122
`
`r------
`
`118
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 5 of 34
`
`u.s. Patent
`
`Juo.27,2017
`
`Sheet 2 of 17
`
`US 9,691,429 B2
`
`202
`
`208---
`
`212
`
`Projects
`
`/1/'200
`
`206
`
`SONG 1
`
`210a
`
`210
`
`[ SONG 7
`
`]lillJ
`
`210b
`
`FIG.2
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 6 of 34
`
`u.s. Patent
`
`Jun. 27, 2017
`
`Sheet 3 of 17
`
`US 9,691,429 B2
`
`/1-/300
`
`Songs
`
`Cancel
`
`302a-----", ~@ B
`-I
`l®-t
`I@
`. [ SONG3
`302d~ ~@ M
`l/
`I@ Q
`-j
`R s
`I@ u
`302f ----..,. -[
`V w
`I@ X
`~I
`y
`Z
`JJ.
`it
`
`@1
`A
`
`C
`D
`E
`
`I
`J
`K
`L
`
`N
`0
`p
`
`T
`
`304
`~
`
`306
`
`i. ~
`
`~ ';
`
`'-Ii'
`
`SONG2
`
`302c-~_
`
`"--'II
`
`SONGS
`
`SONGS
`
`302g--,_
`
`SONG?
`
`FIG.3A
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 7 of 34
`
`u.s. Patent
`
`Jun. 27, 2017
`
`Sheet 4 of 17
`
`US 9,691,429 B2
`
`SELECT SONG
`DURATION
`
`'" , , , , ,
`
`~50
`
`'" , , , , ,
`
`,
`: 3:35
`, ,
`
`364
`
`360
`
`362
`
`FIG. 38
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 8 of 34
`
`u.s. Patent
`
`Juo.27,2017
`
`Sheet 5 of 17
`
`US 9,691,429 B2
`
`404-
`
`402
`
`SONG 2
`
`/V400
`
`406
`
`408
`
`FIG.4
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 9 of 34
`
`u.s. Patent
`
`Juo.27,2017
`
`Sheet 6 of 17
`
`US 9,691,429 B2
`
`Normal
`
`Start
`
`-
`
`/'1./500
`
`502
`
`508
`
`.-------1---506
`
`FIG.5A
`
`550
`~
`
`504
`
`502
`
`518
`
`~
`\
`
`FIG. 58
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 10 of 34
`
`600~
`
`620~
`
`640~
`
`628
`
`648
`
`CAPTURE VIDEO
`
`CAPTURE VIDEO
`
`e •
`
`7Jl
`•
`~
`~
`~
`
`~ = ~
`
`2-
`?
`N
`~-....l
`
`N o ....
`
`-....l
`
`('D
`('D
`
`rFJ =(cid:173)
`.....
`-....l
`o ....
`....
`
`-....l
`
`660
`
`606
`
`624
`
`660
`
`606
`
`614
`
`00:10
`
`602
`
`FIG. 6A
`
`612
`
`602
`
`FIG. 68
`
`626
`
`602
`
`FIG.6C
`
`646
`
`d
`rJl
`,,'..c
`0'1
`'..c
`"""'" ~
`N
`
`'..c = N
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 11 of 34
`
`/(/700
`
`710
`
`7~
`TAKE: 1 TAKE: 3
`TAKE: 2 TAKE: 3
`TAKE: 1 TAKE: 2 TAKE: 1
`TAKE: 3
`TIME: 00:00 TIME:
`TIME:
`TIME: 01 :00 TIME:
`TIME: 02:00 TIME:
`TIME: 03:00
`03:36 I
`02:40
`-00:30
`01 :30
`00:30
`00:50
`01:30
`02:40
`00:50
`01 :00
`02:00
`03:00
`
`712
`
`1722
`
`1732
`
`1742
`
`1752
`
`1772
`
`1782
`
`1792
`
`726
`
`1736
`
`1746
`
`1756
`
`1776 A~
`
`~
`~~~-::=::I I
`I
`•
`" \ \
`TAKE: 3
`TAKE: 1 TAKE: 2
`
`\ I WI
`
`A \~ I 1((
`
`TAKE: 2
`
`I l~l'~I~~l
`TAK~ 1
`
`TAKE: 3
`
`TAKE: 3
`
`TAKE: 3
`
`e •
`
`7Jl
`•
`~
`~
`~
`
`~ = ~
`
`~ = ?
`
`N
`~-....l
`N
`
`0 ....
`
`-....l
`
`('D
`
`QO
`
`rFJ =-('D
`.....
`0 ....
`....
`
`-....l
`
`760
`
`FIG.7A
`
`d
`rJl
`,,'..c
`0'1
`'..c
`"""'" ~
`N
`
`'..c = N
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 12 of 34
`
`u.s. Patent
`
`Juo.27,2017
`
`Sheet 9 of 17
`
`US 9,691,429 B2
`
`a
`a
`0
`C0
`C0
`
`N
`a
`~~
`
`0
`0
`Q) a
`E a
`f- a
`
`IIlIlllIlll
`I:IjttJilru!
`ttU.UII
`IIlIIUII
`IliIIUIi
`III III III
`IliIIHII
`
`Ilillllllll
`III III III
`IliIIHII
`III III III
`lUI III
`IIlIlllIlll
`~~ilru:t
`!['U1Hr:u
`III III III
`Ililllllll
`IIlIllIlll
`Ilillllll
`IIlIlllIlll
`III III III
`Ililllllll
`III III III
`Ililllllll
`IIlIllIlll
`III III III
`IIlIlllIlll
`~~ru:t
`!['U1Hr:u
`III III III
`Ililllllll
`IIlIllIlll
`Ilillllll
`ttU.UII
`I:IjttJilru!
`!['U1Hr:u
`III III III
`Ilillllll
`IIlIllIlll
`III III III
`IIlIlllIlll
`l:Ij~ill1:l't
`!['U1Hr:u
`~~ru:t
`Ilillllll
`IIlIllIlll
`III III III
`IIlIllIlll
`l:Ij~ill1:l't
`
`~tpJElI
`III III III
`Ilillllll
`III III III
`Ilillllll
`IIlIlllIlll
`III III III
`Ilillllll
`IIlIllIlll
`Ilillllll
`IIlIlllIlll
`III III III
`IIlIlllIlll
`~~ru:t
`
`IIlIlllIlll
`III III III
`!['U1Hr:u
`~~ru:t
`Ilillllll
`III III III
`Ilillllll
`IIlIllIlll
`l:Ij~ill1:l't
`
`CO
`f'..
`0
`LL
`
`"'0
`Q)
`fj')
`:.1
`C
`:::>
`I!
`
`"'0
`Q)
`(f)
`=>
`II
`
`Ilillllll D
`
`'!""""
`Q)
`.Y
`
`(\j
`Q)
`.Y
`
`C0
`Q)
`.Y
`
`~ ~ ~ :::J «
`
`0
`""0
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 13 of 34
`
`u.s. Patent
`
`Jun. 27, 2017
`
`Sheet 10 of 17
`
`US 9,691,429 B2
`
`SELECT AN AUDIO TRACK
`
`SELECT A DURATION FOR
`THE AUDIO TRACK
`
`802
`
`804
`
`CAPTURE VIDEO TAKES WHILE
`PLAYING AUDIO TRACK
`
`FINISHED
`CAPTU RI NG
`VIDEO
`TAKES?
`
`)-N_O __ -----!
`
`YES
`
`810
`
`CREATE MUSIC VIDEO
`FEATURING AUDIO TRACKS AND
`AT LEASE A SUBSET OF
`THE VIDEO TAKES
`
`FIG. SA
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 14 of 34
`
`u.s. Patent
`
`Juo.27,2017
`
`Sheet 11 of 17
`
`US 9,691,429 B2
`
`ANALYZE AUDIO TRACK
`
`~852
`
`850
`
`/
`
`856
`;
`ANALYZE VIDEO TAKES
`
`DETERMINE
`AUDIO INTENSITY LEVELS
`OF AUDIO TRACK
`
`~854
`
`DETERM!NE
`VIDEO INTENS!TY LEVELS
`OF VIDEO TAKES
`
`860 ;
`
`PAIR SECTIONS OF AUDIO
`TRACK WTIH PORTIONS OF
`VIDEO TAKES BASED ON
`DETERMINED AUDIO
`INTENS!TY LEVELS AND
`DETERMINED
`VIDEO INTENSITY LEVELS
`
`FIG. 88
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 15 of 34
`
`u.s. Patent
`
`Jun. 27, 2017
`
`Sheet 12 of 17
`
`US 9,691,429 B2
`
`908b
`902b
`908e
`902e
`
`900
`~
`
`908a
`
`906
`
`904
`
`AKE 1
`
`FIG. 9A
`
`920
`~
`
`922a-h
`
`928b------~
`
`928a'-!'--"'8
`
`928d
`
`9289
`
`928e 928h
`FIG. 98
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 16 of 34
`
`u.s. Patent
`
`Juo.27,2017
`
`Sheet 13 of 17
`
`US 9,691,429 B2
`
`940
`/L-/
`CAPTURE VIDEO
`
`948
`
`.----
`
`942
`
`AKE3
`
`FIG.9C
`
`946
`
`944
`
`970b
`
`970e
`970a
`
`970b 970a
`
`,
`
`\ )
`
`944
`
`904
`
`I
`I
`
`,
`
`t:\~'(\~
`944
`904
`924
`\
`\
`944
`904 944
`
`FIG.9D
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 17 of 34
`
`u.s. Patent
`
`Jun. 27, 2017
`
`Sheet 14 of 17
`
`US 9,691,429 B2
`
`980
`
`/
`SELECT AN AUDIO TRACK ~ 982
`t
`CAPTURE A PLURALITY OF ~ 84
`VIDEO TAKES
`
`+
`DETERMINE A NUMBER OF FACES
`WITHIN EACH VIDEO TAKE WHILE
`CAPTURING THE VIDEO TAKES
`
`986
`
`~'
`
`+
`PAIR AT LEAST A SUBSET OF THE
`PLURALITY OF CAPTURED
`VIDEO TAKES TO THE SELECTED ~ 988
`AUDIO TRACK BASED ON THE
`DETERMINED NUMBER OF FACES
`WITHIN THE VIDEO TAKES
`
`+
`CREATE MUSIC VIDEO INCLUDING
`THE SUBSET OF VIDEO TAKES
`SYNCHRONIZED TO THE
`SELECTED AUDIO TRACK
`
`~: 990
`
`FIG.9E
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 18 of 34
`
`u.s. Patent
`
`Juo.27,2017
`
`Sheet 15 of 17
`
`US 9,691,429 B2
`
`i00\).. d1-Z d1 k:i002
`
`1 ooo~1--1 -----"'-,V~--7~"..--~r~-------l
`1002
`FIG.10A
`
`1024
`
`1024
`
`1040~
`
`t1
`
`t2
`
`iII»!.
`
`t1
`
`t2
`iII»!.
`
`-110-1 ...
`
`t1
`
`-110-1
`
`FIG. 100
`
`1062
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 19 of 34
`
`u.s. Patent
`
`Jun. 27, 2017
`
`Sheet 16 of 17
`
`US 9,691,429 B2
`
`1082 ~
`
`1084 ~
`
`~086 ~
`
`RECEIVE AUDIO INPUT VIA
`AN AUDiO INPUT iNTERFACE
`+
`RECORD THE RECEIVED AUDIO ON
`A USER DEVICE
`
`t
`
`DETERMINE THAT THE RECORDED
`AUDIO INCLUDES AT LEAST ONE OF
`A VOCAL PHRASE AND A
`MELODIC PHRASE
`t
`PAIR AT LEAST A PORTION OF AT
`LEAST ONE OF A PLURALITY OF
`CAPTURED VIDEO TAKES TO THE ~ 1088
`RECORDED AUDIO BASED ON THE
`DETERMINED VOCALAND/OR
`MELODIC PHRASE WHILE THE
`VIDEO TAKES ARE CAPTURED
`t
`GENERATE A MUSIC VIDEO INCLUDING
`THE RECORDED AUDIO TRACK AND
`AT LEAST THE PORTION OF THE
`AT LEAST ONE OF THE PLURALITY
`OF CAPTURED VIDEO TAKES
`SYNCRONIZED TO THE
`RECORDED AUDIO TRACK
`FIG.10E
`
`1090 ~
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 20 of 34
`
`u.s. Patent
`
`Jun. 27, 2017
`
`Sheet 17 of 17
`
`US 9,691,429 B2
`
`/l/1100
`
`RECORD AN
`AUDiO TRACK
`
`1102
`
`......... _------
`11 04 ""'+-__
`
`EMPO: 90Bpm
`
`ey: A minor
`
`1106
`
`1108
`
`00 :00 ..... ,"""', ........ ,"""', ......... ,"" ,00 :56
`
`FIG. 11
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 21 of 34
`
`US 9,691,429 B2
`
`1
`SYSTEMS AND METHODS FOR CREATING
`MUSIC VIDEOS SYNCHRONIZED WITH AN
`AUDIO TRACK
`
`FIELD OF THE INVENTION
`
`Various embodiments described herein generally relate to
`systems and methods for creating music videos. In particu(cid:173)
`lar, music videos may be created including portions of one
`or more video takes that are automatically synchronized to
`an audio track.
`
`BACKGROUND OF THE INVENTION
`
`Music videos, whether they involve famous musicians or
`independent artists, are fun and creative mediums for shar(cid:173)
`ing one's music and musical style with the world. While
`most music videos, at least historically, were created in
`support of established musicians for marketing purposes, the
`enhanced capabilities of mobile devices allows almost any
`individual to record and edit music, as well as capture video,
`all using one device. Individuals, whether an experienced
`musician or a novice, young or old, now have the ability to
`create their own music videos using such devices.
`Although music videos often include an individual's own
`music, it is also possible for music videos to be created based
`on an individual's favorite or a popular song. While there are
`presently some applications of this concept, most of these
`applications have several inherent drawbacks.
`In one instance, music videos have been created where an
`audio track plays in the background while a single video is
`captured or recorded. This, however, creates an extremely
`poor quality music video as there is no visual transition
`between various parts of the audio track. For example, a 35
`single video take may be used for the audio track's verse and
`chorus. This leads to extremely unprofessional looking
`music videos that, while potentially entertaining, are aes(cid:173)
`thetically inferior to professional quality music videos,
`which may use multiple video takes captured at one or more 40
`locations.
`Another instance of music videos being created focuses
`on a linear application of a video and audio track. For
`example, a single video take may be captured and multiple
`end points may be applied to that video. However, this
`application is extremely limited in that it does not allow a
`user to use multiple video, and, as such, does not allow the
`user to apply multiple end points to the multiple videos.
`Furthermore, in order to have different locations in the music
`video, a user would need to visit multiple locations in
`chronological order and capture video at each location. The
`editing of the video takes captured at each location would
`then only present the locations in the order that they were
`visited. This, as mentioned previously, creates a music video
`that is unprofessional in appearance, as professional music
`videos may have varying locations throughout the music
`video.
`In another, somewhat similar, instance, music videos have
`been created where a user is required to capture video of
`another individual within a silhouette. A generic silhouette is
`overlaid on the user's display and, while the user records
`video, the individual being recorded must stay generally
`within the bounds of the silhouette. While this may expand
`the manipulative aspects of the video, the user is unfortu(cid:173)
`nately constrained to be within one particular area, and does
`not have a free range of motion to fully capture any desired
`action. Furthermore, as the user is bound by the silhouette,
`
`2
`the ability to transition to different video takes for different
`portions of an audio track is limited, if at all possible.
`In yet another instance, music videos have been created
`that include only a small portion or snippet of the audio track
`with an individual capturing a single video for that portion
`of the audio track. For example, a user may select a song to
`form a music video for, and create a music video based on
`the song's verse or chorus. This may lead to short, dull, and
`unprofessional music videos as the music video may become
`10 nothing more than a video clip for a small tidbit of a song.
`Thus, in light of some of the aforementioned problems, it
`would be beneficial for there to be systems, methods, and
`non-transitory computer readable mediums that allow a user
`to create a professional style music video using portions of
`15 multiple videos takes taken at different times and at different
`locations that are automatically synchronized to a selected
`audio track. Furthermore, it would be beneficial for there to
`be systems, methods, and non-transitory computer readable
`medium that allow video takes to be paired with an audio
`20 track such that sections of the audio track having various
`intensity levels or dynamics are matched with suitable
`portions of the video takes.
`
`SUMMARY OF THE INVENTION
`
`25
`
`This generally relates to systems, methods, and non(cid:173)
`transitory computer readable mediums for creating music
`videos that are synchronized to an audio track.
`In one exemplary embodiment, a method for creating a
`30 music video where an audio track is synchronized with a
`plurality of video takes is described. An audio track, such as
`a song, may be selected. For example, a user may select a
`song stored on their user device, in a music library on an
`external device, or on a music server. A plurality of video
`takes may be captured also using the user device. While the
`plurality of video takes are being captured, they may also be
`synchronized with the selected audio track. The synchroni(cid:173)
`zation allows for the captured video takes to be aesthetically
`and/or musically synchronized with an appropriate section
`or sections of the audio track. A music video may then be
`created including the audio track and at least a subset of the
`plurality of video takes that are already synchronized to the
`selected audio tracks. For example, portions of one or more
`captured video takes may be matched to certain sections of
`45 the audio track based on the audio track's audio intensity
`levels and/or a particular video take's video intensity.
`In another exemplary embodiment, a user device includ(cid:173)
`ing at least one audio input component, at least image
`capturing component, memory, and at least one processor is
`50 described. The memory may store an audio track recorded
`using the at least one audio input component and a plurality
`of video takes captured by the at least one image capturing
`component. For example, a user may record an audio track
`using their user device's microphone as well as record video
`55 takes using one or more cameras resident on the user device.
`The at least one processor of the user device may then
`determine a vocal and/or melodic phrase within the recorded
`audio track, and synchronize at least a portion of one or more
`captured video takes to the recorded audio track based on the
`60 determined vocal and/or melodic phrase while the at least
`one of the plurality of video takes is captured.
`In yet another exemplary embodiment, another method
`for creating a music video is described. An audio track may
`be selected and a plurality of video takes may be captured.
`65 A number of faces within each video take of the captured
`video takes may be determined while the plurality of video
`takes are being captured. Also while the plurality of video
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 22 of 34
`
`US 9,691,429 B2
`
`3
`takes are being captured, at least a subset of the plurality of
`captured video takes may be synchronized to the selected
`audio track based on the number of faces determined to be
`within each video take. A music video may then be created
`including the selected audio track and at least the subset of
`the plurality of captured video takes synchronized to the
`selected audio track.
`In still yet another exemplary embodiment, a user device
`including memory for storing a plurality of video takes, at
`least one image capturing component, and at least one 10
`processor is described. The at least one processor is operable
`to receive a selection of an audio track from the plurality of
`audio tracks stored in memory. The audio track may then
`play and, while playing, at least one video take may be 15
`captured using the at least one image capturing component.
`The at least one captured video take may be synchronized to
`the selected audio track while the selected audio track plays.
`A music video may then be generated that includes the
`selected audio track and at least a subset of the at least one 20
`video take that is already synchronized to the selected audio
`track.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The above and other features of the present invention, its
`nature and various advantages will be more apparent upon
`consideration of the following detailed description, taken in
`conjunction with the accompanying drawings in which:
`FIG. 1 is an illustrative block diagram of a user device in
`accordance with various embodiments;
`FIG. 2 is an illustrative diagram of a user interface
`displayed on a user device in accordance with various
`embodiments;
`FIGS. 3A and 3B are illustrative diagrams of user inter(cid:173)
`faces displayed on a user device for selecting an audio track
`in accordance with various embodiments;
`FIG. 4 is an illustrative diagram of a user interface
`displayed on a user device presenting a selected audio track
`for a music video to be created in accordance with various
`embodiments;
`FIGS. 5A and 5B are illustrative diagrams of various user
`interfaces displayed on a user device for capturing video
`takes for a music video to be created in accordance with
`various embodiments;
`FIGS. 6A-C are illustrative diagrams of various user
`interfaces displaying video takes being captured by a user
`device for a music video in accordance with various embodi(cid:173)
`ments;
`FIGS. 7A and 7B are illustrative diagrams of a created
`music video including a plurality of captured video takes
`synchronized to selected audio track in accordance with
`various embodiments;
`FIGS. 8A and 8B is an illustrative flowchart of a process
`for creating music videos in accordance with various
`embodiments;
`FIGS. 9A-C are illustrative diagrams of user interfaces
`including various video takes having a number of faces or
`images determined to be within the video takes in accor(cid:173)
`dance with various embodiments;
`FIG. 9D is an illustrative diagram of various sections of
`a selected audio track's waveform synchronized with por(cid:173)
`tions of video takes based on the number of faces determined
`to be within each video take in accordance with various
`embodiments;
`FIG. 9E is an illustrative flowchart of a process for
`synchronizing video takes to an audio track based on a
`
`4
`number of faces determined to be within the video takes in
`accordance with various embodiments;
`FIGS. 10A-D are illustrative diagrams of various vocal
`and melodic indicators and phrases for use as transition
`points between video takes for a music video synchronized
`to an audio track in accordance with various embodiments;
`FIG. 10E is an illustrative flowchart of a process for
`creating a music video including various video takes syn(cid:173)
`chronized to a recorded audio track based on a determined
`vocal and/or melodic phrase within the audio track in
`accordance with various embodiments; and
`FIG. 11 is an illustrative diagram of a user interface for
`recording an audio track to be used for creating a music
`video in accordance with various embodiments.
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`
`The present invention may take form in various compo(cid:173)
`nents and arrangements of components, and in various
`techniques, methods, or procedures and arrangements of
`steps. The referenced drawings are only for the purpose of
`illustrated embodiments, and are not to be construed as
`limiting the present invention. Various inventive features are
`25 described below that can each be used independently of one
`another or in combination with other features. Furthermore,
`in at least some embodiments, like referenced numerals refer
`to like parts throughout.
`FIG. 1 is an illustrative block diagram of a user device in
`30 accordance with various embodiments. User device 100, in
`some embodiments, may correspond to any electronic
`device or system. Various types of user devices include, but
`are not limited to, portable media players, cellular tele(cid:173)
`phones or smart phones, pocket-sized personal computers,
`35 personal digital assistants ("PDAs"), desktop computers,
`laptop computers, tablet computers, and/or electronic acces(cid:173)
`sory devices such as smart watches and bracelets. User
`device 100 may communicate with one or more additional
`user devices, networks, and/or servers. For example, user
`40 device 100 may send text messages to other user devices
`across a network, or user device 1 00 may access one or more
`websites located on a server.
`User device 100, in some embodiments, may include one
`or more processors 102, memory 104, storage 106, commu-
`45 nications circuitry 108, an input interface 110, and an output
`interface 118. In some embodiments, input interface 110
`may include one or more cameras 110 or other image
`capturing components, one or more microphones 112 or
`other audio capturing components, and one or more external
`50 device inputs 116. Further, in some embodiments, output
`interface 118 may include display 120 and one or more
`speakers 122 or other audio output components. Persons of
`ordinary skill in the art will recognize that user device 100
`may include any number of components, and one or more
`55 additional components or modules may be added or omitted
`without deviating from the scope of the present disclosure.
`Additionally, one or more components may be combined or
`separated, and multiple instances of various components are
`also possible, however only one of each component is shown
`60 within user device 100 for simplicity.
`Processor(s) 102 may include any suitable processing
`circuitry, such as one or more processors, capable of con(cid:173)
`trolling the operations and functionality of user device 100.
`In some embodiments, processor(s) 102 may facilitate com-
`65 munications between various components within user
`device 100. For example, processor(s) 102 may cause output
`interface 118 to perform an associated output in response to
`
`

`

`Case 6:20-cv-00693-ADA Document 1-2 Filed 07/29/20 Page 23 of 34
`
`US 9,691,429 B2
`
`5
`one or more inputs being detected by input interface 110.
`Processor(s) 102 may run an operating system for user
`device 100, applications resident on user device 100, firm(cid:173)
`ware applications, media application, and/or any other type
`of application, or any combination thereof function on, or in
`conjunction with, user device 100.
`Memory 104 may include any suitable form of memory,
`such as cache memory, semi-permanent memory (e.g.,
`RAM), or any other memory type, or any combination of. In
`some embodiments, memory 104 may be used in place of
`and/or in addition to an external memory or storage unit or
`device for storing data on user device 100.
`Storage 106 may include one or more storage mediums.
`Various types of storage mediums include, but are not
`limited to, hard drives, solid state drives, flash memory, 15
`permanent memory (e.g., ROM), or any other storage type,
`or any combination thereof. Any form of data or content may
`be stored within storage 106, such as photographs, music
`files, videos, contact information, applications, documents,
`or any other file type, or any combination thereof.
`In some embodiments, memory 104 and storage 106 may
`be combined into a single component. For example, a single
`memory component may include memory and storage func(cid:173)
`tions. In other embodiments, multiple instances of memory
`104 and/or storage 106 may be present, however it is also 25
`possible for memory 104 and/or storage 106 to be external
`to user device 100. For example, one or more files may be
`stored remotely on an external hard drive or on a cloud
`storage provider. However, persons of ordinary skill in the
`art will recognize that the aforementioned scenarios are 30
`merely examples.
`Communications circuitry 108 may include any circuitry
`capable of connecting user device 100 to one or more
`additional devices (e.g., laptop computers, smartphones,
`etc.), one or more networks (e.g., local area networks 35
`("LAN"), wide area networks ("WAN"), point-to-point net(cid:173)
`works, etc.), and/or one or more servers (e.g., file manage(cid:173)
`ment systems, music directories, etc.). Communications
`circuitry may support any suitable communications protocol
`including, but not limited to, Wi-Fi (e.g., 802.11 protocol), 40
`Bluetooth®, radio frequency systems (e.g., 900 MHz, 1.4
`GHz, and 5.6 GHz communications systems), infrared,
`GSM, GSM plus EDGE, CDMA, quadband, LTE, VOIP, or
`any other communications protocol, or any combination
`thereof.
`Input interface 110 may include any suitable mechanism
`and/or component for receiving inputs from a user operating
`user device 100. For example, input interface 110, in one
`embodiment, includes one or more cameras 112. Cameras
`112 may correspond to any suitable image capturing com(cid:173)
`ponent capable of capturing images and/or video. For
`example, camera 112 may capture photographs, sequences
`of photographs, rapid shots, videos, or any other type of
`image, or any combination thereof. In some embodiments,
`cameras 112 may be capable of capturing high-definition
`("HD"), 3-D, and/or panoramic images and/or videos. In
`some embodiments, cameras 112 may include one or more
`filters or settings for images and/or video that may be
`captured by cameras 112 (e.g., black and white, monochro(cid:173)
`matic, fades, slow-motion, etc.). In some embodiments, user
`device 100 may include multiple instances of camera 112.
`For example, user device 100 may include a front-facing
`camera and a rear-facing camera. In some embodiments, one
`or more additional image capturing components, such as a
`zoom or add on filter, may be used in connection with, or 65
`instead of, camera 112 to aid in capturing images and/or
`videos.
`
`6
`Microphone(s) 114 may be any component capable of
`detecting and/or receiving audio signals. For example,
`microphone(s) 114 may include one or more sensors for
`generating electrical signals and circuitry capable of pro(cid:173)
`cessing the generated electrical signals. In some embodi(cid:173)
`ments, user device 100 may include multiple instances of
`microphone 114, such as a first microphone and a second
`microphone. In some embodiments, user device 100 may
`include multiple microphones capable of detecting various
`10 frequency levels (e.g., high/low-frequency microphones).
`Furthermore, in some embodiments, one or more external
`microphones may be connected to user device 100 and may
`be used in conjunction with, or instead of, microphone(s)
`114.
`External device input 116 may correspond to any input
`interface or set of input interfaces capable of receiving
`inputs from an external device. For example, one or more
`external microphones, as described above, may be coupled
`to user device 100 through external device input 116. As
`20 another example, a user may couple an electric guitar,
`drums, and/or keyboard to user device 100 via external
`device input 116. However, it is also possible for a user to
`couple one or more external devices, suc