`
`
`
`
`
`
`
`
`
`
`
`
`Exhibit 10
`to the Declaration of Catherine Hartman
`Public Redacted Version
`
`
`
`
`Exhibit 10
`
`to the Declaration of Catherine Hartman
`
`Public Redacted Version
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 2 of 91
`
`Watch Next Eng Review
`
`Brad Froehle, Lukasz Heldt, Su-Lin Wu
`(representing the work of many others)
`5/8/2017
`
`Confidential & Proprietary
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052174
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 3 of 91
`
`Agenda
`
`Goals:
`
`• Overview of current state of Watch Next
`• Details for selected areas in
`o Systems and Infrastructure
`0 Candidate nomination and ranking
`• Future directions
`
`Informative review: no decisions needed at this time
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052175
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 4 of 91
`
`Introduction
`
`Confidential & Proprietary
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052176
`
`Intrfroduction
`
`0 Tube
`
`Confidential & Proprietary
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052176
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 5 of 91
`
`Mission
`
`Help every user find another compelling video to watch next, at scale.
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052177
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 6 of 91
`
`Case Management Systems
`
`Compare Cross Jurisdictional
`Data &Access Suspect Profiles.
`See More
`
`Cart
`
`Watch Next: Desktop
`
`•
`
`re
`
`•
`
`•111
`
`4
`
`0
`
`a
`
`•
`
`•
`
`(7,
`,,
`
`[.]
`
`n
`o
`
`Nexus 5X vs Moto X Pure Edition
`Android Authority
`
`CI &rime. 1 el 1 • MI
`
`107,50
`
`tip next
`
`T
`
`jpN
`L
`IFIL'NI
`
`1211
`
`1313
`
`Autepley o eriN
`
`Neus, 5
`
`slt Worth It?
`
`9.5 v.e.s
`
`exus SP vs Samsung Galaxy Node 5
`
`X Pure Edition (Style) VS iPhone OS
`
`Plus
`Andro, Au,hordy
`154,3 views
`
`Nestle 6P Review!
`Andro,d Au,finr.ty
`41743.66
`
`Nexus 5X vs Moto X Pure Edition
`Pacl.now
`195,92,iews
`
`You Tube
`
`Published on Oct 25, 2015
`Buy Nexus 55, Mtp://genims/Nex EX I Buy Moto %Pure: htiongeni.us/rnotexpu ,II comparison: http://go,VDGgdF9
`
`Today we take a look at the Nexus 5% and Moto X Pure Edition, how. these two highly affordable devices compare?
`
`* ft
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052178
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 7 of 91
`
`Watch Next: Mobile/Tablet/TV
`
`re
`
`11,08
`
`Nexus 5X Review!
`
`212.060 views
`
`■Subeixibx
`
`Nexus 6P vs Nexus 5%
`Argrold Authority
`views
`
`Illii Atus,0.5XzeZ,,,to KN. Edoun
`
`'ION views
`
`Nexue6P Revlawl
`
`ne .Pl
`
`Android Authority
`1,77,00 subscribers
`
`Nexus OP vs Samsung Galaxy Nola
`
`I
`
`Nexus 5X Second Opinion
`
`dr
`
`Vs,
`
`p.20,5 Buy mqr,..on nup
`
`ustNe.51, I Reed mare 1111.9 //good/
`
`Nexus 5X vs Moto X Pure Eallon
`And.o.ak.thar,
`lencnewa
`
`- ngs you his thoughts,. News 5X
`
`forumS http.e/wWW arldraidauthOnlytOrn/Cbron.
`
`Suggestions
`Nexus 5X vs Moto X Pure Edition by Android Authority 107,522 views
`
`•
`
`10:36
`
`A psi*.
`
`11:04
`
`12:46
`
`418
`
`5X
`•
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052179
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 8 of 91
`
`Watch Next Beyond Main App
`
`• Music
`• Kids
`• Gaming
`• Externally available API
`• Embedded video endscreen (offsite suggestions)
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052180
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 9 of 91
`
`Slide 7
`
`1
`
`is it available for YT Go?
`Rongmei Zhang, 5/9/2017
`
`My understanding is that YT Go would need related items restricted
`1
`to the YT Go corpus, is that correct? It might be possible to filter
`suggestions, but some analysis might be needed to see if there is enough
`coverage to provide a reasonable user experience.
`+bfroehle@google.com
`Su-Lin Wu, 5/9/2017
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052181
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 10 of 91
`
`Challenges
`
`User Intents
`
`• Finding -- continuation of search
`• Browsing -- continuation of homepage, direct watch
`• Navigation -- lean back autoplay, episodic series, more from uploader
`
`Creator/Advertiser Expectations
`
`• Large driver of traffic
`• Part of their watch page experience
`• Brand perception for Creators and YouTube
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052182
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 11 of 91
`
`Goals
`uccess metrics
`
`• Watchtime
`• Freshness
`• DAVs
`• Satisfaction (panel bad rate)
`
`Secondary considerations
`
`• Effect on partner traffic
`• Exposure for paid movies, live events, Red originals, playlists etc
`• Perception of YouTube and creators
`• Embed strategy
`• Related graph data
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052183
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 12 of 91
`
`Slide 9
`
`1
`
`And also perception of regulators and governments.
`Cristos Goodrow, 5/8/2017
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052184
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 13 of 91
`
`YouTube and Watch Next are Growing
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052185
`
`YouTube and Watch Next are Growin
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052185
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 14 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052186
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052186
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 15 of 91
`
`go/sequences
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052187
`
`
`
`
`
`go/sequences
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052187
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 16 of 91
`
`Recent Accomplishments (Q4 through today)
`
`• Systems and infrastructure simplifications
`o Flattened nominator graph
`o Experiments auto-pushed
`
`(in collaboration with many, including
`Home/Recs, Research, Content Id, Abuse)
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052188
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 17 of 91
`
`Slide 13
`
`Please spell-out abbreviations like WTR (which I think it Watch
`2
`Through Rate) and MFU (More From Uploader)
`Cristos Goodrow, 5/9/2017
`
`2
`
`Done, thanks for pointing those out.
`Su-Lin Wu, 5/9/2017
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052189
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 18 of 91
`
`Infrastructure
`
`Confidential & Proprietary
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052190
`
`Infrastructure
`
`0 Tube
`
`Confidential & Proprietary
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052190
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 19 of 91
`
`High Level View
`
`4.2)44
`
`Billion Videos 11,14',
`
`Wit,
`
`1. Candidate
`Nomination
`
`2. Scoring
`
`-100 Candidates
`
`A Few Winners
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052191
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 20 of 91
`
`Request Flow
`
`Generate candidates.
`
`Nomination
`
`Attach video/playlist metadata, and
`other annotations.
`
`Annotation
`
`Remove candidates.
`
`Score candidates with Brain.
`
`Make the final output decisions:
`autoplay candidate, shelves
`
`Filtering
`
`Scoring
`
`Packing
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052192
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 21 of 91
`
`System Overview
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052193
`
`System Overview
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052193
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 22 of 91
`
`Scale & Resource Costs
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052194
`
`Scale & Resource Costs
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052194
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 23 of 91
`
`Slide 18
`
`+heldt@google.com
`1
`+liwei@google.com
`
`Approximate current plans:
`Brain => Still on Axon (not Servo). Working on Servo migration first (Q3)
`and then plan to start evaluating SeaStar. Note SeaStar will require some
`model changes (discretization) with unknown, potentially negative, quality
`implications. Will require experimentation.
`Jellyfish => No plans yet.
`Brad Froehle, 5/6/2017
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052195
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 24 of 91
`
`Slide 18
`
`Lastly, one issue w/ SeaStar is that a naive implementation may
`2
`strand a large amount of CPU resources on the dedicated machines. It's my
`understanding that we'll need to pack other jobs on those machines for them
`to be cost effective.
`
`These solutions are being explored jointly w/ p13n, since they have similar
`use case. Planning a shared YT Discovery pool of brain servers (or possibly
`Servomatic, run by ML SRE).
`Brad Froehle, 5/6/2017
`
`Brain = plans to move to SeaStar?
`1
`Training = plans to move to gpu or Jellyfish?
`Danner Stodolsky, 5/8/2017
`
`1
`
`We are planning on switching to Servo in Q2.
`Lukasz Heldt, 5/8/2017
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052196
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 25 of 91
`
`Infrastructure Trends
`
`• Batch —> Streaming pipelines
`o Reduce latency of user action to incorporation in our systems. Freshness.
`• Pay down technical debt / improve code health / increase team velocity
`o Rewrite async producer or callback code with Fibers (Superroot Flows).
`o Release & push automation.
`• Simpler shared components (more "microservices")
`o Discovery Engine:
`■ Video Metadata Service (to replace Muppet).
`■
`("Co-watch") Candidate Service.
`■ Eventual migration from Superroot to Browse/Recs Platform?
`o Build on standard services like Bigtable, Laelaps, & Mapess.
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052197
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 26 of 91
`
`Freshness via infrastructure
`
`a
`
`• 2016Q1 Instant Muppet
`o Ensure fresh videos are in corpus and can be suggested
`o Fresher ranking & abuse signals (-12h —> 1 Os for important updates)
`• 2016Q3 Eggbeater + Candidate Service
`o Streaming co-watch related graph computations + new serving infrastructure
`o Co-watch latency: 22h—>E3h
`• 2017Q1 Intraday v3
`o Co-watch latency: 8h->3h
`• 2017Q1 Docfilter
`o React faster to private/unlisted—*public transitions
`o
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052198
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 27 of 91
`
`Slide 20
`
`3
`
`+bfroehle@google.com
`Su-Lin Wu, 5/9/2017
`
`The previous system only recorded public->private transitions;
`3
`there was no way to "refresh" a video, except waiting 12h for the deletion
`marker ttl to expire.
`Brad Froehle, 5/9/2017
`
`Please remind me how docfilter helps with unlisted->public (I see
`3
`that it would help with public->unlisted/delete).
`Cristos Goodrow, 5/9/2017
`
`4
`
`https://goto.google.com/wn-docfilter
`Brad Froehle, 5/9/2017
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052199
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 28 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052200
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052200
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 29 of 91
`
`Team Velocity
`
`Candidate Service
`
`• Make it easier to experiment with new, streaming, candidate sources.
`• Simple API; efficient at serving time.
`
`Experiments
`
`• Cookie-day experiments always running (no wait for preperiod); daily shuffled
`cookies reduce bias.
`• New experiment configurations pushed every 30 minutes automatically.
`• Easy to train & serve experimental Brain models.
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052201
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 30 of 91
`
`Challenges
`
`•
`
`"Full stack" development (backend + InnerTube / UI changes)
`•
`• Non-uniformity between Home & Watch Next (& Search)
`o Propagating scoring signals is unique on each surface; different proto representations of a
`video complicate code reuse.
`o Different approaches towards decoration.
`• Resource availability & management
`• Legacy code
`o Legacy RPC interface; ultra-legacy clients (GData)
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052202
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 31 of 91
`
`Nomination
`
`Confidential & Proprietary
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052203
`
`INOmlnation
`
`0 Tube
`
`Confidential & Proprietary
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052203
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 32 of 91
`
`Confidential & Proprietary
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052204
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052204
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 33 of 91
`
`Deep Dive:
`Related Graph ("Co-watch")
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052205
`
`Deep Dive:
`Related Graph ("Co-watch")
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052205
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 34 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052206
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052206
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 35 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052207
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052207
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 36 of 91
`
`Raw Stats (as of 2017-05)
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052208
`
`Raw Stats (as of 2017-05)
`
` HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052208
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 37 of 91
`
`(External) Related Graph Customers
`
`• Personalization
`• Video Inventory Analytics
`• Legos
`• Abuse
`• Unplugged
`• Ads
`• Search
`• Viewer Model
`• Research
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052209
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 38 of 91
`
`Future Directions
`
`TicTac
`
`• Streaming visits, targeting co-watch latency <1 h
`
`Discovery Engine
`
`• Power P13N "related-on-the-fly" nominator with Candidate Service infra.
`
`Matchmaker
`
`• Predictive co-watch at publish time.
`• Find ideal places to recommend newly published videos.
`•
`Improve recommendations after watching fresh videos.
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052210
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 39 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052211
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-0005221 1
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 40 of 91
`
`Online Ranking
`
`Confidential & Proprietary
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052212
`
`Online Ranking
`
`0 Tube
`
`Confidential & Proprietary
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052212
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 41 of 91
`
`Objective
`
`Lifetime watch time per user
`
`Watch time per session
`
`Watch time per panel
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052213
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 42 of 91
`
`Objective: Watch Time Per Panel
`
`• Given the watch video and user
`... how much watch time will candidate generate
`
`• Rank candidates by expected watch time
`
`P(ctr) * E(watch ratio I click) * video length
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052214
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 43 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052215
`
`Ranking Model
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052215
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 44 of 91
`
`Training
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052216
`
`Trainin
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052216
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 45 of 91
`
`Slide 37
`
`How exactly is a +ve / -ve example defined in this context? Is
`1
`there a doc describing this? +liwei@google.com +jhe@google.com
`Hariharan Chandrasekaran, 7/17/2018
`
`1
`Positive examples are the impressions that are clicked by the
`users. See this doc
`(https://docs.google.com/document/d/lmUFcByRMVfX3olGtEq_2yywqeQ4qSR(40VFvAMq
`MCNvA/edit#heading=h.pnz9paci6xOb) for details.
`Li Wei, 7/17/2018
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052217
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 46 of 91
`
`Training
`
`Brain Trainer
`(continuous)
`
`validate and
`push periodically
`
`1
`
`Brain
`Serving
`
`c
`
`......
`
`......,
`
`•
`
`• E
`
`-40 models (prod + exps)
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052218
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 47 of 91
`
`Features
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052219
`
`Features
`
` HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052219
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 48 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052220
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052220
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 49 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052221
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052221
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 50 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052222
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052222
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 51 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052223
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052223
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 52 of 91
`
`Sitewide Watch Time
`
`Launches since 5/2016:
`
`+3.14%
`
`Including autonav on mobile:
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052224
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 53 of 91
`
`Tuning Objective
`
`expected watch time per impression
`
`video length
`
`CTR
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052225
`
`Tuning Objective
`
`expected watch time per impression
`
`SS ~
`
`一
`
`es
`
`
`Video length
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052225
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 54 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052226
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052226
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 55 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052227
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052227
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 56 of 91
`
`Tuning Objective
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052228
`
`Tuning Objective
`
` HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052228
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 57 of 91
`
`Tuning Objective - future direction
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052229
`
`Tuning Objective - future direction
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052229
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 58 of 91
`
`source
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052230
`
`
`
`
`
`source
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052230
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 59 of 91
`
`Filtering & Safety
`
`Confidential & Proprietary
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCH N DR-00052231
`
`Filtering & Safety
`
`0 Tube
`
`Confidential & Proprietary
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052231
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 60 of 91
`
`Trashy Demotion
`
`• For trashy candidates:
`
`• Trashiness today:
`
`10 of the Worst Parents in the World
`leRichest 131
`ago • 8.439227 views
`narenting is an understatement. These are the top
`orrible mothers and fathers ever to have children
`
`5'27
`
`GIANT ANACONDA ATTACKS COW I
`ANACONDA ATTACK ANIMAL I ANACONDA...
`by Jala Mose
`1 month ago • 59,564 views
`GIANT ANACONDA ATTACKS COW I ANACONDA
`ATTACK ANIMAL I ANACONDA ATTACK AMAZING...
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052232
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 61 of 91
`
`MFA Demotion
`
`SPAM
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052233
`
`MFA Demotion
`
` HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052233
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 62 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052234
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052234
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 63 of 91
`
`Content Rating Filtering
`
`Exception made for series playlists (though safety mode continues to be respected).
`
`Documentation
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052235
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 64 of 91
`
`Content Rating Filtering
`
`Exception made for series playlists (though safety mode continues to be respected).
`
`Documentation
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052236
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 65 of 91
`
`Additional Filtering
`
`• Dishonor
`• Playability (rights management)
`• Rating (thumbs up/down ratio)
`• Previously dismissed
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052237
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 66 of 91
`
`Themes for 2017
`
`Serving
`
`Automation.
`
`Ranking
`
`TFX/Servo/Seastar
`
`Nomination
`
`model
`
`Quality
`
`Utilize & contribute Discovery Engine pieces.
`
`Move to shared scoring service; adopt
`
`Simplify architecture; move to fully streaming
`
`Move beyond immediate Watch Time (long ter
`.ull Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052238
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 67 of 91
`
`Appendix
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052239
`
`Appendix
`
`Tube
`
`Confidential & Proprietary
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052239
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 68 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052240
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052240
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 69 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052241
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052241
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 70 of 91
`
`Autonav Quality
`
`0 • A • 9,24
`
`Why don't we just take the first
`watch next result?
`
`Jimmy Interviews Peyton Manning
`(Kristen Wiig)
`2,514 285 views
`
`1611K 911K
`
`The Tonight Show Starring Ji..
`9,519,215 sLb.:rlber,
`
`IN Subscribe
`
`Jimmy Interviews Khaleesi from
`Game of Thrones (Kristen Wiig)
`The Tonight Show Starring
`7NA views
`
`Freestylin' with The Roots:
`Valentine's Day Hearts and Merd...
`The Tonight Show Starring Jimm...
`375K views
`
`Egg Russian Roulette with Peyton
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052242
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 71 of 91
`
`OK to suggest,
`not to play
`
`Paid videos, sensitive topics,
`live events, etc
`
`Higher bar
`for relevance
`
`Rank "next in series" or similar topics
`ahead of personal recs, more from
`creator, etc
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052243
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 72 of 91
`
`Google I You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052244
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052244
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 73 of 91
`
`A sequence, not just a video
`
`TODAY: Topic Drift
`
`Each pair makes sense, but the sequence
`does not.
`
`SOLUTION: Utilize watch history more
`effectively
`
`Google Chrome: Dear Sophie (1:32)
`https://www.youtube.corn/watch?y=R4vkVHijdOk
`by Google Chrome
`10,504,212 views
`wntube l doctube
`
`Related Videos
`
`Show results in vmi (videos only).
`
`0
`
`1
`
`2
`
`3
`
`-, 6iiiii 's there a dead
`pakistani on my couch
`ir.,, ,d,
`
`I ,
`
`30 Worst Google Auto-Completes Ever (10:41)
`https://www.youtube.com/watch?v=EKQEIImY_iU
`by Danger Dolan
`2,600,787 views
`INLINKS + TOPIC_SEARCH
`wntube I doctube
`
`----
`
`A VOLCANO ,
`WHERE?
`
`40 Most Amazing Last Words Ever Spoken (13:40)
`https://www.youtube.com/watch?v=t4tMtel87Vw
`by Danger Dolan
`2,297,352 views
`INLINKS
`wntube i doctube
`
`15 Weirdest Beaches on Earth (11:08)
`https://www.youtube.com/watch2v=LUgmKOLJUSs
`by Danger Dolan
`1,475,110 views
`INLINKS
`wntube I dOctube
`
`15 Most Dangerous Places on Earth (7:48)
`https://www.youtube.com/watch?y=sty9dGG8JY4
`by Danger Dolan
`6,198,744 views
`INLINKS
`vvntube i doctube
`
`.....
`
`"
`
`15 Magical Places You're Not Allowed To Visit (9:04)
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCH N DR-00052245
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 74 of 91
`
`Why don't we just autoplay the first result?
`
`J R
`
`III
`
`11, Follow
`
`If youtube Autoplay had a crossfader it'd actually
`be better than alot of djs
`
`Filter out
`• paid content, age-gated
`• playlists
`•
`recently watched
`
`Leanback considerations
`•
`less tolerance for WTF's
`• control for drifting off-topic
`• use Radio for music autoplay
`
`Different model for autoplay?
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052246
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 75 of 91
`
`Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052247
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052247
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 76 of 91
`
`Future Directions
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052248
`
`Future Directions
`
` HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052248
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 77 of 91
`
`Ranking Model
`
`• One of the biggest models at Google (-210 billion training examples)
`• Using latest technology
`
`08/2016
`
`01/2017
`
`Sibyl —> Distbelief
`
`Distbelief —> Tensorflow
`
`• Close collaboration with the research
`o DB -> TF transition findings shared with many other teams at Google
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052249
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 78 of 91
`
`Features
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052250
`
`Features
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052250
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 79 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052251
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052251
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 80 of 91
`
`Q4-Today Accomplishments
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052252
`
`Q4-Today Accomplishments
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052252
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 81 of 91
`
`Slide 72
`
`4
`+bfroehle@google.com
`What else should we mention here?
`Su-Lin Wu, 5/2/2017
`
`+heldt@google.com
`5
`Yi's user state launch might be worth talking about in detail, because
`there is some evidence that the watch page can move dav's.
`Su-Lin Wu, 5/2/2017
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052253
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 82 of 91
`
`Notes Summary:
`
`Slide 4: 'Maybe remove the second part'
`Slide 5: 'Note -- clean up screenshots in next iteration'
`Slide 6: 'Also kids, music, gaming.'
`Slide 8: 'The watch page is not a destination that people revisit (often);
`WN is a continuation of the user experience, making it easier for the user
`to find another video to watch.
`The suggestion panel attempts to satisfy multiple user intents and balance
`with creator/advertiser interests such as brand safety.
`Tail content is important, poor experiences that end a user's session are
`more memorable than average ones.'
`Slide 10: 'https://video-analytics-
`demo.corp.google.com/#dt=c,fe=17285,fr=lw-
`001,fs=15584;fc=0,fcr=0,r=trafficsources,rpa=a,rpbm=7-93-
`110,rpd=112,rpg=93,rpm=a,rpp=0,rppc=O,rpr=w,rps=93,rpsd=1,rpt=0'
`Slide 13: '* Trashy impact updated to reflect human eval scoring.'
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052254
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 83 of 91
`
`Notes Summary:
`
`Slide 15: 'Candidate generation is a spectrum: related to video watched <-
`
`> related to user
`
`Some nominators target video, some target user. Over time, working more
`towards a blend.'
`Slide 17: 'Instant doc updates: http://vi/throttler/domain?domain=youtube-
`related-pipeline.corpus-updater-instant-dirty
`mentos, content labels, new, first legos, delete, title/description change,
`live event, news cluster, dishonor, series'
`Slide 18:
`'https://resourceweather.corp.google.com/#costs?cells=&charge=dollar&cluste
`rs=&dateFrom=2017-05-02&dateTo=2017-05-
`02&excludeManualCharges=true&excludeTiSupplier=false&groupBy=principal.mdb&
`grouping=true&mdbGroups=&measures=net,costutil,allocations,utilization,usag
`e,asstock&mode=table&predefinedDateRangeWithViewMode=userDefinedlday&princi
`palId=50040647&services=summary&sortBy=totalComputeCostSweHpq&sortOrder=des
`cending&utilizationType=service'
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052255
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 84 of 91
`
`Notes Summary:
`
`Slide 20: 'Lots of ways to accidentally lose freshness:
`
`Docfilter: https://ariane.googleplex.com/launch/181356'
`Slide 25:
`'https://dasnay.corp.google.com/dnstai85/#1abel id=1r79b&view=default
`
`Playlists?
`
`Some non-co-watch important for bootstrapping the co-watch graph.'
`Slide 27: 'Search strong signal because it reflects user intent. Others
`provide coverage for things people aren't searching for. Others act to
`reinforce.'
`Slide 28: 'Annotate w/ end-to-end chart. Describe the evolution of a panel
`over a week.'
`Slide 29: 'Add slide on glassbox model? (only if time allows).
`Call out update rate.'
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052256
`
`
`
`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 85 of 91
`
`Notes Summary:
`
`Slide 30: 'http://go/wn-related-graph-table.
`
`Personalization: "Related-on-the-Fly" nominator
`VIA: Unfiltered graph for content labels
`Legos: Smear entities across related graph
`Abuse:
`Un