throbber
Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 1 of 91
`
`
`
`
`
`
`
`
`
`
`
`
`Exhibit 10
`to the Declaration of Catherine Hartman
`Public Redacted Version
`
`
`
`
`Exhibit 10
`
`to the Declaration of Catherine Hartman
`
`Public Redacted Version
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 2 of 91
`
`Watch Next Eng Review
`
`Brad Froehle, Lukasz Heldt, Su-Lin Wu
`(representing the work of many others)
`5/8/2017
`
`Confidential & Proprietary
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052174
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 3 of 91
`
`Agenda
`
`Goals:
`
`• Overview of current state of Watch Next
`• Details for selected areas in
`o Systems and Infrastructure
`0 Candidate nomination and ranking
`• Future directions
`
`Informative review: no decisions needed at this time
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052175
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 4 of 91
`
`Introduction
`
`Confidential & Proprietary
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052176
`
`Intrfroduction
`
`0 Tube
`
`Confidential & Proprietary
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052176
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 5 of 91
`
`Mission
`
`Help every user find another compelling video to watch next, at scale.
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052177
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 6 of 91
`
`Case Management Systems
`
`Compare Cross Jurisdictional
`Data &Access Suspect Profiles.
`See More
`
`Cart
`
`Watch Next: Desktop
`
`•
`
`re
`
`•
`
`•111
`
`4
`
`0
`
`a
`
`•
`
`•
`
`(7,
`,,
`
`[.]
`
`n
`o
`
`Nexus 5X vs Moto X Pure Edition
`Android Authority
`
`CI &rime. 1 el 1 • MI
`
`107,50
`
`tip next
`
`T
`
`jpN
`L
`IFIL'NI
`
`1211
`
`1313
`
`Autepley o eriN
`
`Neus, 5
`
`slt Worth It?
`
`9.5 v.e.s
`
`exus SP vs Samsung Galaxy Node 5
`
`X Pure Edition (Style) VS iPhone OS
`
`Plus
`Andro, Au,hordy
`154,3 views
`
`Nestle 6P Review!
`Andro,d Au,finr.ty
`41743.66
`
`Nexus 5X vs Moto X Pure Edition
`Pacl.now
`195,92,iews
`
`You Tube
`
`Published on Oct 25, 2015
`Buy Nexus 55, Mtp://genims/Nex EX I Buy Moto %Pure: htiongeni.us/rnotexpu ,II comparison: http://go,VDGgdF9
`
`Today we take a look at the Nexus 5% and Moto X Pure Edition, how. these two highly affordable devices compare?
`
`* ft
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052178
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 7 of 91
`
`Watch Next: Mobile/Tablet/TV
`
`re
`
`11,08
`
`Nexus 5X Review!
`
`212.060 views
`
`■Subeixibx
`
`Nexus 6P vs Nexus 5%
`Argrold Authority
`views
`
`Illii Atus,0.5XzeZ,,,to KN. Edoun
`
`'ION views
`
`Nexue6P Revlawl
`
`ne .Pl
`
`Android Authority
`1,77,00 subscribers
`
`Nexus OP vs Samsung Galaxy Nola
`
`I
`
`Nexus 5X Second Opinion
`
`dr
`
`Vs,
`
`p.20,5 Buy mqr,..on nup
`
`ustNe.51, I Reed mare 1111.9 //good/
`
`Nexus 5X vs Moto X Pure Eallon
`And.o.ak.thar,
`lencnewa
`
`- ngs you his thoughts,. News 5X
`
`forumS http.e/wWW arldraidauthOnlytOrn/Cbron.
`
`Suggestions
`Nexus 5X vs Moto X Pure Edition by Android Authority 107,522 views
`
`•
`
`10:36
`
`A psi*.
`
`11:04
`
`12:46
`
`418
`
`5X
`•
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052179
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 8 of 91
`
`Watch Next Beyond Main App
`
`• Music
`• Kids
`• Gaming
`• Externally available API
`• Embedded video endscreen (offsite suggestions)
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052180
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 9 of 91
`
`Slide 7
`
`1
`
`is it available for YT Go?
`Rongmei Zhang, 5/9/2017
`
`My understanding is that YT Go would need related items restricted
`1
`to the YT Go corpus, is that correct? It might be possible to filter
`suggestions, but some analysis might be needed to see if there is enough
`coverage to provide a reasonable user experience.
`+bfroehle@google.com
`Su-Lin Wu, 5/9/2017
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052181
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 10 of 91
`
`Challenges
`
`User Intents
`
`• Finding -- continuation of search
`• Browsing -- continuation of homepage, direct watch
`• Navigation -- lean back autoplay, episodic series, more from uploader
`
`Creator/Advertiser Expectations
`
`• Large driver of traffic
`• Part of their watch page experience
`• Brand perception for Creators and YouTube
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052182
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 11 of 91
`
`Goals
`uccess metrics
`
`• Watchtime
`• Freshness
`• DAVs
`• Satisfaction (panel bad rate)
`
`Secondary considerations
`
`• Effect on partner traffic
`• Exposure for paid movies, live events, Red originals, playlists etc
`• Perception of YouTube and creators
`• Embed strategy
`• Related graph data
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052183
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 12 of 91
`
`Slide 9
`
`1
`
`And also perception of regulators and governments.
`Cristos Goodrow, 5/8/2017
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052184
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 13 of 91
`
`YouTube and Watch Next are Growing
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052185
`
`YouTube and Watch Next are Growin
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052185
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 14 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052186
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052186
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 15 of 91
`
`go/sequences
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052187
`
`
`
`
`
`go/sequences
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052187
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 16 of 91
`
`Recent Accomplishments (Q4 through today)
`
`• Systems and infrastructure simplifications
`o Flattened nominator graph
`o Experiments auto-pushed
`
`(in collaboration with many, including
`Home/Recs, Research, Content Id, Abuse)
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052188
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 17 of 91
`
`Slide 13
`
`Please spell-out abbreviations like WTR (which I think it Watch
`2
`Through Rate) and MFU (More From Uploader)
`Cristos Goodrow, 5/9/2017
`
`2
`
`Done, thanks for pointing those out.
`Su-Lin Wu, 5/9/2017
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052189
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 18 of 91
`
`Infrastructure
`
`Confidential & Proprietary
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052190
`
`Infrastructure
`
`0 Tube
`
`Confidential & Proprietary
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052190
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 19 of 91
`
`High Level View
`
`4.2)44
`
`Billion Videos 11,14',
`
`Wit,
`
`1. Candidate
`Nomination
`
`2. Scoring
`
`-100 Candidates
`
`A Few Winners
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052191
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 20 of 91
`
`Request Flow
`
`Generate candidates.
`
`Nomination
`
`Attach video/playlist metadata, and
`other annotations.
`
`Annotation
`
`Remove candidates.
`
`Score candidates with Brain.
`
`Make the final output decisions:
`autoplay candidate, shelves
`
`Filtering
`
`Scoring
`
`Packing
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052192
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 21 of 91
`
`System Overview
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052193
`
`System Overview
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052193
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 22 of 91
`
`Scale & Resource Costs
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052194
`
`Scale & Resource Costs
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052194
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 23 of 91
`
`Slide 18
`
`+heldt@google.com
`1
`+liwei@google.com
`
`Approximate current plans:
`Brain => Still on Axon (not Servo). Working on Servo migration first (Q3)
`and then plan to start evaluating SeaStar. Note SeaStar will require some
`model changes (discretization) with unknown, potentially negative, quality
`implications. Will require experimentation.
`Jellyfish => No plans yet.
`Brad Froehle, 5/6/2017
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052195
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 24 of 91
`
`Slide 18
`
`Lastly, one issue w/ SeaStar is that a naive implementation may
`2
`strand a large amount of CPU resources on the dedicated machines. It's my
`understanding that we'll need to pack other jobs on those machines for them
`to be cost effective.
`
`These solutions are being explored jointly w/ p13n, since they have similar
`use case. Planning a shared YT Discovery pool of brain servers (or possibly
`Servomatic, run by ML SRE).
`Brad Froehle, 5/6/2017
`
`Brain = plans to move to SeaStar?
`1
`Training = plans to move to gpu or Jellyfish?
`Danner Stodolsky, 5/8/2017
`
`1
`
`We are planning on switching to Servo in Q2.
`Lukasz Heldt, 5/8/2017
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052196
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 25 of 91
`
`Infrastructure Trends
`
`• Batch —> Streaming pipelines
`o Reduce latency of user action to incorporation in our systems. Freshness.
`• Pay down technical debt / improve code health / increase team velocity
`o Rewrite async producer or callback code with Fibers (Superroot Flows).
`o Release & push automation.
`• Simpler shared components (more "microservices")
`o Discovery Engine:
`■ Video Metadata Service (to replace Muppet).
`■
`("Co-watch") Candidate Service.
`■ Eventual migration from Superroot to Browse/Recs Platform?
`o Build on standard services like Bigtable, Laelaps, & Mapess.
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052197
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 26 of 91
`
`Freshness via infrastructure
`
`a
`
`• 2016Q1 Instant Muppet
`o Ensure fresh videos are in corpus and can be suggested
`o Fresher ranking & abuse signals (-12h —> 1 Os for important updates)
`• 2016Q3 Eggbeater + Candidate Service
`o Streaming co-watch related graph computations + new serving infrastructure
`o Co-watch latency: 22h—>E3h
`• 2017Q1 Intraday v3
`o Co-watch latency: 8h->3h
`• 2017Q1 Docfilter
`o React faster to private/unlisted—*public transitions
`o
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052198
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 27 of 91
`
`Slide 20
`
`3
`
`+bfroehle@google.com
`Su-Lin Wu, 5/9/2017
`
`The previous system only recorded public->private transitions;
`3
`there was no way to "refresh" a video, except waiting 12h for the deletion
`marker ttl to expire.
`Brad Froehle, 5/9/2017
`
`Please remind me how docfilter helps with unlisted->public (I see
`3
`that it would help with public->unlisted/delete).
`Cristos Goodrow, 5/9/2017
`
`4
`
`https://goto.google.com/wn-docfilter
`Brad Froehle, 5/9/2017
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052199
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 28 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052200
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052200
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 29 of 91
`
`Team Velocity
`
`Candidate Service
`
`• Make it easier to experiment with new, streaming, candidate sources.
`• Simple API; efficient at serving time.
`
`Experiments
`
`• Cookie-day experiments always running (no wait for preperiod); daily shuffled
`cookies reduce bias.
`• New experiment configurations pushed every 30 minutes automatically.
`• Easy to train & serve experimental Brain models.
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052201
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 30 of 91
`
`Challenges
`
`•
`
`"Full stack" development (backend + InnerTube / UI changes)
`•
`• Non-uniformity between Home & Watch Next (& Search)
`o Propagating scoring signals is unique on each surface; different proto representations of a
`video complicate code reuse.
`o Different approaches towards decoration.
`• Resource availability & management
`• Legacy code
`o Legacy RPC interface; ultra-legacy clients (GData)
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052202
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 31 of 91
`
`Nomination
`
`Confidential & Proprietary
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052203
`
`INOmlnation
`
`0 Tube
`
`Confidential & Proprietary
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052203
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 32 of 91
`
`Confidential & Proprietary
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052204
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052204
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 33 of 91
`
`Deep Dive:
`Related Graph ("Co-watch")
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052205
`
`Deep Dive:
`Related Graph ("Co-watch")
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052205
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 34 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052206
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052206
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 35 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052207
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052207
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 36 of 91
`
`Raw Stats (as of 2017-05)
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052208
`
`Raw Stats (as of 2017-05)
`
` HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052208
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 37 of 91
`
`(External) Related Graph Customers
`
`• Personalization
`• Video Inventory Analytics
`• Legos
`• Abuse
`• Unplugged
`• Ads
`• Search
`• Viewer Model
`• Research
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052209
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 38 of 91
`
`Future Directions
`
`TicTac
`
`• Streaming visits, targeting co-watch latency <1 h
`
`Discovery Engine
`
`• Power P13N "related-on-the-fly" nominator with Candidate Service infra.
`
`Matchmaker
`
`• Predictive co-watch at publish time.
`• Find ideal places to recommend newly published videos.
`•
`Improve recommendations after watching fresh videos.
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052210
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 39 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052211
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-0005221 1
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 40 of 91
`
`Online Ranking
`
`Confidential & Proprietary
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052212
`
`Online Ranking
`
`0 Tube
`
`Confidential & Proprietary
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052212
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 41 of 91
`
`Objective
`
`Lifetime watch time per user
`
`Watch time per session
`
`Watch time per panel
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052213
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 42 of 91
`
`Objective: Watch Time Per Panel
`
`• Given the watch video and user
`... how much watch time will candidate generate
`
`• Rank candidates by expected watch time
`
`P(ctr) * E(watch ratio I click) * video length
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052214
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 43 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052215
`
`Ranking Model
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052215
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 44 of 91
`
`Training
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052216
`
`Trainin
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052216
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 45 of 91
`
`Slide 37
`
`How exactly is a +ve / -ve example defined in this context? Is
`1
`there a doc describing this? +liwei@google.com +jhe@google.com
`Hariharan Chandrasekaran, 7/17/2018
`
`1
`Positive examples are the impressions that are clicked by the
`users. See this doc
`(https://docs.google.com/document/d/lmUFcByRMVfX3olGtEq_2yywqeQ4qSR(40VFvAMq
`MCNvA/edit#heading=h.pnz9paci6xOb) for details.
`Li Wei, 7/17/2018
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052217
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 46 of 91
`
`Training
`
`Brain Trainer
`(continuous)
`
`validate and
`push periodically
`
`1
`
`Brain
`Serving
`
`c
`
`......
`
`......,
`
`•
`
`• E
`
`-40 models (prod + exps)
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052218
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 47 of 91
`
`Features
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052219
`
`Features
`
` HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052219
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 48 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052220
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052220
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 49 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052221
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052221
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 50 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052222
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052222
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 51 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052223
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052223
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 52 of 91
`
`Sitewide Watch Time
`
`Launches since 5/2016:
`
`+3.14%
`
`Including autonav on mobile:
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052224
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 53 of 91
`
`Tuning Objective
`
`expected watch time per impression
`
`video length
`
`CTR
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052225
`
`Tuning Objective
`
`expected watch time per impression
`
`SS ~
`
`一
`
`es
`
`
`Video length
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052225
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 54 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052226
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052226
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 55 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052227
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052227
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 56 of 91
`
`Tuning Objective
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052228
`
`Tuning Objective
`
` HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052228
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 57 of 91
`
`Tuning Objective - future direction
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052229
`
`Tuning Objective - future direction
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052229
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 58 of 91
`
`source
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052230
`
`
`
`
`
`source
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052230
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 59 of 91
`
`Filtering & Safety
`
`Confidential & Proprietary
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCH N DR-00052231
`
`Filtering & Safety
`
`0 Tube
`
`Confidential & Proprietary
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052231
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 60 of 91
`
`Trashy Demotion
`
`• For trashy candidates:
`
`• Trashiness today:
`
`10 of the Worst Parents in the World
`leRichest 131
`ago • 8.439227 views
`narenting is an understatement. These are the top
`orrible mothers and fathers ever to have children
`
`5'27
`
`GIANT ANACONDA ATTACKS COW I
`ANACONDA ATTACK ANIMAL I ANACONDA...
`by Jala Mose
`1 month ago • 59,564 views
`GIANT ANACONDA ATTACKS COW I ANACONDA
`ATTACK ANIMAL I ANACONDA ATTACK AMAZING...
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052232
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 61 of 91
`
`MFA Demotion
`
`SPAM
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052233
`
`MFA Demotion
`
` HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052233
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 62 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052234
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052234
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 63 of 91
`
`Content Rating Filtering
`
`Exception made for series playlists (though safety mode continues to be respected).
`
`Documentation
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052235
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 64 of 91
`
`Content Rating Filtering
`
`Exception made for series playlists (though safety mode continues to be respected).
`
`Documentation
`
`You Tube
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052236
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 65 of 91
`
`Additional Filtering
`
`• Dishonor
`• Playability (rights management)
`• Rating (thumbs up/down ratio)
`• Previously dismissed
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052237
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 66 of 91
`
`Themes for 2017
`
`Serving
`
`Automation.
`
`Ranking
`
`TFX/Servo/Seastar
`
`Nomination
`
`model
`
`Quality
`
`Utilize & contribute Discovery Engine pieces.
`
`Move to shared scoring service; adopt
`
`Simplify architecture; move to fully streaming
`
`Move beyond immediate Watch Time (long ter
`.ull Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052238
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 67 of 91
`
`Appendix
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052239
`
`Appendix
`
`Tube
`
`Confidential & Proprietary
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052239
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 68 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052240
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052240
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 69 of 91
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052241
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052241
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 70 of 91
`
`Autonav Quality
`
`0 • A • 9,24
`
`Why don't we just take the first
`watch next result?
`
`Jimmy Interviews Peyton Manning
`(Kristen Wiig)
`2,514 285 views
`
`1611K 911K
`
`The Tonight Show Starring Ji..
`9,519,215 sLb.:rlber,
`
`IN Subscribe
`
`Jimmy Interviews Khaleesi from
`Game of Thrones (Kristen Wiig)
`The Tonight Show Starring
`7NA views
`
`Freestylin' with The Roots:
`Valentine's Day Hearts and Merd...
`The Tonight Show Starring Jimm...
`375K views
`
`Egg Russian Roulette with Peyton
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052242
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 71 of 91
`
`OK to suggest,
`not to play
`
`Paid videos, sensitive topics,
`live events, etc
`
`Higher bar
`for relevance
`
`Rank "next in series" or similar topics
`ahead of personal recs, more from
`creator, etc
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052243
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 72 of 91
`
`Google I You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052244
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052244
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 73 of 91
`
`A sequence, not just a video
`
`TODAY: Topic Drift
`
`Each pair makes sense, but the sequence
`does not.
`
`SOLUTION: Utilize watch history more
`effectively
`
`Google Chrome: Dear Sophie (1:32)
`https://www.youtube.corn/watch?y=R4vkVHijdOk
`by Google Chrome
`10,504,212 views
`wntube l doctube
`
`Related Videos
`
`Show results in vmi (videos only).
`
`0
`
`1
`
`2
`
`3
`
`-, 6iiiii 's there a dead
`pakistani on my couch
`ir.,, ,d,
`
`I ,
`
`30 Worst Google Auto-Completes Ever (10:41)
`https://www.youtube.com/watch?v=EKQEIImY_iU
`by Danger Dolan
`2,600,787 views
`INLINKS + TOPIC_SEARCH
`wntube I doctube
`
`----
`
`A VOLCANO ,
`WHERE?
`
`40 Most Amazing Last Words Ever Spoken (13:40)
`https://www.youtube.com/watch?v=t4tMtel87Vw
`by Danger Dolan
`2,297,352 views
`INLINKS
`wntube i doctube
`
`15 Weirdest Beaches on Earth (11:08)
`https://www.youtube.com/watch2v=LUgmKOLJUSs
`by Danger Dolan
`1,475,110 views
`INLINKS
`wntube I dOctube
`
`15 Most Dangerous Places on Earth (7:48)
`https://www.youtube.com/watch?y=sty9dGG8JY4
`by Danger Dolan
`6,198,744 views
`INLINKS
`vvntube i doctube
`
`.....
`
`"
`
`15 Magical Places You're Not Allowed To Visit (9:04)
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCH N DR-00052245
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 74 of 91
`
`Why don't we just autoplay the first result?
`
`J R
`
`III
`
`11, Follow
`
`If youtube Autoplay had a crossfader it'd actually
`be better than alot of djs
`
`Filter out
`• paid content, age-gated
`• playlists
`•
`recently watched
`
`Leanback considerations
`•
`less tolerance for WTF's
`• control for drifting off-topic
`• use Radio for music autoplay
`
`Different model for autoplay?
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052246
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 75 of 91
`
`Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052247
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052247
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 76 of 91
`
`Future Directions
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052248
`
`Future Directions
`
` HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052248
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 77 of 91
`
`Ranking Model
`
`• One of the biggest models at Google (-210 billion training examples)
`• Using latest technology
`
`08/2016
`
`01/2017
`
`Sibyl —> Distbelief
`
`Distbelief —> Tensorflow
`
`• Close collaboration with the research
`o DB -> TF transition findings shared with many other teams at Google
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052249
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 78 of 91
`
`Features
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052250
`
`Features
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052250
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 79 of 91
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052251
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052251
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 80 of 91
`
`Q4-Today Accomplishments
`
`You Tube
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052252
`
`Q4-Today Accomplishments
`
`
`
`HIGHLY CONFIDENTIAL — ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052252
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 81 of 91
`
`Slide 72
`
`4
`+bfroehle@google.com
`What else should we mention here?
`Su-Lin Wu, 5/2/2017
`
`+heldt@google.com
`5
`Yi's user state launch might be worth talking about in detail, because
`there is some evidence that the watch page can move dav's.
`Su-Lin Wu, 5/2/2017
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052253
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 82 of 91
`
`Notes Summary:
`
`Slide 4: 'Maybe remove the second part'
`Slide 5: 'Note -- clean up screenshots in next iteration'
`Slide 6: 'Also kids, music, gaming.'
`Slide 8: 'The watch page is not a destination that people revisit (often);
`WN is a continuation of the user experience, making it easier for the user
`to find another video to watch.
`The suggestion panel attempts to satisfy multiple user intents and balance
`with creator/advertiser interests such as brand safety.
`Tail content is important, poor experiences that end a user's session are
`more memorable than average ones.'
`Slide 10: 'https://video-analytics-
`demo.corp.google.com/#dt=c,fe=17285,fr=lw-
`001,fs=15584;fc=0,fcr=0,r=trafficsources,rpa=a,rpbm=7-93-
`110,rpd=112,rpg=93,rpm=a,rpp=0,rppc=O,rpr=w,rps=93,rpsd=1,rpt=0'
`Slide 13: '* Trashy impact updated to reflect human eval scoring.'
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052254
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 83 of 91
`
`Notes Summary:
`
`Slide 15: 'Candidate generation is a spectrum: related to video watched <-
`
`> related to user
`
`Some nominators target video, some target user. Over time, working more
`towards a blend.'
`Slide 17: 'Instant doc updates: http://vi/throttler/domain?domain=youtube-
`related-pipeline.corpus-updater-instant-dirty
`mentos, content labels, new, first legos, delete, title/description change,
`live event, news cluster, dishonor, series'
`Slide 18:
`'https://resourceweather.corp.google.com/#costs?cells=&charge=dollar&cluste
`rs=&dateFrom=2017-05-02&dateTo=2017-05-
`02&excludeManualCharges=true&excludeTiSupplier=false&groupBy=principal.mdb&
`grouping=true&mdbGroups=&measures=net,costutil,allocations,utilization,usag
`e,asstock&mode=table&predefinedDateRangeWithViewMode=userDefinedlday&princi
`palId=50040647&services=summary&sortBy=totalComputeCostSweHpq&sortOrder=des
`cending&utilizationType=service'
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052255
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 84 of 91
`
`Notes Summary:
`
`Slide 20: 'Lots of ways to accidentally lose freshness:
`
`Docfilter: https://ariane.googleplex.com/launch/181356'
`Slide 25:
`'https://dasnay.corp.google.com/dnstai85/#1abel id=1r79b&view=default
`
`Playlists?
`
`Some non-co-watch important for bootstrapping the co-watch graph.'
`Slide 27: 'Search strong signal because it reflects user intent. Others
`provide coverage for things people aren't searching for. Others act to
`reinforce.'
`Slide 28: 'Annotate w/ end-to-end chart. Describe the evolution of a panel
`over a week.'
`Slide 29: 'Add slide on glassbox model? (only if time allows).
`Call out update rate.'
`
`HIGHLY CONFIDENTIAL - ATTORNEYS' EYES ONLY
`
`GOOG-SCHNDR-00052256
`
`

`

`Case 3:20-cv-04423-JD Document 209-1 Filed 12/08/22 Page 85 of 91
`
`Notes Summary:
`
`Slide 30: 'http://go/wn-related-graph-table.
`
`Personalization: "Related-on-the-Fly" nominator
`VIA: Unfiltered graph for content labels
`Legos: Smear entities across related graph
`Abuse:
`Un

This document is available on Docket Alarm but you must sign up to view it.


Or .

Accessing this document will incur an additional charge of $.

After purchase, you can access this document again without charge.

Accept $ Charge
throbber

Still Working On It

This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.

Give it another minute or two to complete, and then try the refresh button.

throbber

A few More Minutes ... Still Working

It can take up to 5 minutes for us to download a document if the court servers are running slowly.

Thank you for your continued patience.

This document could not be displayed.

We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.

Your account does not support viewing this document.

You need a Paid Account to view this document. Click here to change your account type.

Your account does not support viewing this document.

Set your membership status to view this document.

With a Docket Alarm membership, you'll get a whole lot more, including:

  • Up-to-date information for this case.
  • Email alerts whenever there is an update.
  • Full text search for other cases.
  • Get email alerts whenever a new case matches your search.

Become a Member

One Moment Please

The filing “” is large (MB) and is being downloaded.

Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.

Your document is on its way!

If you do not receive the document in five minutes, contact support at support@docketalarm.com.

Sealed Document

We are unable to display this document, it may be under a court ordered seal.

If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.


Access Government Site

We are redirecting you
to a mobile optimized page.





Document Unreadable or Corrupt

Refresh this Document
Go to the Docket

We are unable to display this document.

Refresh this Document
Go to the Docket