Skip to main content

Design a Global Video Streaming Service (e.g., YouTube, Netflix)

 

Design a Global Video Streaming Service (e.g., YouTube, Netflix)

Question: Design a scalable and fault-tolerant video streaming platform that can:

  • Stream videos globally with low latency.

  • Allow users to upload videos.

  • Handle millions of users simultaneously.

Requirements (functional and non-functional)

functional requirement: video upload, video streaming, handle the network bandwidth, video for the different devices like mobile, smart tv, computer non-function : high availability, fault tolerance


1. High-Level Requirements

Functional Requirements:

  • Video Upload: Users can upload videos in various formats.

  • Video Streaming: Provide smooth playback with adaptive streaming for different network conditions.

  • Network Bandwidth Handling: Adjust video quality dynamically based on bandwidth.

  • Device Compatibility: Support multiple devices (e.g., mobile, smart TV, computer).

Non-Functional Requirements:

  • High Availability: The service should handle millions of concurrent viewers with minimal downtime.

  • Fault Tolerance: The system should recover gracefully from failures like server crashes or network issues.

2. High-Level Design

Here's the architectural breakdown:

  1. Frontend: Provides user interface for uploading, browsing, and watching videos.

  2. Backend Services:

    • Upload Service: Handles video uploads and metadata storage.

    • Processing Service: Transcodes videos into multiple resolutions and formats.

    • Streaming Service: Delivers videos to users with adaptive bitrate streaming.

  3. Content Delivery Network (CDN): Caches videos close to users for low-latency streaming.

  4. Database:

    • Metadata storage (e.g., title, description, resolution info).

    • User data (e.g., watch history, preferences).

  5. Storage: Distributed storage for original and transcoded videos.

  6. Load Balancer: Distributes requests across multiple servers to ensure availability.

3. Capacity Planning

Let’s estimate resource requirements for a system handling 10 million daily users:

Storage:

  • Assume 1 million uploads daily, average video size = 100 MB.

  • Original videos = 1 million x 100 MB = 100 TB/day.

  • Transcoded versions (3 resolutions) = 3 x 100 TB = 300 TB/day.

  • For 1 month of storage: 300 TB x 30 days = ~9 PB (Petabytes).

Traffic:

  • Assume 10 million users, each streaming an average of 1 hour/day.

  • Bitrate for 1080p video: 5 Mbps.

  • Total bandwidth required: 10 million x 5 Mbps = 50 Tbps.

  • A CDN can offload 80% of traffic, so backend bandwidth = 10 Tbps.

Processing:

  • Each video is transcoded into 3 resolutions.

  • Average transcoding time per video = 5 minutes.

  • Total processing required: 5 minutes x 1 million videos/day = ~83,333 hours/day.

  • With 100 servers handling 50 videos/hour, you’ll need ~1,667 servers for transcoding.

4. Detailed Design

Upload Workflow:

  1. User uploads video.

  2. Upload Service stores the video in temporary storage (e.g., S3 bucket).

  3. Metadata (e.g., title, uploader info) is stored in a relational database like PostgreSQL.

  4. Processing Service fetches the video, transcodes it into multiple resolutions (e.g., 1080p, 720p, 480p), and stores them in distributed storage (e.g., HDFS).

Streaming Workflow:

  1. User requests a video.

  2. The Streaming Service retrieves the video metadata.

  3. CDN serves the video, reducing load on the backend.

  4. Adaptive streaming adjusts resolution based on the user’s available bandwidth.

Device Compatibility:

  • Transcode videos into formats like H.264 or H.265 to support multiple devices.

  • Use HTML5 players for web and SDKs for smart TVs and mobile devices.

5. Handling Edge Cases

Video Uploads:

  • Large Files: Use chunked uploads to handle interruptions.

  • Invalid Formats: Validate video format during upload.

Streaming:

  • Low Bandwidth: Use adaptive bitrate streaming to lower resolution for slow connections.

  • Server Outages: Use replicated storage to serve videos from a different region.

High Traffic:

  • Use CDNs to cache popular videos geographically closer to users.

  • Auto-scale backend servers to handle traffic spikes.

6. Trade-Offs

1. Storage Cost vs. Quality:

  • Storing multiple resolutions increases costs but improves device compatibility.

  • You may decide to limit resolutions for infrequently accessed videos.

2. Caching vs. Latency:

  • CDNs reduce latency but introduce cache invalidation challenges for newly uploaded videos.

3. Consistency vs. Availability:

  • For highly available systems, some metadata (e.g., view counts) may be eventually consistent.

7. Final System Diagram

Here’s what the architecture looks like:

User -> CDN -> Load Balancer -> Streaming Service -> Video Storage
       -> Upload Service -> Processing Service -> Distributed Storage
       -> Metadata DB


Labels:

Comments

Post a Comment

Popular posts from this blog

Azure key vault with .net framework 4.8

Azure Key Vault  With .Net Framework 4.8 I was asked to migrate asp.net MVC 5 web application to Azure and I were looking for the key vault integrations and access all the secrete out from there. Azure Key Vault Config Builder Configuration builders for ASP.NET  are new in .NET Framework >=4.7.1 and .NET Core >=2.0 and allow for pulling settings from one or many sources. Config builders support a number of different sources like user secrets, environment variables and Azure Key Vault and also you can create your own config builder, to pull in configuration from your own configuration management system. Here I am going to demo Key Vault integrations with Asp.net MVC(download .net framework 4.8). You will find that it's magical, without code, changes how your app can read secretes from the key vault. Just you have to do the few configurations in your web config file. Prerequisite: Following resource are required to run/complete this demo · ...
Post a Comment
Read more

How to Make a Custom URL Shortener Using C# and .Net Core 3.1

C# and .Net Core 3.1:  Make a Custom URL Shortener Since a Random URL needs to be random and the intent is to generate short URLs that do not span more than 7 - 15 characters, the real thing is to make these short URLs random in real life too and not just a string that is used in the URLs Here is a simple clean approach to develop custom solutions Prerequisite:  Following are used in the demo.  VS CODE/VISUAL STUDIO 2019 or any Create one .Net Core Console Applications Install-Package Microsoft.AspNetCore -Version 2.2.0 Add a class file named ShortLink.cs and put this code: here we are creating two extension methods. public   static   class   ShortLink {      public   static   string   GetUrlChunk ( this   long   key ) =>            WebEncoders . Base64UrlEncode ( BitConverter . GetBytes ( key ));      public   static   long   GetK...
Post a Comment
Read more

Azure Logic Apps Send Email Using Send Grid Step by Step Example

Azure Logic Apps Send Email Using Send Grid Step by Step     Step 1- Create Send Grid Account Create a SendGrid Account  https://sendgrid.com/ Login and Generate Sendgrid Key and keep it safe that will be used further to send emails You can use Free service. it's enough for the demo purpose Step 2- Logic App Design Login to  https://portal.azure.com Go to Resources and Create Logic App Named "EmailDemo" Go To Newly Created Rosoure Named "EmailDemo" and Select a Trigger "Recurrence", You can choose according to your needs like HTTP, etc. Note* Without trigger you can not insert new steps or Actions Click on Change Connection and add Send Grid Key  Click on Create and Save Button on the Top. As we have recurrence so it will trigger according to our setup(every 3 months) so just for the test click on "RUN" button  Finally, you should get an email like below one:
Post a Comment
Read more