High-Level Design (HLD) is like creating a blueprint for a software system before actually building it. It gives an overview of how different parts of the system will work together. HLD is created before Low-Level Design (LLD) and actual coding.
- Gives a Clear Picture – Helps developers, architects, and stakeholders understand how the system will be structured.
- Saves Time & Effort – Prevents mistakes by planning the system before writing code.
- Ensures Scalability & Performance – Ensures the system can handle growth in users and data.
HLD typically consists of the following elements:
A visual representation of how different components interact.
- Example: A web application’s HLD may include frontend (React), backend (Spring Boot), database (MySQL), and caching system (Redis).
Explains different modules in the system and their purpose.
- Example: An e-commerce system may have:
- User Module (Handles login, signup)
- Product Module (Stores product details)
- Order Module (Manages orders and payments)
Specifies the programming languages, frameworks, and databases used.
- Example:
- Backend: Spring Boot
- Frontend: React
- Database: PostgreSQL
- Authentication: JWT & OAuth2
Describes how data moves between different components.
- Example:
- A user requests product details.
- The request goes to the backend (Spring Boot).
- The backend fetches data from the database.
- The response is sent back to the frontend.
Lists the main APIs that different components will use.
- Example:
POST /login→ To authenticate users.GET /products→ To fetch product list.
How the system ensures data protection and secure access.
- Example:
- Encryption for storing sensitive data.
- Role-based access for different users.
How the system will handle high traffic and future growth.
- Example:
- Load Balancers to distribute traffic.
- Caching (Redis) to speed up data retrieval.
Imagine you're designing a URL Shortener like bit.ly. Here's how HLD might look:
User → Load Balancer → API Gateway → URL Shortener Service → Database (MongoDB) ↳ Cache (Redis)
- User Service → Handles authentication.
- Shortener Service → Generates short URLs.
- Analytics Service → Tracks clicks.
- Backend: Spring Boot
- Database: MongoDB
- Cache: Redis
- Security: OAuth2
- User enters a long URL.
- Backend generates a short URL and stores it.
- When the short URL is clicked, backend retrieves the original URL from the database or cache.
POST /shorten→ Converts long URL to short.GET /{shortCode}→ Redirects to original URL.
| Feature | High-Level Design (HLD) | Low-Level Design (LLD) |
|---|---|---|
| Focus | Big picture of the system | Detailed implementation |
| Diagrams Used | Architecture diagrams | Class diagrams, sequence diagrams |
| Target Audience | Architects, Managers | Developers, Engineers |
| Example | “We’ll use a cache for faster access” | “We’ll use Redis with an expiry of 1 hour” |
- HLD is like a roadmap for software development.
- It ensures that the system is structured, scalable, and secure.
- Once HLD is finalized, developers proceed with Low-Level Design (LLD) and coding.
When designing software applications, you can choose between Monolithic and Distributed architectures. Let’s break them down in an easy-to-understand way.
A Monolithic system is a single, unified application where all parts (frontend, backend, database, etc.) are tightly coupled and run as a single unit.
Imagine you’re building an e-commerce website where users can buy products. A monolithic approach means:
- Frontend (UI), Backend (Business Logic), and Database are all in one big codebase.
- If one part fails, the whole system might crash.
- Everything is deployed together in one server.
User → Monolithic Server (Handles everything: UI, APIs, Database queries)
- ✔ Simple to develop & deploy – One codebase, one deployment.
- ✔ Easier debugging & testing – Everything runs in a single environment.
- ✔ Better performance (for small apps) – No network latency between services.
- ✖ Difficult to scale – Scaling means deploying everything, even if only one feature needs more resources.
- ✖ Slower development – As the codebase grows, it becomes harder to manage.
- ✖ Single point of failure – If one module crashes, the whole system might fail.
- ✅ Small or Medium-sized applications.
- ✅ Startups or MVPs (Minimal Viable Products).
- ✅ Teams with limited DevOps resources.
A Distributed system is where different parts of an application are split into separate services that communicate over a network.
Instead of a single monolithic system, we break it into multiple services:
- User Service → Handles login/signup.
- Product Service → Manages products.
- Order Service → Handles orders and payments.
- Notification Service → Sends emails and messages. Each of these runs independently and can be deployed separately.
User → API Gateway → [User Service] | [Product Service] | [Order Service] | [Database]
- ✔ Scalability – Services can scale independently.
- ✔ Fault Tolerance – If one service fails, others continue working.
- ✔ Faster Development – Teams can work on different services in parallel.
- ✔ Technology Flexibility – Each service can use a different technology stack.
- ✖ Complex Communication – Services talk to each other via APIs, increasing network overhead.
- ✖ Harder Debugging – More logs and services make it harder to trace errors.
- ✖ Deployment & Maintenance Overhead – Requires advanced DevOps setup (Docker, Kubernetes, etc.).
- ✅ Large applications with high traffic.
- ✅ Businesses that need high availability and fault tolerance.
- ✅ Teams working on different features in parallel.
| Feature | Monolithic System | Distributed System |
|---|---|---|
| Architecture | All-in-one codebase | Separated into multiple services |
| Scalability | Hard to scale | Easy to scale individual services |
| Performance | Faster (no network calls) | Slightly slower (inter-service communication) |
| Fault Tolerance | Low (failure crashes the app) | High (only one service might fail) |
| Deployment | Single deployment | Independent deployments |
| Tech Stack | One stack for all | Can use different stacks per service |
| Best for | Small/medium apps | Large, high-traffic apps |
- Monolithic Systems are simple, easy to develop, and best for small to medium projects.
- Distributed Systems offer better scalability, flexibility, and fault tolerance but add complexity.
Choosing the Right Approach depends on your team size, scalability needs, and project complexity.
Latency is the time it takes for a system to respond to a request. It is a key performance metric in system design, especially for high-performance applications like web services, databases, and distributed systems.
There are multiple types of latency depending on where the delay occurs:
- Network Latency – The time taken for data to travel across a network.
- Disk Latency – The time taken to read/write data from/to a disk.
- Memory Latency – The delay in accessing data from RAM.
- Processing Latency – The time taken by a CPU to process a request.
- Database Latency – The delay in retrieving data from a database.
- Application Latency – The time taken by the application logic to process and respond to a request.
Latency is caused by several factors, including:
- Physical Distance – Data takes time to travel over long distances.
- Congestion – Too many requests in a system slow down response times.
- I/O Delays – Reading from disks or databases takes time.
- Context Switching – When a CPU switches between tasks, there is overhead.
- Serialization & Deserialization – Converting data formats adds delay.
Latency is usually measured in milliseconds (ms) and can be analyzed using:
- Average Latency – The mean time taken for requests.
- P99, P95, P50 (Percentiles) – Helps analyze worst-case performance (P99 means 99% of requests are faster than this value).
- Tail Latency – The slowest response times, often affecting user experience.
To improve performance, we can apply the following techniques:
-
Caching
- Store frequently accessed data in memory (e.g., Redis, Memcached).
- Reduces database and disk read latency.
-
Load Balancing
- Distribute requests across multiple servers.
- Prevents overload on a single machine.
-
CDN (Content Delivery Network)
- Stores copies of static content closer to users.
- Reduces network latency.
-
Database Optimization
- Use indexing to speed up searches.
- Partition data to distribute load.
-
Asynchronous Processing
- Use background jobs instead of waiting for tasks to complete.
- Example: Sending an email asynchronously instead of making users wait.
-
Efficient Data Serialization
- Use lightweight formats like Protocol Buffers (Protobuf) instead of JSON.
-
Reducing Network Hops
- Minimize the number of times data needs to be transferred between services.
-
Edge Computing
- Process data closer to users instead of sending it to a central server.
- Google Search: Uses caching and distributed indexing to return results in milliseconds.
- Netflix: Uses CDNs to store videos closer to users, reducing buffering time.
- Amazon: Optimizes database queries and uses distributed databases to reduce checkout time.
Latency is a critical factor in system design that affects user experience and performance. Understanding different types of latency and applying optimization techniques can help build scalable and efficient systems.
A Content Delivery Network (CDN) is a system of distributed servers that helps deliver content (like images, videos, and web pages) quickly to users by caching it in multiple locations worldwide.
CDNs reduce latency and improve load times by serving content from a server closer to the user rather than always relying on the origin server.
CDNs operate using a network of edge servers, which cache content from an origin server. When a user requests content:
- Request Routing – The CDN determines the nearest server to the user.
- Cache Lookup – If the requested content is available on the edge server, it is delivered instantly.
- Origin Fetch – If the content is not cached, the edge server requests it from the origin server and stores it for future requests.
This mechanism significantly reduces the time needed to serve content.
CDNs improve system performance in multiple ways:
- Without a CDN: A user in India accessing a website hosted in the US experiences higher latency.
- With a CDN: The content is cached in India, reducing round-trip time.
- Less data needs to be fetched from the origin server.
- Optimized delivery of assets like images, CSS, and JavaScript.
- The origin server handles fewer requests.
- CDNs distribute traffic, preventing overload.
- If the origin server fails, cached content is still served.
- Load balancing ensures continuous availability.
- CDNs absorb traffic spikes and protect against Distributed Denial of Service (DDoS) attacks.
- They act as a buffer between attackers and the origin server.
CDNs can be categorized based on their functionality:
- Caching CDNs – Store and serve static content like images, videos, CSS, and JavaScript.
- Dynamic Content CDNs – Optimize dynamic content like personalized web pages.
- Streaming CDNs – Deliver high-quality live and on-demand videos.
- Security CDNs – Protect against cyber threats like DDoS and bot attacks.
CDN architecture consists of several key components:
- Origin Server – The main server where the original content is stored.
- Edge Servers – Distributed servers that cache content closer to users.
- PoPs (Points of Presence) – Data centers where edge servers are located.
- Load Balancers – Distribute traffic efficiently to reduce congestion.
- Anycast DNS – Routes users to the nearest server automatically.
To maximize CDN efficiency, companies use several techniques:
- Time-to-Live (TTL): Determines how long content stays in the cache.
- Cache Invalidation: Ensures outdated content is removed or updated.
- Stale-While-Revalidate: Serves old content while fetching a fresh copy.
- Gzip & Brotli: Compress text-based files to reduce transfer size.
- Image Optimization: WebP or JPEG compression reduces image load time.
- Geo Load Balancing: Directs users to the nearest CDN server.
- Traffic Routing: Uses real-time traffic conditions to optimize delivery.
- HTTPS Everywhere: Encrypts data for security.
- Token Authentication: Ensures authorized users access premium content.
Some well-known CDN providers include:
- Cloudflare – Security-focused, free-tier available.
- Akamai – Enterprise-grade, widely used by large organizations.
- AWS CloudFront – Integrates with Amazon Web Services.
- Google Cloud CDN – Works with Google Cloud Platform.
- Fastly – Optimized for real-time streaming.
- Netflix: Uses CDNs to deliver high-definition video with minimal buffering.
- Amazon: Uses CloudFront to speed up e-commerce site load times.
- Facebook & Instagram: Use CDNs to load images and videos instantly.
While CDNs are beneficial, they may not be ideal in certain situations:
- Highly Dynamic Content – If content changes frequently, caching is ineffective.
- Low-Traffic Websites – The cost may outweigh the benefits.
- Strict Data Residency Rules – Some organizations must store data in specific locations.
CDNs play a crucial role in reducing latency, improving performance, and increasing availability. They are essential for handling global traffic efficiently and securing web applications.
Throughput is the rate at which a system processes requests or data over a given time period. It is usually measured in:
- Requests per second (RPS) – Web servers and APIs.
- Transactions per second (TPS) – Databases and financial systems.
- Megabits per second (Mbps) or Gigabits per second (Gbps) – Network speed.
- Latency is the delay in processing a single request.
- Throughput is the total amount of work the system handles over time.
- A system with low latency (quick response) but low throughput can only handle a few users at a time.
- A system with high throughput can process many requests per second, even if each request takes time.
Several components impact throughput in a system:
- A slow CPU limits the number of requests it can handle per second.
- Multi-core CPUs and parallel processing increase throughput.
- Limited bandwidth causes bottlenecks in data transmission.
- CDNs and compression help optimize throughput.
- HDDs have lower throughput due to mechanical parts.
- SSDs (Solid State Drives) improve throughput by providing faster read/write speeds.
- Indexes, query optimization, and caching improve database throughput.
- High-throughput databases include Apache Cassandra, Redis, and Amazon DynamoDB.
- More threads/processes increase throughput by handling multiple requests simultaneously.
- Load balancing distributes traffic, preventing one server from becoming a bottleneck.
- Redis, Memcached, and CDNs reduce database and API calls, improving throughput.
Throughput is measured using different tools and metrics:
- Used for APIs, databases, and web applications.
- Tools: Apache JMeter, Locust, wrk.
- Used for network speed tests.
- Tools: iPerf, Wireshark.
- Measured in MB/s (Megabytes per second).
- Tools: iostat, fio.
To scale a system for higher throughput, consider:
- Distribute requests across multiple servers.
- Use Round Robin, Least Connections, or Hash-based routing.
- Redis/Memcached: Store frequently accessed data in memory.
- CDNs: Serve static content closer to users.
- Indexing, query tuning, and partitioning reduce database load.
- Read replicas improve read throughput.
- Offload tasks using message queues (Kafka, RabbitMQ).
- Use event-driven architectures instead of synchronous requests.
- Gzip/Brotli compression reduces bandwidth usage.
- HTTP/2 & QUIC protocols improve data transfer efficiency.
- Add more servers instead of upgrading a single machine.
- Stateless microservices allow easy scaling.
- Handles hundreds of thousands of queries per second.
- Uses distributed databases and caching for high throughput.
- Streams high-quality video to millions of users simultaneously.
- Uses CDNs, parallel data pipelines, and microservices.
- Process thousands of transactions per second.
- Use sharding, failover mechanisms, and caching.
- Batching requests increases throughput but introduces delay.
- Databases like Cassandra favor availability & throughput over strict consistency (CAP theorem).
Throughput is a key performance metric in web applications, databases, and network systems. Optimizing throughput requires techniques like caching, load balancing, parallelism, and scaling.
Availability in system design refers to a system's ability to remain operational and accessible for users without downtime. It is typically measured as a percentage of uptime over a given period.
Ensures a system is always accessible, even during failures.
[ \text{Availability} = \left( \frac{\text{Uptime}}{\text{Uptime} + \text{Downtime}} \right) \times 100 ]
- A system with 99.99% availability experiences ~52 minutes of downtime per year.
- A system with 99.9% availability has ~8.76 hours of downtime per year.
| Availability % | Downtime Per Year | Downtime Per Month |
|---|---|---|
| 99% (Two Nines) | 3.65 days | 7.2 hours |
| 99.9% (Three Nines) | 8.76 hours | 43.8 minutes |
| 99.99% (Four Nines) | 52 minutes | 4.38 minutes |
| 99.999% (Five Nines) | 5.26 minutes | 26.3 seconds |
Several factors influence a system’s availability:
- Server crashes, disk failures, or power outages reduce availability.
- Solution: Redundant servers and failover mechanisms.
- Bugs in applications, OS crashes, or memory leaks cause downtime.
- Solution: Automated testing and rolling deployments.
- ISP outages, DNS failures, or DDoS attacks can make a system unavailable.
- Solution: Multiple network providers, CDNs, and load balancers.
- Deadlocks, corruption, or overloads can lead to downtime.
- Solution: Replication, partitioning, and caching.
- Misconfigurations, accidental deletions, or security breaches.
- Solution: Automated backups, version control, and monitoring.
- Distributes traffic across multiple servers to avoid overloading any single one.
- Examples: Nginx, HAProxy, AWS ELB.
- Active-Passive Failover: A backup server takes over if the primary one fails.
- Active-Active Failover: Multiple servers handle requests simultaneously.
- Master-Slave Replication: One master database for writes, multiple replicas for reads.
- Sharding: Splitting data across multiple databases to prevent overload.
- Reduces database load by storing frequently accessed data.
- Examples: Redis, Memcached, Cloudflare CDN.
- Automatically adds or removes servers based on demand.
- Examples: AWS Auto Scaling, Kubernetes Horizontal Pod Autoscaler (HPA).
- Using multiple data centers across different regions improves fault tolerance.
- Example: Netflix runs on AWS across multiple regions.
- Regular backups prevent data loss in case of failures.
- Disaster Recovery (DR) Plans ensure quick recovery after major failures.
- A single server or data center means if it fails, the entire system is down.
- Example: Small startups with limited resources.
- Deploying in multiple availability zones within the same cloud provider.
- Example: AWS Multi-AZ RDS (Database replication across regions).
- Deploying in multiple regions across the globe ensures zero downtime.
- Example: Google Search, Netflix, Amazon.
- CAP Theorem states that a distributed system cannot have 100% availability and consistency at the same time.
- Example: AP systems (Amazon DynamoDB, Cassandra) prioritize availability over strict consistency.
- Higher availability means more servers, backups, and redundancy, increasing costs.
- Example: AWS multi-region deployments are expensive but ensure 99.999% uptime.
- Google Search: Uses multi-region deployments, load balancing, and failover mechanisms.
- Netflix: Deploys across multiple AWS regions to avoid downtime.
- Amazon AWS: Provides 99.999% availability for critical services like S3.
High availability is crucial for mission-critical applications and can be achieved through load balancing, redundancy, database replication, caching, and distributed systems.
Consistency in system design means that all nodes in a distributed system have the same data at any given time. When a client reads data, it should get the most recent and correct value.
According to CAP Theorem, a distributed system can only guarantee two out of the three properties:
- Consistency (C): Every read gets the latest write.
- Availability (A): Every request receives a response, even if some nodes fail.
- Partition Tolerance (P): The system continues to function despite network failures.
| System Type | Prioritizes |
|---|---|
| CP (Consistency + Partition Tolerance) | Guarantees correct data but might sacrifice availability. Example: Relational Databases (PostgreSQL, MySQL with strong consistency). |
| AP (Availability + Partition Tolerance) | Always responds but might return stale data. Example: NoSQL Databases like DynamoDB, Cassandra. |
| CA (Consistency + Availability) | Works only in ideal conditions without network failures (impractical in distributed systems). |
Different systems choose different levels of consistency based on their use case.
Guarantees that all nodes see the same data immediately after a write.
- Used in: Banking, financial transactions, and critical applications.
- Example: In a banking system, if you transfer ₹500, your updated balance should reflect immediately.
- Implementations: Distributed locking (Paxos, Raft), Single-leader databases (PostgreSQL, MySQL with primary-replica setup).
- Trade-off: Slower performance because all writes must be acknowledged before proceeding.
Guarantees that all replicas will eventually be consistent, but reads might see stale data.
- Used in: Social media, messaging apps, and content delivery networks (CDNs).
- Example: A Facebook status update might take a few seconds to appear on another user’s timeline.
- Implementations: NoSQL Databases (Cassandra, DynamoDB, MongoDB with replica sets).
- Trade-off: Fast reads and writes, but data might not be up-to-date immediately.
Guarantees that a user always sees their own updates immediately.
- Used in: User dashboards, profile updates.
- Example: If you change your LinkedIn profile picture, you should see the new image immediately, even if others still see the old one.
Ensures that if one event affects another, the order is preserved.
- Example: If Alice sends a message to Bob, Bob should see it before replying.
Ensures that a user never sees older data after reading a newer version.
- Example: If a stock price updates to ₹100, you won’t see ₹95 again.
Used to keep all replicas in sync.
- Examples: Paxos, Raft (Used in Apache Zookeeper, Kubernetes, etcd).
Quorum (N, R, W) model:
- N = Total number of replicas.
- R = Minimum replicas required for a read.
- W = Minimum replicas required for a write. Ensures consistency by requiring enough nodes to agree on updates.
- Example: Cassandra uses tunable consistency with quorum reads/writes.
Used in eventual consistency models to resolve differences between nodes.
- Techniques:
- Last Write Wins (LWW) – Latest timestamp is used.
- Version Vector (Vector Clocks) – Tracks different versions to detect conflicts.
- Application-Level Resolution – Manual merging of conflicting data.
One node (leader) handles writes, and others (followers) replicate data.
- Example: MySQL with primary-replica replication.
- Drawback: Leader failure can cause downtime unless failover mechanisms exist.
Each update creates a new version instead of overwriting data.
- Used in: Databases like PostgreSQL, MySQL (InnoDB), and MongoDB.
| Consistency Model | Pros | Cons |
|---|---|---|
| Strong Consistency | Guarantees correct, up-to-date data | Slower performance, increased latency |
| Eventual Consistency | Fast reads/writes, better availability | Stale reads, possible conflicts |
| Quorum-Based Consistency | Balances speed and correctness | Adds complexity, requires tuning |
Higher consistency means slower performance (due to synchronization overhead). Lower consistency improves speed but might return outdated data.
If a network partition occurs, choosing consistency means rejecting requests until synchronization is complete.
- Example: Banks prefer consistency over availability (incorrect balances are unacceptable).
- Example: Social media prefers availability over consistency (eventual consistency is fine).
| System | Consistency Model | Reason |
|---|---|---|
| Banking (PayPal, Visa) | Strong Consistency | Transactions must be accurate |
| Google Search Index | Eventual Consistency | Faster updates, slight delay is acceptable |
| WhatsApp Message Delivery | Causal Consistency | Message order must be maintained |
| Amazon DynamoDB | Eventual or Strong Consistency (Configurable) | Balances availability and consistency |
| Git Version Control | Multi-Version Consistency | Branching and merging ensure correctness |
- For financial transactions → Strong consistency (Correctness is more important).
- For social media feeds → Eventual consistency (Availability is more important).
- For e-commerce inventory → Quorum-based consistency (Ensures valid stock levels).
- For real-time messaging → Causal consistency (Ensures correct order of messages).
Consistency is crucial for ensuring data correctness in distributed systems. The choice between strong, eventual, and quorum-based consistency depends on the use case. Optimizing consistency often involves trade-offs with performance and availability.
Partition Tolerance (P) in system design means that a distributed system continues to operate even if some nodes cannot communicate due to network failures.
A network partition occurs when communication between nodes is lost due to failures (e.g., data center outages, server crashes, network congestion).
A partition-tolerant system ensures that it still functions correctly despite network partitions.
The CAP Theorem states that a distributed system can have at most two out of three properties:
- Consistency (C) – Every read gets the latest write.
- Availability (A) – Every request gets a response, even during failures.
- Partition Tolerance (P) – The system remains operational despite network failures.
| Type | Prioritizes |
|---|---|
| CP (Consistency + Partition Tolerance) | Guarantees correct data but sacrifices availability. Example: MongoDB with strong consistency. |
| AP (Availability + Partition Tolerance) | Ensures uptime but may return stale data. Example: DynamoDB, Cassandra. |
| CA (Consistency + Availability) | Works only if there are no partitions (not practical in distributed systems). |
Partition tolerance is necessary in distributed systems because networks are unreliable. Common causes include:
-
Data Center Failures: A power outage in one region makes that part of the system unreachable.
Example: AWS us-east-1 outage affects applications globally. -
Network Congestion & Latency Issues: High traffic can cause delays or dropped connections between nodes.
Example: Microservices in Kubernetes failing due to slow network responses. -
Hardware Failures: Server crashes, faulty network cables, or router failures can partition the network.
Example: A faulty switch disconnects half of a distributed database. -
DDoS Attacks: Malicious traffic overloads the network, making some nodes unreachable.
Example: Cloudflare mitigates DDoS attacks to maintain partition tolerance.
When a partition occurs, the system must decide how to handle unavailable nodes. Different strategies exist:
-
AP Systems (Availability + Partition Tolerance)
The system continues to respond but might return outdated data.
Example: Amazon DynamoDB, Apache Cassandra, CouchDB.
Trade-off: Reads might not be consistent, but the system remains online. -
CP Systems (Consistency + Partition Tolerance)
The system ensures consistency but may reject requests until all nodes sync.
Example: MongoDB (when configured with strong consistency), HBase, Zookeeper.
Trade-off: Slower responses or downtime during partitions. -
Quorum-Based Reads/Writes (Balanced Approach)
Uses majority voting to decide if a request is valid.
Example: Cassandra uses quorum-based writes to ensure data correctness.
Trade-off: Improves consistency but adds latency.
-
Data Replication
Keeps copies of data across multiple nodes.
Example: Google Spanner replicates data globally to handle failures. -
Multi-Region Deployment
Distributes services across different data centers.
Example: Netflix runs in multiple AWS regions to stay online. -
Leader Election (Consensus Protocols)
Elects a new leader if the current one fails.
Example: Raft & Paxos in Kubernetes leader election. -
Conflict Resolution Strategies
When network partitions heal, conflicts must be resolved.
Example: Last Write Wins (LWW) in DynamoDB, Vector Clocks in Riak.
| System | Partition-Tolerant? | Type |
|---|---|---|
| Amazon DynamoDB | Yes | AP (Eventual Consistency) |
| Google Spanner | Yes | CP (Strong Consistency) |
| Apache Cassandra | Yes | AP (Eventual Consistency) |
| MongoDB | Yes | CP (Strong Consistency Mode) |
| Kafka/Zookeeper | Yes | CP (Ensures data integrity) |
| Approach | Pros | Cons |
|---|---|---|
| AP Systems (DynamoDB, Cassandra) | High availability | Data may be inconsistent temporarily |
| CP Systems (MongoDB, Zookeeper) | Strong consistency | Might reject requests during failure |
| Quorum-Based (Cassandra, Spanner) | Balances availability and consistency | Adds latency |
- For real-time applications (e.g., stock trading, banking) → CP System (Strong Consistency).
- For social media, caching, or logs → AP System (High Availability).
- For a balance between consistency & availability → Quorum-based model (Cassandra, Spanner).
Partition tolerance is essential for any distributed system because network failures are inevitable. The system design must choose between Availability (AP) or Consistency (CP) based on the business needs.
The CAP theorem (also called Brewer’s Theorem) states that in a distributed system, you can only achieve two out of the following three properties:
- Consistency (C) – Every read returns the most recent write.
- Availability (A) – Every request gets a response, even if some nodes fail.
- Partition Tolerance (P) – The system continues to function despite network failures.
Since network partitions always exist in distributed systems, we must choose between Consistency (C) and Availability (A) when a partition occurs.
Let's consider a banking system:
If a network partition occurs and a customer withdraws ₹1000 from an ATM in one city and checks their balance in another city:
- Strong Consistency (C) → The system blocks the second request until all nodes sync.
- High Availability (A) → The system returns stale data but remains online.
- Partition Tolerance (P) → The system handles failures gracefully and continues working.
-
CP Systems (Consistency + Partition Tolerance)
Guarantees correct data but may reject requests during network failures.
Example: MongoDB (with strong consistency mode), HBase, Zookeeper
Use Case: Banking, financial transactions, stock trading
Example Scenario:
You transfer ₹5000 from one bank account to another.
CP System: If the network fails, the transfer is delayed but data remains correct. -
AP Systems (Availability + Partition Tolerance)
Guarantees uptime but may return stale or inconsistent data.
Example: Amazon DynamoDB, Cassandra, CouchDB, Akamai CDN
Use Case: Social media, caching, content delivery networks (CDNs)
Example Scenario:
You post a tweet, but due to network partition, some users see the tweet immediately while others see it a few seconds later. -
CA Systems (Consistency + Availability) (Impractical in Distributed Systems)
Works only in ideal conditions with no network failures (i.e., no partition tolerance).
Example: Single-node relational databases (MySQL, PostgreSQL in a single instance).
Use Case: Local databases (not distributed).
Example Scenario:
A single-node MySQL database can maintain both consistency and availability because there's no network partition.
| Database | CAP Type | Explanation |
|---|---|---|
| MongoDB | CP | Prioritizes consistency but may reject requests during failures. |
| Apache Cassandra | AP | Ensures availability but allows stale reads. |
| Google Spanner | CP | Strong consistency across regions, but might delay responses. |
| Amazon DynamoDB | AP | High availability with eventual consistency. |
| Zookeeper | CP | Used in leader election and coordination, prioritizing correctness. |
| Approach | Pros | Cons |
|---|---|---|
| CP (Strong Consistency, Tolerates Failures) | Guarantees accurate data | May reject requests or be slow |
| AP (High Availability, Tolerates Failures) | Always online, handles failures well | Might return outdated data |
| CA (Consistency + Availability, No Partition Tolerance) | Strong consistency & availability | Breaks in a distributed system |
- For banking and transactions → Choose CP (Consistency + Partition Tolerance).
- For social media and e-commerce → Choose AP (Availability + Partition Tolerance).
- For local applications → CA (Consistency + Availability) is fine.
The CAP theorem helps architects decide between consistency, availability, and partition tolerance when designing distributed systems. Since network failures are inevitable, most distributed databases sacrifice consistency (AP) or availability (CP), but never partition tolerance.
Scaling is the ability of a system to handle increasing workload by adding resources (hardware or software) efficiently. A well-designed system can scale without performance degradation.
- A system that can't scale will crash or slow down under heavy traffic.
- Websites like Amazon, Netflix, and Facebook must handle millions of users simultaneously.
- Scaling allows high availability, low latency, and better user experience.
There are two main ways to scale a system:
| Type | Description | Pros | Cons |
|---|---|---|---|
| Vertical Scaling (Scaling Up) | Adding more resources (CPU, RAM, SSD) to a single machine. | Simple to implement, no code changes | Expensive, hardware limits, single point of failure |
| Horizontal Scaling (Scaling Out) | Adding more machines (servers, databases, load balancers). | Infinite scalability, high availability | Complex, requires distributed system design |
How it works:
- Upgrade the CPU, RAM, or SSD of an existing server.
- Suitable for monolithic applications and relational databases (MySQL, PostgreSQL).
Example:
- Upgrading a 2-core CPU to an 8-core CPU to handle more users.
Limitations:
- Expensive: High-end servers cost more.
- Single Point of Failure: If the machine fails, everything crashes.
- Hardware Limits: There is an upper limit (e.g., max RAM = 1 TB).
When to Use Vertical Scaling?
- When data consistency is critical (banking, transactions).
- When handling moderate traffic growth.
How it works:
- Add more servers instead of upgrading one machine.
- Requires a load balancer to distribute traffic.
- Used in microservices, NoSQL databases, cloud services.
Example:
- Instead of upgrading a single database server, add multiple smaller servers in a cluster (e.g., MongoDB sharding).
Advantages:
- No hardware limit: Can add unlimited servers.
- Fault tolerance: If one server fails, others take over.
- Better cost efficiency: Commodity hardware can be used.
Disadvantages:
- More complexity: Requires load balancers, distributed databases, and consistency mechanisms.
- Network overhead: Communication between nodes adds latency.
When to Use Horizontal Scaling?
- High traffic applications (Netflix, Twitter, Instagram).
- Distributed databases (Cassandra, DynamoDB, Elasticsearch).
- Microservices architectures.
-
Load Balancing:
Distributes incoming traffic across multiple servers.
Popular Load Balancers: Nginx, HAProxy, AWS Elastic Load Balancer (ELB). -
Caching:
Stores frequently accessed data in memory (RAM) to reduce database load.
Examples: Redis and Memcached for in-memory caching. CDN (Content Delivery Network) for caching static content. -
Database Sharding:
Splits a large database into smaller partitions (shards).
Each shard handles a subset of queries → Improves read/write performance.
Example: Twitter shards users based on user ID ranges. -
Replication:
Read replicas improve performance by distributing read queries.
Master-slave replication (e.g., MySQL, PostgreSQL).
Eventual consistency is used in NoSQL databases (e.g., Cassandra). -
Asynchronous Processing:
Offloads heavy tasks to background workers.
Message Queues (Kafka, RabbitMQ) help handle spikes in traffic.
Example: Amazon uses Lambda functions for processing orders asynchronously. -
Microservices Architecture:
Breaks a monolithic app into smaller independent services.
Each service can scale independently.
Example: Netflix uses microservices for handling different functionalities like streaming, recommendations, and billing. -
CDN (Content Delivery Network):
Distributes static content (images, videos) across global edge servers.
Reduces latency by serving content from the nearest location.
Examples: Cloudflare, AWS CloudFront, Akamai.
| System | Scaling Method |
|---|---|
| Amazon (E-commerce) | CDN + Load Balancing + Sharding |
| Netflix (Streaming) | CDN + Microservices + Auto Scaling |
| Facebook (Social Media) | Load Balancing + Caching + Replication |
| Google Search | Distributed Indexing + Load Balancing |
| Asynchronous Processing + Caching |
-
High CPU Usage (>80%)
Solution: Vertical scaling (increase CPU) or horizontal scaling (add servers). -
High Database Load
Solution: Replication (read replicas) or Sharding. -
Slow Response Time (Latency > 200ms)
Solution: Caching (Redis, CDN) or Load Balancing. -
Increased User Base (Traffic Growth)
Solution: Horizontal Scaling + Auto Scaling (AWS, Kubernetes).
- Automatically adds or removes servers based on real-time traffic.
- Used by AWS Auto Scaling, Google Kubernetes Engine (GKE), Azure Scale Sets.
- Reduces costs by only using extra resources when needed.
| Challenge | Solution |
|---|---|
| Data Consistency | Use strong consistency models (CP databases) or eventual consistency (AP databases). |
| Network Latency | Use caching (Redis), CDNs, and edge computing. |
| Cost Optimization | Use auto-scaling to reduce costs. |
| Concurrency Issues | Implement locking mechanisms and event-driven architectures. |
Scaling is essential for handling growing traffic efficiently. Choosing between vertical vs. horizontal scaling depends on cost, complexity, and fault tolerance needs.
Redundancy in system design refers to having backup resources (hardware, software, data, or network components) to improve fault tolerance, availability, and reliability. It ensures that the system continues to function even if some components fail.
- Prevents single points of failure (SPOF).
- Improves system reliability and availability.
- Ensures data durability in case of failures.
- Minimizes downtime for critical systems (e.g., banking, cloud services).
Using multiple physical components (servers, disks, network devices) to ensure continuous operation.
- Multiple power supplies in a data center to prevent power failure.
- RAID (Redundant Array of Independent Disks) for data storage redundancy.
- Dual network connections (active-passive failover).
| Pros | Cons |
|---|---|
| Prevents hardware failure impact | Expensive |
| Ensures high availability | More maintenance required |
Having multiple copies of the same data to prevent data loss.
- Database Replication → Keep multiple copies of a database across different servers.
- Backup Systems → Regular snapshots stored in remote locations (AWS S3, Google Cloud Storage).
| Pros | Cons |
|---|---|
| Prevents data loss | Increased storage cost |
| Enables quick recovery | Can cause synchronization issues |
Using multiple network paths to ensure connectivity even if one path fails.
- BGP (Border Gateway Protocol) enables traffic rerouting if a path is down.
- Multiple ISPs to prevent internet outage.
- Load balancers with multiple backend servers.
| Pros | Cons |
|---|---|
| Prevents network outages | Higher bandwidth costs |
| Ensures connectivity | Requires network management |
Running multiple instances of the same software to handle failures gracefully.
- Microservices replication → Running multiple instances of a service.
- Hot Standby Servers → A backup server takes over immediately if the primary server fails.
- Kubernetes Auto-Healing → If a container crashes, it is restarted automatically.
| Pros | Cons |
|---|---|
| Prevents software crashes | Requires extra compute power |
| Improves system uptime | Synchronization challenges |
Storing data and services in multiple geographic locations to survive regional failures (e.g., earthquakes, power outages).
- Cloud Providers (AWS, Azure, GCP) replicate data across multiple regions.
- CDNs (Content Delivery Networks) serve data from the closest location to users.
- Multi-region databases (Google Spanner, AWS DynamoDB Global Tables).
| Pros | Cons |
|---|---|
| Survives regional disasters | Expensive to maintain |
| Improves latency | Complex synchronization |
A technique to store data redundantly across multiple disks to improve reliability.
| RAID Level | Description | Pros | Cons |
|---|---|---|---|
| RAID 1 (Mirroring) | Copies data on two disks | High fault tolerance | Doubles storage cost |
| RAID 5 (Striping with Parity) | Distributes data & parity across disks | Efficient storage | Requires at least 3 disks |
| RAID 10 (RAID 1 + RAID 0) | Mirrors & stripes data for speed & redundancy | High performance & redundancy | Expensive |
Keeping multiple copies of a database to ensure availability.
| Type | Description | Example |
|---|---|---|
| Master-Slave Replication | Writes to the master, reads from replicas | MySQL, PostgreSQL |
| Master-Master Replication | Both nodes handle writes & reads | MongoDB, CouchDB |
| Multi-Region Replication | Distributes copies globally | Google Spanner, AWS DynamoDB |
Distributes traffic across multiple servers to prevent overload and failures.
| Load Balancer Type | Description | Example |
|---|---|---|
| DNS Load Balancing | Routes users to the nearest data center | Cloudflare, AWS Route 53 |
| Application Load Balancing | Balances traffic at the HTTP level | AWS ALB, Nginx |
| Network Load Balancing | Distributes TCP/UDP traffic | AWS NLB |
Periodic backups ensure that data can be restored after failure.
| Backup Type | Description | Example |
|---|---|---|
| Full Backup | Copies all data | Daily backups in AWS S3 |
| Incremental Backup | Only backs up changed data | Rsync, Google Drive Sync |
| Snapshot Backup | Creates point-in-time snapshots | AWS EBS Snapshots |
| System | Redundancy Method |
|---|---|
| Google Search | Geo-redundant data centers, load balancing |
| Netflix | Multi-region microservices, CDN caching |
| AWS | Availability zones, database replication |
| Message queues, database replication | |
| Banking Systems | RAID storage, high-availability clusters |
| Challenge | Solution |
|---|---|
| High Cost | Optimize resource allocation, use cloud auto-scaling |
| Synchronization Issues | Implement consistency models (e.g., CAP theorem) |
| Complexity in Management | Use orchestration tools (Kubernetes, Terraform) |
| Data Corruption | Implement checksums and versioning |
Redundancy is critical for designing highly available and fault-tolerant systems. A well-planned redundancy strategy balances cost, complexity, and reliability.
Replication and Redundancy are both strategies used in system design to improve fault tolerance, availability, and reliability. Though they seem similar, they have distinct purposes and implementations.
Replication is the process of making multiple copies of data, services, or components to improve availability, performance, and fault tolerance.
- Ensures real-time or periodic synchronization between copies.
- Improves read performance by allowing multiple nodes to serve requests.
- Commonly used in databases, storage, and distributed systems.
| Type | Description | Example |
|---|---|---|
| Database Replication | Keeps multiple copies of a database synchronized | MySQL Master-Slave, PostgreSQL Replication |
| File Replication | Copies files across multiple storage nodes | Google Drive, Dropbox |
| Service Replication | Runs multiple instances of a service for fault tolerance | Kubernetes, Load Balancing |
| Multi-Region Replication | Distributes data globally for lower latency | AWS DynamoDB Global Tables, Google Spanner |
✔ Improves read performance (read queries can be handled by replicas).
✔ Ensures high availability (failover if a primary node goes down).
✔ Reduces network latency by placing replicas closer to users.
✖ Consistency challenges (data may not be immediately updated across all replicas).
✖ Increased storage and maintenance costs.
✖ Synchronization overhead (ensuring all copies are up to date).
Redundancy refers to having extra (backup) components, resources, or data to prevent failures and ensure reliability.
- Redundant components may or may not be active at all times.
- Used for disaster recovery, fault tolerance, and load balancing.
- Commonly found in hardware, networks, power supplies, and storage systems.
| Type | Description | Example |
|---|---|---|
| Hardware Redundancy | Duplicate hardware components to prevent failures | Dual power supplies, RAID storage |
| Network Redundancy | Multiple network paths to prevent downtime | Dual ISPs, BGP routing |
| Data Redundancy | Extra copies of data for backup and recovery | RAID, Backups |
| Software Redundancy | Running multiple instances of software | Hot Standby Servers, Kubernetes Auto-Healing |
✔ Prevents single points of failure (SPOF).
✔ Ensures high availability and disaster recovery.
✔ Improves fault tolerance (the system keeps running even if a component fails).
✖ Resource wastage (extra components may never be used).
✖ Higher costs (extra hardware, maintenance, and storage).
✖ More complexity in managing redundant systems.
| Feature | Replication | Redundancy |
|---|---|---|
| Definition | Creating multiple copies of data or services | Having extra components or backups for fault tolerance |
| Purpose | Improves performance & availability | Ensures reliability & fault tolerance |
| Usage | Used for load balancing, database scaling, and disaster recovery | Used to prevent failures and ensure business continuity |
| Examples | Database replication, multi-region file storage, microservice instances | RAID storage, dual power supplies, backup data centers |
| Active vs Passive | Replicas are usually active and in use | Redundant systems may be passive (backup) or active |
| Performance Impact | Improves read performance but increases write complexity | No direct performance benefits, but prevents downtime |
| Cost | High (due to storage and synchronization overhead) | Higher (due to unused resources) |
| Scenario | Use Replication | Use Redundancy |
|---|---|---|
| Scaling database reads | ✅ | ❌ |
| Preventing single points of failure | ❌ | ✅ |
| Ensuring quick disaster recovery | ✅ | ✅ |
| Minimizing network downtime | ✅ (multi-region replication) | ✅ (redundant ISPs) |
| Handling hardware failures | ❌ | ✅ (dual power supplies, RAID) |
| Caching frequently accessed data | ✅ (CDN, Redis replication) | ❌ |
| Ensuring 24/7 uptime for critical services | ✅ | ✅ |
| Company | Replication Usage | Redundancy Usage |
|---|---|---|
| Google Drive syncs files across devices | Data centers in multiple locations | |
| Netflix | Microservices are replicated across multiple AWS regions | Uses redundant cloud infrastructure to prevent service downtime |
| Replicates user posts and messages globally | Uses redundant load balancers and failover systems | |
| AWS | DynamoDB Global Tables replicate data across continents | Multiple Availability Zones for redundancy |
- Replication is about making multiple copies of data/services to improve availability and performance.
- Redundancy is about having extra (backup) components to prevent failures and ensure reliability.
In most modern system designs, both replication and redundancy are used together for optimal performance and fault tolerance.
Replication can be broadly categorized into Active Replication and Passive Replication, depending on how requests are processed and how failure recovery is managed.
In Active Replication, all replicas process the same request simultaneously and return responses independently.
- A request is broadcast to all replicas using a coordination protocol (e.g., Atomic Broadcast).
- Each replica executes the request independently.
- Responses are consolidated, and the system ensures consistency.
- ✔ High availability and fault tolerance → No single point of failure.
- ✔ Fast failover → If a replica fails, others continue to operate normally.
- ✔ Strong consistency needed → All replicas must be synchronized.
- Financial transactions (e.g., stock trading, banking).
- Distributed databases (e.g., Google Spanner, Raft consensus).
- Cloud services with high availability (e.g., Kubernetes with multi-master nodes).
| Pros | Cons |
|---|---|
| High availability | More processing overhead |
| No single point of failure | More complex synchronization |
| Faster failover time | Requires strong consistency mechanisms |
In Passive Replication, only one replica (the primary) processes the request, while other replicas stay idle (passive) and update their state based on the primary’s execution.
- The Primary Replica processes all requests.
- After processing, it sends the updated state to all backup replicas.
- If the primary fails, a backup is promoted to become the new primary.
- ✔ Easier to implement → Only one replica handles requests at a time.
- ✔ Less processing overhead → Only the primary executes transactions.
- ✔ Failover takes time → A backup must take over when the primary fails.
- Databases with primary-backup architecture (e.g., PostgreSQL, MySQL Replication).
- Leader-based distributed systems (e.g., Zookeeper, etcd, Raft).
- Cloud storage systems (e.g., AWS RDS primary-read replicas).
| Pros | Cons |
|---|---|
| Lower processing overhead | Failover introduces downtime |
| Easier to implement | Single point of failure in primary |
| Works well for read-heavy workloads | Slower recovery in case of failure |
| Feature | Active Replication | Passive Replication |
|---|---|---|
| Execution | All replicas process the request independently | Only the primary executes; backups receive updates |
| Consistency Model | Requires strong consistency | Eventual consistency is common |
| Failover Speed | Fast failover (since all replicas are active) | Slower failover (backup must take over) |
| Fault Tolerance | High (no single point of failure) | Medium (primary is a single point of failure) |
| Processing Overhead | High (all replicas execute requests) | Low (only primary executes) |
| Example Use Cases | Financial transactions, Kubernetes multi-master | Databases (MySQL, PostgreSQL), AWS RDS |
| Scenario | Use Active Replication | Use Passive Replication |
|---|---|---|
| Low-latency, high-availability systems | ✅ | ❌ |
| Leader-based coordination needed | ❌ | ✅ |
| Write-heavy workloads | ✅ | ❌ |
| Read-heavy workloads | ❌ | ✅ |
| Fast failover required | ✅ | ❌ |
| Lower cost and simpler implementation | ❌ | ✅ |
| System | Active Replication | Passive Replication |
|---|---|---|
| Google Spanner | ✅ | ❌ |
| Kubernetes Multi-Master | ✅ | ❌ |
| PostgreSQL Streaming Replication | ❌ | ✅ |
| AWS RDS (Read Replicas) | ❌ | ✅ |
| Raft Consensus Algorithm | ❌ | ✅ |
| Stock Trading Platforms | ✅ | ❌ |
- Active Replication is used when high availability and fast failover are required, but it has higher processing overhead.
- Passive Replication is used for leader-based systems where failover is rare but needed, and it has lower overhead but higher failover time.
A load balancer is a system that distributes incoming network traffic across multiple servers to ensure no single server is overwhelmed. It helps with:
- ✔ High Availability → Ensures servers don’t crash due to overload.
- ✔ Scalability → Handles more users by spreading traffic efficiently.
- ✔ Fault Tolerance → Redirects traffic if a server fails.
Imagine a restaurant with only one cashier. If 100 people arrive, the single cashier will get overloaded. But if there are 5 cashiers, customers can be distributed among them, reducing waiting time.
A load balancer acts like a restaurant manager, directing customers (requests) to different cashiers (servers).
- A user makes a request (e.g., visiting a website).
- The load balancer receives the request and selects a healthy server.
- It forwards the request to that server.
- The server processes the request and sends back a response.
| Type | Works At | Example |
|---|---|---|
| Network Load Balancer (NLB) | TCP/UDP (Layer 4) | AWS NLB, HAProxy |
| Application Load Balancer (ALB) | HTTP/HTTPS (Layer 7) | AWS ALB, Nginx, Traefik |
| Type | Description | Example |
|---|---|---|
| Hardware Load Balancer | Physical device handling traffic | F5 Big-IP, Citrix NetScaler |
| Software Load Balancer | Software installed on servers | Nginx, HAProxy |
| Cloud Load Balancer | Managed by cloud providers | AWS ELB, Azure LB, GCP Load Balancer |
Load balancers decide which server should handle a request using different strategies:
| Algorithm | How It Works | Best For |
|---|---|---|
| Round Robin | Sends requests to servers in a circular order | Equal workload distribution |
| Least Connections | Sends requests to the server with the fewest active connections | Dynamic workloads (e.g., chat apps) |
| Least Response Time | Chooses the server responding the fastest | Performance-sensitive apps |
| IP Hash | Assigns users to specific servers based on IP address | Session persistence (e.g., shopping carts) |
| Weighted Round Robin | Servers get different weights based on their capacity | Servers with different power levels |
- One load balancer distributes traffic across multiple servers.
- Issue: If the load balancer fails, the entire system goes down.
- Uses two or more load balancers in active-passive or active-active mode.
- Fix: Resolves the single point of failure problem.
- Used by companies like Google, Facebook, Netflix.
- Distributes traffic across multiple data centers worldwide to reduce latency.
- Example: Cloudflare, AWS Route 53, Google Cloud Load Balancer.
- ✔ E-commerce Websites (Amazon, Flipkart) → Handles thousands of users shopping at the same time.
- ✔ Streaming Services (Netflix, YouTube) → Ensures smooth video playback without buffering.
- ✔ Social Media (Facebook, Twitter) → Distributes requests across data centers worldwide.
- ✔ Banking Apps → Prevents downtime during peak transaction periods.
- ✔ Prevents Overloading → Ensures smooth performance.
- ✔ Handles Failures → Automatically reroutes traffic if a server crashes.
- ✔ Improves Speed → Distributes traffic efficiently.
- ✔ Enables Scalability → Allows adding more servers easily.
- ✖ Single Point of Failure (if only one load balancer is used).
- ✖ Extra Cost (especially hardware and cloud-based load balancers).
- ✖ Latency (if not configured properly).
A load balancer is a traffic controller that ensures servers don’t get overloaded.
Different types (NLB, ALB) and algorithms help distribute traffic efficiently.
It is essential for scalable, high-performance web applications.
A streaming service (like Netflix, YouTube, or Spotify) allows users to watch videos or listen to audio without downloading the entire file. Instead, data is sent in small chunks over the internet, allowing playback to start almost instantly.
A streaming service consists of the following major components:
- User Requests a Video/Audio → Sent to a Load Balancer.
- Load Balancer Distributes Traffic → Chooses a nearby Streaming Server.
- Content Delivery Network (CDN) Delivers Chunks → Ensures fast playback.
- Playback Buffering & Adaptive Streaming → Adjusts quality based on internet speed.
- The video is stored in a Storage System (e.g., AWS S3, Google Cloud Storage).
- The video is converted into multiple formats & resolutions (e.g., 144p, 360p, 1080p).
- Why? → Different devices (mobile, PC, TV) need different formats.
- Large services use Distributed Object Storage (e.g., AWS S3, Google Cloud Storage).
- A Metadata Database stores video details (title, duration, owner, etc.).
- If multiple servers exist, a Load Balancer directs the request to the best server.
- Load Balancers use Least Latency, Round Robin, or Geo-location based routing.
- A CDN caches the video close to the user (Edge Servers).
- Example CDNs: Cloudflare, Akamai, AWS CloudFront.
- Why? → Reduces latency & prevents server overload.
- Instead of downloading the full file, the video is sent in small parts (HLS, DASH protocols).
- The video player buffers a few seconds ahead to prevent pauses.
- If the user’s internet is slow, the quality drops to 480p or 360p automatically.
- If the internet improves, it increases to 1080p or 4K.
- Database for Metadata → Stores video details (PostgreSQL, MongoDB, Cassandra).
- Object Storage → Stores actual video files (AWS S3, Google Cloud Storage).
- Transcoding Services → Converts videos to different resolutions (FFmpeg, AWS MediaConvert).
- Content Indexing & Recommendation Engine → Suggests videos based on user behavior.
- CDN & Edge Servers → Cache popular content close to users.
- Load Balancer → Distributes traffic across multiple streaming servers.
- WebSocket & TCP/IP Protocols → Ensures smooth live streaming.
| Component | Technologies Used |
|---|---|
| Storage | AWS S3, Google Cloud Storage, HDFS |
| Database | PostgreSQL, MongoDB, Cassandra |
| Transcoding | FFmpeg, AWS Elemental |
| CDN | Cloudflare, Akamai, AWS CloudFront |
| Load Balancer | Nginx, HAProxy, AWS ELB |
| Streaming Protocols | HLS, DASH, WebRTC |
| Monitoring & Logging | Prometheus, Grafana, ELK Stack |
Live streaming is slightly different from video-on-demand:
- User Sends a Live Video Feed → Streamed via RTMP Protocol to a streaming server.
- Live Encoding → Video is compressed and converted into chunks in real-time.
- CDN Distribution → Live chunks are sent to multiple edge locations.
- Viewer Playback → Users receive low-latency streams based on internet speed.
| Challenge | Solution |
|---|---|
| High Latency | Use CDNs & Adaptive Streaming (HLS/DASH) |
| Scalability | Use Load Balancers & Microservices |
| Storage Costs | Store in Cloud (AWS S3, GCS) & delete old content |
| Data Consistency | Use distributed databases like Cassandra |
| Personalized Recommendations | Use Machine Learning & AI |
| Streaming Service | Key Features |
|---|---|
| Netflix | Uses AWS for storage & CDNs for faster delivery |
| YouTube | Stores billions of videos & transcodes in real-time |
| Twitch | Focuses on low-latency live streaming |
| Spotify | Uses CDNs to reduce buffering in music playback |
- ✔ Streaming services use CDNs, Load Balancers, and Adaptive Bitrate Streaming to deliver content smoothly.
- ✔ Storage, processing, and real-time delivery are key components.
- ✔ Scalability & low-latency are the biggest challenges.
Caching is a technique used to store frequently accessed data in a fast storage layer (like RAM) to reduce the time required to fetch it.
Think of it like a bookmark in a book—instead of flipping through all pages to find a chapter, you jump directly to your bookmark.
✔ Faster Data Access → Reduces database and API response time.
✔ Reduces Load on Servers → Improves scalability and performance.
✔ Handles High Traffic → Prevents database overload.
Caching can be applied at different layers of a system:
| Type of Cache | Where It’s Used | Example |
|---|---|---|
| Database Cache | Stores frequently accessed queries in memory | Redis, Memcached |
| Application Cache | Stores API responses or computed values | In-memory caching, local storage |
| Content Delivery Network (CDN) Cache | Stores static content close to users | Cloudflare, AWS CloudFront |
| Browser Cache | Stores web resources locally on user’s browser | JavaScript, CSS, images |
| Operating System Cache | Speeds up disk and memory operations | Page cache, disk buffer |
- User requests data (e.g., fetching a product from an e-commerce site).
- Cache is checked first → If data is found (cache hit), return it instantly.
- If not found (cache miss), query the database and store the result in the cache.
- Next time, the data is served from the cache instead of the database.
📌 The application first checks the cache. If data is not found, it fetches it from the database and stores it in the cache for future use.
- ✔ Pros → Simple and efficient, updates automatically.
- ❌ Cons → First request is slow (cache miss).
📌 Example:
A user searches for "iPhone 15" on an e-commerce website.
If it's not in cache, fetch from the database and store in Redis.
Next time, it is served instantly from Redis.
📌 Every write operation updates both the database and the cache.
- ✔ Pros → Ensures cache and database are always in sync.
- ❌ Cons → Slower writes due to double writing (cache + DB).
📌 Example:
When a user adds a product review, it's stored in both Redis and MySQL.
This ensures the cache is always updated.
📌 The application writes data only to the cache first, and later syncs it to the database asynchronously.
- ✔ Pros → Super fast writes, reduces DB load.
- ❌ Cons → Risk of data loss if cache crashes before syncing to DB.
📌 Example:
User adds an item to the cart, it's first stored in Redis.
Later, it syncs to MySQL in batches.
📌 The application always queries the cache. If the cache doesn’t have data, it fetches it from the DB and updates the cache.
- ✔ Pros → Automates cache population, no manual writes needed.
- ❌ Cons → Adds extra complexity to the system.
📌 Example:
Netflix recommendations are preloaded in the cache so users get instant results.
Since caches have limited storage, we must remove old data intelligently.
| Policy | How It Works | When to Use? |
|---|---|---|
| Least Recently Used (LRU) | Removes the least accessed item first | General-purpose caching |
| Least Frequently Used (LFU) | Removes items used the least over time | When some data is always needed |
| First In, First Out (FIFO) | Removes the oldest cached item first | Simple, but not very efficient |
| Time-To-Live (TTL) | Data expires after a fixed time (e.g., 10 min) | When data changes frequently |
📌 Example:
Instagram stories expire after 24 hours, so caching them for a few hours makes sense.
Stock prices need frequent updates, so TTL-based caching is used.
✔ Google Search → Frequently searched queries are cached for instant results.
✔ YouTube & Netflix → Popular videos are cached on CDN edge servers.
✔ E-commerce (Amazon, Flipkart) → Product details & prices are stored in Redis for quick access.
✔ Social Media (Facebook, Twitter) → User timelines and notifications are cached for speed.
✔ Stock Market Apps → Real-time stock prices are cached to reduce API load.
| Purpose | Technology Used |
|---|---|
| In-Memory Cache | Redis, Memcached |
| Database Cache | PostgreSQL, MySQL Query Cache |
| CDN Cache | Cloudflare, AWS CloudFront, Akamai |
| Application Cache | Spring Cache, Ehcache, Guava Cache |
| Browser Cache | Service Workers, IndexedDB |
| Problem | Solution |
|---|---|
| Cache Inconsistency | Use Write-Through or Write-Back caching |
| Cache Eviction Issues | Use LRU, LFU, or TTL policies |
| Stale Data in Cache | Use TTL (Time-To-Live) expiration |
| Cache Stampede (Too Many Requests at Once) | Implement locking & request collapsing |
| Security Issues | Encrypt cache data if sensitive |
✔ Caching reduces latency and improves performance.
✔ Different caching strategies (Cache-Aside, Write-Through, etc.) are used based on needs.
✔ Cache eviction policies (LRU, TTL, etc.) help manage storage efficiently.
✔ Used in databases, APIs, web browsers, CDNs, and application layers.
Caching is essential in system design to improve performance, reduce latency, and handle high traffic efficiently. Let’s go deep into different types of caching, applications, advantages, disadvantages, data invalidation strategies, and cache eviction policies.
Caching is a technique used to store frequently accessed data in a fast storage layer (like RAM) so that future requests can be served quickly without querying the primary data source (database, API, etc.).
Imagine a library where students frequently request the same books. Instead of fetching the book from the store every time, the librarian keeps the most requested books on the front desk for quick access. This is caching.
- Description: Stores data directly in RAM for ultra-fast access.
- Use Case: User sessions in web applications.
- Example Technologies: Redis, Memcached
- ❌ Downside: If the system crashes, all data in memory is lost.
- Description: Caches frequently queried database results in memory to reduce DB load.
- Use Case: E-commerce websites caching product details.
- Example Technologies: PostgreSQL Query Cache, MySQL Query Cache, Redis as a DB cache
- ❌ Downside: If the cache is not properly invalidated, stale data may be served.
- Description: Stores static content like images, videos, and HTML files close to users.
- Use Case: Netflix, YouTube storing videos in CDN servers.
- Example Technologies: Cloudflare, AWS CloudFront, Akamai
- ❌ Downside: Difficult to handle real-time updates if content changes frequently.
- Description: Stores computed results or API responses within the application itself.
- Use Case: Caching API responses to avoid redundant API calls.
- Example Technologies: Spring Cache, Ehcache
- ❌ Downside: Requires manual cache invalidation when data changes.
- Description: Stores web assets (CSS, JavaScript, images, etc.) on the user’s device.
- Use Case: Facebook, Twitter storing images, stylesheets, and scripts in the browser cache.
- ❌ Downside: Users may see old content if the cache is not refreshed properly.
- Description: Stores frequently accessed files on disk instead of RAM.
- Use Case: Operating systems caching frequently used programs and files on SSD/HDD.
- ❌ Downside: Slower than RAM but helps in persistent caching.
- Description: Cache is stored across multiple servers in a cluster.
- Use Case: Amazon stores user sessions across multiple data centers for uninterrupted service.
- Example Technologies: Redis Cluster, Amazon ElastiCache
- ❌ Downside: Complex synchronization needed between cache nodes.
- Description: A cache shared across multiple applications or services.
- Use Case: Global authentication services like Google Auth caching user login sessions.
- ❌ Downside: High network latency if the cache is too far from users.
- Web Applications: Reduce database load by caching user sessions, authentication tokens, and frequently accessed data.
- E-commerce: Speed up product searches and recommendations by caching catalog data.
- Social Media: Cache user profiles, friend lists, and timelines.
- Video Streaming: CDN caching helps reduce buffering times.
- Gaming: Cache game assets to minimize load times.
- Reduces latency: Faster response times.
- Decreases server load: Prevents database overload.
- Improves scalability: Handles more users efficiently.
- Enhances user experience: Quick page loads.
- Cache Inconsistency: Stale data may be returned if not properly invalidated.
- Cache Misses: If the data is not in the cache, a database query is needed.
- Storage Limitations: Cache space is finite.
- Security Risks: Sensitive data in cache must be encrypted.
Cache invalidation ensures that stale data is removed from cache when the underlying data changes.
| Strategy | How It Works | Example |
|---|---|---|
| Write-Through | Updates cache immediately when data is written to DB | Updating product stock in cache and DB together |
| Write-Back | Data is first written to cache, then asynchronously updated in DB | Storing user cart items in cache before persisting in DB |
| Write-Around | Writes data directly to DB but skips cache | Reducing unnecessary cache updates for rarely accessed data |
| TTL (Time-to-Live) | Sets expiration time for cache entries | Cache expires after 10 minutes, forcing a fresh database fetch |
Eviction policies manage how old data is removed from the cache:
| Policy | How It Works | Use Case |
|---|---|---|
| Least Recently Used (LRU) | Removes the least accessed item first | General-purpose caching (Redis default) |
| Least Frequently Used (LFU) | Removes items used the least over time | Long-term caching needs |
| First In, First Out (FIFO) | Removes the oldest cached item first | Queued processing systems |
| TTL (Time-to-Live) | Data expires after a set time | Stock prices, real-time data |
- Netflix uses LRU caching to keep popular movies in memory while removing least-watched ones.
- Reduces response time, improves scalability, and enhances user experience.
- Different caching techniques (In-Memory, CDN, Distributed, etc.) serve different use cases.
- Proper invalidation and eviction policies prevent stale data issues.
- Caching is used in databases, APIs, browsers, CDNs, applications, and storage systems.
Cache expiration is essential for maintaining data freshness and preventing stale data from being served to users. Two primary strategies for cache expiration are Time-Driven Expirations and Event-Driven Expirations.
Time-driven expiration is based on a fixed time duration. The cache entry is removed after a specific period, regardless of whether the data is still relevant.
- Each cache entry is assigned a Time-to-Live (TTL) value.
- When the TTL expires, the cache entry is automatically deleted.
- Used when data changes periodically but not frequently.
- ✔ User session caching → A user’s session expires after 30 minutes of inactivity.
- ✔ Stock market prices → Prices expire every 10 seconds to refresh market data.
- ✔ Weather updates → Cached weather reports expire every 15 minutes to fetch new data.
- ✔ DNS caching → Domain name lookups have a TTL to refresh IP addresses.
- ✅ Easy to implement → Just set a TTL and let the cache manage expiration.
- ✅ Prevents memory overflow → Old data gets removed automatically.
- ✅ Efficient for predictable expiration times.
- ❌ Might remove useful data if it is still valid beyond its TTL.
- ❌ Can lead to cache misses if too many entries expire at once.
- ❌ Not suitable for real-time updates when data changes unpredictably.
Event-driven expiration happens when a specific event triggers the cache invalidation, instead of relying on a fixed time.
- When data changes in the database or another source, the cache is updated or cleared.
- Used when data changes unpredictably.
- Data update in DB → Invalidate the corresponding cache entry.
- User logout → Invalidate session cache immediately.
- ✔ E-commerce product inventory → If stock count changes, immediately invalidate product cache.
- ✔ Social media posts → When a user edits a post, the cache entry is refreshed.
- ✔ Real-time leaderboards → Scores update after each match, not on a timer.
- ✔ User profile updates → When a user changes their email, their cached profile is updated instantly.
- ✅ Ensures data consistency → No risk of stale data.
- ✅ More efficient than time-based expiration when data updates are unpredictable.
- ✅ Works well for real-time systems like gaming leaderboards, social media feeds.
- ❌ More complex to implement → Requires event listeners and cache synchronization.
- ❌ If the event system fails, stale data might persist.
- ❌ Increased processing overhead when many cache entries need to be invalidated quickly.
| Feature | Time-Driven Expiration | Event-Driven Expiration |
|---|---|---|
| Trigger | Fixed time (TTL) | Data updates/events |
| Best for | Periodic data refreshes | Real-time updates |
| Use case | Weather data, session caching | Product stock, user profiles |
| Implementation | Simple (just set TTL) | Complex (needs event handling) |
| Risk | Stale data before TTL expires | Event system failures |
Many systems use both strategies together for better efficiency.
- Twitter timelines → Cached for 10 minutes (TTL) but updated instantly when a user posts a new tweet (event-driven).
- E-commerce inventory → Cached for 1 hour, but updated immediately when stock changes.
- Time-driven expiration is simpler but may serve stale data.
- Event-driven expiration ensures fresh data but is harder to implement.
- A hybrid approach is often the best choice in large-scale systems.