1.1 The Eight Fallacies of Distributed Computing#

In the 1990s, L. Peter Deutsch and others at Sun Microsystems compiled a list of fallacies; assumptions that programmers new to distributed applications invariably make, to their peril.

CAUTION

1. The network is reliable. (It is not.)
2. Latency is zero. (It is not.)
3. Bandwidth is infinite. (It is not.)
4. The network is secure. (It is not.)
5. Topology doesn’t change. (It does.)
6. There is one administrator. (There are many.)
7. Transport cost is zero. (It is not.)
8. The network is homogeneous. (It is not.)

Every pattern, protocol, and architecture discussed in this volume is, in some way, a strategy for mitigating the consequences of these fallacies.

1.2 A Roadmap for Advanced Backend Engineering#

Building on our foundation, we explore the advanced topics that define modern systems architecture:

• Section 2: Advanced Data Management and Consistency
• Section 3: Resilient System Design Patterns
• Section 4: Advanced Asynchronous Communication
• Section 5: Performance Engineering at Scale
• Section 6: Advanced API and Security Architectures

2.1 The Consistency Spectrum and the PACELC Theorem#

The CAP theorem describes behavior during network partitions, but the PACELC theorem provides a more complete picture:

NOTE

“If there is a Partition (P), a distributed system must choose between Availability (A) and Consistency (C). Else (E), when the system is running normally, it must choose between Latency (L) and Consistency (C).”

This forces nuanced architectural discussion. A system might sacrifice consistency for availability during failures but prioritize consistency over latency during normal operation.

2.2 Distributed Transactions: The Saga Pattern#

Two-Phase Commit is synchronous and unsuitable for microservices. The Saga Pattern manages data consistency across services through local transactions and compensating actions.

TIP

1. Order Service: Creates order in PENDING state, publishes ORDER_CREATED event
2. Payment Service: Processes payment, publishes PAYMENT_PROCESSED on success
3. Inventory Service: Updates stock, publishes INVENTORY_UPDATED on success
4. Order Service: Updates order to CONFIRMED

Failure Handling: If inventory fails, Payment Service compensates with refund, Order Service cancels.

Implementation Styles:

NOTE

• Choreography: Services publish/subscribe to events with no central coordinator
• Orchestration: Central orchestrator manages saga state and compensating transactions

2.3 Event Sourcing and CQRS#

These patterns build highly scalable and auditable systems.

• Event Sourcing: Store immutable events instead of current state. Current state is derived by replaying events.

NOTE

1
// Instead of storing balance: 80
2
// Store event sequence:
3
[
4
  {"type": "AccountCreated", "initialBalance": 0},
5
  {"type": "DepositMade", "amount": 100},
6
  {"type": "WithdrawalMade", "amount": 20}
7
]
8
// Current balance = replay events

• CQRS (Command Query Responsibility Segregation): Separates the write model from the read model.

TIP

• Different models optimized for writes vs reads
• Independent scaling of command and query sides
• Better domain modeling with separate contexts

2.4 Database Internals for the Backend Engineer#

Understanding storage engines and replication strategies is crucial for performance and reliability.

NOTE

• InnoDB: Transactional, ACID-compliant, row-level locking for OLTP workloads
• MyISAM: Fast for reads, table-level locking, no transactions (deprecated for new apps)

Replication Strategies:

TIP

• Leader-Follower: All writes to leader, reads from replicas (most common)
• Multi-Leader: Multiple nodes accept writes, replicate conflicts must be resolved
• Leaderless (Cassandra style): Writes to multiple nodes simultaneously, quorum reads

Transaction Isolation Levels (SQL):

NOTE

1. Read Uncommitted: Can read uncommitted changes (dirty reads)
2. Read Committed: Only reads committed changes (non-repeatable reads possible)
3. Repeatable Read: Row value consistent within transaction (phantom reads possible)
4. Serializable: Full serial execution (highest consistency, lowest performance)

3.1 The Circuit Breaker Pattern (In-Depth)#

The Circuit Breaker monitors failures and prevents cascading failures in distributed systems.

NOTE

• Closed: Normal operation, requests flow through, monitoring failures
• Open: Fail fast for downstream issues, timeout before retrying
• Half-Open: Test downstream recovery with single probe request

3.2 The Bulkhead Pattern#

Isolate application components into pools to prevent single failures from affecting the entire system.

TIP

Use separate thread/connection pools for each downstream service. A slow Service A won’t impact Service B’s pool, preventing complete system failure.

3.3 The Retry and Timeout Patterns#

Essential for handling transient failures in distributed systems.

CAUTION

• Timeouts: Aggressive timeouts prevent resource exhaustion
• Exponential Backoff: Increase retry intervals (1s, 2s, 4s, 8s)
• Jitter: Add randomness to prevent thundering herd problems

3.4 Rate Limiting and Load Shedding#

Protect services from overload and implement graceful degradation.

NOTE

• Token Bucket: Tokens accumulate for requests, removed on use
• Leaky Bucket: Requests processed at fixed rate, excess discarded
• Load Shedding: Reject low-priority requests under extreme load

4.1 Message Brokers vs. Event Logs#

Different approaches to messaging with distinct trade-offs.

TIP

• RabbitMQ: Smart routing, work queuing, message broker model
• Apache Kafka: Event streaming, persistent logs, multiple consumers

4.2 Idempotent Consumers#

Critical for handling “at-least-once” delivery in messaging systems.

NOTE

1
function processMessage(message) {
2
  if (processedMessages.contains(message.id)) {
3
    return; // Skip duplicate
4
  }
5

6
  // Process message
7
  processBusinessLogic(message);
8

9
  // Track as processed (atomic with business logic)
10
  processedMessages.add(message.id);
11
}

4.3 The Transactional Outbox Pattern#

Solve atomic database updates and event publishing in event-driven systems.

TIP

1. Update business entity AND insert event into outbox in single local transaction
2. Message relay asynchronously publishes events and marks as sent
3. Guarantees atomicity without distributed transactions
4. Provides “at-least-once” delivery semantics

5.1 Caching Patterns (In-Depth)#

Advanced caching strategies beyond basic cache-aside.

NOTE

• Cache-Aside: App code manages cache, lazy loading
• Read-Through: Cache handles data loading from DB
• Write-Through: Cache updates synchronously update DB
• Write-Back: Cache updates asynchronously flush to DB

Thundering Herd Mitigation:

CAUTION

When cached item expires, thousands of requests simultaneously miss cache and overwhelm DB. Solution: lock-based re-fetching where only first request loads data while others wait.

5.2 Concurrency vs. Parallelism#

Fundamental concepts for performance optimization.

TIP

• I/O-Bound Workloads: Asynchronous models (Node.js, asyncio) handle many concurrent requests
• CPU-Bound Workloads: Parallelism (Go, Java) leverages multiple cores

5.3 Profiling and Performance Tuning#

You cannot optimize what you cannot measure.

NOTE

Use profilers to generate flame graphs identifying:

• CPU hotspots in code execution paths
• Memory allocation patterns and leaks
• I/O bottlenecks and waiting time

6.1 The API Gateway Pattern#

Single entry point managing communication between clients and services.

TIP

• Routing: Direct requests to appropriate microservices
• Authentication/Authorization: Verify credentials at the edge
• Rate Limiting: Enforce usage policies and throttling
• Request Transformation: Adapt requests for downstream services
• Observability: Centralized logging and monitoring

6.2 Service Mesh#

Infrastructure layer for secure, fast, and reliable service-to-service communication.

NOTE

• Sidecar Proxy: (Envoy) handles all inbound/outbound traffic per service
• Control Plane: (Istio, Linkerd) configures all sidecar proxies
• Features: mTLS, traffic management, distributed tracing, observability

6.3 Zero Trust Security#

“Never trust, always verify” security model for distributed systems.

CAUTION

• Identity-Based Authentication: Verify every request regardless of origin
• Least Privilege Access: Grant minimum necessary permissions
• Assume Breach: Design expecting internal compromise

6.4 JWTs (In-Depth): Risks and Mitigation#

Understanding JWT vulnerabilities and secure implementation.

CAUTION

• Algorithm Confusion Attacks: Trick server into weak algorithms
  • Mitigation: Configure library to accept only strong algorithms (RS256)
• Token Revocation: Stateless tokens can’t be invalidated
  • Mitigation: Maintain revocation denylist in fast cache