Redpanda

Retaining historical streaming logs on local NVMe drives turns storage costs into a linear tax on throughput. On cloud instances like AWS i4i.2xlarge (featuring NVMe drives), storage costs roughly $0.16 to $0.24 per GB-month when factored into instance pricing. Storing 50 TB of event history locally costs roughly $10,000 per month just for disk footprint, while also forcing massive broker rebalancing delays when a node fails and its local partition replicas must be re-replicated over the network.

Redpanda addresses this with Tiered Storage (Shadow Indexing). It decouples storage from compute by offloading immutable log segments to object stores like AWS S3 or Google Cloud Storage (GCS). Object storage costs $0.023 per GB-month for standard access and under $0.0125 for infrequent access. Tiered storage slashes storage overhead by up to 90% while turning node recovery into an $O(1)$ metadata swap rather than an $O(N)$ disk hydration process.

Architecture: Local NVMe vs. Object Storage Path

Redpanda is written in C++ using the Seastar asynchronous framework, utilizing a thread-per-core model. Each CPU core controls a subset of partitions (shards) and manages its own memory and disk I/O without lock contention.

When Tiered Storage is enabled:

1. Hot Path (Writes & Tailing Readers): incoming writes hit the active segment on local NVMe. Consumers reading real-time data fetch directly from local NVMe cache or memory buffers with sub-millisecond p99 latencies.
2. Segment Sealing & Upload: Once a log segment reaches a size threshold (e.g., 128 MB) or age limit, it is sealed. A background worker thread per core uploads the sealed segment along with its index file to AWS S3.
3. Local Truncation: Once uploaded and verified, the segment is eligible for local deletion based on retention.local.target.bytes or retention.local.target.ms.
4. Cold Path (Historical Readers): When a batch consumer requests offsets no longer present on NVMe, Redpanda intercepts the fetch request, streams the required segment chunk from S3 into a controlled local cache, serves the reader, and managed cache retention policies kick in.

+-----------------------------------------------------------------------+
|                          REDPANDA BROKER                              |
|                                                                       |
|  +------------+   Write    +-------------------+                      |
|  | Producer   | ---------> | Active Segment    |                      |
|  +------------+            | (Local NVMe)      |                      |
|                            +-------------------+                      |
|                                      |                                |
|  +------------+   Read (0ms)         v Seal                           |
|  | Hot        | <--------- +-------------------+                      |
|  | Consumer   |            | Closed Segment    |                      |
|  +------------+            +-------------------+                      |
|                                      |                                |
|                                      | Async Upload                   |
|                                      v                                |
|                            +-------------------+                      |
|                            | Cloud Storage     |                      |
|                            | Background Worker |                      |
|                            +-------------------+                      |
+--------------------------------------|--------------------------------+
                                       |
                                       v HTTP PUT
                            +---------------------+
                            | AWS S3 / GCS        |
                            | Object Storage      |
                            +---------------------+
                                       |
  +------------+   Hydrate (30-100ms)  | HTTP GET
  | Historical | <---------------------+
  | Consumer   |
  +------------+

System Trade-Off Matrix

Production Configuration

To deploy Tiered Storage, set global storage credentials via redpanda.yaml or cluster configuration settings using rpk.

Cluster-Wide Configuration (`redpanda.yaml`)

redpanda:
  data_directory: /var/lib/redpanda/data
  node_id: 1
  seed_servers:
    - host:
        address: 10.0.1.10
        port: 33145

  # Enable Cloud Storage Infrastructure
  cloud_storage_enabled: true
  cloud_storage_region: "us-east-1"
  cloud_storage_bucket: "prod-redpanda-tiered-storage"
  cloud_storage_credentials_source: "config_file"
  cloud_storage_access_key: "AKIAXXXXXXXXXXXXXXXX"
  cloud_storage_secret_key: "WjalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

  # Connection tuning per CPU core
  cloud_storage_max_connections: 20
  cloud_storage_segment_max_bytes: 134217728 # 128 MB

  # Cache controls for cold historical fetches
  cloud_storage_cache_directory: /var/lib/redpanda/data_cache
  cloud_storage_cache_size_bytes: 107374182400 # 100 GB local NVMe cache

Applying Topic-Level Tiering Policies

Configure tiered storage parameters dynamically per topic using rpk topic alter-config:

# Enable Shadow Indexing on an existing high-throughput topic
rpk topic alter-config telemetry-events \
  --set redpanda.remote.write=true \
  --set redpanda.remote.read=true \
  --set retention.bytes=5497558138880 \ # 5 TB total retention (S3 + NVMe)
  --set retention.local.target.bytes=107374182400 # Keep only 100 GB locally on NVMe

Parameter Breakdown

• redpanda.remote.write=true: Enables background offloading of closed local segments to S3.
• redpanda.remote.read=true: Directs consumer read requests for truncated local offsets to fetch from S3 transparently.
• retention.local.target.bytes: Soft limit for local NVMe partition capacity. Redpanda removes local segments that have been uploaded to S3 once local utilization passes this limit.

Production Failure Modes & Operational Diagnostics

1. AWS S3 Rate Limiting (`503 Slow Down`)

• Root Cause: S3 imposes limits of 3,500 PUT/POST/DELETE and 5,500 GET requests per second per prefix. If hundreds of shards write to or read from a single prefix (e.g., s3://bucket/topic/partition/), S3 throttles traffic, causing thread backpressure in Redpanda.
• Symptom: Redpanda logs show cloud_storage - S3 API error: 503 Slow Down. Local disk retention targets are violated because background segment uploads stall.
• Fix: Enable hashed prefixes in cluster configs to distribute partition uploads across multiple S3 prefix hash rings:

rpk cluster config set cloud_storage_enable_segmented_prefix_upload true

2. Disk Exhaustion due to Upload Lag

• Root Cause: Network throughput to S3 is lower than incoming write throughput, or cloud_storage_max_connections is set too low. Local NVMe fills up with closed segments faster than they can be uploaded and cleared.
• Symptom: Local NVMe disk usage approaches 90% space, risking broker write stalls.
• Fix:
  1. Increase the maximum outgoing connection pool size per core:

     rpk cluster config set cloud_storage_max_connections 50
     

  2. Increase network link capacity or adjust cloud_storage_segment_max_bytes to 64MB (67108864) to create smaller, faster unit transfers if instance egress is constrained.

3. Historical Consumer Hydration Storms

• Root Cause: Multiple analytical consumers (e.g., Apache Flink, Trino, Spark) issue large historical reads simultaneously. The cluster attempts to stream gigabytes of historical data into cloud_storage_cache_directory, driving up local disk I/O and thrashing the cache.
• Symptom: NVMe disk read latency spikes on hot paths. Cold read request latencies blow past 2,000 ms. Cache miss rate approaches 100%.
• Fix:
  1. Enforce strict consumer group fetch byte limits to throttle concurrent hydration requests.
  2. Isolate analytical read traffic to dedicated read-replica clusters