Building Buraq: A Distributed Task Queue Processing 12K+ Tasks/Sec

The Challenge

At BitSic Product Studio, we needed a task queue that could:

1. Process 10,000+ tasks per second
2. Maintain sub-10ms latency
3. Guarantee zero data loss
4. Scale horizontally

Existing solutions (Celery, Bull, Sidekiq) couldn't meet our performance requirements. So we built Buraq.

Architecture

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Producer  │────▶│   Gateway   │────▶│    Redis    │
└─────────────┘     └─────────────┘     └─────────────┘
                                                │
                                                ▼
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Result    │◀────│   Workers   │◀────│    Stream   │
│   Handler   │     │  (N Pods)   │     │   Groups    │
└─────────────┘     └─────────────┘     └─────────────┘

Core Components

1. Gateway Service

• Validates and enqueues tasks
• Handles rate limiting
• Implements backpressure

2. Redis Backend

• Redis Streams for task queuing
• Consumer groups for parallel processing
• Persistent storage for durability

3. Worker Pods

• Auto-scaling based on queue depth
• Graceful shutdown handling
• Dead letter queue for failures

Performance Optimizations

1. Binary Protocol

Instead of JSON, we use a custom binary protocol over gRPC:

type Task struct {
    ID      uint64
    Type    uint8
    Payload []byte
}

This reduced serialization overhead by ~70%.

2. Connection Pooling

pool := &redis.Pool{
    MaxIdle:     100,
    MaxActive:   500,
    IdleTimeout: 240 * time.Second,
    Dial: func() (redis.Conn, error) {
        return redis.Dial("tcp", redisAddr)
    },
}

3. Worker Preallocation

Workers maintain persistent Redis connections and use pipelining:

pipe, _ := conn.Send("XREADGROUP", ...)
pipe.Send("XACK", ...)
conn.Flush()

Results

• Throughput: 12,000+ tasks/second
• Latency: P50 < 3ms, P99 < 5ms
• Reliability: Zero data loss in 24 months
• Scale: Powers $50M+/year in payment processing

Lessons Learned

1. Redis Streams are underrated - Perfect for high-throughput queues
2. Protocol matters - Binary serialization pays off at scale
3. Connection pooling is critical - Redis connection overhead is real
4. Horizontal scaling is essential - Kubernetes HPA handles traffic spikes

Buraq demonstrates that with careful architecture, Go can handle massive workloads that typically require more complex systems.