Scaling Voice Analysis Systems: Your Complete Production Scale Guide

TL;DR: Scaling voice analysis from prototype to production requires strategic architecture decisions across compute, storage, and data layers. This guide covers horizontal scaling patterns, load balancing strategies (round-robin, least-connection, compute-aware), distributed job queues (Redis, RabbitMQ, SQS), caching for feature vectors and model outputs, database sharding for time-series voice data, CDN integration for audio delivery, auto-scaling policies, cost optimization ($0.001-0.10 per analysis), monitoring & observability, and production architecture blueprints for 10-10,000 requests/second. By the end, you'll know how to build voice analysis infrastructure that scales economically while maintaining <2s latency.

---

The Scaling Challenge: Why Voice Analysis is Different

Unique constraints:

• Large payloads: Audio files are 1-10 MB (100-1000× larger than typical API requests)
• CPU-intensive: ML inference uses 100-1000× more compute than CRUD operations
• Memory-hungry: Models require 2-8 GB RAM per worker process
• Time-series data: Voice features grow unbounded (users record weekly → millions of sessions)
• Bursty traffic: Usage spikes during business hours (3-5× baseline)

Scaling dimensions:

Layer	Bottleneck	Scaling Strategy
API Gateway	Request handling (1,000-10,000 req/s)	Horizontal scaling + load balancer
Audio Upload	Network bandwidth (10-100 Gbps)	Direct-to-S3 uploads (presigned URLs)
Job Queue	Message throughput (10,000-100,000 msg/s)	Redis Streams / SQS with partitioning
ML Workers	GPU/CPU compute (1-10 req/s per worker)	Auto-scaling worker pool (10-100 instances)
Database	Write throughput (1,000-10,000 writes/s)	Time-series partitioning + read replicas
Storage	IOPS (100,000-1M operations/s)	Object storage (S3) + CDN

---

Architecture Evolution: From Monolith to Microservices

Stage 1: Monolith (0-100 req/day)

Architecture: Single server running everything

┌─────────────────────────────────────┐
│         Single Server (AWS EC2)     │
│                                     │
│  ┌──────────────────────────────┐  │
│  │  FastAPI (Python)            │  │
│  │  - Upload endpoint           │  │
│  │  - Synchronous analysis      │  │
│  │  - ML inference (in-process) │  │
│  └──────────────────────────────┘  │
│                                     │
│  ┌──────────────────────────────┐  │
│  │  PostgreSQL (local)          │  │
│  │  - Stores features + results │  │
│  └──────────────────────────────┘  │
│                                     │
│  ┌──────────────────────────────┐  │
│  │  Local Disk                  │  │
│  │  - Temporary audio storage   │  │
│  └──────────────────────────────┘  │
└─────────────────────────────────────┘

Cost: $50-100/month
Capacity: ~100 requests/day (3-4/hour)
Latency: 10-30 seconds per request

Limitations:

• Single point of failure (server crash = complete outage)
• Can't handle concurrent requests (synchronous processing)
• Memory limits (8 GB RAM = 2-3 concurrent ML models max)

Stage 2: Async Workers (100-1,000 req/day)

Architecture: Separate API and worker processes

┌──────────────────┐      ┌─────────────────┐      ┌──────────────────┐
│   API Server     │      │   Redis Queue   │      │   ML Workers     │
│   (FastAPI)      │─────▶│                 │─────▶│   (3x Python)    │
│   - Upload       │      │   Job Queue     │      │   - Feature      │
│   - Queue jobs   │      │   (FIFO)        │      │     extraction   │
│                  │      │                 │      │   - ML inference │
└──────────────────┘      └─────────────────┘      └──────────────────┘
         │                                                    │
         │                                                    │
         ▼                                                    ▼
┌──────────────────────────────────────────────────────────────────────┐
│                        PostgreSQL                                    │
│                   (RDS, single instance)                             │
└──────────────────────────────────────────────────────────────────────┘
         ▲
         │
┌──────────────────┐
│   S3 Storage     │
│   (audio files)  │
└──────────────────┘

Cost: $200-400/month
Capacity: ~1,000 requests/day (40/hour)
Latency: 5-15 seconds per request
Concurrency: 3-10 simultaneous analyses

Improvements:

• Non-blocking API (returns immediately, processes async)
• Horizontal scaling of workers (add more when busy)
• Persistent storage (S3 = 99.999999999% durability)

Stage 3: Microservices (1,000-10,000 req/day)

Architecture: Specialized services with auto-scaling

                    ┌─────────────────────┐
                    │   Load Balancer     │
                    │   (ALB / Nginx)     │
                    └──────────┬──────────┘
                               │
              ┌────────────────┼────────────────┐
              │                │                │
         ┌────▼────┐      ┌───▼────┐      ┌───▼────┐
         │ API #1  │      │ API #2 │      │ API #3 │
         │(FastAPI)│      │(FastAPI)│      │(FastAPI)│
         └────┬────┘      └───┬────┘      └───┬────┘
              │               │               │
              └───────┬───────┴───────┬───────┘
                      │               │
            ┌─────────▼──────┐  ┌────▼────────────┐
            │  Redis Cluster │  │  S3 / CloudFront│
            │  (job queue +  │  │  (audio + CDN)  │
            │   caching)     │  └─────────────────┘
            └────────┬───────┘
                     │
        ┌────────────┼────────────┐
        │            │            │
   ┌────▼─────┐ ┌───▼──────┐ ┌──▼───────┐
   │ Worker #1│ │ Worker #2│ │ Worker #N│
   │ (Python) │ │ (Python) │ │ (Python) │
   │ - STT    │ │ - STT    │ │ - STT    │
   │ - Features│ │ - Features│ │ - Features│
   │ - ML     │ │ - ML     │ │ - ML     │
   └────┬─────┘ └───┬──────┘ └──┬───────┘
        │           │           │
        └───────────┼───────────┘
                    │
         ┌──────────▼──────────┐
         │  PostgreSQL Cluster │
         │  - Primary (writes) │
         │  - Replica (reads)  │
         │  - Partitioned by   │
         │    timestamp        │
         └─────────────────────┘

Cost: $800-2,000/month
Capacity: ~10,000 requests/day (400/hour, 7/min)
Latency: <2 seconds (cached), 3-8 seconds (cold)
Concurrency: 10-50 simultaneous analyses
Availability: 99.9% (multi-AZ deployment)

Stage 4: Enterprise Scale (10,000+ req/day)

Architecture: Multi-region, geo-distributed

┌─────────────────────────────────────────────────────────────┐
│                   Global CDN (CloudFront)                   │
│             (audio delivery + static assets)                │
└────────────────────────┬────────────────────────────────────┘
                         │
        ┌────────────────┼────────────────┐
        │                │                │
┌───────▼──────┐  ┌──────▼──────┐  ┌─────▼──────┐
│  US-EAST-1   │  │  EU-WEST-1  │  │ AP-SOUTH-1 │
│              │  │             │  │            │
│ ┌──────────┐ │  │ ┌─────────┐ │  │ ┌────────┐ │
│ │ API (×10)│ │  │ │ API (×5)│ │  │ │API (×3)│ │
│ └────┬─────┘ │  │ └────┬────┘ │  │ └───┬────┘ │
│      │       │  │      │      │  │     │      │
│ ┌────▼─────┐ │  │ ┌────▼────┐ │  │ ┌───▼────┐ │
│ │Workers   │ │  │ │ Workers │ │  │ │Workers │ │
│ │ (×50)    │ │  │ │  (×20)  │ │  │ │ (×10)  │ │
│ └────┬─────┘ │  │ └────┬────┘ │  │ └───┬────┘ │
│      │       │  │      │      │  │     │      │
│ ┌────▼─────┐ │  │ ┌────▼────┐ │  │ ┌───▼────┐ │
│ │PostgreSQL│ │  │ │PostgreSQL│ │  │ │PostgreSQL│
│ │  Primary │ │  │ │ Replica │ │  │ │ Replica│ │
│ └──────────┘ │  │ └─────────┘ │  │ └────────┘ │
└──────────────┘  └─────────────┘  └────────────┘

Cost: $5,000-20,000/month
Capacity: 100,000+ requests/day (4,000/hour, 70/min, >1/sec)
Latency: <500ms (geo-routed), 1-3 seconds (analysis)
Concurrency: 100-500 simultaneous analyses
Availability: 99.99% (multi-region failover)

---

Load Balancing Strategies for ML Workloads

1. Round-Robin (Naive)

Algorithm: Distribute requests equally across workers

# Nginx configuration
upstream ml_workers {
    server worker1:8000;
    server worker2:8000;
    server worker3:8000;
}

server {
    listen 80;
    location /analyze {
        proxy_pass http://ml_workers;
    }
}

Problem: Doesn't account for worker load (some requests take 5s, others 30s)

Result: Worker 1 might be analyzing 60-second audio while Worker 2 is idle

2. Least Connections (Better)

Algorithm: Send request to worker with fewest active connections

upstream ml_workers {
    least_conn;  # Use least-connection algorithm
    server worker1:8000;
    server worker2:8000;
    server worker3:8000;
}

Improvement: Accounts for concurrent load, but not processing time

3. Compute-Aware Load Balancing (Best)

Algorithm: Weight workers by available compute capacity

import redis
import time

redis_client = redis.Redis()

def get_best_worker():
    """
    Select worker with lowest compute utilization.

    Returns:
        str: Worker URL with most available capacity
    """
    workers = ['worker1:8000', 'worker2:8000', 'worker3:8000']

    # Get current compute load for each worker
    loads = {}
    for worker in workers:
        # Each worker reports: (active_jobs, total_compute_seconds_queued)
        active_jobs = int(redis_client.get(f'worker:{worker}:active_jobs') or 0)
        queued_seconds = float(redis_client.get(f'worker:{worker}:queued_seconds') or 0)

        # Compute load score (lower = better)
        # Factor in both number of jobs and total compute time
        loads[worker] = active_jobs * 10 + queued_seconds

    # Select worker with lowest load
    best_worker = min(loads, key=loads.get)
    return f'http://{best_worker}'

# Usage in API gateway
@app.post("/api/voice/analyze")
async def analyze(audio: UploadFile):
    # Select best worker based on current load
    worker_url = get_best_worker()

    # Forward request to selected worker
    response = requests.post(f'{worker_url}/analyze', files={'audio': audio})
    return response.json()

Workers report their load:

# Worker process
class VoiceAnalysisWorker:
    def __init__(self, worker_id):
        self.worker_id = worker_id
        self.redis = redis.Redis()
        self.active_jobs = 0
        self.queued_seconds = 0.0

    async def analyze_audio(self, audio_path):
        # Get audio duration
        audio_duration = get_audio_duration(audio_path)

        # Update load metrics
        self.active_jobs += 1
        self.queued_seconds += audio_duration * 2  # Analysis takes ~2× audio duration
        self.update_redis_metrics()

        try:
            # Perform analysis
            result = await self.run_analysis(audio_path)
            return result
        finally:
            # Update metrics after completion
            self.active_jobs -= 1
            self.queued_seconds -= audio_duration * 2
            self.update_redis_metrics()

    def update_redis_metrics(self):
        """Report current load to Redis for load balancer."""
        self.redis.set(f'worker:{self.worker_id}:active_jobs', self.active_jobs)
        self.redis.set(f'worker:{self.worker_id}:queued_seconds', self.queued_seconds)
        self.redis.expire(f'worker:{self.worker_id}:active_jobs', 60)  # TTL for failure detection

Performance comparison:

Strategy	Avg Latency	p95 Latency	Worker Utilization
Round-robin	12.3s	28.5s	60% (uneven)
Least connections	8.7s	18.2s	75%
Compute-aware	5.4s	11.8s	88% (even)

---

Distributed Job Queue: Managing Analysis Pipeline

Option 1: Redis Streams (Simple, Fast)

Architecture:

┌──────────────┐     ┌─────────────────┐     ┌──────────────┐
│  API Server  │────▶│  Redis Stream   │────▶│  Worker #1   │
│              │     │  (job queue)    │     │              │
│              │     │                 │────▶│  Worker #2   │
│              │     │                 │     │              │
│              │     │                 │────▶│  Worker #N   │
└──────────────┘     └─────────────────┘     └──────────────┘

Producer (API server):

import redis

redis_client = redis.Redis()

def queue_analysis_job(audio_id, user_id, analysis_types):
    """Add analysis job to Redis stream."""
    job_data = {
        'audio_id': audio_id,
        'user_id': user_id,
        'analysis_types': ','.join(analysis_types),
        'queued_at': time.time()
    }

    # Add to stream (atomic operation)
    job_id = redis_client.xadd(
        'analysis_jobs',  # Stream name
        job_data,
        maxlen=10000  # Limit stream size (keep last 10K jobs)
    )

    return job_id.decode('utf-8')  # e.g., "1705334842000-0"

# Usage
job_id = queue_analysis_job('aud_xyz', 'user_123', ['age', 'emotion'])

Consumer (worker):

def process_jobs(worker_id, batch_size=10):
    """Worker process: consume jobs from Redis stream."""
    # Create consumer group (once, idempotent)
    try:
        redis_client.xgroup_create('analysis_jobs', 'workers', id='0', mkstream=True)
    except redis.exceptions.ResponseError:
        pass  # Group already exists

    print(f"Worker {worker_id} starting...")

    while True:
        # Read batch of jobs (blocks until available)
        jobs = redis_client.xreadgroup(
            groupname='workers',
            consumername=worker_id,
            streams={'analysis_jobs': '>'},  # '>' = unprocessed messages
            count=batch_size,
            block=5000  # Wait up to 5 seconds for new jobs
        )

        if not jobs:
            continue  # No jobs available

        # Process each job
        for stream_name, messages in jobs:
            for job_id, job_data in messages:
                try:
                    # Decode job data
                    audio_id = job_data[b'audio_id'].decode('utf-8')
                    user_id = job_data[b'user_id'].decode('utf-8')
                    analysis_types = job_data[b'analysis_types'].decode('utf-8').split(',')

                    print(f"Processing job {job_id}: audio={audio_id}")

                    # Run analysis
                    result = analyze_audio(audio_id, analysis_types)

                    # Store results in database
                    store_results(user_id, audio_id, result)

                    # Acknowledge job completion (remove from pending)
                    redis_client.xack('analysis_jobs', 'workers', job_id)

                except Exception as e:
                    print(f"Error processing job {job_id}: {e}")
                    # Job remains in pending list for retry

# Start worker
process_jobs(worker_id='worker-1')

Benefits:

• Simple (Redis only, no extra dependencies)
• Fast (100,000+ messages/second)
• Consumer groups (multiple workers share queue)
• Automatic retries (unacknowledged messages redelivered)

Limitations:

• No dead-letter queue (failed jobs accumulate)
• Limited observability (hard to track job status)

Option 2: Celery + RabbitMQ (Feature-Rich)

Setup:

# Install
pip install celery[redis]  # or celery[amqp] for RabbitMQ

# celery_app.py
from celery import Celery

app = Celery('voice_analysis', broker='redis://localhost:6379/0')

@app.task(bind=True, max_retries=3, default_retry_delay=60)
def analyze_audio_task(self, audio_id, user_id, analysis_types):
    """Celery task: analyze audio file."""
    try:
        # Download audio from S3
        audio_path = download_from_s3(audio_id)

        # Run analysis
        result = run_voice_analysis(audio_path, analysis_types)

        # Store results
        store_results(user_id, audio_id, result)

        return {'status': 'success', 'audio_id': audio_id}

    except Exception as e:
        # Retry with exponential backoff
        self.retry(exc=e)

Producer:

from celery_app import analyze_audio_task

# Queue job (returns immediately)
task = analyze_audio_task.delay('aud_xyz', 'user_123', ['age', 'emotion'])

print(f"Job queued: {task.id}")  # e.g., "8k2n4p6q-9s1t-3v5w-7y9a-1c3e5g7i9k1m"

Worker:

# Start worker (command line)
celery -A celery_app worker --loglevel=info --concurrency=4

# Output:
# [2025-01-15 14:30:00,123: INFO] Connected to redis://localhost:6379/0
# [2025-01-15 14:30:00,456: INFO] Ready to accept tasks (4 processes)

Benefits:

• Auto-retry with exponential backoff
• Dead-letter queue (failed jobs isolated)
• Task prioritization (high-priority users first)
• Monitoring UI (Flower dashboard)

Option 3: AWS SQS (Managed, Scalable)

Producer:

import boto3

sqs = boto3.client('sqs', region_name='us-east-1')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/voice-analysis-jobs'

def queue_job_sqs(audio_id, user_id, analysis_types):
    """Send message to SQS queue."""
    response = sqs.send_message(
        QueueUrl=queue_url,
        MessageBody=json.dumps({
            'audio_id': audio_id,
            'user_id': user_id,
            'analysis_types': analysis_types
        }),
        MessageAttributes={
            'Priority': {
                'DataType': 'Number',
                'StringValue': '1'  # High priority
            }
        }
    )
    return response['MessageId']

Consumer:

def process_sqs_jobs():
    """Worker: poll SQS queue for jobs."""
    while True:
        # Receive messages (long polling, waits up to 20 seconds)
        response = sqs.receive_message(
            QueueUrl=queue_url,
            MaxNumberOfMessages=10,  # Batch size
            WaitTimeSeconds=20,       # Long polling
            VisibilityTimeout=300     # Hide message for 5 minutes while processing
        )

        messages = response.get('Messages', [])

        for message in messages:
            try:
                # Parse job data
                job_data = json.loads(message['Body'])
                audio_id = job_data['audio_id']

                # Process job
                result = analyze_audio(audio_id, job_data['analysis_types'])
                store_results(job_data['user_id'], audio_id, result)

                # Delete message (acknowledge)
                sqs.delete_message(
                    QueueUrl=queue_url,
                    ReceiptHandle=message['ReceiptHandle']
                )

            except Exception as e:
                print(f"Error: {e}")
                # Message will reappear in queue after VisibilityTimeout

Benefits:

• Fully managed (no Redis/RabbitMQ servers)
• Auto-scaling (handles 10-100,000 messages/second)
• Dead-letter queue (automatic after N retries)
• Pay per message ($0.40 per million messages)

---

Caching Strategies: Avoiding Redundant Computation

1. Feature Vector Caching

Concept: Cache expensive feature extraction (openSMILE output)

import hashlib
import pickle

def get_cached_features(audio_path):
    """Get features from cache or compute."""
    # Compute hash of audio file
    with open(audio_path, 'rb') as f:
        audio_hash = hashlib.sha256(f.read()).hexdigest()

    cache_key = f'features:{audio_hash}'

    # Check cache
    cached = redis_client.get(cache_key)
    if cached:
        print(f"Cache HIT: {audio_hash[:8]}")
        return pickle.loads(cached)

    # Cache MISS: compute features
    print(f"Cache MISS: {audio_hash[:8]}, computing...")
    features = extract_features(audio_path)  # openSMILE (5-10 seconds)

    # Store in cache (7-day TTL)
    redis_client.setex(
        cache_key,
        7 * 24 * 60 * 60,  # 7 days
        pickle.dumps(features)
    )

    return features

Cache hit rate: 15-30% (users re-analyze same audio, duplicate uploads)

Savings:

• Feature extraction: 5-10 seconds saved per cache hit
• Compute cost: $0.01 → $0.00 per cached request
• At 30% hit rate with 10,000 req/day → save $30/day = $900/month

2. Model Output Caching

Concept: Cache ML predictions for identical inputs

def get_cached_prediction(features, model_name):
    """Get ML prediction from cache or compute."""
    # Hash features (deterministic input)
    features_hash = hashlib.sha256(
        str(sorted(features.items())).encode()
    ).hexdigest()

    cache_key = f'prediction:{model_name}:{features_hash}'

    # Check cache
    cached = redis_client.get(cache_key)
    if cached:
        return json.loads(cached)

    # Compute prediction
    prediction = model.predict(features)

    # Cache for 30 days
    redis_client.setex(cache_key, 30 * 24 * 60 * 60, json.dumps(prediction))

    return prediction

Cache hit rate: 5-10% (lower because features vary slightly even for same audio)

3. CDN for Audio Delivery

Setup (CloudFront + S3):

# terraform/cloudfront.tf
resource "aws_cloudfront_distribution" "audio_cdn" {
  origin {
    domain_name = aws_s3_bucket.voice_recordings.bucket_regional_domain_name
    origin_id   = "S3-voice-recordings"

    s3_origin_config {
      origin_access_identity = aws_cloudfront_origin_access_identity.default.cloudfront_access_identity_path
    }
  }

  enabled = true

  default_cache_behavior {
    target_origin_id = "S3-voice-recordings"

    allowed_methods = ["GET", "HEAD"]
    cached_methods  = ["GET", "HEAD"]

    forwarded_values {
      query_string = false
      cookies { forward = "none" }
    }

    viewer_protocol_policy = "redirect-to-https"
    min_ttl                = 86400   # 1 day
    default_ttl            = 604800  # 7 days
    max_ttl                = 2592000 # 30 days
  }

  price_class = "PriceClass_100"  # US, Canada, Europe
}

Benefits:

• Latency: 200ms (S3 direct) → 20ms (CDN edge location)
• Bandwidth cost: $0.09/GB (S3) → $0.085/GB (CloudFront) for first 10 TB
• S3 request cost: $0.0004/1000 requests → $0 (CDN handles)

---

Database Optimization: Time-Series Partitioning

Problem: Voice Data Grows Unbounded

-- Without partitioning:
SELECT * FROM voice_sessions WHERE user_id = 'user_123';
-- Scans entire table (10 million rows) to find 50 user sessions
-- Query time: 3-5 seconds

Solution: Monthly Partitioning

-- Create partitioned table
CREATE TABLE voice_sessions (
    id UUID NOT NULL,
    user_id UUID NOT NULL,
    started_at TIMESTAMP NOT NULL,
    audio_duration_seconds INT,
    analysis_status TEXT,
    PRIMARY KEY (id, started_at)
) PARTITION BY RANGE (started_at);

-- Create monthly partitions
CREATE TABLE voice_sessions_2025_01 PARTITION OF voice_sessions
    FOR VALUES FROM ('2025-01-01') TO ('2025-02-01');

CREATE TABLE voice_sessions_2025_02 PARTITION OF voice_sessions
    FOR VALUES FROM ('2025-02-01') TO ('2025-03-01');

-- Auto-create future partitions (pg_partman extension)
SELECT partman.create_parent(
    p_parent_table := 'public.voice_sessions',
    p_control := 'started_at',
    p_type := 'native',
    p_interval := '1 month',
    p_premake := 3  -- Pre-create 3 months ahead
);

-- Query optimization:
SELECT * FROM voice_sessions
WHERE user_id = 'user_123'
AND started_at >= '2025-01-01' AND started_at < '2025-02-01';
-- Scans only voice_sessions_2025_01 partition (~300,000 rows)
-- Query time: <100ms (30-50× faster)

Index Strategy for Partitioned Tables

-- Create indexes on each partition (automatically inherited)
CREATE INDEX idx_voice_sessions_user_id
    ON voice_sessions (user_id, started_at DESC);

CREATE INDEX idx_voice_sessions_status
    ON voice_sessions (analysis_status, started_at DESC)
    WHERE analysis_status IN ('pending', 'processing');  -- Partial index

Partition Maintenance: Auto-Drop Old Data

-- Drop partitions older than 2 years (GDPR compliance)
SELECT partman.run_maintenance('public.voice_sessions', p_retention => '2 years');

-- Cron job: run daily at 3am
SELECT cron.schedule('partition-maintenance', '0 3 * * *',
    $$SELECT partman.run_maintenance('public.voice_sessions', p_retention => '2 years')$$
);

---

Auto-Scaling Policies: Elastic Compute

Metrics-Based Auto-Scaling (AWS ECS)

# Auto-scaling policy for ML workers
resource "aws_appautoscaling_policy" "ml_workers_cpu" {
  name               = "ml-workers-cpu-scaling"
  service_namespace  = "ecs"
  resource_id        = "service/voice-analysis/ml-workers"
  scalable_dimension = "ecs:service:DesiredCount"

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageCPUUtilization"
    }

    target_value       = 70.0  # Keep CPU at ~70%
    scale_in_cooldown  = 300   # Wait 5 min before scaling down
    scale_out_cooldown = 60    # Scale up quickly (1 min)
  }
}

# Min/max worker count
resource "aws_appautoscaling_target" "ml_workers" {
  service_namespace  = "ecs"
  resource_id        = "service/voice-analysis/ml-workers"
  scalable_dimension = "ecs:service:DesiredCount"
  min_capacity       = 5    # Always 5 workers minimum
  max_capacity       = 50   # Max 50 workers (cost cap)
}

Queue-Based Auto-Scaling (More Accurate)

# Scale based on queue depth
resource "aws_appautoscaling_policy" "ml_workers_queue" {
  name               = "ml-workers-queue-scaling"
  service_namespace  = "ecs"
  resource_id        = "service/voice-analysis/ml-workers"
  scalable_dimension = "ecs:service:DesiredCount"

  step_scaling_policy_configuration {
    adjustment_type         = "ChangeInCapacity"
    cooldown                = 60
    metric_aggregation_type = "Average"

    # Scale out rules
    step_adjustment {
      metric_interval_lower_bound = 0
      metric_interval_upper_bound = 100
      scaling_adjustment          = 1  # Add 1 worker if queue < 100
    }

    step_adjustment {
      metric_interval_lower_bound = 100
      metric_interval_upper_bound = 500
      scaling_adjustment          = 3  # Add 3 workers if 100 < queue < 500
    }

    step_adjustment {
      metric_interval_lower_bound = 500
      scaling_adjustment          = 10  # Add 10 workers if queue > 500
    }
  }
}

# CloudWatch alarm: trigger scaling when queue depth > 50
resource "aws_cloudwatch_metric_alarm" "queue_depth_high" {
  alarm_name          = "ml-workers-queue-depth-high"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "ApproximateNumberOfMessagesVisible"
  namespace           = "AWS/SQS"
  period              = 60  # 1 minute
  statistic           = "Average"
  threshold           = 50
  alarm_actions       = [aws_appautoscaling_policy.ml_workers_queue.arn]

  dimensions = {
    QueueName = "voice-analysis-jobs"
  }
}

Scaling behavior:

• Queue depth < 50: 5 workers (minimum)
• Queue depth 50-100: Scale to 6 workers (+1)
• Queue depth 100-500: Scale to 9 workers (+3)
• Queue depth > 500: Scale to 19 workers (+10)
• Queue depth returns to 0: Scale down to 5 workers after 5-minute cooldown

---

Cost Optimization: From $0.10 to $0.001 Per Analysis

Cost Breakdown by Component

Component	Naive Cost	Optimized Cost	Optimization
Compute (ML)	$0.05	$0.008	Spot instances (-84%)
Storage (audio)	$0.02	$0.0002	S3 Glacier + auto-delete
Database writes	$0.01	$0.0001	Batch inserts (100× fewer)
Network (upload)	$0.02	$0.00	Direct-to-S3 (no API bandwidth)
Total	$0.10	$0.0083	92% reduction

1. Use Spot Instances (50-90% Cheaper)

# AWS ECS with Spot instances
resource "aws_ecs_capacity_provider" "spot" {
  name = "ml-workers-spot"

  auto_scaling_group_provider {
    auto_scaling_group_arn = aws_autoscaling_group.ml_workers_spot.arn

    managed_scaling {
      maximum_scaling_step_size = 10
      minimum_scaling_step_size = 1
      status                    = "ENABLED"
      target_capacity           = 90  # Use 90% spot instances
    }
  }
}

resource "aws_autoscaling_group" "ml_workers_spot" {
  mixed_instances_policy {
    instances_distribution {
      on_demand_base_capacity                  = 2   # 2 on-demand (always available)
      on_demand_percentage_above_base_capacity = 10  # 10% on-demand, 90% spot
      spot_allocation_strategy                 = "price-capacity-optimized"
    }

    launch_template {
      launch_template_specification {
        launch_template_id = aws_launch_template.ml_worker.id
      }

      # Multiple instance types (increases spot availability)
      override {
        instance_type = "c6i.2xlarge"  # 8 vCPU, 16 GB RAM
      }
      override {
        instance_type = "c5.2xlarge"
      }
      override {
        instance_type = "c5a.2xlarge"
      }
    }
  }
}

Spot vs On-Demand pricing (us-east-1):

• c6i.2xlarge on-demand: $0.34/hour
• c6i.2xlarge spot: $0.10/hour (70% cheaper)
• For 10 workers × 24/7 = $816/month → $240/month with spot

2. Batch Database Writes

class BatchDatabaseWriter:
    """Buffer writes and flush in batches."""
    def __init__(self, batch_size=100, flush_interval_seconds=5):
        self.batch_size = batch_size
        self.flush_interval = flush_interval_seconds
        self.buffer = []
        self.last_flush = time.time()

    def write(self, record):
        """Add record to buffer."""
        self.buffer.append(record)

        # Flush if buffer full or interval elapsed
        if len(self.buffer) >= self.batch_size:
            self.flush()
        elif time.time() - self.last_flush >= self.flush_interval:
            self.flush()

    def flush(self):
        """Write all buffered records to database."""
        if not self.buffer:
            return

        # Batch insert (single transaction)
        db.execute("""
            INSERT INTO voice_acoustic_features (session_id, f0_mean, f0_std, ...)
            VALUES %s
        """, self.buffer)

        print(f"Flushed {len(self.buffer)} records")
        self.buffer = []
        self.last_flush = time.time()

# Usage
writer = BatchDatabaseWriter()

for session in sessions:
    features = extract_features(session.audio_path)
    writer.write((session.id, features['f0_mean'], features['f0_std'], ...))

# Final flush
writer.flush()

Performance improvement:

• Individual inserts: 100 inserts × 5ms = 500ms total
• Batch insert: 1 insert × 10ms = 10ms total (50× faster)
• Database cost: $0.01 (100 writes) → $0.0001 (1 write) = 100× cheaper

3. Auto-Delete Audio After Analysis

# S3 Lifecycle Policy (Terraform)
resource "aws_s3_bucket_lifecycle_configuration" "voice_recordings" {
  bucket = aws_s3_bucket.voice_recordings.id

  rule {
    id     = "delete-after-30-days"
    status = "Enabled"

    # Delete audio files after 30 days
    expiration {
      days = 30
    }

    # Move to cheaper storage after 7 days (optional)
    transition {
      days          = 7
      storage_class = "GLACIER_INSTANT_RETRIEVAL"  # 68% cheaper
    }
  }
}

Storage cost comparison (per GB per month):

• S3 Standard: $0.023/GB
• S3 Glacier Instant Retrieval: $0.004/GB (83% cheaper)
• Delete after 30 days: $0.023 × (30/30) = $0.023 per audio file (baseline)
• Glacier after 7 days + delete after 30: $0.023 × (7/30) + $0.004 × (23/30) = $0.0054 + $0.0031 = $0.0085 (63% cheaper)

---

The Bottom Line: Scaling Checklist

For production voice analysis at scale:

1. Architecture evolution:
   • ✅ Start: Monolith (0-100 req/day, $50/mo)
   • ✅ Grow: Async workers (100-1K req/day, $200/mo)
   • ✅ Scale: Microservices (1K-10K req/day, $800/mo)
   • ✅ Enterprise: Multi-region (10K+ req/day, $5K+/mo)
2. Load balancing:
   • ✅ Compute-aware routing (not round-robin)
   • ✅ Worker load reporting (Redis metrics)
   • ✅ Health checks (remove failed workers)
3. Job queue:
   • ✅ Redis Streams (simple) or Celery (feature-rich) or SQS (managed)
   • ✅ Consumer groups (multiple workers)
   • ✅ Auto-retry with exponential backoff
   • ✅ Dead-letter queue (isolate failures)
4. Caching:
   • ✅ Feature vectors (15-30% hit rate, 5-10s saved)
   • ✅ Model outputs (5-10% hit rate)
   • ✅ CDN for audio delivery (10× faster, 50% cheaper)
5. Database:
   • ✅ Monthly partitioning (30-50× faster queries)
   • ✅ Read replicas (3-5× more capacity)
   • ✅ Batch writes (100× fewer transactions)
   • ✅ Auto-delete old partitions (GDPR + cost)
6. Auto-scaling:
   • ✅ Queue-based (scale on backlog depth)
   • ✅ Step scaling (aggressive scale-out, gentle scale-in)
   • ✅ Min 5, max 50 workers (cost cap)
7. Cost optimization:
   • ✅ Spot instances (70-90% cheaper compute)
   • ✅ S3 Glacier Instant Retrieval after 7 days (83% cheaper)
   • ✅ Auto-delete after 30 days (GDPR + free storage)
   • ✅ Batch database writes (100× fewer transactions)
   • ✅ Target: $0.001-0.01 per analysis

Expected performance at scale:

• Throughput: 1-10 requests/second sustained (100,000/day)
• Latency: p50 <2s (cached), p95 <8s (cold), p99 <15s
• Availability: 99.9% (multi-AZ) to 99.99% (multi-region)
• Cost: $0.001-0.01 per analysis (90%+ reduction from naive)

The key to scaling: Start simple (monolith), measure bottlenecks, optimize one layer at a time. Premature optimization wastes time—scale when you have real traffic, not theoretical load.

Voice Mirror scales from prototype (single server, 100 req/day, $50/month) to production (microservices, 10,000 req/day, $800/month) using compute-aware load balancing, Redis Streams job queue, feature vector caching (30% hit rate), monthly-partitioned PostgreSQL (50× faster queries), spot instances (70% cost reduction), and S3 Glacier lifecycle policies. Our architecture handles 1-10 req/second sustained with <2s p50 latency and 99.9% availability, at $0.001-0.01 per analysis.