Production engineering

The problem

Small e-commerce teams don’t have an SRE department. When traffic spikes during a product drop, or a bot farm starts hammering checkout, someone has to notice, find the cause and fix it fast, without getting in the way of real customers placing real orders. For the platforms I run through Studio110, that someone is me.

What I did

• High availability. Load-balanced web nodes behind HAProxy, a replicated Galera database cluster and replicated search, so losing one machine doesn’t take the shop down.
• Observability first. Self-hosted Prometheus and Grafana with alerting, plus self-hosted error tracking. Every fix and every bot rule ships with a metric and a dashboard panel, and with an alert wherever a silent failure would hurt.
• Bot defence. During one busy product drop, automated traffic from bot farms pushed server load to many times its normal level. I traced it, added rate limits at the load balancer, tied the CAPTCHA check to the visitor’s real browser session, and kept tightening the detection rules afterwards, checking each time that real customers still got through.
• Payments that can’t double up. Payment-provider callbacks are idempotent, so a retry or a race can’t turn into a duplicate order, and unpaid orders are reconciled automatically instead of by hand.
• A health sweep that never sleeps. A scheduled check runs every 30 minutes, looks at orders, errors, load and bot activity, and opens a ticket when something drifts.
• Fleet hygiene. Patched PHP security releases across every client server in one sweep, moved CI onto self-hosted runners, and moved mail onto a self-hosted mail stack with a proper relay.

Impact

• Problems are spotted on a dashboard or by an alert, and every fix leaves a metric behind, so the same issue is caught earlier the next time.
• Bot rules are re-checked against real traffic on every sweep, so the defence tightens without locking customers out.
• One person runs production for several shops with the tooling a much larger team would use.