▲ 15 ▼ Scaling data ingestion systems from perl to go part 1

A consequence of MediaMath’s astronomical growth over the past few years is dealing with huge growth in service usage. Rapid growth sometimes means that systems are built quickly, without making hard plans for the future. Systems with headroom can now often become insufficient in as little as six months, and so technical debt becomes a tough challenge to address. We deal with the question, “Do we try to re-write this, or do we modify what we already have to scale with the load we expect to see?” Nowhere has this been clearer than in ingesting user data, which since 2011 has more than tripled in volume to greater than 500k QPS. In this post I’ll discuss MediaMath’s history of ingesting user data, and how we lowered latency, simplified concurrency, and increased capacity by taking an experimental leap and switching to Go.