▲ 12 ▼ Lessons from Building and Scaling Reddit's Ad Serving Platform with Go

about.sourcegraph.com posted by kenny 2264 days ago

The Reddit engineering team recently introduced Go into their stack to write a new ad-serving system to replace a third party system. Deval Shah talks us through the architecture of the new service, the Reddit team's experience using Go for the first time, and all the lessons they've learned from using Go to build this ad server. Slides for the talk are available on github.

▲ tomf 2264 days ago ▼
Didn't know reddit used Go, everyone seems to now!
TLDR;
1. Use a framework/toolkit
2. Go makes rapid iteration easy and safe
3. Distributed tracing with Thrift and Go is hard
4. Use deadlines within and across services
5. Use load testing and benchmarking
Contributing
▲ kenny 2264 days ago ▼
I love that they call adding adverts inline into content enrichment :)

Some interesting notes here on how replacing the third party service was done.
It was like changing airplanes mid-flight. They slowly added the new infrastructure around their third-party service, and when it was ready, they would rip it out.:
- They started by injecting the ad selector into the request path, having it purely as a proxy. The system did the same thing it was doing before, but with the ad selector in place. This allowed them to scale up requests through the ad selector without without actually doing anything.
- Then, instead of just being a proxy, they implemented and rolled out native ad selection in the ad selector service. Now, the ad selector would process the requests internally, but still act as a proxy and pass the request to the third party. The system would still use the third party response.
- Then, they added Event Logger to implement logging for native responses, and set up Kafka.
- They continued building out the rest of services, starting with stub services, and added logic along the way.
- Eventually, once everything was in place, they cut off the third party ad server.
I also like that they recommend using a framework/libraries - as long as the packages used are lightweight, importable as distinct units (no interdependencies) and don't get in the way, I think this is a really good idea. The main argument for this as I see it is consistency - if you have a large codebase or several services, the last thing you want is people choosing which library to use without constraints. It's better to have a common foundation for all the code, which typically is or becomes a framework (whether out of house or in-house). Low dependencies is great, but for things like logging services, it's essential that everyone uses the same thing. It's easier to navigate the code base, easier to get new hires up to speed, and easier to write new code, as decisions have been made for you. I suspect companies/people who don't want to use a framework end up writing their own anyway in piecemeal fashion - which is fine, but it's still a framework.
Contributing