IX Traffic Filter: Meeting 2020’s Business Challenges with Machine Learning
Let’s rewind to a time before quarantine and face masks – the year 2019. As we set our sights on 2020, one of our goals at IX was to invest in efficiency on behalf of our publishers and buyers. We could never have anticipated how important this goal would become, and realised the time was right to meet 2020’s businesses challenges head-on via Machine Learning.
Surprise Challenges In 2020
With the rise of COVID-19 and the resulting economic downturn, we saw massive increases in traffic as quarantined users spent more time browsing, while many programmatic advertisers paused their campaigns. At the same time, travel restrictions made it more difficult to install additional hardware in our global data centres. We had already anticipated growing our volume of auctions this year, but these circumstances made it increasingly important that our infrastructure costs did not grow linearly with increased supply.
Case in point – a pretty typical day on the exchange today has IX running over 120,000,000,000 auctions (120 billion!) and peaking at over 1.2 million QPS.
Our buy-side partners were also challenged by this traffic growth. Many DSPs have QPS constraints that were being pushed to their limits. To be a good partner, we wanted to deliver solutions that optimised the supply we sent to DSPs, without overwhelming them with supply they weren’t interested in – or simply could not handle given the unexpected surges we were seeing.
Predicting the Future with Intelligent Traffic Filtration
Our challenge was straightforward: what if we could predict which impression opportunities wouldn’t ever be likely to receive any bids, and effectively save the infrastructure that would be tasked with operating an unproductive auction? More importantly, could we accomplish this without any impact on our sellers’ revenue?
This was the vision behind IX Traffic Filter, a machine-learning initiative that consists of separate but interlocking projects: Supply Traffic Filter and Buyer Traffic Filter.
Supply Traffic Filter
Supply Traffic Filter has two goals: ensure the exchange can scale to new heights, and make it more efficient. If we only run auctions that are likely to produce bids, we can maximise efficiency to buyers at no cost to publishers, while protecting our infrastructure against traffic spikes (hopefully we won’t be seeing another COVID-19 any time soon).
Here’s how it works: we apply machine learning to historical auction datasets, building models that predict how buyers will bid on future ad requests. These models are deployed to every exchange node and are used in real-time to filter out requests that are anticipated to receive a “no-bid” from buyers. These models are accurate enough (and are constantly retrained!) to ensure that we can preserve revenue at rates >99% while reducing exchange node load by double-digit percentages.
For more information on how Supply Traffic Filter works, check out our engineering blog post.
Buyer Traffic Filter
While Supply Traffic Filter is focused on the exchange overall, the aim of Buyer Traffic Filter is to provide efficiency to buyers: reduce their infrastructure costs, while sending them the most relevant traffic. Within each auction, we want to only send bid requests to buyers which actually want that traffic – without impacting their ability to buy (as in, we should not filter out requests that a buyer would have bid on). Buyer Traffic Filter sits further down the exchange than Seller Traffic Filter and makes many filtration decisions per impression opportunity to make sure we route that auction to each buyer who wants to bid on it.
Buyer Traffic Filter works similarly to Supply Traffic Filter, but we use DSP-specific bidding data to make filtration decisions. The benefit to buyers depends on the flavour of DSP integration: QPS-uncapped DSPs will see efficiency gains, whereas QPS-constrained DSPs will see bid rates and fill rates increase as they will receive a more valuable traffic stream. This also affords them the opportunity to continue scaling as IX scales without having to make aggressive infrastructure investments at each stage of their growth.
What We’ve Learned
So far, we’ve learned we can reduce our volume of auctions by double-digit percentages, with <1% impact to spend. We’ve learned we can pass through a small percentage of traffic to monitor our performance in real-time, thus constantly retraining the system. And we’ve learned that this problem is the ideal candidate for machine learning techniques. We’re continuing to invest in IX Traffic Filter, to be the most efficient exchange for buyers and the most resilient one for publishers. Index Exchange is grateful to be a safe haven for our partners, and we will continue to double-down in the face of uncertainty and push the envelope to keep the ecosystem strong.