Polymorph is an ad-tech company that sells an ad-serving suite for publishers. We used previous ad request data to model and train various machine learning models, in order to predict CTR, the probability an ad is actually clicked given that it is loaded on a user’s page.
Polymorph is an ad-tech startup based in SF, and it focuses on creating a white-label suite for publishers who want to create, manage, and display ads on their own websites and applications. Polymorph focuses on native ads, which are ads that fit within the context of the page. So clients who use Polymorph want to use certain areas of their page to display ads to earn revenue, and Polymorph facilitates this process of choosing which ads to display to maximize revenue. Polymorph’s platform is white-labelled, meaning that the ads appear as if they were created by the site displaying the ads, meaning these ad insertions feel natural for users. The company’s suite currently serves 15,000 + sites/apps, and can process 100,00 ad requests per second.
In order to serve both advertisers and publishers who want to sell ad space, Polymorph holds auctions for advertisers to bid on ad spaces to place their ads. Polymorph conducts both direct auctions and third-party auctions. For direct auctions, Polymorph handles the bidding for the advertisers, and uses stats and metrics from these advertisers to calculate a bidding price. For third-party auctions, Polymorph receives bids from third-party advertisers and uses the bids set by these third-parties.
There are multiple units used for bidding, two of which are CPC (cost per click) and CPM (cost per thousand ads loaded). Direct auctions allow bids in both CPC and CPM, while third-party only allow bids in CPM. Our job was to predict CTR, which is the probability a user clicks on an ad given that the ad is loaded on their webpage. The purpose of this project is to efficiently and accurately convert between the above-mentioned two units of measurement for bidding since direct auctions has bids using both units.
This being an ML project, we first looked at existing papers on calculating CPC. Luckily, there have already been previous competitions and attempts to calculate CPC and we tried creating models based on previous literature. Before getting started, we analyzed the data given to us with exploratory data analysis to visually determine useful features and randomly subsampled features hundreds of times for feature selection. We experimented with 4 models: field-aware factorization machines, random forests, logistic regression, and gradient boosting. In order to measure accuracy, we used various units of measurements, such as log-loss and f1 score, and also used grid search for hyperparameter tuning.
The biggest technical challenge was both the lack of data and the poor quality of the data. The initial data set given had too few positive samples (data for clicked ads), so models always returned a prediction of no-click since it didn’t have enough data for clicked ads. Additionally, there weren’t too many useful features in that many of the features given (such as the country and browser of the user) did not tell us much in terms of whether or not the user would click the ad. Thus, our models simply didn’t have enough quality data to train on and this threw them off considerably. After the mid-semester deliverable, we were able to get more and more specific data, and this helped our models.
Another technical challenge was standardizing feature selection and hyperparameter values. Everyone took ownership of a separate model and worked in a separate ipython notebook, so near the end, people had different procedures for selecting features and tuning hyperparameters. Making sure that we standardized these procedures was essential in ensuring final results were comparable.
In retrospect, the project was intriguing conceptually, and we had a lot of fun trying to predict CTR, but there were still many roadblocks we faced given the data issues and the fact that with such a large dataset, iterative changes took days.
Everyone on the team still learned a lot from the experience. Many members did not have much previous exposure to machine learning, and this was a very practical and face-paced, albeit rocky, introduction to machine learning. Learning about the pipeline for creating a working machine learning model was incredibly intellectually stimulating, in that the actual model is only the tip of the iceberg-to create such a model. Hours and hours of data cleaning and examination, parameter tuning, and model training were required. Often times, doing this randomly until the best possible result emerge actually worked the best.