Can You Find the Bias In Modeling?

The brain loves puzzles and patterns. When we solve them, we get a nice dopamine hit. This is why data analysts – and those who love data storytelling – can fall into bias traps: Our brains rush to connect the dots, with erroneous results. Correlation is not causation. And confirmation bias is a real thing.

Recently Udemy had a great blog on digital media analytics “traps,” with a few great tips on avoiding them. These bias traps covered such corollary examples as conversion rate/device type; website traffic /social media posts; email opens/sales. All great examples of biases that media mix modeling solves. Here are a couple of tips for avoiding the sand pit:

  • Ask “Why” repeatedly, questioning the relationship of variables. And asking what other factors (called exogenous variables) might be at play. For example, email opens might coincidentally align with sales seasonality. In fact, a previous history of peak seasonal sales may have trained the marketing team to heavy up on email during that season.
  • Use controlled experiments. Some media mix models (e.g. Bayesian) start with priors. Priors are assumptions as a starting point. Controlled experiments deliver a smarter starting point.
  • Segment and contextualize data. Factors that go into a model as independent variables need to align. For example, regional media impressions should be modeled against same-region sales – a common sense example, but you’d be surprised how often teams don’t think about proper segmentation of data.
  • Communicate uncertainty. Bad data in = bad data out. It’s worth the time to interrogate data going into a model to ensure its influence is properly captured. For example, billboard impressions are not equivalent to TV impressions.

Finding patterns, as satisfying as it may be, is not the north star role of modelers. Testing patterns and challenging them is key to reliable outputs.

controll