Vaguely Right or Precisely Wrong?

This week Peter Daboll, Chief of Creative Strategy and Insights at iSpot.tv, published a social media post saying that AI and machine learning have created a dynamic, especially with digital media, where metrics can be measured within 6 decimals. But that doesn’t mean the measurement is right.

Why?

Models are created from inputs of assumptions. And when the assumptions vary, the output can vary wildly. My colleague posits that being “precisely wrong” is as dangerous as a guess. Instead, marketers should embrace “vaguely right.”

We have a way of collaboratively and transparently addressing “vaguely right.” In FutureSight’s platform, we include a visualization of the MAPE, Mean Absolute Percent Error. The MAPE is the difference between the key performance indicator  forecast and the key performance indicator actual. Guess what? The forecast never exactly matches the observed. The scientific protocol is to adjust the data until the MAPE is an acceptable margin. This is called “fitting” the model.

“Fitting” the model is an exercise in quality assurance – does the model’s forecast mirror the observed, within a reasonable margin of error. Why or why not? This exercise necessitates a patient and skilled data scientist. As my colleague says, “Data without human oversight is like a ship without a captain.”

And having a MAPE a good thing, because if forecast and actual  match too closely, you may have actually “overfit” the model. In this case, the data input assumptions that are causing that non-existent gap, may, in the long run, be erroneous assumptions. This is actually quite common.

Common sense tells us that there is always some variable, for which we cannot account, that will influence a key performance indicator. We want that unknown represented in the MAPE.  As another colleague says, “All models are wrong; some of them are useful.”

All that said, don’t let MAPEs and fits intimidate you, because at least getting started measuring, and evolving your model to include more variables, is far better than doing nothing.