“It’s hard to get analysis right,” says Gary Angel in a post at the SemAngel blog. “[And] even when you do, it’s hard to get it consistently right.” You increase your chances, he argues, with a good process that prevents common pitfalls like these:
Failing to establish measurements of success before deployment. “[I]t’s always possible to find some evidence of success when you are allowed to choose the measure of success after the fact,” he notes.
Allowing vendors to measure their own performance. It isn’t an issue of questioning a vendor’s honesty; rather, says Angel, any self-interested party is naturally inclined to read data in the most positive—and, likely, least helpful—light.
Conducting analysis without the aid of a professional statistician. According to Angel, this doesn’t mean you need an entire team. “But if you have a team generating reporting and analysis on a regular basis,” he says, “you need at least one gate-keeper reviewing it and quashing the most abusive practices.”
Neglecting to tell an analyst that something has changed. If your technology and marketing managers don’t coordinate with those who study your data, critical insights might be lost.
“Good process is very much about protecting ourselves from the things that cause mistakes so that we have a chance to be consistently correct,” says Angel.
Source: SemAngel.