I am quoting Chris from http://www.thefoldblog.com/ because he reminded me of something today that I think is really important.  When it comes to analytics, “you don’t get too many tries to get it right.”  His blog post about Obama’s recovery.org drew a bit of a knee-jerk defense from me (can’t help it), but ultimately I think Chris has it right.

To bring you up to speed on the debate, ABC ran a story yesterday about the number of errors on recovery.org, a website put in place by the Obama administration to provide transparency into stimulus spending.  The article cites examples of congressional districts that don’t exist, due in large part to self-reported data points and recipients who don’t know their district.

The article drew out all sorts of hellfire from conservatives, as you would expect, claiming that the site was bogus if not maliciously bent on deceit.

While my first instincts were to defend and explain, Chris has reminded me of every other analytics-oriented project.  People get fired up about this stuff because it is important.  Analytics can lead to government spending, but in the private sector it can also lead to bonuses, lay-offs, promotions, and plant closures, to name a few.  People are understandably vested in the outcomes.

So when you put a BI solution in place, or when you’re about to release a “study,” you should bear this in mind, because in analytics you don’t get too many tries to get it right.

The details

In the case of recovery.org, in the administration’s haste, they seem to have never put any stops in place should any respondent enter an invalid district (how easy a lookup is that, and wouldn’t that have saved a headache?!).  More, they could possibly have looked the district up by zip code and saved false positives.

Now there is a natural question around what other sorts of checks should have been in place but were not.  Is there a data analyst in the trenches looking at the $/job figures to identify outliers and push back on the survey respondents for more accurate numbers?  Has someone looked at top-down and bottom-up to make sure these numbers are sound?  That’s something about which the end user doesn’t have any visibility, but now it would seem not.

The point is, once you’ve lost legitimacy, you still might be able to win back your supporters, but you’ve given your detractors a lot of ammunition, and the undecided middle has reason to distance.  It has happened in industry untold times, and it seems to have hit hard in this case.  Great lesson, and thanks, Chris, for helping me get on the right side of this after all.