Hidden Data

Related to the Inverse Gambler's Fallacy mentioned over at Maggie's is the hidden data of retrospective analysis.

Start with 100 businesses (or schools, or churches), but come back in a year and check out which 3 have done the best.  Interview those three in detail.  Ask them what they thought was they key to their success.  Then write $24.95 management books and hope to start a new business fad.  That's a good model for book sales.

But it's not a good model for finding out what makes businesses successful.  Those three top businesses will each identify a few factors they are sure brought them success.  And how can we argue?  They're rich and we aren't.  Except that if you had collected data on all 100 businesses beforehand, you would find that there were other firms that used exactly the same strategies but did not succeed. 

The strategy isn't entirely useless, of course.  It might lead us in a correct direction, and give us a testable hypothesis.  But we don't see the businesses that did the same thing, but it didn't work, somehow.  This is the reasoning behind the idea of requiring pharmaceutical companies to also report the studies that showed little or no benefit to their product. 

If you pick only the studies that show your pill works...
If you only write books about the churches that grew...
If you only adopt the mission statements of schools that had more Ivy-acceptances...

You might still fail.  Those institutions that succeeded will, irritatingly, give themselves credit for being so smart, and be even more convinced their TQI or Prosperity Gospel or pop-psych fad works, but you should not be.