The Closet Moderate: Selection bias is FUNdamental!

Friday, October 15, 2010

Selection bias is FUNdamental!

As usual, I have been stirred from my usual state of slothful non-bloggitude by the statistical illiteracy of someone at Slate. Only this time, its worse, because its a write-up of a study by some legitimate economists, and they're really the ones to blame.
The study is a randomized control trial of management consulting. The authors went to a bunch (66) of midsize textile companies around Mumbai and offered them free consulting provided by Accenture. Then, of those that accepted (17), half got the consulting right away (the "treatment" group), and half got the consulting six months later (the "control" group). By comparing improvements in the treatment group to improvements in the control group, we can get an estimate of the impact of the consulting itself. In this case, the authors found that profitability in the companies increased by 16.8% on average. Wow, that's awesome, time to whip that into a little counter-intuitive parable of the importance of management, send it in to your editor, and cash that fat business reporter paycheck, right? Right, except that the real headline here should be: "75% of Indian firms turn down free consulting (market value: $500,000) from internationally renowned firm". In fancy pants economics terms, what the authors have estimated is the treatment on treated (ToT); that is to say, the effect of the treatment on those who complied with the treatment process. But Mr. Fisman is implicitly interpreting this as being the average treatment effect (ATE); that is, what the effect would be of forcing compliance on everyone. Thus he writes:

And what of the cubicle dweller lamenting the injustices of the modern office? When the 38 principles of good management meet the realities of running an organization with tens or hundreds of thousands of employees, what results is a rigid set of rules, regulations, and constraints that can seem designed to make office life a pointless misery. But it's also what allows the modern corporation to avoid the chaos of the unmanaged cotton weavers of Mumbai.

But you can't look at the effect of consulting on those who accepted it and say "wow, this consulting really works." Presumably, the minority of companies that said yes were the companies with the poorest management among the original 66. If we really believe in the intelligence of the other 49 company owners, the fact that they rejected the help means that it would've had a small or even negative impact on their business.
This paper shows us that, yes, management matters for some, but it strongly implies that it only matters up to a certain point. You have to go to the worst 25% of companies in a country with bad management practices in order to get big gains from improvements in management. If you made some bold assumptions about the relationship between baseline management quality and the effect of consulting, you could even conclude that management consulting is counterproductive for all but the very worst companies in America, and that, just as the cubicle dweller suspects, "best management practices" are a total crock of shit.
To be fair: the paper itself does not claim to be estimating an ATE, and they apparently didn't start out intending to find an ATE; however, given that the paper doesn't appear to be geared towards a statistically-sophisticated audience (i.e. Slate columnists), they should do more to emphasize that fact. And on the whole the paper is worth reading; there is a lot of interesting material in it beyond the headline result.


Fake Steve Hawking said...

You tell 'em, Waldorf.

Statler said...

Shut up and file your TPS report, FSH.