Graduate statistics is bad for your peace of mind

Here is what I wrote on Jay’s blog about the answers to some of my earlier questions about the statistics and numbers in the Call to Action report. People with more expertise in statistics and research design are welcome to point out where my points and questions are stupid or ill informed. I have been out of graduate school for five years now.

Mr. Alexander directs us to the TW report, but I’ve looked through that and I have not seen the actual reporting of the regression coefficients, effects sizes, or p values that are absolutely essential in judging whether a particular statistical correlation is a big deal or a tiny blip. When the report says factors have a significant effect on vitality (as defined in the report) how big an effect are we talking about? What are the numbers?

In addition, the TW report (appendix 1) used a statistical technique called Factor Analysis to create the categories that then determined the high, average, and low vitality churches, but they do not report any of the coefficients that were used to create these three factors. The use of ANOVA to do hypothesis testing raises the same sorts of questions, but in some ways more so.

All statistical tests are based on a whole host of assumptions and requirements to be valid. And the results of most real world statistical analysis are subject to much interpretation. What we seem to have in both the TW report and then amplified in the Steering Committee report is one interpretation without considering possible alternative interpretations.

My experience with social data like this tends to be that the case that can actually be defended by the data analysis is highly nuanced and often quite limited in its scope. You have to hedge and qualify your claims heavily because the statistics indicate a lot but also leave open lots of questions. Of course, all the hedging and qualifying does not lend itself to “firing up the troops” very well.

The TW report has almost nothing but bare percentages. I’m sure the consultants did a lot of number crunching that they did not put in that report, but to treat their conclusions as holy writ without the underlying statistical report does not make sense to me.