Ok. Brit posted about it. Apparently some people claim that the Faculty Scholarly Productivity Index (FSP) shows something about the rankings produced by the Philosophical Gourmet Report (PGR) (e.g., that they’re off). But it doesn’t.
That is not because the PGR is actually the best possible way to measure program or even faculty quality. It is also not because the FSP is a bad way to measure program or faculty quality. In fact, I think if you’re going to measure faculty quality other than by doing a reputation survey, the methodology of the FSP is the way to go: (a) figure out who’s in what department (b) count books and journal articles of those people (c) count citations. What the FSP is aiming to do is to measure the productivity of individual people and then produce an aggregate ranking. There are some things that are questionable. For instance, why exactly should you count a book the same as 5 journal articles? In general: the specific weighting of parameters is questionable–what would probably be more indicative of faculty quality would be to publish individual and mean rankings on each dimension (articles, books, citations).
But what really sinks the FSP in its current form is the quality of the data: (1) books from Amazon, i.e., any book regardless of publisher is included. (2) Journal articles from scopus.com. Scopus’s selection of philosophy journals is, well, atrocious. I could find about half a dozen philosophy journals that I recognized the title of. As far as I can tell, it only includes a couple of top journals, notably not: PPR, Phil Q, JPhil, Phil Review, Synthese. So the only reason that some good departments show up on FSP’s top 10 is that they have faculty who publish books and many papers in Ethics, BJPS, and Linguistics and Philosophy. As far as I can tell, there isn’t even a bias for analytic philosophy and against continental: Kant Studien is not in scopus, and neither is any other continental journal I know of. There are no logic journals at all.
The short of it: if you’re going to measure faculty productivity, but only count journal articles in an arbitrary selection of journals only a few of which are generally of high quality, you’re going to get unreliable results.