Automatic Publications Ranking for Research Institutions

Communications of the ACM, 50(9), September 2007, p. 14

In their paper entitled “Automatic and Versatile Publications Ranking for Research Institutions and Scholars,” in Commun. ACM (June 2007, pp. 81 ff.), Jie Ren and Richard N. Taylor show that automatic publications ranking can lead to results similar to those from manual processes [1]. Although the authors warn for the sensitivity of the measurement for parameter choices, they suggest that the reproduction of the rankings validates the use of the instrument for quality assessment.

In addition to numbers of publications, citations can be used for the ranking. Ten of the seventeen journals used by these authors are also included in the Science Citation Index-Expanded Version. A publication and citation count of these ten journals for precisely the same period (1995-2003) leads to completely different rankings. Table 1 provides the extension of the table with publication and citation rates for the 50 programs under discussion; the (Spearman) correlation coefficients between the original rankings [1, 2] and these new ones are of the order of 0.5.

Table 1: Top 50 US computing graduate programs as provided in Ren & Taylor (2007, at p. 84)

extended with citation and publication rates using the ISI journal set.

[1]	[4]	Score		ISI Publications	Citations	c/p ratio
1	2	87	Massachusetts Institute of Technology	73	1186	16.2
2	1	79	University of Maryland, College Park	38	955	25.1
3	6	78	Carnegie Mellon University	43	617	14.3
4	19	73	Georgia Institute of Technology	20	425	21.3
5	7	70	Stanford University	51	850	16.7
6	3	64	University of Illinois, Urbana-Champaign	23	187	8.1
7	4	63	University of Michigan, Ann Arbor	5	41	8.2
8	5	61	University of Texas, Austin	23	226	9.8
9	10	59	Purdue University	7	101	14.4
10	11	47	University of California, Berkeley	38	652	17.2
11	26	46	University of California, San Diego	14	123	8.8
12	12	45	University of Massachusetts, Amherst	12	179	14.9
13	30	44	Rutgers University, New Brunswick	22	245	11.1
14	9	43	University of Southern California	16	300	18.8
14	15	43	University of Washington, Seattle	32	445	13.9
16	21	40	Cornell University	40	1277	31.9
16	13	40	University of California, Santa Barbara	12	129	10.8
18	32	39	Michigan State University	0	0
19	20	38	University of California, Irvine	14	231	16.5
20	16	36	University of Minnesota, Minneapolis	2	31	15.5
21	8	35	University of Wisconsin, Madison	21	342	16.3
22	22	31	Princeton University	29	627	21.6
22	31	31	Columbia University	14	246	17.6
24	14	29	Ohio State University	2	15	7.5
24	34	29	University of Florida, Gainesville	3	5	1.7
26	37	28	University of Pennsylvania	14	211	15.1
26	27	28	Texas A&M University	5	56	11.2
26	18	28	Pennsylvania State University	1	10	10.0
29	50	27	University of Texas, Dallas	23	226	9.8
30	29	26	SUNY, Stony Brook	10	137	13.7
31	93	25	Oregon State University	4	24	6.0
31	17	25	University of California, Los Angeles	7	33	4.7
31	51	25	University of Virginia	2	5	2.5
34	65	24	California Institute of Technology	12	185	15.4
34	25	24	University of Arizona	13	190	14.6
34	24	24	University of Illinois, Chicago	19	187	9.8
37	43	23	State University of New York, Buffalo	2	22	11.0
38	38	21	Rice University	18	231	12.8
38	40	21	Louisiana State University	2	20	10.0
38	68	21	Washington University in St. Louis	10	73	7.3
41	56	20	Harvard University	13	118	9.1
42	63	19	Southern Methodist University	0	0
42	51	19	University of Iowa	0	0
42	87	19	University of South Florida, Tampa	0	0
45	69	18	Rensselaer Polytechnic Institute	1	2	2.0
45	53	18	Boston University	0	0
47	71	17	North Carolina State University	2	31	15.5
47	41	17	University of California, Davis	5	43	8.6
49	36	16	University of Colorado, Boulder	14	203	14.5
50	23	16	New York University	9	69	7.7

My rankings are based on the attribution of one full point to an institution for each (co-authored) publication and its corresponding citations (on May 27, 2007). However, proportional attribution does not significantly affect my results [2]. Not only does the order change, but five of the fifty institutions under study have not a single publication attributed in the ISI selection during these nine years.

I don’t wish to claim that the citation-based rankings are any better than those published previously. The reliability of bibliometric constructs and their validity as indicators of quality are two different issues.

Loet Leydesdorff

Amsterdam School of Communications Research (ASCoR),

University of Amsterdam, The Netherlands.

loet@leydesdorff.net; http://www.leydesdorff.net.

References

1. Ren, J. and Taylor, R. N., Automatic and versatile publications ranking for research institutions and scholars, Commun. ACM 50, 6 (2007), 81-85.

2. Geist, R., Chemparambil, M., Hedemeimi, S., and Turner, A. J., Computing research programs in the U.S., Commun. ACM 39, 12 (1996), 96-99.

3. Leydesdorff, L., Caveats for the Use of Citation Indicators in Research and Journal Evaluations, Journal of the American Society for Information Science

and Technology (forthcoming)

return