Conclusions | Appendix A - E
Through analyzing four different perspectives—research output and impact, research focus, research inputs, and knowledge transfer and collaboration, this report outlines a process that states can undertake to identify and showcase their research strengths—those areas in which they have a comparative research advantage.
Research Output and Impact
Within the U.S., the production of research is not evenly balanced but rather concentrated in a few states. The
combined absolute number of research publications of the top five states (California, New York, Massachusetts,
Texas and Maryland) comprised more than 50 of the total
Over the past decade, some states have greatly expanded their research enterprise. For example, Florida achieved both a high level of publication output (210,016 publications, ninth overall and in the top quintile of all states) and a high compound annual growth rate (5.1 percent per year, seventh among all states).
The U.S. produces highly impactful research, which is cited 49 percent more than the world average. Within the
U.S., the states that produce the most impactful research are Massachusetts and California.
The U.S. as a whole specializes in research on medicine and engineering. 28.7 percent and 17.4 percent of the total U.S. research output was in medicine and engineering, respectively.
North Carolina is an example of one state that particularly specialized in medicine. It ranked in the top five among all states in both the relative volume and the relative citation impact of its research in medicine.
Research Inputs and Efficiency
Similar to research outputs, the geographic balance of research inputs is not balanced across all states. The top six states in terms of research and development expenditures (California, New York, Texas, Maryland, Pennsylvania and Massachusetts) accounted for 42.8 percent of all U.S. higher education R&D expenditures from 2004–2013.
The U.S. as a whole produced 6.5 publications per million $ USD of research and development expenditures. Some states, such as Massachusetts, were very efficient, producing 12.7 publications per million $USD of research and development expenditures.
Knowledge Transfer and Collaboration
Although universities produce the majority of research output, it is important to have a critical mass of research
occurring across different sectors, such as government and business, to facilitate knowledge transfer. About 8.5 percent of all published U.S. research was conducted by corporate institutions. Some states have a particularly strong corporate research sector. For example, 20.8 percent of New Jersey’s total output was from corporate researchers, the highest among all states in the country
Knowledge from U.S. research is transferred not only across academic-corporate boundaries but also national
boundaries. 68.4 percent of all U.S. research published in 2004–2013 was downloaded by readers from outside the U.S. Some states’ research drew particularly strong interest from international audiences. For examples, research from Mississippi and Nevada had the highest rates of international readership at 73.5 percent and 73.3 percent, respectively.
Citations of academic research in industry patents represent another way of measuring knowledge transfer. The top states in terms of patent citation share relative to U.S. publication share were Massachusetts (1.69), Maryland (1.54), Maine (1.45) Michigan (1.40) and Washington (1.38). This suggests that those states’ research outputs had a much greater impact on innovation than their national publication volume would otherwise suggest.With this report, CSG and Elsevier hope to contribute to discussion of how states and institutions can identify and leverage their comparative research strengths. Investing in those strengths helps drive innovation and future economic growth.
1. Weinberg, B. A., Owen-Smith, J., Rosen, R. F., Schwarz, L., Allen, B. M., Weiss, R. E., & Lane, J. (2014). Science Funding and Short-Term Economic Activity.
Science, 344(6179), 41–43. doi:10.1126/science.1250055; Lane, J., & Bertuzzi, S. (2011). Research funding. Measuring the results of science investments.
Science, 331(6018), 678–80. doi:10.1126/science.1201865
2. Florida, R. (2014). The Rise of the Creative Class--Revisited: Revised and Expanded. Basic Books.
3. Roberts, E., & Eesley, C. E. (2011). Entrepreneurial Impact: The Role of MIT. Foundations and Trends in Entrepreneurship, 7(1-2), 1–149. Retrieved from
4. Eesley, C. E., & Miller, W. F. (2013). Impact: Stanford University’s Economic Impact via Innovation and Entrepreneurship. SSRN Electronic Journal. doi:10.2139/ssrn.2227460
5. Mazzucato, M. (2013). The Entrepreneurial State: Debunking Public vs. Private Sector Myths (p. 264). Anthem Press. Retrieved from http://www.amazon.com/The-Entrepreneurial-State-Debunking-Private/dp/0857282522
6. Moretti, E. (2012). The New Geography of Jobs. Houghton Mifflin Harcourt.; Delgado, M., Porter, M. E., & Stern, S. (2014). Clusters, convergence, and economic performance. Research Policy, 43(10), 1785–1799. doi:10.1016/j.respol.2014.05.007
7. For more information on what types of records are included in our definition of peer-reviewed publications, please see Appendix C: Glossary.
8. Thus, the percentile associated with 1.0 (California) indicates that that state produced more publications than all other states, a percentile associated with 0.5 (Alabama) indicates that that state produced the median number of publications among all states, and a percentile associated with 0.0 (South Dakota—not displayed in the following scatterplot) indicates that the state produced fewer publications than all other states.
9. For more details on how institutions were categorized into the different sectors, please see Appendix C: Glossary.
10. Peer-reviewed publications in the Scopus database can be categorized as one of three different types – journal articles, review papers, and conference proceedings. See Appendix B: Document Types for more details. For more theoretical considerations of citation based indicators, see e.g., Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. J. (2011). Towards a new crown indicator: an empirical analysis. Scientometrics, 87(3),
11. For a full list of research fields and how research publications are categorized into fields, please see Appendix D: Field Classification Systems.
12. See Appendix C: Glossary for a more technical description of how this is calculated.
13. This organization is based on based on Klavans and Boyack’s uni-dimensional map of science. As a general rule, adjacent subject areas (such as computer science and mathematics, or energy and environmental science) are “closer” to one another in terms of cross-disciplinary collaboration and epistemic overlap. See Klavans, R., & Boyack, K. W. (2009). Toward a Consensus Map of Science. Journal of the American Society for Information Science and
Technology, 60(3), 455–476. doi:10.1002/asi.20991
14. In this and subsequent figures, names of fields are shortened for ease of view. For full list and names of fields, please see Appendix D: Scopus 27 Subject Classification.
15. Katz, B., & Bradley, J. (2013). The Metropolitan Revolution: How Cities and Metros Are Fixing Our Broken Politics and Fragile Economy. Washington, DC: Brookings Institution Press.
16. According to the US Bureau of Labor Statistics (http://www.bls.gov/oes/current/oes_ar.htm) occupational employment surveys (OES), the location quotient for farming, fishing, and forestry occupations in Arkansas in 2013 was 1.44. This means that the relative percentage of workers in that occupation in Arkansas was 44% than the relative percentage of workers in that occupation throughout the US. For more information, see http://www.bls.gov/help/
17. Britt, R. (2015). Higher Education R&D Expenditures Resume Slow Growth in FY 2013. Arlington, VA. Retrieved from http://www.nsf.gov/statistics/2015/nsf15314/#data.
18. See http://report.nih.gov/nihdatabook/index.aspx.
19. Basken, P. (2015). Team science: Research cooperation grows as federal money tightens. The Chronicle of Higher Education. Retrieved from http://chronicle.com/article/Team-science-Research/228175/
20. This is a high-level proxy for research efficiency. For a longer discussion on the limitations of this approach, please see Appendix B: Measuring Research Efficiency and Productivity.
21. Since HERD only covers R&D spending by higher education institutions, we restrict our normalization to just research output from the academic sector. If we instead normalized total research output (across all sectors) by total HERD, Tennessee would still be the most efficient state, producing 23.8 publications per million $ R&D.
22. It is important to note, however, that different fields of research have different space requirements – for example, research in the biological and biomedical sciences tend to require much more space than that in the social sciences. More in-depth studies of research space usage and efficiency can normalize for these distortions. See Appendix B: Measuring Research Efficiency and Productivity for more details.
23. Maine technically had the highest number of medical publications from universities per 1000 NASF of medical school research space (58.6), but based on the 2011 survey, it had only 9504 NASF of research space in medical schools, the sixth lowest among all states. By comparison, Massachusetts had 1,693,485 NASF of research space in its medical schools.
24. Please see Appendix B:Measuring Research Efficiency and Productivity for a longer discussion on limitations of this measure.
25. Saxenian, A. L. (1996). Regional Advantage: Culture and Competition in Silicon Valley and Route 128. Cambridge, Mass.: Harvard University Press. Retrieved from http://www.amazon.com/Regional-Advantage-Culture-Competition-Silicon/dp/0674753402
26. Leydesdorff, L., & Meyer, M. (2003). The Triple Helix of university – industry – government relations. Scientometrics, 58(2), 191–203; Mowery, D. C., & Ziedonis, A. A. (2015). Markets versus spillovers in outflows of university research. Research Policy, 44(1), 50–66. doi:10.1016/j.respol.2014.07.019
27. D’Este, P., Guy, F., & Iammarino, S. (2012). Shaping the formation of university-industry research collaborations: what type of proximity does really matter? Journal of Economic Geography, 13(4), 537–558. doi:10.1093/jeg/lbs010 ; Hewitt-Dundas, N. (2011). The role of proximity in university-business cooperation for innovation. The Journal of Technology Transfer, 38(2), 93–115. doi:10.1007/s10961-011-9229-4; Bishop, K., D’Este, P., & Neely, A. (2011). Gaining from interactions with universities: Multiple methods for nurturing absorptive capacity. Research Policy, 40(1), 30–40. doi:doi:10.1016/j.respol.2010.09.009
28. For more information on how we classify research output into the corporate, government, medical, and other sectors, please see Appendix C: Glossary.
29. Kurtz, M.J., & Bollen, J. (2012). Usage Bibliometrics. Annual Review of Information Science and Technology
30. Volume 44, Issue 1. Retrieved online from http://onlinelibrary.wiley.com/doi/10.1002/aris.2010.1440440108/pdf.; Moed, H. F. (2005). Statistical relationships between downloads and citations at the level of individual documents within a single journal. Journal of the American Society for Information Science and Technology, 56(10), 1088–1097. doi:10.1002/asi.20200; Schlögl, C., Gorraiz, J., Gumpenberger, C., Jack, K., & Kraker, P. (2014).
Comparison of downloads, citations and readership data for two information systems journals. Scientometrics, 101(2), 1113–1128. doi:10.1007/s11192-014-1365-9; Schloegl, C., & Gorraiz, J. (2010). Comparison of citation and usage indicators: the case of oncology journals. Scientometrics, 82(3), 567–580. doi:10.1007/s11192-010-0172-1; Schloegl, C., & Gorraiz, J. (2011). Global usage versus global citation metrics: The case of pharmacology journals. Journal
of the American Society for Information Science and Technology, 62(1), 161–170. doi:10.1002/asi.21420; Wang, X., Wang, Z., & Xu, S. (2012). Tracing scientist’s research trends realtimely. Scientometrics, 95(2), 717–729. doi:10.1007/s11192-012-0884-5
31. Bornmann, L. (2013). What is societal impact of research and how can it be assessed? a literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217–233. doi:10.1002/asi.22803; Tijssen, R. J. . (2001). Global and domestic utilization of industrial relevant science: patent citation analysis of science–technology interactions and knowledge flows. Research Policy, 30(1), 35–54. doi:10.1016/S0048-7333(99)00080-3
32. For a more precise definition of what counts as a “download,” please see Appendix C: Glossary.
33. For more information about the data sources for patent data, please see Appendix B: Data Sources.
34. D’Este, P., & Patel, P. (2007). University–industry linkages in the UK: What are the factors underlying the variety of interactions with industry? Research Policy, 36(9), 1295–1313. doi:10.1016/j.respol.2007.05.002; Schartinger, D., Rammer, C., Fischer, M. M., & Fröhlich, J. (2002). Knowledge interactions between universities and industry in Austria: sectoral patterns and determinants. Research Policy, 31, 303–328.
35. Similar to research usage (downloads), we normalize a patent citations to a state’s research output (either overall or by subject) relative to its national publication share since states with higher levels of research output have a higher probability” of having any research output being cited in patents.
36. Wuchty, S., Jones, B. F., & Uzzi, B. (2007). The Increasing Dominance of Teams in Production of Knowledge. Science, 316(5827), 1036–1039. doi:10.1126/science.1136099
37. In 2013, Elsevier and Science Europe collaborated on a joint report that analyzed and comparatively benchmarked collaboration rates amongst European countries and US states. For more information, see http://www.elsevier.com/about/press-releases/science-and-technology/european-and-us-research-collaboration-and-mobility-patterns-science-europe-and-elsevier-release-comprehensive-study.
38. For more information, see Appendix C: Glossary.
39. The average Salton index between any two U.S. states is 0.016.
40. Moed, H. F., Glänzel, W., & Schmoch, U. (Eds.). (2005). Handbook of Quantitative Science and Technology Research. Dordrecht: Kluwer Academic Publishers. doi:10.1007/1-4020-2755-9
41. Price, D. J. de S. (1977). Foreword. In Essays of an Information Scientist (pp. v–ix).
42. Garfield, E. (1979). Is citation analysis a legitimate evaluation tool? Scientometrics, 1(4), 359–375. doi:10.1007/BF02019306
43. Pinski, G., & Narin, F. (1976). Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics. Information Processing & Management, 12(5), 297–312. doi:10.1016/0306-4573(76)90048-0.
44. Irvine, J., Martin, B. R., Abraham, J., & Peacock, T. (1987). Assessing basic research: Reappraisal and update of an evaluation of four radio astronomy observatories. Research Policy, 16(2-4), 213–227. doi:10.1016/0048-7333(87)90031-X
46. Van Eck, N. J., Waltman, L., van Raan, A. F. J., Klautz, R. J. M., & Peul, W. C. (2013). Citation analysis may severely underestimate the impact of clinical researchas compared to basic research. PloS One, 8(4), e62395. doi:10.1371/journal.pone.0062395
47. http://projectcounter.org/, http://usagereports.elsevier.com/asp/main.aspx
48. Glänzel, W. (2001) “National characteristics in international scientific co-authorship relations” Scientometrics 51 (1) pp. 69–115.
49. Leydesdorff, L. (2008) “On the normalization and visualization of author co-citation data: Salton’s Cosine versus the Jaccard Index” Journal Of The American Society For Information Science And Technology 59 (1) pp. 77–85.
< Prev Next >