How to conduct Social science Research

how to write social science research paper and how to write social science research proposal and how to write a social science research question| free pdf download
HelenaColins Profile Pic
HelenaColins,New Zealand,Professional
Published Date:06-07-2017
Your Website URL(Optional)
Comment
The Impacts of Humanities and Social Science Research Working Paper October 2014 Contents Summary ....................................................................................................................................................... 4 Section 1: Introduction ................................................................................................................................. 6 1.1 About this document .............................................................................................................................. 6 1.2 Defining Impact ................................................................................................................................... 7 1.3 Why evaluate impact .......................................................................................................................... 9 1.4 Building on International and Canadian experiences ....................................................................... 11 Section 2: Description of Indicators ............................................................................................................ 13 HSS research has impacts on SCHOLARSHIP ........................................................................................... 14 HSS research has impacts on CAPACITY.................................................................................................. 24 HSS research has impacts on the ECONOMY .......................................................................................... 31 HSS research has impacts on SOCIETY AND CULTURE ............................................................................ 37 HSS research has impacts on PUBLIC POLICY ......................................................................................... 45 Acknowledgements……………………………………………………………………………………………………………………………….51 3 Summary  Research in the humanities, social sciences, and fine and creative arts has impact but defining, measuring, and comparing these impacts is an enormous challenge.  This research contributes to a free and democratic society, public policy, quality of life, social cohesion, business innovation, the environment, creativity, commercial and economic activity, administrative and institutional development. Studying this impact leads to a better appreciation of the role of the humanities and social sciences in enriching society.  This document proposes five broad ways this research has impact and a basket of indicators for each that can be used to measure them. These are presented on the second page of this summary.  The intent is that organizations involved in measuring the impact of research in the humanities, social sciences, and fine and creative arts will be able to draw from these indicators to develop their methodology. Not all indicators will necessarily be used at any one point in time, but instead they provide a suite of indicators which can be deployed in various combinations depending on the type of impact being studied and the expertise, time, and budget available.  The indicators in this document are not intended to be used to compare individual researchers, or to be relevant to tenure and promotion decisions, but to allow the exploration and, where appropriate, the comparison of research impact at aggregated levels - whether at the university, regional, national, or international scale.  This is a living document. This draft focuses most on the conceptual background. With feedback and experience in implementation it is hoped that future versions of this document will be able to include some Canadian-specific “how-to” guidance. 4 Summary HSS research has impacts on HSS research has impacts on SCHOLARSHIP CAPACITY through teaching and mentoring that can be measured using indicators at the graduate and such as: undergraduate levels  Bibliometric indicators  Downloads from Open Access that can be measured using indicators repositories such as:  Citations in grant applications  Number and quality of  Acknowledgements experiential learning/ research  Prizes and awards opportunities for students  Reputation as measured by  Surveys of students and alumni survey  Employer surveys  Post-publication peer-review  Integration of research as a (book reviews, dedicated learning outcome in courses symposia)  Juried exhibitions and performances HSS research has impacts on the HSS research has impacts on HSS research has impacts on ECONOMY SOCIETY AND CULTURE PRACTICE AND POLICY that can be measured using indicators that can be measured using indicators that can be measured using indicators such as: such as: such as:  Invitations to participate as an  Advisory roles and board  Number and quality of expert witness, an advisor, on an memberships partnerships between researchers expert panel or committee  Revenue opportunities and cost and community groups  Citations in government savings in the public, private and  Requests for consultancy/advice documents not-for-profit sectors resulting from community groups from research applied in practice  Consulting for governments or  Media coverage of research think tanks  Income derived from patents, (newspapers/ TV/ online)  Commissioned reports patent licensing, copyright and  Requests for media appearances trademarks  Engagement of the public at events  Consulting contracts  Research-related social media  Public use of research-based web resources on social and cultural issues 5 The Impacts of Humanities and Social Science Research Section 1: Introduction HSS research has impacts on HSS research has HSS research has HSS research has HSS research has impacts on impacts on the impacts on impacts on CAPACITY SCHOLARSHIP through ECONOMY SOCIETY AND PRACTICE AND teaching and CULTURE POLICY mentoring Academic Impacts Impacts on Society Although the evidence of these impacts is visible every day, documenting and defining these impacts is an enormous challenge. Identifying the impact of any research is very difficult. But HSS research in particular is recognized as being particularly difficult to evaluate as accessible measures commonly used in scientific and technical disciplines, such as citations, patents and licensing revenues, are less relevant. Here, we propose a series of indicators with the potential to document, compare and aggregate at levels beyond an individual research project. In an era of ranking and impact measurement, it is increasingly important to develop an accepted and as comprehensive as possible set of indicators to document the impacts of HSS research, and for these to serve as a platform upon which further indicators can be grafted as they are identified 1.1 About this document This document delves into some of the ways that HSS research has impact and draws from international practices in proposing indicators that can be used to illuminate the impacts of HSS research in Canada. It is important to note that this document deals with the impacts of HSS research. There are many benefits and impacts of other activities in the humanities and social sciences that will not be captured here. 6 This is not intended to be an academic document advancing the theory of impact measurement, but a plain-language practical document that makes use of advances in the field to propose ways of assessing the impact of HSS research in Canada including the all-important caveats as to the limitations of such indicators. The intent is that organizations involved in measuring the impact of HSS research will be able to draw from these indicators to inform their methodology and help better position their studies. It is not expected that all of these indicators will be used all of the time, but rather that a subset of indicators will be selected as appropriate. As organizations measuring impact have different aims, scope, time, budget, and study different disciplines or types of impact, relevant indicators can be chosen as needed. HSS research impact is often demonstrated by case studies, which are extremely valuable and can be 1 compelling. Case studies demonstrate the impact of individual projects and the potential of all HSS research to have impact. They are undoubtedly an important way of communicating impact, but they are not dealt with in this document. The indicators in this document are not intended to be used to compare individual researchers, or to be relevant to tenure and promotion decisions, but to allow the exploration and where appropriate the comparison of research efforts whether at the university, regional, national, or international level. Put simply, it is up to others to decide whether impact should be measured; and this document simply lays out some of the indicators of impact. This is a living document. This draft focuses most on the conceptual background to the indicators proposed and provides examples of how they have been used internationally so that users can draw from current practice to develop their own methodologies. With Canadian experience it is hoped that future versions of this document will be able to include some Canadian-specific “how-to” guidance on methodologies as well as case studies of how these indicators have been applied. This document is divided into two sections. Section one provides an introduction and situates the discussion within the ongoing international discussion and practice of impact measurement. Section 2 describes the proposed Canadian framework. It describes five ways that HSS research has impact along with a preliminary list of indicators that can be used to identify impact. Each indicator is then examined to describe the advantages and limitations; where possible or appropriate, links are provided to where and how the indicators have been used previously to measure impact. 1.2 Defining Impact Research impact refers to the influence scholarly and creative enquiry has upon wider society, intended as well as unintended, immediate as well as protracted. It includes the influence such research has upon 1 For example: ESRC 2009 http://www.esrc.ac.uk/_images/Taking%20Stock_tcm8-4545.pdf 7 future researchers within the discipline as well as in other disciplines and on public policy, quality of life, social cohesion, business innovation, the environment, artistic and creative practices, commercial and economic activity, administrative and institutional development, and political and cultural understanding. There are as many definitions of research impact as there are types of research. Canada’s International Development Research Centre (IDRC) points out that there “is no universal definition for research impacts.” In its recent report on “research excellence,” the IRDC draws upon Sandra Nutley et al.’s cross- disciplinary study of research impact models, which differentiates between research that “brings about changes in levels of understanding, knowledge and attitude” and research that “results in changes in practice and policy making.” These distinctions suggest multiple varieties of research impact, including the generation of new knowledge, new insights, changes in attitudes, beliefs, and behaviours, references to and citations in research, increased access to research, more research and the extension of research beyond disciplinary boundaries. The IDRC study also draws attention to the Impact of Social Sciences project by the London School of Economics (LSE), which defines research impact as “an occasion of influence.” For the LSE, research impact is defined by its influence on people, ideas, organizations, and industry, not by the outcome of the influence itself. The LSE differentiates and classifies research impact into academic impacts and non- academic impacts or impacts inside and outside universities. Academic impacts, based on the LSE’s definition, are typically illuminated by citation indicators in other academic work, while non-academic impacts tend to be measured by references to research in anything from government documents and trade press materials to the media and the web. Citation indicators, i.e. “occasions of influence,” the LSE points out, generally demonstrate academic impacts and might include bibliometric (citations) and non-bibliometric (discussions, consultations, surveys of opinion) records as well as the “digital footprints” of research products. Research, in this instance, though, has an academic impact when the influence is upon another researcher and finds its way into his or her work. But research, the LSE impact project also points out, has an external impact when its influence reaches beyond academic audiences and universities. The LSE cites business, government, non-government, and the media as potential non-academic venues for HSS research impact. In this case, citation indicators can take many forms, from references to and citations or discussions of a person or work in media, meetings, conferences, seminars, working groups, speeches or statements, to web links and the provision of external access to research materials. A researcher might also demonstrate research impact through direct involvement in the decision-making processes of non-academic institutions. Tracking down external references, though, or following the “footprints,” digital or otherwise, of research influence outside academic work can be as difficult as ascertaining impact in external decision-making processes. 8 In a review for RAND of international practices related to research impacts, Jonathan Grant et al. argue that neither quantitative nor qualitative measures entirely suffice, so a combination of both, including case studies, narratives, questionnaires, self-evaluation, and other proxy indicators, should be employed. RAND, the IRDC, along with a number of related studies do arrive at one consensus, however, which is that no matter how well impact is defined “it takes an uncertain amount of time for research to have influence,” making it “difficult to evaluate impacts in the short-term.” Moreover, research impact in external areas, such as policy, might be indirect, while the subjectivity and level of expertise of the 2 research users and reviewers can skew the way research is received, evaluated, and used. In other words, any definition of research impact must take the long-view of research influence into account as well as the contradictory or inconsistent views of users and reviewers. But research that generates new knowledge both inside and outside universities, research that can improve subsequent research, as well as research that influences the decisions (regardless of outcome) that shape people’s lives, communities, governance, the environment, and elsewhere can be defined as having impact. 1.3 Why evaluate impact HSS research is a driver of the “knowledge economy.” HSS research has proven qualitative benefits, such as contributing to culture and identity or more generally enriching and improving people’s lives, and proven quantitative benefits, such as contributing to economic growth, job creation, community development, and student learning. HSS research, though, is principally conducted in universities and there are increasing calls for demonstrations of the impact of HSS research both within and beyond the academy. HSS research may be narrowly focused, intellectually-inclined, and discipline specific or it may be socially-engaged, problem-focused, and cross-disciplinary. It may serve the needs of one community or many. Increasingly, though, HSS research is being called upon by non-academics to solve complex problems outside the disciplines and institutions where HSS researchers typically work. This has led to a widening view of what constitutes research as well as how it should be used and is generally referred to as the “democratizing” of knowledge. Related to this phenomenon is the increasingly important role that HSS research plays in social innovation, or the development of new ideas, concepts, strategies, and 2 IDRC, “Evaluating Research Excellence: Main Debates.“ URL: http://www.idrc.ca/EN/Programs/Evaluation/Pages/evaluating-excellence.aspx; LSE Public Policy Group, “Maximizing the impacts of your research: A Handbook for Social Scientists.” (London: LSE Public Policy Group, 2011). URL: http://www.lse.ac.uk/government/research/resgroups/LSEPublicPolicy/Docs/LSE_Impact_Handbook_April_2011.p df; S. Nutley, et al., “Models of research impact: a cross-sector review of literature and practice.” (London: Learning and Skills Research Centre, 2003). URL: http://www.st- andrews.ac.uk/ruruweb/pdf/LSDA%20literature%20review%20final.pdf; Jonathan Grant, et al., “Capturing Research Impacts.” (RAND, 2010). URL: http://www.rand.org/pubs/documented_briefings/DB578.html 9 organizations to meet social needs and social goals. Researchers whose work fosters innovation, including new ideas, technologies, patterns of growth, alliances, and relationships, across disciplinary and organizational boundaries have the opportunity to not only extend their own research networks but 3 to play a critical role in the kind of innovation that addresses social needs and improves people’s lives. These processes, though, also lead to new challenges when it comes to evaluating or measuring the impact of HSS research. Traditional (namely bibliometric) methods alone, for example, fall short of accounting for the qualitative or societal benefits of HSS research and do little to suggest potential returns on public investment. More robust, multi-pronged evaluation frameworks will improve our 4 understanding of the value and impact of HSS research and help us find new ways to expand its scope. Evidence of impact is presumed to be a sign of value; and as universities, research institutions, funding bodies, and governments assess and reassess their research budgets, the ability to substantiate impact is key to preserving, protecting, and increasing HSS research funding. Evaluating impact helps HSS researchers demonstrate the wider social and economic contribution of their work as well as explore new avenues, audiences, venues, and applications for their research. A recent study of the value of arts and humanities research in an era of austerity by the Dublin Institute of Technology points out that while “universities differ in the emphasis they place on different disciplines, arts and humanities research has continued to remain at the heart of the belief that society benefits from the pursuit of knowledge and the scholarship generated by universities.” The study emphasizes that the social and economic application of HSS research, not unlike the sciences, has underpinned university-based research agendas and defined the social contract between researchers, governments, and taxpayers since the Second World War. For these reasons, then, a framework to evaluate research impact is needed. To traditional performance and productivity measures, such as bibliometrics and research grants, are being added newer measures, such as patents, licenses, and consultancy contracts. Material published in peer-reviewed journals is increasingly assessed for its potential spillover or economic benefit. Can it be commercialized? Can it be mobilized to benefit society more widely? Can it contribute to economic recovery? Evaluating research impact, in other words, has evolved from a linear model, usually assessed incompletely by bibliometric means, to a more holistic measure taking into account less obvious, often non-disciplinary but still interconnected factors. The fact that HSS research can be so difficult to measure and judge in the first place constitutes one of the most compelling reasons for evaluating its impact. A less than robust rubric for evaluating HSS 3 Geoff Mulgan, “Social Innovation: What It Is, Why It Matters And How It Can Be Accelerated,” Working Paper, Oxford’s Said School of Business (2007). URL: http://eureka.bodleian.ox.ac.uk/761/1/Social_Innovation.pdf 4 Ellen Hazelkorn, et al., “Recognising the Value of the Arts and Humanities in a Time of Austerity.” (Dublin Institute of Technology, 2013). URL: http://arrow.dit.ie/cgi/viewcontent.cgi?article=1042&context=cserrep 10 research impact leads to the risk of overlooking or undervaluing the important contribution that HSS research makes to knowledge production, to society, culture, and the economy. HSS researchers cover an enormous breadth of disciplines – from history, literature, philosophy and the creative arts to law, linguistics, and policy and child studies – and while certain disciplines lend themselves more straightforwardly to social and economic impact measures, creative or what the Dublin study referred to as “curiosity-inspired” projects can be more difficult to assess, particularly in terms of their flow-through into society and the economy. But we know it is there and so finding new and non-traditional ways to understand and document the importance and purpose of HSS research as well as its audience is critical to a better appreciation of the role of the social sciences, humanities and creative and fine arts in 5 enriching society. 1.4 Building on International and Canadian experiences Research agencies and institutions throughout the world are exploring and experimenting with a variety of techniques to define and assess research impact. They are struggling with many of the same challenges we face in Canada, from acknowledging the relative paucity of measures for impact outside the academy to the increasingly diversified venues for scholarly dissemination. Whereas peer-reviewed articles, books, and conference papers were once considered the main examples of measurable research outputs, new types and categories are quickly being added to evaluation frameworks, including audio- visual material, original creative works, software, reports, legal cases, maps, newspaper and magazine articles, and works of translation. To meet the growing need for new, effective, cross-disciplinary evaluation metrics, Canada can both contribute to and build upon existing international examples of frameworks and tools for evaluating research. 6 In 2013, RAND performed a comparison of key international research evaluation frameworks. RAND’s approach accounted for the origins and rationale that led to the development of each system, the scope of each system’s approach, including timescales used, the ways in which data was aggregated and analyzed, how the data was used, including audience, and was followed by RAND’s own analysis of the effectiveness and wider application of each approach. RAND itemized the most common tools used for research evaluation in each of the international frameworks. They included: bibliometrics, data mining, data visualization, logic models, case studies, document review, peer review, surveys, interviews, site visits, and economic analysis (see Fig. 1). 5 Ellen Hazelkorn, et al., “Recognising the Value of the Arts and Humanities in a Time of Austerity.” (Dublin Institute of Technology, 2013). URL: http://arrow.dit.ie/cgi/viewcontent.cgi?article=1042&context=cserrep 6 The eight research evaluation frameworks surveyed by RAND included the Canadian Academy of Health Science Payback Framework (CAHS), Excellence in Research for Australia (ERA), the National Institute of Health Research Dashboard (England), the Research Excellence Framework (REF) (UK), Productive Interactions (Netherlands and European Commission), Science and Technology for America’s Reinvestment: Measuring the Effect of Research on Innovation, Competitiveness and Science (STAR METRICS) (US). 11 7 Fig. 1. Comparison of tools used in international evaluation frameworks. From RAND 2013. This draft report draws from indicators used in a variety of international reports, such as the ones identified in RAND 2013. It also proposes new indicators that have not yet been successfully used. As a living document this is one step in a long conversation. Experience and feedback will help refine subsequent drafts of this document and will allow specificity to be added in terms of how these indicators can be rigorously applied in Canada. The Federation hopes that this document, and future iterations of it, will be a useful contribution to understanding, comparing, and articulating the impact of humanities and social science research in Canada. 7 We gratefully acknowledge RAND for granting permission to use the figure from document MG-1217- AAMC. Full citation: Susan Guthrie, et al, Measuring Research: A guide to research evaluation frameworks and tools (RAND, 2013). URL: http://www.rand.org/pubs/monographs/MG1217.html 12 Section 2: Description of Indicators This section of the report describes five baskets where HSS research has impact, and provides detail on the types of indicators that can be used to identify impact, along with examples of studies that have used the indicator. A summary of the types of impact and the basket of indicators appears below. The list of indicators in each basket is not exhaustive – these are examples and over time we expect many more will be identified and added to each. It is not expected that any study will use all of these indicators, rather a selection of indicators will be chosen based on the type of impact of interest, and the time, expertise and finances available for the study. This section of the document is not intended to be read in a linear way: each item is designed to stand alone, click to immediately access any part of the document. HSS research has impacts on HSS research has impacts on SCHOLARSHIP CAPACITY through teaching and mentoring that can be measured using indicators at the undergraduate and such as: graduate levels  Bibliometric indicators that can be measured using indicators  Downloads from Open Access such as: repositories  Number and quality of experiential  Citations in grant applications learning/ research opportunities for  Acknowledgements students  Prizes and awards  Surveys of students and alumni  Reputation as measured by survey  Employer surveys  Post-publication peer-review (book  Integration of research as a learning reviews, dedicated symposia) outcome in courses  Juried exhibitions and performances HSS research has impacts on the HSS research has impacts on HSS research has impacts on ECONOMY SOCIETY AND CULTURE PRACTICE AND POLICY that can be measured using indicators that can be measured using indicators that can be measured using indicators such as: such as: such as:  Invitations to participate as an expert  Advisory roles and board memberships  Number and quality of partnerships witness, an advisor, on an expert  Revenue opportunities and cost savings between researchers and community panel or committee in the public, private and not-for-profit groups  Citations in government documents sectors resulting from research applied  Requests for consultancy/advice from in practice community groups  Consulting for governments or think- tanks  Income derived from patents, patent  Media coverage of research  Commissioned reports licensing, copyright and trademarks (newspapers/ TV/ online)  Consulting contracts  Requests for media appearances  Engagement of the public at events  Research-related social media  Public use of research-based web resources on social and cultural issues 13 HSS research has impacts on SCHOLARSHIP HSS scholarship covers an enormous range of disciplines, encompassing everything from health, wellness, happiness, and cultural identity to history, literature, the fine arts, family life, democracy, civic engagement, and international affairs. The purpose of HSS research is to build upon existing knowledge as well as create new knowledge, insight, and understanding in the process. HSS researchers who make maximizing the impact of their work part of their research agenda will extend the influence of their ideas beyond disciplinary or university boundaries and into the public sphere. The impact that HSS research has on scholarship is fundamental. It is what allows knowledge to move forward. Researchers are accustomed to maintaining a record of their scholarship for the purposes of demonstrating how their research has contributed to both the disciplinary and the wider social context in which he or she is working. To validate their work within their disciplines and home institutions as well as to governments and funding bodies, HSS researchers often quantify the influence of their work, usually through the use of agreed-upon indicators or metrics. However, these measures of scholarly impact are not always captured in discussions of impact. Impacts on scholarship might not be the focus of many studies of impact, but is an important type of impact to be considered. Indicators for measuring the impact of HSS research on research include:  Bibliometric indicators  Downloads from Open Access repositories  Citations and/or references in grant applications  Published Acknowledgements  Prizes and awards  Reputation (as measured by survey among appropriate expert cohort)  Post-publication peer-review (e.g., book reviews, dedicated symposia) Bibliometric Indicators SUMMARY TABLE: Type of research output measured: Peer-reviewed journal articles. Most relevant for: Social science research. Time lag between research output and measureable impact: 4 years+ Key advantages: ARC index provides a quantitative, replicable, comparable measure of citations. Data is readily available. Key limitations: Only relevant when peer-reviewed journal article is the primary means of research communication, results depend on database used, and time lag. 14 Bibliometrics refer to quantitative or statistical means used to assess patterns in journal article publications within specific fields, literatures, and databases. They are a common way to study the impact of research. Thompson Reuter’s Social Science Citation Index (SSCI), Arts and Humanities Citation Index (AHCI) and Journal Citation Reports (JCR), part of its Web of Science Core Collection, are the most commonly used sources of bibliometric data, but Google Scholar (GS) is gaining in popularity, in part because it provides a more comprehensive coverage of citations. Citation analysis programs such as H- Index and Publish or Perish both retrieve and analyze citations. These programs typically have the following goals: to return metrics on the total number of papers, the total number of citations, average citations per paper, citations per author, papers per author, and citations per year. In other words, bibliometric methods can be used to ascertain individual performance or to compare relationships between two or more authors or bodies of work. If an academic’s work is well cited, it is likely that he or she has made a significant impact. Where the aim is to develop a set of metrics to study groups of researchers, however, the most relevant measure is currently the Average Relative Citations (ARC), which measure research impact based on how many times an article has been referenced relative to other research in a field. The benefit of bibliometrics is that they provide a quantitative and comparable 8 measure of research impact. It is rarely appropriate, though, to use bibliometrics as the only measure of research impact. Bibliometrics are useful to measure research fields where the means of communicating research is through English-language peer-reviewed journal articles, as is the case in some social sciences. But in the humanities, books and in some instances book chapters tend to be more important, making bibliometrics less valuable. If an academic shows weak citation metrics, it may be the result of little or no impact in a field, but it may also be caused by an otherwise limited or small field to start with, or one who is publishing in languages other than English. Likewise, bibliometrics fail to accommodate for disciplinary and career-length differences and to delineate individual contributions to multi-authored articles. Misclassification of journal articles containing original research into auxiliary categories constitutes another shortcoming of automated bibliometrics; and since different databases include different journals, it is important to consider the type of database being used. Finally, it is important to recognize the four to five year time lag between research being performed, published, cited, and then measured. Examples of bibliometrics being used to measure the impact of HSS research: Excellence for Research in Australia 2015 Submission Guidelines: http://www.arc.gov.au/era/era_2015/era_2015.htm 8 Anne-Wil Harzing and Ron van der Wal, A.W. Harzing, Publish or Perish (2007). URL: http://www.harzing.com/pop.htm. See also “Google Scholar: the democratization of citation analysis?,” Ethics in Science and Environmental Politics 8/1 (2007): 61-73; Harzing and van der Wal, “A Google Scholar h-index for journals: An alternative metric to measure journal impact in economics and business,” Journal of the American Society for Information Science and Technology 60/1 (2009): 41–46. 15 International Comparative Performance of the UK Research Base, 2013: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/263729/bis-13-1297- international-comparative-performance-of-the-UK-research-base-2013.pdf The State of S&T in Canada, 2012: http://www.scienceadvice.ca/uploads/eng/assessments%20and%20publications%20and%20news%20re leases/SandT_II/StateofST2012_fullreportEN.pdf Times Higher Education World University Rankings: http://www.timeshighereducation.co.uk/world- university-rankings/2013-14/subject-ranking/subject/arts-and-humanities/methodology QS World University Rankings: http://www.iu.qs.com/university-rankings/world-university- rankings/?__hstc=238059679.9f3f58d109786dad29026d26e71d88b3.1389386050938.1389386050938. 1389386050938.1&__hssc=238059679.5.1389386050939&__hsfp=3540311731 Leiden Ranking 2013: http://www.leidenranking.com/methodology/datacollection LSE Handbook for social scientists: http://www.lse.ac.uk/government/research/resgroups/LSEPublicPolicy/Docs/LSE_Impact_Handbook_A pril_2011.pdf CAHS 2011: http://www.cahs-acss.ca/wp-content/uploads/2011/09/ROI_FullReport.pdf Downloads from Open Access repositories SUMMARY TABLE: Type of research output measured: All HSS research that can be made available online. Most relevant for: HSS research. Time lag between research output and measureable impact: Immediate. Key advantages: Strong indicator of research impact and individual reputation. Key limitations: No formalized system currently exists. Data is not readily available and can be both unreliable and unrepresentative. Contingent on non-specialist users. Open access publishing offers new opportunities to increase the visibility and citation of HSS research products. And the extent to which downloads from open access repositories can serve as indicators of research impact constitutes one of the new questions in the impact debate. Studies conducted over the past 10 years have generally concluded that free online availability substantially increases a paper's impact. The key, though, is that articles need to be available online for free, not on a subscription or pay-per-use structure. A 2004 study found that peer-reviewed articles that had been deposited in an open-source repository “generated a citation impact up to 400 per cent higher than papers in the same journals that had not been posted in ArXiv,” a Cornell-based open-source server. The same study also 16 found that researchers tended “to post their best articles freely on the web” and that pre-print posting 9 on open access databases also translated into more citations. Using downloads from open access repositories as indicators of research impact, though, can pose challenges. Citation indexes tend to be selective and reflect mainly the better-known journals in the various fields. Authors who publish in an otherwise select list of open-access journals will likely have access to these journals in the first place and will benefit from the increased visibility that these journals already afford. In other words, the open-access repository itself might be less of an indicator of research impact as the journal in which the article is originally published. But the open-access database is a necessary tool and one with comprehensive coverage might enable HSS researchers to better ascertain the value of open-access on citation and research impact. Different open-access databases provide different download statistics and some counting approaches are more reliable than others, while others yet are difficult to compare across databases. Complicating matters are hybrid databases that offer some pay-per-use and some free content. Also problematic is the fact that it remains difficult to determine post-download “usage,” which is to say that, like the problems associated with bibliometrics, 10 downloads alone do not necessarily represent influence or impact. Examples of downloads being used to measure the impact of HSS research: None found yet. Citations in grant applications SUMMARY TABLE: Type of research output measured: Peer-reviewed journal articles and books. Most relevant for: Social science and humanities research. Time lag between research output and measureable impact: 4 years+. Key advantages: Strong indicator of research impact and individual reputation. Key limitations: No formalized system currently exists. Data is not readily available and can be both unreliable and unrepresentative. Most grant applications require applicants to demonstrate their research track record and to situate their proposed research project within a body of scholarship. Through this process, applicants indicate their contribution to a field or discipline and highlight the research of other scholars that has had an impact in a field or discipline. Citation metrics for grant applications in HSS can be used to validate or quantify the claims applicants make about the quality of their work, the extent of their relationships, and the status of their research in a field. Similarly, citations in grant applications to an applicant’s 9 Stevan Harnad and Tim Brody, “Comparing the impact of open access (OA) vs. non-OA articles in the same journals,” D-Lib Magazine (2004): 10/6. URL: http://www.dlib.org/dlib/june04/harnad/06harnad.html as cited in Henk Moed, “Does open access publishing increase citation or download rates?,” Research Trends Issue 28 (May 2012). URL: http://www.researchtrends.com/issue28-may-2012/does-open-access-publishing-increase- citation-or-download-rates/ 10 See Moed and OPCIT: The Open Citation Project, “The effect of open access and downloads ('hits') on citation impact: a bibliography of studies” (2012). URL: http://opcit.eprints.org/oacitation-biblio.html. 17 academic peers can be used as evidence of the national or international reputation in a field of those whose work is highlighted in the application. Citation metrics for grant applications, like reputational surveys, are useful in assessing the extent to which research has set an agenda in a field or constituted a breakthrough. Citations in grants reveal the journals and books that are often considered to be the most influential and/or representative, while also revealing the cross-disciplinary use or application of research beyond immediate areas of interest and expertise. Citations in grant applications are a viable way for applicants to both quantify and qualify the impact of their research and for reviewers to measure it. But citations in grant applications are not usually exhaustive, they do not necessarily confirm intellectual debt, and they can be misleading or misrepresentative as some scholars may choose their references for tactical rather than scholarly reasons. Some grants limit the number of references/citations an applicant may use, which further circumscribes the data. Meanwhile, recent publications may be under-represented, if not excluded altogether, owing to the time lag between publication and citation. Examples of citations in grant applications being used to measure the impact of research: Canada Graduate Scholarships-Master's Program: http://www.nserc-crsng.gc.ca/ResearchPortal-PortailDeRecherche/Instructions-Instructions/CGS_M- BESC_M_eng.asp Fulbright: http://us.fulbrightonline.org/application-tips/academic National Institute of Health: http://grants.nih.gov/grants/oer.htm CAHS 2011: http://www.cahs-acss.ca/wp-content/uploads/2011/09/ROI_FullReport.pdf Acknowledgements SUMMARY TABLE: Type of research output measured: Peer-reviewed journal articles and books. Most relevant for: Social science and humanities research. Time lag between research output and measureable impact: 4 years+ Key advantages: Valuable tool for analyzing relational impacts of acknowledged research, comparable measure of citations. Applicable to books. Key limitations: Data is not readily available and requires costly technical software and coding. Acknowledgements in research publications and books indicate contributions, in the form of hands-on help, financial support, or influence, to a research project. Granting bodies and institutional sponsors often expect expressions of appreciation for research funding in publications. Whereas citations denote specific intellectual debts, acknowledgments can also be qualitative and of a more personal nature. Both manual and automated acknowledgment extraction efforts have shown acknowledgements to be a valuable tool for analyzing the relational impacts of acknowledged research contributions. The sciences 18 11 refer to this process as “mapping knowledge domains.” But to be a viable analytic tool to measure HSS research, cost-effective, easy-to-use automated methods to distinguish substantive expressions of appreciation from obligatory expressions of debt need to be developed. Although not yet in use to assess the impact of HSS research, the computer science community has developed automatic acknowledgment extraction and indexing tools to measure scientific contributions. Their algorithms can be extended to any documents that have acknowledgments, but the code is not designed for nontechnical users. The argument in favour of using acknowledgement metrics, particularly in combination with citation indexing, is that acknowledgement analysis measures not only the research impact of individuals or groups, funding agencies, government, corporate, and university sponsors, but also trends in individual or group research, trends within research communities, as well as trends in institutional or agency sponsorship. Acknowledgements uncover relationships between research stakeholders; they reveal indirect contributions to projects and they provide additional context for research. They are also indicative of the informal collaborative networks which are more common than formal partnerships in some disciplines. There are, however, generational and cultural differences in how scholars acknowledge their intellectual debts, while some monograph publishers actively discourage extensive acknowledgements. To date, the complex coding and cost of automating data 12 extraction and indexing has prohibited the widespread use of acknowledgment metrics. Examples of acknowledgements being used to measure the impact of scientific research: CiteSeer X Beta: http://citeseer.ist.psu.edu/index Database Systems and Logic Programming (DBLP): http://dblp.uni-trier.de Prizes and awards SUMMARY TABLE: Type of research output measured: Lifetime or high profile contributions. Most relevant for: All HSS research, but mainly peer-reviewed journal articles and books and creative works, installations, performances. Time lag between research output and measureable impact: 2-4 years+. Key advantages: Promotes innovation and rewards influential research. Key limitations: Subject to potentially non-expert decision panels and could overshadow otherwise significant research contributions. Prizes either recognize research achievement or promote research innovation. Occasionally they aspire to do both. The role of prizes and shortlisting for prizes as a means to assess research impact is the subject of considerable debate, namely because it remains difficult to quantify the role that prizes and 11 Richard M. Shiffrin and Katy Borner, “Mapping Knowledge Domains,” PNAS 101/1 (2004). URL: http://www.pnas.org/content/101/suppl_1/5183.full. 12 C. Lee Giles and Isaac G. Councill, “Who gets acknowledged: Measuring scientific contributions through automatic acknowledgment indexing,” PNAS 101/51 (2004). URL: http://www.pnas.org/content/101/51/17599.long. 19 shortlists play in shaping a research profile or extending research impact beyond an uptick in popularity or sales. Certain prizes, such as the Nobel, Pulitzer, and MacArthur Fellowships, offer considerable prestige and come with profile-raising media coverage, while smaller prizes, regardless of prestige, may earn the winner little or no attention at all. Prizes may be most helpful in garnering individual and research groups further prizes, grant money, and professional advancement. Prizes may be antidotes to an overdependence on bibliometrics or high-profile journals as measures of research impact when those prizes identify or draw attention to important research regardless of where or how it was published. Prizes can help to establish or identify research agendas and can be used as predictors of whom or which research groups will be effective. Conversely, they provide opportunities to recognize high quality research by new or non-brand-name scholars. General article or book prizes can promote inter- disciplinarity and collaboration. Limitations associated with decision panels can include the following: they may be composed of non-experts; they may be unmotivated, polarized, subject to “group-think,” or pre-occupied with what’s “on trend;” and the prizes themselves may have political or corporate 13 agendas. BookNet Canada tracks book sales, capturing roughly 75% of all trade sales at key times in the season, including awards season, and confirms that awards and the publicity surrounding them can have a significant impact on the title, namely because awards are seen as indicators of quality. The announcement of a prize shortlist, particularly for notable prizes such as the Scotiabank Giller Prize and CBC’s Canada Reads, is enough to spur attention and sales. Book publishers have dubbed this the “Giller Effect” or the “Canada Reads Effect.” But to date there is no metric to quantify or assess the role that 14 prizes play in measuring research impact. Examples of prizes being used to measure the impact of research: SSHRC Impact Awards: http://prezi.com/zxrdimzboxay/sshrc-impact-awards/ Canadian Historical Association Prizes: http://www.cha-shc.ca/english/what-we-do/cha-prizes/cha-prizes.htmlsthash.4PJgzjt4.dpbs Canada Council Prizes (including Killam, Molson, Diefenbaker, and Governor General’s Awards): http://www.canadacouncil.ca/council/grants.aspx Book Net Canada: http://www.booknetcanada.ca 13 Bob Bruner, “What Were They Thinking? And Other Questions for a Prize Committee,” University of Virginia, Darden School of Business Blog (17 October 2013). URL: http://blogs.darden.virginia.edu/deansblog/2013/10/what-were-they-thinking-and-other-questions-for-a-prize- committee/; Rebecca Mann, “Using Google to gauge impact: the Nobel Prize in Economics,” LSE Impact Blog (2012). URL: http://blogs.lse.ac.uk/impactofsocialsciences/2012/10/18/mann-using-google-impact-nobel/; Ian Sample, “Nobel winner declares boycott of top science journals,” The Guardian (9 December 2013). URL: http://www.theguardian.com/science/2013/dec/09/nobel-winner-boycott-science-journals 14 “BookNet Explains the ‘Canada Reads Effect,’” CBC Books (30 November 2010). URL: http://www.cbc.ca/books/canadareads/2010/11/booknet-canada-explains-the-canada-reads-effect.html; Vinay Minon, “The Giller Effect,” Toronto Star (1 November 2013). URL: http://www.thestar.com/entertainment/books/2013/11/01/the_giller_effect.html 20

Advise: Why You Wasting Money in Costly SEO Tools, Use World's Best Free SEO Tool Ubersuggest.