Keywords
i10-Index - H-Index - bibliometrics - altmetrics - altmetric attention score
Introduction
Evaluation of a researcher's productivity has historically been measured by the most widely accepted bibliometric tool, the Hirsch index (H-index), first reported by Jorge E. Hirsch in 2005.[1] The H-index is defined as the highest number of publications of a researcher that each have been cited at least that many times.[1] For example, in a simplistic form, if a researcher has produced ‘x’ number of publications, which have each been cited ‘x’ number of times, then that author would have an H-index of ‘x’. While the formal H-index describes a researcher's publications over their entire career, it can also be described for a discrete time period. For instance, a researcher's H-index over the past 5 years only, (known as the H5-index) can be a measure of more recent productivity. The H-index combines two traditional measures of research yield: publication number (quantity) and citations (quality), and theoretically evaluates both the academic productivity and influence of a given researcher. Unfortunately, the H-index has come under scrutiny as it can be manipulated by self-citations and does not measure the impact of the research on the public.[2]
[3]
[4]
[5]
Altmetric Attention Score
This has led to the development of newer alternative metrics such as the Altmetric Attention Score (AAS), which uses a proprietary algorithm that calculate the volume of attention received by a research output across numerous online sources (e.g., Twitter, blogs) ([Table 1]). The AAS is commonly presented as the Altmetric donut that uses different colors to represent different sources of attention.[6] Additionally, criticism of the H-index has also revealed that it can be exhausting to calculate, as it requires access to an author's entire bibliography and citation history, which can be difficult to obtain. Many web sites widely accessible to researchers and the public, such as Web of Science and Google, are able to determine this number electronically.[7]
[8] Though this would seemingly make calculating the H-index extremely simple, these platforms cannot be fully relied upon as there are discrepancies in the number of citations and publications reported on each platform. The accuracy of the reported H-index found with electronic searches can also be skewed by the presence of authors with identical names.[9]
Table 1
Altmetric attention score default weightings for research outputs
|
Research output
|
Default weight
|
|
News
|
8
|
|
Blog
|
5
|
|
Policy document (per source)
|
3
|
|
Patent
|
3
|
|
Wikipedia
|
3
|
|
Twitter (tweets and retweets)
|
1
|
|
Peer review (Publons, PubPeer)
|
1
|
|
Weibo (not trackable since 2015, but historical data kept)
|
1
|
|
Google+ (not trackable since 2019, but historical data kept)
|
1
|
|
F1000
|
1
|
|
Syllabi (open syllabus)
|
1
|
|
LinkedIn (not trackable since 2014, but historical data kept)
|
0.5
|
|
Facebook (only a curated list of public pages)
|
0.25
|
|
Reddit
|
0.25
|
|
Pinterest (not trackable since 2013, but historical data kept)
|
0.25
|
|
Q&A (stack overflow)
|
0.25
|
|
YouTube
|
0.25
|
|
Number of Mendeley readers
|
0
|
|
Number of dimensions and Web of Science citations
|
0
|
The i10-index
For these reasons, simpler and more convenient bibliometrics have been of recent interest. The i10-index, defined as the number of publications with greater than 10 citations, has been promoted by Google Scholar, which is free and accessible to the general public.[10]
[11] Because of the power and ubiquity of this search engine, the potential for the i10-index is extremely high given the sheer number of people who look up articles, topics, and citations via Google. Additionally, the power of the Google search engine also makes it highly likely that the i10-index is more accurate and inclusive of researchers' total outputs than the H-index computed by more limited search engines. Additionally, the i10-index holds promise as a metric in that it is easier to calculate manually from the established PubMed database or other websites if needed,[12] for example, to quickly compare two researchers or institutions.
Relationship between Article Metrics
While there is no doubt that the i10-index is a more convenient bibliometric, it is unclear whether this metric better reflects a researcher's accomplishments than the H-index or AAS. These metrics (AAS and h-index) have been looked at in relationship to each other but there has not been clear data on the i10 index and its potential interplay with these other more established metrics. Previous studies have demonstrated a weak correlation between the Altmetric Attention Score and the senior author's H5-index in the plastic surgery literature.[13] The potential benefits of the i10 index when compared with the h-index are that it might take into account more the impact of research in addition to the output, as it takes into account the number of papers that have been cited by 10 other different papers. Additionally, it is unclear if an author with a high i10-index is predictive of the impact of an individual study or its reach to the general population. However, relationship data and application recommendations for the i10-index is substantially lacking despite its ease of accessibility. This study seeks to correlate the i10-index and the H5-index with various author bibliometrics, article metrics, and the AAS to determine the utility of the i10-index in evaluating the impact of plastic surgery research. The superiority or inferiority of the i10 index as a bibliometric tool, as well as its potential interplay with the more established h-index and AAS metrics remains unknown.
Materials and Methods
Identification of Article and Author Metrics
All journal articles published over the span of 2 years from January 1, 2017 to January 1, 2019 in the plastic surgery journal with the highest impact factor, Plastic and Reconstructive Surgery (PRS), were identified. The journal was chosen due to its early adoption of the AAS for providing article-level metrics, which is easily accessible through the journal website. The years 2017 and 2018 were selected to ensure there was appropriate time (at least 1 year after publication) for articles to be discovered and shared, and for the automated Altmetric algorithm to pick up online attention and calculate the AAS for articles.
The following information was extracted from each article: title, senior author's name and institution, month of publication, and article metrics. The specific article metrics gathered included abstract views, full-text views, PDF downloads, times tweeted, times emailed, percentile of tracked articles of similar age in all journals, percentile of tracked articles of similar age in PRS, number of citations by post, and the Altmetric Attention Score. The senior author was identified as the last author listed. Using the citation report tool in the Web of Science website,[7] senior author bibliometrics, including the 5-year H-index (H5-index), total publications (2014–2018), average citations per item, and the sum of times cited with and without self-citations were obtained for each senior author. Author names were cross-referenced by institution to minimize inaccuracy. The H5-index was chosen for analysis instead of the lifetime H-index to minimize skewing of bibliometric data by authors who had longer academic careers and prevent discrepancies in data due to the relatively recent adoption of the AAS. Additionally, the i10-index of each senior author from years 2014 to 2018 was calculated by sorting senior author publications by number of citations to identify the number of publications with at least ten citations. Articles without AAS data were excluded from this study, while articles with an AAS score of 0 were included as such.
Statistical Analysis
The i10-index, H5-index, and AAS were paired with all other author bibliometrics and article metric extensions for correlation analysis. The Shapiro–Wilk test was used to assess for normality of data, and correlation studies were performed using Spearman's rank correlation coefficient (rs). A correlation coefficient value (rs) of less than 0.10 was defined as no correlation, 0.10 to 0.39 as weak correlation, 0.40 to 0.69 as moderate correlation, 0.70 to 0.89 as strong correlation, and 0.90 to 1.00 as very strong correlation.[14] A value of p < 0.05 was defined as statistically significant. All calculations were conducted using the IBM SPSS Statistics for Mac, Version 25 (Armonk, NY: IBM Corp.).
Results
Overall Results
A total of 1,668 articles were published from January 1, 2017 to January 1, 2019. Of these, 971 articles were included for analysis and 697 articles were excluded due to unavailable AAS score data at time of data collection. The i10-index ranged from 0 to 85 with a mean 13.83 ± 16.57, whereas the H5-index ranged from 0 to 26 (6.34 ± 4.49) and the AAS ranged from 1 to 525 (19.13 ± 43.56). The overall characteristics of the included articles are summarized in [Table 2].
Table 2
General characteristics and statistics of article metrics and author bibliometrics from articles in plastic and reconstructive surgery from 2017 to 2019
|
Average
|
Range
|
|
Bibliometrics
|
|
i10-Index
|
102.932
|
1–526
|
|
H5-Index
|
7.763
|
0–25
|
|
Total publications (2014–2018)
|
62.093
|
2–227
|
|
Sum of times cited
|
412.890
|
0–2,420
|
|
Sum of times cited without self-citation
|
375.669
|
0–2,331
|
|
Average citations per item
|
8.961
|
0–319
|
|
Article Metrics
|
|
Altmetric Attention Score
|
12.246
|
1–228
|
|
Number of citations by posts
|
18.864
|
1–113
|
|
Full-text views
|
446.322
|
37–5,215
|
|
Abstract views
|
332.958
|
0–6,646
|
|
PDF downloads
|
147.229
|
14–1,462
|
|
Times emailed
|
1.000
|
0–45
|
|
Times tweeted
|
12.763
|
0–72
|
|
Percentile of tracked articles of similar age (all journals)
|
56.449
|
1–99
|
|
Percentile of tracked articles of similar age (PRS)
|
53.729
|
1–99
|
Correlation of i10-index with Senior Author Bibliometrics
[Table 3] demonstrates the correlation of the i10-index and H5-index with the other senior author bibliometrics. There was weak positive correlation between the i10-index and H5-index (rs = 0.31, p < 0.01). The i10-index also correlated weakly with total publications from 2014 to 2018 (rs = 0.28, p < 0.01) and sum of times cited with and without self-citations (both rs = 0.25, p < 0.01). However, there was no correlation between i10-index and average citations per item (rs = 0.08, p = 0.02). In comparison, when evaluating the H5-index with other senior author bibliometrics, the H5-index correlated very strongly with total publications (rs = 0.91) and sum of times cited with and without self-citations (both rs = 0.97). There was moderate correlation with average citations per item (rs = 0.66). All H5-index bibliometric correlations were statistically significant (p < 0.01) ([Table 3]). Overall, the H5-index correlated more strongly than the i10-index with all measures of senior author bibliometrics.
Table 3
Spearman coefficients and p-values of the i10-Index and H5-Index vs senior author bibliometrics
|
Correlation with i10-Index (ρ)
|
p-Value
|
Correlation with H5-Index (ρ)
|
p-Value
|
|
Bibliometrics
|
|
H5-Index
|
0.74
|
<0.01
|
—
|
—
|
|
Total publications (2014–2018)
|
0.71
|
<0.01
|
0.94
|
<0.01
|
|
Sum of times cited
|
0.73
|
<0.01
|
0.97
|
<0.01
|
|
Sum of times cited without self-citation
|
0.74
|
<0.01
|
0.95
|
<0.01
|
|
Average citations per item
|
0.44
|
<0.01
|
0.68
|
<0.01
|
Correlation of i10-index with Article Metrics
The correlations of both major author bibliometrics (the i10-index and the H5-index) with various article metrics, including the AAS, are listed in [Table 4]. There was moderate positive correlation between the i10-index and number of times emailed (rs = 0.47, p < 0.01). However, the i10-index did not correlate with any other article metrics, including the AAS, number of citations by posts, full-text views, abstract views, PDF downloads, times tweeted, and percentile of tracked articles of similar age in all journals and PRS. The H5-index demonstrated moderate positive correlation with number of times emailed (rs = 0.41, p < 0.01), and weak correlation with the AAS (rs = 0.14, p < 0.01), number of citations by posts (rs = 0.10, p < 0.01), and times tweeted (rs = 0.10, p < 0.01) ([Table 4]). No correlations were seen with full-text views, abstract views, PDF downloads, and percentile of tracked articles of similar age in all journals and PRS.
Table 4
Spearman coefficients and p-values of the i10-Index and H5-Index vs article metrics
|
Correlation with i10-Index (ρ)
|
p-Value
|
Correlation with H5-Index (ρ)
|
p-Value
|
|
Article metrics
|
|
Altmetric Attention Score (AAS)
|
0.15
|
n.s.
|
0.25
|
<0.01
|
|
Number of citations by posts
|
0.20
|
0.03
|
0.26
|
0.01
|
|
Full text views
|
−0.13
|
n.s.
|
0.01
|
n.s.
|
|
Abstract views
|
0.05
|
n.s.
|
0.23
|
0.01
|
|
PDF downloads
|
−0.14
|
n.s.
|
0.05
|
n.s.
|
|
Times emailed
|
0.67
|
<0.01
|
0.69
|
<0.01
|
|
Times tweeted
|
0.20
|
0.03
|
0.25
|
0.01
|
|
Percentile of tracked articles of similar age (all journals)
|
0.18
|
n.s.
|
0.24
|
0.01
|
|
Percentile of tracked articles of similar age (PRS)
|
0.17
|
n.s.
|
0.23
|
0.01
|
Abbreviation: n.s., not significant.
Discussion
Correlation of i10 with H-Index and AAS
Given the popularity of Google as a search engine, the i10-index has been promoted as a simple and easily calculated metric of research output, possibly as a metric to rival the H-index of Altmetric Attention Score (AAS).[15] Despite these advantages, and the fact that Google Scholar (the primary source of the i10-index) is free to access, to date, there have not been any studies that evaluated the utility of the i10-index for plastic surgery research by comparing it to the more investigated H-index or AAS.
Previously, Robinson found a very strong correlation between the H-index and the i10-index in a cohort of 151 general surgeons.[16] This contrasts with weak correlation that we identified in our study (rs = 0.31, p < 0.01). Because bibliometrics have been found to vary between specialties and the index scores do not exist in isolation,[17]
[18]
[19]
[20]
[21] it is critical to evaluate how meaningful these scores are within the greater scientific context. In the field of plastic surgery, the H-index and the i10-index were both found to be strongly correlated with academic rank,[22]
[23] whereas in fields of hand surgery and radiology, academic rank correlated better with the H-index.[24]
[25] In addition, NIH funding was shown to correlate weakly with the i10-index and not at all with the H-index.[26] Other authors have attempted to combined these two measurements; Kozak proposed an algorithm utilizing both of these indexes, and concluded that indexes can be constructed for a particular evaluation task and purpose, but might not be generalizable to all fields.[27] Additionally, our study did not show any meaningful correlations between the i10-index and the AAS and most other article metrics, suggesting that while it reflects the academic productivity of the author, it is insufficient alone in reflecting the impact of individual studies among the lay public.
Overall Utility of the i10 Index
The lack of correlation is not particularly surprising, as the i10-index is a measure of the author while the AAS is a measure of the article. However, even as an author bibliometric, the H5-index demonstrated correlation with AAS (rs= 0.14, p < 0.01). Overall, the slightly higher correlation strength with article metrics over the i10-index suggests that the H-index may play a more complimentary role alongside the AAS in evaluating scientific merit in the plastic surgery community. Using bibliometrics (i10-index and H-index) and article metrics (AAS) alongside each other allows for the ability to evaluate both research productivity and public impact, which may more comprehensively assess a researcher's overall influence. These findings are consistent with results from other studies that analyzed metrics of research impact in the field of plastic surgery and similarly concluded that citation-based metrics and article metrics should be utilized in tandem, rather than in isolation.[13]
[28]
[29]
Overall, the weak correlations of the i10-index and all other measures suggest that the i10-index yields information that is unique to itself as a metric that the H-index and AAS do not. In addition to the differences between how the i10-index and H-index measure impact, there are some external factors that may favor selection for the i10-index in particular circumstances. For example, access to the Web of Science and Scopus are by paid subscriptions,[7]
[30] whereas Google Scholar is free.[8] These databases also produce different search and metric results as their coverage, content, and detection method of research outputs vary.[9]
[31]
[32]
[33]
[34] Google Scholar provides a broader range of citations than curated databases as it includes all research fields and types of output (e.g., posters, presentations).[31]
[34]
[35]
[36] As a result, the i10-index may be a more suitable metric to utilize when seeking to evaluate all aspects of a researcher's impact, as opposed to only those indexed in databases with scholarly inclusion criteria set by review committees.[4]
[37]
[38]
Limitations
This study is limited by the focus on a single journal, generalizing the findings reported in this study to other fields should be conducted with caution. Additionally, the i10-index and H-index calculations are based on a 5-year study period rather than an unrestricted time period. This was done to prevent discrepancies in data, as the Altmetric company was founded in 2011,[39] our study demonstrated that the i10-index may be more suitable for evaluating longer-term impact. In addition, given that author bibliometrics such as citations and publications traditionally take much longer to accumulate compared with article metrics, it is unclear what timing would allow for achievement of their full metric scoring potentials and for optimal comparison between the two types of metric systems. Additionally, our study depended on available website databases to search for senior author and bibliometric data. Despite our efforts to cross reference all authors with their associated academic institutions, it is difficult to guarantee that all relevant publications were included during the bibliometric data extraction process.
Conclusion
Scholarly output has typically been measured by citation-based metrics such as the Hirsch index (H-index). Recently, however, other research impact metrics have been of interest to the scientific community, which are easier to calculate or more accessible to the public. While this study demonstrates the H-index to be more predictive of the impact of researchers in plastic surgery compared with the i10-index, it is also evident that it is insufficient to use only one index as each metric system has its unique strengths and limitations.
