Evaluation of the i10-Index in Plastic Surgery Research and its Correlation with Altmetric Attention Scores and Traditional Author Bibliometrics: An Evaluation of a Single Journal

Abstract

Background Although the Hirsch index (H-index) has become one of the most accepted measures of scholarly output, its limitations have led to the proposition of newer alternative metrics. The i10-index, notable for being easy to calculate and free to access, has potential, given its association with the power and ubiquity of Google. This study aims to evaluate the utility of the i10-index for plastic surgery research by examining its relationship with author bibliometrics and article metrics, including the H-index and Altmetric Attention Score (AAS).

Methods Article metrics were extracted from articles published in the highest impact plastic surgery journal, Plastic and Reconstructive Surgery, over a 2-year period (2017–2019). Senior author bibliometrics, including i10-index and H5-index, were obtained from Web of Science. Correlation analysis was performed using Spearman's rank correlation coefficient (r_s).

Results A total of 1,668 articles were published and 971 included. Senior author i10-index measurements demonstrated moderate correlation with times emailed (r_s = 0.47), and weak correlations with H5-index, total publications, and sum of times cited with and without self-citations. The H5-index correlated very strongly with total publications (r_s = 0.91) and sum of times cited (both r_s = 0.97), moderately with average citations per item (r_s = 0.66) and times emailed (r_s = 0.41), and weakly with number of citations by posts, AAS, and times tweeted.

Conclusions Although the i10 strongly correlates with the H5-index, it fails to prove superior to the H5-index in predicting the impact of specific research studies in the field of plastic surgery.

Keywords

i10-Index - H-Index - bibliometrics - altmetrics - altmetric attention score

Introduction

Evaluation of a researcher's productivity has historically been measured by the most widely accepted bibliometric tool, the Hirsch index (H-index), first reported by Jorge E. Hirsch in 2005.[1] The H-index is defined as the highest number of publications of a researcher that each have been cited at least that many times.[1] For example, in a simplistic form, if a researcher has produced ‘x’ number of publications, which have each been cited ‘x’ number of times, then that author would have an H-index of ‘x’. While the formal H-index describes a researcher's publications over their entire career, it can also be described for a discrete time period. For instance, a researcher's H-index over the past 5 years only, (known as the H5-index) can be a measure of more recent productivity. The H-index combines two traditional measures of research yield: publication number (quantity) and citations (quality), and theoretically evaluates both the academic productivity and influence of a given researcher. Unfortunately, the H-index has come under scrutiny as it can be manipulated by self-citations and does not measure the impact of the research on the public.[2] [3] [4] [5]

Altmetric Attention Score

This has led to the development of newer alternative metrics such as the Altmetric Attention Score (AAS), which uses a proprietary algorithm that calculate the volume of attention received by a research output across numerous online sources (e.g., Twitter, blogs) ([Table 1]). The AAS is commonly presented as the Altmetric donut that uses different colors to represent different sources of attention.[6] Additionally, criticism of the H-index has also revealed that it can be exhausting to calculate, as it requires access to an author's entire bibliography and citation history, which can be difficult to obtain. Many web sites widely accessible to researchers and the public, such as Web of Science and Google, are able to determine this number electronically.[7] [8] Though this would seemingly make calculating the H-index extremely simple, these platforms cannot be fully relied upon as there are discrepancies in the number of citations and publications reported on each platform. The accuracy of the reported H-index found with electronic searches can also be skewed by the presence of authors with identical names.[9]

Table 1 Altmetric attention score default weightings for research outputs

Research output	Default weight
News	8
Blog	5
Policy document (per source)	3
Patent	3
Wikipedia	3
Twitter (tweets and retweets)	1
Peer review (Publons, PubPeer)	1
Weibo (not trackable since 2015, but historical data kept)	1
Google+ (not trackable since 2019, but historical data kept)	1
F1000	1
Syllabi (open syllabus)	1
LinkedIn (not trackable since 2014, but historical data kept)	0.5
Facebook (only a curated list of public pages)	0.25
Reddit	0.25
Pinterest (not trackable since 2013, but historical data kept)	0.25
Q&A (stack overflow)	0.25
YouTube	0.25
Number of Mendeley readers	0
Number of dimensions and Web of Science citations	0

The i10-index

For these reasons, simpler and more convenient bibliometrics have been of recent interest. The i10-index, defined as the number of publications with greater than 10 citations, has been promoted by Google Scholar, which is free and accessible to the general public.[10] [11] Because of the power and ubiquity of this search engine, the potential for the i10-index is extremely high given the sheer number of people who look up articles, topics, and citations via Google. Additionally, the power of the Google search engine also makes it highly likely that the i10-index is more accurate and inclusive of researchers' total outputs than the H-index computed by more limited search engines. Additionally, the i10-index holds promise as a metric in that it is easier to calculate manually from the established PubMed database or other websites if needed,[12] for example, to quickly compare two researchers or institutions.

Relationship between Article Metrics

While there is no doubt that the i10-index is a more convenient bibliometric, it is unclear whether this metric better reflects a researcher's accomplishments than the H-index or AAS. These metrics (AAS and h-index) have been looked at in relationship to each other but there has not been clear data on the i10 index and its potential interplay with these other more established metrics. Previous studies have demonstrated a weak correlation between the Altmetric Attention Score and the senior author's H5-index in the plastic surgery literature.[13] The potential benefits of the i10 index when compared with the h-index are that it might take into account more the impact of research in addition to the output, as it takes into account the number of papers that have been cited by 10 other different papers. Additionally, it is unclear if an author with a high i10-index is predictive of the impact of an individual study or its reach to the general population. However, relationship data and application recommendations for the i10-index is substantially lacking despite its ease of accessibility. This study seeks to correlate the i10-index and the H5-index with various author bibliometrics, article metrics, and the AAS to determine the utility of the i10-index in evaluating the impact of plastic surgery research. The superiority or inferiority of the i10 index as a bibliometric tool, as well as its potential interplay with the more established h-index and AAS metrics remains unknown.

Materials and Methods

Identification of Article and Author Metrics

All journal articles published over the span of 2 years from January 1, 2017 to January 1, 2019 in the plastic surgery journal with the highest impact factor, Plastic and Reconstructive Surgery (PRS), were identified. The journal was chosen due to its early adoption of the AAS for providing article-level metrics, which is easily accessible through the journal website. The years 2017 and 2018 were selected to ensure there was appropriate time (at least 1 year after publication) for articles to be discovered and shared, and for the automated Altmetric algorithm to pick up online attention and calculate the AAS for articles.

The following information was extracted from each article: title, senior author's name and institution, month of publication, and article metrics. The specific article metrics gathered included abstract views, full-text views, PDF downloads, times tweeted, times emailed, percentile of tracked articles of similar age in all journals, percentile of tracked articles of similar age in PRS, number of citations by post, and the Altmetric Attention Score. The senior author was identified as the last author listed. Using the citation report tool in the Web of Science website,[7] senior author bibliometrics, including the 5-year H-index (H5-index), total publications (2014–2018), average citations per item, and the sum of times cited with and without self-citations were obtained for each senior author. Author names were cross-referenced by institution to minimize inaccuracy. The H5-index was chosen for analysis instead of the lifetime H-index to minimize skewing of bibliometric data by authors who had longer academic careers and prevent discrepancies in data due to the relatively recent adoption of the AAS. Additionally, the i10-index of each senior author from years 2014 to 2018 was calculated by sorting senior author publications by number of citations to identify the number of publications with at least ten citations. Articles without AAS data were excluded from this study, while articles with an AAS score of 0 were included as such.

Statistical Analysis

The i10-index, H5-index, and AAS were paired with all other author bibliometrics and article metric extensions for correlation analysis. The Shapiro–Wilk test was used to assess for normality of data, and correlation studies were performed using Spearman's rank correlation coefficient (r_s). A correlation coefficient value (r_s) of less than 0.10 was defined as no correlation, 0.10 to 0.39 as weak correlation, 0.40 to 0.69 as moderate correlation, 0.70 to 0.89 as strong correlation, and 0.90 to 1.00 as very strong correlation.[14] A value of p < 0.05 was defined as statistically significant. All calculations were conducted using the IBM SPSS Statistics for Mac, Version 25 (Armonk, NY: IBM Corp.).

Results

Overall Results

A total of 1,668 articles were published from January 1, 2017 to January 1, 2019. Of these, 971 articles were included for analysis and 697 articles were excluded due to unavailable AAS score data at time of data collection. The i10-index ranged from 0 to 85 with a mean 13.83 ± 16.57, whereas the H5-index ranged from 0 to 26 (6.34 ± 4.49) and the AAS ranged from 1 to 525 (19.13 ± 43.56). The overall characteristics of the included articles are summarized in [Table 2].

Table 2 General characteristics and statistics of article metrics and author bibliometrics from articles in plastic and reconstructive surgery from 2017 to 2019

	Average	Range
Bibliometrics
i10-Index	102.932	1–526
H5-Index	7.763	0–25
Total publications (2014–2018)	62.093	2–227
Sum of times cited	412.890	0–2,420
Sum of times cited without self-citation	375.669	0–2,331
Average citations per item	8.961	0–319
Article Metrics
Altmetric Attention Score	12.246	1–228
Number of citations by posts	18.864	1–113
Full-text views	446.322	37–5,215
Abstract views	332.958	0–6,646
PDF downloads	147.229	14–1,462
Times emailed	1.000	0–45
Times tweeted	12.763	0–72
Percentile of tracked articles of similar age (all journals)	56.449	1–99
Percentile of tracked articles of similar age (PRS)	53.729	1–99

Correlation of i10-index with Senior Author Bibliometrics

[Table 3] demonstrates the correlation of the i10-index and H5-index with the other senior author bibliometrics. There was weak positive correlation between the i10-index and H5-index (r_s = 0.31, p < 0.01). The i10-index also correlated weakly with total publications from 2014 to 2018 (r_s = 0.28, p < 0.01) and sum of times cited with and without self-citations (both r_s = 0.25, p < 0.01). However, there was no correlation between i10-index and average citations per item (r_s = 0.08, p = 0.02). In comparison, when evaluating the H5-index with other senior author bibliometrics, the H5-index correlated very strongly with total publications (r_s = 0.91) and sum of times cited with and without self-citations (both r_s = 0.97). There was moderate correlation with average citations per item (r_s = 0.66). All H5-index bibliometric correlations were statistically significant (p < 0.01) ([Table 3]). Overall, the H5-index correlated more strongly than the i10-index with all measures of senior author bibliometrics.

Table 3 Spearman coefficients and p-values of the i10-Index and H5-Index vs senior author bibliometrics

	Correlation with i10-Index (ρ)	p-Value	Correlation with H5-Index (ρ)	p-Value
Bibliometrics
H5-Index	0.74	<0.01	—	—
Total publications (2014–2018)	0.71	<0.01	0.94	<0.01
Sum of times cited	0.73	<0.01	0.97	<0.01
Sum of times cited without self-citation	0.74	<0.01	0.95	<0.01
Average citations per item	0.44	<0.01	0.68	<0.01

Correlation of i10-index with Article Metrics

The correlations of both major author bibliometrics (the i10-index and the H5-index) with various article metrics, including the AAS, are listed in [Table 4]. There was moderate positive correlation between the i10-index and number of times emailed (r_s = 0.47, p < 0.01). However, the i10-index did not correlate with any other article metrics, including the AAS, number of citations by posts, full-text views, abstract views, PDF downloads, times tweeted, and percentile of tracked articles of similar age in all journals and PRS. The H5-index demonstrated moderate positive correlation with number of times emailed (r_s = 0.41, p < 0.01), and weak correlation with the AAS (r_s = 0.14, p < 0.01), number of citations by posts (r_s = 0.10, p < 0.01), and times tweeted (r_s = 0.10, p < 0.01) ([Table 4]). No correlations were seen with full-text views, abstract views, PDF downloads, and percentile of tracked articles of similar age in all journals and PRS.

Table 4 Spearman coefficients and p-values of the i10-Index and H5-Index vs article metrics

	Correlation with i10-Index (ρ)	p-Value	Correlation with H5-Index (ρ)	p-Value
Article metrics
Altmetric Attention Score (AAS)	0.15	n.s.	0.25	<0.01
Number of citations by posts	0.20	0.03	0.26	0.01
Full text views	−0.13	n.s.	0.01	n.s.
Abstract views	0.05	n.s.	0.23	0.01
PDF downloads	−0.14	n.s.	0.05	n.s.
Times emailed	0.67	<0.01	0.69	<0.01
Times tweeted	0.20	0.03	0.25	0.01
Percentile of tracked articles of similar age (all journals)	0.18	n.s.	0.24	0.01
Percentile of tracked articles of similar age (PRS)	0.17	n.s.	0.23	0.01

Abbreviation: n.s., not significant.

Discussion

Correlation of i10 with H-Index and AAS

Given the popularity of Google as a search engine, the i10-index has been promoted as a simple and easily calculated metric of research output, possibly as a metric to rival the H-index of Altmetric Attention Score (AAS).[15] Despite these advantages, and the fact that Google Scholar (the primary source of the i10-index) is free to access, to date, there have not been any studies that evaluated the utility of the i10-index for plastic surgery research by comparing it to the more investigated H-index or AAS.

Previously, Robinson found a very strong correlation between the H-index and the i10-index in a cohort of 151 general surgeons.[16] This contrasts with weak correlation that we identified in our study (r_s = 0.31, p < 0.01). Because bibliometrics have been found to vary between specialties and the index scores do not exist in isolation,[17] [18] [19] [20] [21] it is critical to evaluate how meaningful these scores are within the greater scientific context. In the field of plastic surgery, the H-index and the i10-index were both found to be strongly correlated with academic rank,[22] [23] whereas in fields of hand surgery and radiology, academic rank correlated better with the H-index.[24] [25] In addition, NIH funding was shown to correlate weakly with the i10-index and not at all with the H-index.[26] Other authors have attempted to combined these two measurements; Kozak proposed an algorithm utilizing both of these indexes, and concluded that indexes can be constructed for a particular evaluation task and purpose, but might not be generalizable to all fields.[27] Additionally, our study did not show any meaningful correlations between the i10-index and the AAS and most other article metrics, suggesting that while it reflects the academic productivity of the author, it is insufficient alone in reflecting the impact of individual studies among the lay public.

Overall Utility of the i10 Index

The lack of correlation is not particularly surprising, as the i10-index is a measure of the author while the AAS is a measure of the article. However, even as an author bibliometric, the H5-index demonstrated correlation with AAS (r_s= 0.14, p < 0.01). Overall, the slightly higher correlation strength with article metrics over the i10-index suggests that the H-index may play a more complimentary role alongside the AAS in evaluating scientific merit in the plastic surgery community. Using bibliometrics (i10-index and H-index) and article metrics (AAS) alongside each other allows for the ability to evaluate both research productivity and public impact, which may more comprehensively assess a researcher's overall influence. These findings are consistent with results from other studies that analyzed metrics of research impact in the field of plastic surgery and similarly concluded that citation-based metrics and article metrics should be utilized in tandem, rather than in isolation.[13] [28] [29]

Overall, the weak correlations of the i10-index and all other measures suggest that the i10-index yields information that is unique to itself as a metric that the H-index and AAS do not. In addition to the differences between how the i10-index and H-index measure impact, there are some external factors that may favor selection for the i10-index in particular circumstances. For example, access to the Web of Science and Scopus are by paid subscriptions,[7] [30] whereas Google Scholar is free.[8] These databases also produce different search and metric results as their coverage, content, and detection method of research outputs vary.[9] [31] [32] [33] [34] Google Scholar provides a broader range of citations than curated databases as it includes all research fields and types of output (e.g., posters, presentations).[31] [34] [35] [36] As a result, the i10-index may be a more suitable metric to utilize when seeking to evaluate all aspects of a researcher's impact, as opposed to only those indexed in databases with scholarly inclusion criteria set by review committees.[4] [37] [38]

Limitations

This study is limited by the focus on a single journal, generalizing the findings reported in this study to other fields should be conducted with caution. Additionally, the i10-index and H-index calculations are based on a 5-year study period rather than an unrestricted time period. This was done to prevent discrepancies in data, as the Altmetric company was founded in 2011,[39] our study demonstrated that the i10-index may be more suitable for evaluating longer-term impact. In addition, given that author bibliometrics such as citations and publications traditionally take much longer to accumulate compared with article metrics, it is unclear what timing would allow for achievement of their full metric scoring potentials and for optimal comparison between the two types of metric systems. Additionally, our study depended on available website databases to search for senior author and bibliometric data. Despite our efforts to cross reference all authors with their associated academic institutions, it is difficult to guarantee that all relevant publications were included during the bibliometric data extraction process.

Conclusion

Scholarly output has typically been measured by citation-based metrics such as the Hirsch index (H-index). Recently, however, other research impact metrics have been of interest to the scientific community, which are easier to calculate or more accessible to the public. While this study demonstrates the H-index to be more predictive of the impact of researchers in plastic surgery compared with the i10-index, it is also evident that it is insufficient to use only one index as each metric system has its unique strengths and limitations.

Conflict of Interest

None declared.

Prior Presentations

This work was previously presented at 2020 Plastic Surgery The Meeting (Virtual).

Authors' Contributions

E.S. and A.H. contributed to data collection, data analysis and manuscript preparation. J.R. contributed to data analysis, manuscript preparation and project oversight.

Ethical Approval

No human or animal subjects were involved in this study. Ethical approval was not required.

• References

• 1 Hirsch JE. An index to quantify an individual's scientific research output. Proc Natl Acad Sci U S A 2005; 102 (46) 16569-16572
• 2 Bartneck C, Kokkelmans S. Detecting h-index manipulation through self-citation analysis. Scientometrics 2011; 87 (01) 85-98
• 3 Bornmann L, Mutz R, Daniel H-D. Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine. J Am Soc Inf Sci Technol 2008; 59 (05) 681-685
• 4 Yong A. Critique of Hirsch's citation index: a combinatorial Fermi problem. Not Am Math Soc 2014; DOI: 10.1090/noti1164.
• 5 Aksnes DW. A macro study of self-citation. Scientometrics 2003; 56: 235-246
• 6 Shearman CP, Hamer JD, Grimshaw G, Crow A. Screening for abdominal aortic aneurysms. BMJ 1992; 305 (6864): 1291-1292
• 7 Sandor A, Modlin IM. A retrospective analysis of 1570 appendiceal carcinoids. Am J Gastroenterol 1998; 93 (03) 422-428
• 8 Deans GT, Spence RA. Neoplastic lesions of the appendix. Br J Surg 1995; 82 (03) 299-306
• 9 Kulkarni AV, Aziz B, Shams I, Busse JW. Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. JAMA 2009; 302 (10) 1092-1096
• 10 Connor J. Google Scholar Citations Open To All. Accessed August 16, 2020 at: https://scholar.googleblog.com/2011/11/google-scholar-citations-open-to-all.html
• 11 Connor J. Google Scholar Citations. Accessed August 16, 2020 at: https://scholar.googleblog.com/2011/07/google-scholar-citations.html
• 12 Berardi RS. Carcinoid tumors of the colon (exclusive of the rectum): review of the literature. Dis Colon Rectum 1972; 15 (05) 383-391
• 13 Shiah E, Heiman AJ, Ricci JA. Analysis of alternative metrics of research impact: a correlation comparison between Altmetric attention scores and traditional bibliometrics among plastic Surgery Research. Plast Reconstr Surg 2020; 146 (05) 664e-670e
• 14 Schober P, Boer C, Schwarte LA. Correlation coefficients: appropriate use and interpretation. Anesth Analg 2018; 126 (05) 1763-1768
• 15 Bornmann L, Mutz R, Hug SE. et al. A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants. J Informetrics 2011; 5 (03) 346-359
• 16 Robinson DBT, Hopkins L, Brown C. et al. Relative value of adapted novel bibliometrics in evaluating surgical academic impact and reach. World J Surg 2019; 43 (04) 967-972
• 17 Svider PF, Pashkova AA, Choudhry Z. et al. Comparison of scholarly impact among surgical specialties: an examination of 2429 academic surgeons. Laryngoscope 2013; 123 (04) 884-889
• 18 Schreiber WE, Giustini DM. Measuring scientific impact with the h-index: a primer for pathologists. Am J Clin Pathol 2019; 151 (03) 286-291
• 19 Nowak JK, Lubarski K, Kowalik LM, Walkowiak J. H-index in medicine is driven by original research. Croat Med J 2018; 59 (01) 25-32
• 20 Malesios CC, Psarakis S. Comparison of the h-index for different fields of research using bootstrap methodology. Qual Quant 2014; 48: 521-545
• 21 Alonso S, Cabrerizo FJ, Herrera-Viedma E. et al. h-Index: a review focused in its variants, computation and standardization for different scientific fields. J Informetrics 2009; 3: 273-289
• 22 Gast KM, Kuzon Jr WM, Waljee JF. Bibliometric indices and academic promotion within plastic surgery. Plast Reconstr Surg 2014; 134 (05) 838e-844e
• 23 Susarla SM, Lopez J, Swanson EW. et al. Are quantitative measures of academic productivity correlated with academic rank in plastic surgery? A national study. Plast Reconstr Surg 2015; 136 (03) 613-621
• 24 Lopez J, Susarla SM, Swanson EW, Calotta N, Lifchez SD. The association of the H-Index and academic rank among full-time academic hand surgeons affiliated with fellowship programs. J Hand Surg Am 2015; 40 (07) 1434-1441
• 25 Jiang A, Ginocchio LA, Rosenkrantz AB. Associations between academic rank and advanced bibliometric indices among United States academic radiologists. Acad Radiol 2016; 23 (12) 1568-1572
• 26 Rosenkrantz AB, Jiang A. Associations between NIH funding and advanced bibliometric indices among radiological investigators. Acad Radiol 2016; 23 (06) 669-674
• 27 Kozak M, Bornmann L. A new family of cumulative indexes for measuring scientific performance. PLoS One 2012; 7 (10) e47679
• 28 Ruan QZ, Chen AD, Cohen JB, Singhal D, Lin SJ, Lee BT. Alternative metrics of scholarly output: the relationship among Altmetric acore, Mendeley reader score, citations, and downloads in plastic and reconstructive surgery. Plast Reconstr Surg 2018; 141 (03) 801-809
• 29 Asaad M, Howell SM, Rajesh A, Meaike J, Tran NV. Altmetrics in plastic surgery journals: does it correlate with citation count?. Aesthet Surg J 2020; 40 (11) NP628-NP635
• 30 Godwin II JD. Carcinoid tumors. An analysis of 2,837 cases. Cancer 1975; 36 (02) 560-569
• 31 Harzing A-W, Alakangas S. Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison. Scientometrics 2016; 106: 787-804
• 32 Bakkalbasi N, Bauer K, Glover J, Wang L. Three options for citation tracking: Google Scholar, Scopus and Web of Science. Biomed Digit Libr 2006; 3: 7
• 33 Bar-Ilan J. Which h-index?—A comparison of WoS, Scopus and Google Scholar. Scientometrics 2008; 74: 257-271
• 34 Harzing A-W, Wal R. Google Scholar as a New Source for Citation Analysis. Ethics Sci Environ Polit 2008; 8: 61-73
• 35 Halevi G, Moed H, Bar-Ilan J. Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation - Review of the Literature. J Informetrics 2017; 11: 823-834
• 36 Teixeira da Silva JA, Dobránszki J. Multiple versions of the h-index: cautionary use for formal academic purposes. Scientometrics 2018; 115: 1107-1113
• 37 Waltman L, van Eck N. The inconsistency of the h-index. J Am Soc Inf Sci Technol 2012; 63: 406-415
• 38 Barnes C. The h-index debate: an introduction for librarians. J Acad Librariansh 2017; 43: 487-494
• 39 Pawlik TM, Shah S, Eckhauser FE. Carcinoid tumor of the biliary tract: treating a rare cause of bile duct obstruction. Am Surg 2003; 69 (02) 98-101

Publication History

Article published online:
02 February 2023