Inpatient Communication Networks: Leveraging Secure Text-Messaging Platforms to Gain Insight into Inpatient Communication Systems

Keywords

hospital communication systems - interdisciplinary communication - text messaging - hospital information systems

Background and Significance

Hospitals are increasingly complex entities comprising health care teams, working in parallel. The team's goal of providing excellent clinical care to patients is highly dependent upon each team member's ability to communicate efficiently and effectively.[1] Unfortunately, communication proves more difficult as teams and facilities grow in size and complexity. Unsurprisingly, faulty communication is frequently cited as contributing to medical errors in the health care setting.[2] [3] [4] [5] These errors have real effects not only on the clinical outcomes of care and satisfaction of patients but also impact quality of care, patient safety, and now carry financial repercussions for hospitals and health care organizations.[6] Currently, there exist significant and persistent barriers to connecting providers in complex inpatient systems quickly and reliably meaning communication remains a central problem in improving in health care systems.

Paging systems have served as a long-standing solution to this problem but are reliant on ever-aging protocols and hardware architecture. As a result, these systems have several critical shortcomings when applied to the field of health care where high-reliability and closed-loop communication are paramount. For this reason, paging systems are being phased out in favor of secure text-messaging platforms that offer fuller message context, read-status notification, and the framework for more effective, closed-loop communication.[7] [8] Many vendors offer competing but ultimately similar products that allow users to leverage Wi-Fi or cellular networks to log into hospital-owned or personal mobile devices and exchange messages or media and place calls between users logged into the same institutional ecosystem (i.e., Halo Communications, Cincinnati, Ohio, United States; TigerConnect, Santa Monica, California, United States; DrFirst, Rockville, Maryland, United States; Voalte Inc, Sarasota, Florida, United States; Epic Systems, Verona, Wisconsin, United States). Each product uses various approaches to ensure that these exchanges are Health Insurance Portability and Accountability Act compliant, relying on some combination of application and/or network-driven encryption.[9] While it is clear that these new platforms offer advantages over legacy paging systems, there is also emerging evidence that secure text-messaging systems are subject to new and different vulnerabilities such as silent failures and unrecognized user overload, especially when automated systems are leveraged to generate notifications to users.[10] [11]

The risks of introducing automated systems in the field of medicine, particularly when user notifications are concerned, have been most broadly examined in the context of computerized physician order entry systems where studies demonstrate that providers override drug safety alerts as a result of apparent “alarm fatigue.”[12] [13] There is also emerging consensus that this concept applies more broadly to hospital environments, that is, the overall burden of alarms presented to providers desensitizes them to genuinely critical or actionable notifications and has the potential to lead to medical errors and harm.[14] [15] [16] [17]

Secure text-messaging serves as yet another potential source of distraction for health care providers and the introduction of trigger-based or other automated notification systems into these communication platforms only exacerbates this risk in the absence of monitoring and surveillance.[18] [19] The current paucity of tools and methods to monitor secure-text messaging platforms in the health care setting means that we have introduced yet another distracting technology which can be inferentially correlated to an increased risk of patient safety-related issues without the capacity to study it effectively. Most work evaluating inpatient communication systems has relied on qualitative and workflow analysis methods to understand actors and systems involved, with few efforts rigorously examining quantitative aspects of these systems.[3] [20] [21] [22] Yet, as these secure-text messaging solutions mature, they are increasingly data-rich and lend themselves to other approaches for developing insight and understanding of inpatient communication systems. In particular, the fields of graph theory and network science provide researchers with conceptual models and methods to study interconnected systems. Historically, graph theory and network science have been leveraged to provide insight into systems such as the world-wide-web, molecular networks, and scholarly citation patterns.[23] [24] [25] [26] Applied to secure text-messaging data, these frameworks allow investigators to examine hospital communication networks in ways that may offer insight into the nature of communication structures, user behavior, and even the network's ability to tolerate disruption.

This project was conceived to build a local framework for understanding our system's inpatient hospital communication network. Our group first created a robust and flexible data pipeline to clean and transform secure text-messaging database data into network models. These models enable study of communication networks using network-specific analysis to (1) identify the characteristics of the inpatient communication network, (2) understand key users and roles within the communication network, and (3) help identify vulnerable users and populations and understand the impact of emerging automated messaging systems on the communication network of a hospital system.

Methods

Cincinnati Children's Hospital Medical Center (CCHMC) is an academic pediatric medical center with 628 inpatient beds. Inpatient clinical work at CCHMC is largely unit-based, meaning that patients admitted to a subspecialty service reside primarily on that subspecialty's home unit. Health providers staffing each team are heterogeneous and might include resident or mid-level providers (nurse practitioner or physician assistant [PA]) in addition to supervising attending physicians. Overseeing hospital-wide clinical operations is a hierarchy of physician and nursing leadership with the latter assuming responsibility for hospital flow (admissions, transfers, discharges, bed capacity, etc.) and active mitigation of issues that arise within units. The local secure text-messaging environment is provided by Voalte (Voalte, Inc.) and users message others via mobile devices, desktop, or laptop computers running the VoalteOne application or personal devices running the VoalteMe application.

All inpatient nursing and ancillary staff are granted access to VoalteOne or VoalteMe. Physician and mid-level providers participate in messaging through the VoalteMe application running on personal devices. Not all physicians and mid-level providers have access to the system with penetrance varied across the medical center depending on divisional and personal preferences. There is no institution-wide mandate for faculty or staff to enroll in the messaging system.

There is no institutional protocol for exchanging messages with the Voalte system. Communication on the system mirrors previous institutional standards of nursing and ancillary bedside staff interacting primarily with frontline providers on medical and surgical teams (i.e., residents and or mid-level providers) with escalation to fellows and attendings as warranted by the clinical scenario. While messages sent in the system include a read receipt, there are no requirements or standards for response.

Messages sent via the Voalte ecosystem are temporarily stored within a rolling 30-day window on a central enterprise server administered by Voalte, Inc. A query to return individual messaging data including sending user, receiving user, and message timestamp was run by an information services analyst on CCHMC's messaging database and the resulting data set was provided to the study team. Notably, data provided for the study was free of any message content. The reasoning for this decision was twofold: (1) message content or other additional protected health information was not relevant to this project, and (2) given the exploratory nature of this work, maintaining privacy and security for health care providers, patients, and the system was deemed paramount.

Using the Pandas Python library, quantitative analysis of the message logs was performed ([Fig. 1]).[27] [28] [29] A network model was then built using the NetworkX library for Python with each user identity (ID) representing a node and a message between two users forming an edge ([Fig. 1]).[30] For this initial exploratory work, an undirected, unweighted network model, an undirected weighted network model, and a directed network model were utilized. These models were leveraged to generate quantitative data about the network. The models were further utilized to generate network-specific metrics to grant insight into role definition and characteristics. Metrics generated examined the number of connections associated with each user (degrees distribution), the relative “importance” of a user within the communication network (page rank), and the ingoing (i.e., hubs) and outgoing (i.e., authorities) connections from a given user in the network. Finally, NetworkX, Matplotlib, and Seaborn libraries were utilized to produce visualizations of quantitative data.[31] [32]

Fig. 1 Illustration demonstrating analysis pipeline, translation from message log data into network model.

Results

The data retrieved from the Voalte system included message sent among 4,442 users from 00:00 on March 30, 2018 to 20:38 on April 22, 2018. Over the course of this timeframe, 4,327 unique senders generated 499,999 messages to 4,413 unique recipients. The bulk of sender/receiver pairs exchanged well under 50 messages with the most active pair of users sending and receiving 356 and 452 messages, respectively, to one another ([Fig. 2]).

Fig. 2 Kernel density estimate of messages per unique sender/receiver combinations in the available messaging data. Red ticks on the x-axis represent unique sender/receiver pairs and demonstrate outliers in the distribution of pairs.

The bulk of individual users sent and received fewer than 250 messages over the time period ([Fig. 3]). However, there were clear outliers to this trend with 18 and 14 users sending and receiving over 1,000 messages, respectively, over the time period. Three users stood out even from this group of outliers sending and receiving 2,230/2,853, 2,519/3,081, and 2,710/3,283, respectively. A linear regression was performed to investigate the correlation between number of messages sent and number of messages received per user. The model demonstrated a Pearson's correlation coefficient of 0.95 and p-value of < 0.001. The slope of the model was 0.935 ([Fig. 3]).

Fig. 3 Scatter plot of messages sent and received per user with overlying linear regression model. Both axes are accompanied by a histogram illustrating the distribution of nodes (or users) along the scale of messages sent and received.

When translated into a network model by assigning each sender and receiver as a node and a message between the two as an edge, the resulting graphs had 4,442 nodes, or unique users sending and/or receiving messages, and 59,913 edges, or messaging connections between each user in the network. There were three distinct, or unconnected, networks identified in the model. One large, highly interlinked network that contained the vast majority of users (4,433 nodes/59,899 edges) and two unconnected networks comprising a smaller number of users that interacted only within the smaller group (7 nodes/13 edges and 2 nodes/1 edge, respectively).

When plotted on log scale, the degree distribution of the fraction of users or nodes indicates the distribution is heavy-tailed and that, while there are some users who send and receive messages in high volumes, most users send and receive very few in our inpatient system ([Fig. 4]).

Fig. 4 Log scale plot of degrees distribution for each node (or user) within the network.

[Table 1] shows the top 10 users, masked by provider role, ranked by three metrics: page rank, hub score, and authority score.

Table 1 Individual's user type ranked by network-specific metrics

Rank	Page rank (user roles, value)	Hub score (user roles, value)	Authority score (user roles, value)
1	Generic Resident Login, 0.00207	Generic RN Login, 0.00255	MPS, 0.00245
2	Generic Resident Login, 0.00179	MPS, 0.00254	Generic RN Login, 0.00239
3	Generic Resident Login, 0.00163	MPS, 0.00249	MPS, 0.00235
4	MPS, 0.00124	HUC, 0.00229	RN, 0.00222
5	HUC, 0.00123	RN, 0.00229	MPS, 0.00213
6	Generic RN Login, 0.00119	HUC, 0.00226	MPS, 0.00213
7	RN, 0.00107	MPS, 0.0222	MPS, 0.00205
8	Generic RN Login, 0.00105	MPS, 0.00217	MPS, 0.00204
9	Generic APRN Login, 0.00105	RN, 0.00209	RN, 0.00198
10	MPS, 0.00104	MPS, 0.00208	Generic Resident Login, 0.00186

Abbreviations: HUC, health unit coordinator; MPS, manager of patient services; PICU, pediatric intensive care unit; RN, registered nurse.

Note: Generic denotes single login and device that multiple users of the same role use on a rolling basis (i.e., PICU team resident device handed from one provider to the next at shift change).

Discussion

The network model of secure text-messaging data reveals the complexity of an inpatient communication network at a large academic medical center even though the data spans less than a month's time. [Fig. 3] demonstrates the user-specific activity with the vast majority of individuals over the course of the timeframe studied exchanging a small number of messages with other users. However, there were a clear cohort of outliers. The identities of these outliers fell into three main thematic buckets: (1) generic users (residents, charge registered nurses) residing at the center of busy acute care areas where multiple users utilize the same login on a continual basis, (2) manager of patient services (MPS) who are nursing personnel responsible for coordinating patient flow, nursing staffing, and situational awareness throughout the inpatient wards, and (3) health unit coordinators (HUCs) who serve as unit-based managers of flow and communication. The group of generic users presents both an interesting dilemma and potential insight. The artificial grouping of multiple users through the generic login means that these “outliers” are not truly individuals within the communication system but multiple individuals sharing a device or login. Thus, the metrics such as aggregate messages are not comparable to other true individual users. However, the aggregate data and network metrics may be more representative of the role these individuals fill when utilizing these generic logins. The combining of multiple users to represent a role may, in fact, be a more appropriate approach to answer some questions, particularly when addressing unit- or role-centric workflows or messaging volumes and patterns. Perhaps most importantly, unlike our MPS and HUC roles, these generic users were not assigned to roles that align with a traditional “dispatch” role where an individual's sole responsibility is to serve as a hub for communication and manage multiple incoming and outgoing communications. Instead, these generic roles occupy providers on inpatient units responsible for patient care tasks including emergency response. The high volume of incoming and outgoing messages means these providers are burdened with significant messaging responsibilities in addition to their patient-oriented duties. This phenomenon begs the question: how can we unburden these providers from nonessential messages and let them perform their critical duties at the bedside?

The slope of our linear regression model indicates that there is a nearly 1:1 correlation between messages sent and received. Further deviation from a slope of 1 might indicate that users are receiving more system-wide or broadcasted messages from system administrators, more messages from automated systems, or that messaging/communication patterns have shifted to more “FYI (for your information)” messages or less frequent user reply to received messages. Our current data does not reflect nodes that are responsible for asymmetric volumes of outgoing messages or large volumes over very short periods of time. The emergence of such nodal behavior would indicate the implementation of increasingly automated messaging nodes in our communication system. Surveillance should continue for this behavior given the risks associated with overloading users with automated messages.

Network analysis of this data reveals a dense, highly connected central network organized around key users described above with two outlying, disconnected networks. Most users within the larger, central network were connected via message to fewer than 50 other users throughout the time period of the study, but again the outliers (those connected to 150 or more users) were notable for the roles highlighted above. The degree distribution of the network resembles that of a scale-free network, or a network with many nodes with few links and few nodes with many links, which helps us understand the nature and function of our institution's inpatient communication network. This confirms a prior suspicion that the communication network is, like our unit-based structure, dependent on highly centralized actors that serve as hubs within clinical units and operational efforts. While the scale-free nature of our communication network is highly efficient and centralized, this also renders it particularly vulnerable to disruption should any of the critical hubs lose connectivity, become task-fixated, or otherwise incapacitated. This knowledge can be applied to our current understanding of users and specific roles to enhance operational efficiencies and to create contingency communication/workflow plans should hubs within the network fail at critical moments. One example of building fault tolerance into the communication network might be to establish role redundancy or processes to offload communication/roles during critical periods. For instance, the role of flow coordinator or pediatric intensive care unit resident might be purposefully split during peak hours where relying on one point of contact risks leading to user overload. This division of labor during periods of high volumes of communication would not only build fault tolerance into the system but allow these previously overwhelmed roles to spend more time and attention to communicating effectively with other members of the team.

Network-specific metrics such as page rank, hub score, and authority score all reinforce the general inference from basic quantitative analysis that the roles mentioned above (resident, MPS, and HUC) serve as important nodes within the inpatient communication system. It is important to note that these metrics, calculated using the system at-large, are likely missing the granularity of the same analytical approach applied to meso- and microsystems within the communication network. While a few users operate at the system level, bridging communication gaps between multiple units and communicating with unit directors, the system-wide analysis may overlook critical roles in these smaller, more enclosed systems.

This work is subject to several limitations. First and foremost, given the highly variable staffing models in an inpatient system, more longitudinal data would help further establish patterns of communication in the network. While these data are analyzed in bulk, analysis looking at user burden over shift times, or between different shifts (i.e., night vs. day, week vs. weekend), might grant insight into user burden over discrete work periods as is common for nursing shifts and, increasingly, inpatient physician shift work. This would require joining the messaging data set with existing employee scheduling data or automated inference of shifts based on user activity over certain periods of time. Joining this data with other data streams/sources was outside the scope of this project given the complexity of such a task within the time constraints of the project team but would undoubtedly augment future insights. For instance, while the data are currently based on user ID within our network, associating user type, role, or location might lead to additional discovery and insight into the different activities, structures, and functions of communities within the inpatient system. Filling in this missing metadata would assist in providing granular insight into the system, allowing investigation into smaller subsystems of the hospital and the differences between communication, structure, and function between them. While this may be possible from the underlying messaging data with the use of regular expressions and natural language processing to extract semantic data from each message, the message metadata are not ideal for several reasons. First, it is user-curated in that users assign themselves to roles and locations when signing into the ecosystem. While users are not free to choose any combination of role and location, many have multiple choices which may or may not reflect the true state of their location and role in the inpatient system at the time a message is sent. Second, the metadata are duplicative in that one user may be signed in under multiple roles and locations according to the metadata. Addressing these discovered challenges were deemed out of scope for this initial, exploratory work. This role/location limitation, combined with the lack of message content means that the insights from our network model are limited to sender, user, and timestamp. Furthermore, it is likely that system engagement is variable across the inpatient system and this messaging data are only part of the communication system in full (e.g., not taking into account other messaging channels and formats). While this data set grants us more quantitative and visual insight than previously available for inpatient communication systems, there are multiple other avenues in which inpatient teams can communicate such as pagers, public announcements, and, as well as the most fundamental and technologically naive way, face-to-face. A more comprehensive study would thus require integration of other data sets, such as paging data, PA logs, employee location/role information, defined shift hours for all roles in the hospital, and dates for resident/provider rotation between services or floors, in addition to extensive observational data.

It should be emphasized that prior literature examining inpatient and health care communication systems has been primarily qualitative in nature and this work, despite its limitations, has spurred more questions worthy of investigation. Locally, the insight gained through this analysis has led to questions about how we design clinical systems, workflows, and prompted discussions of messaging algorithms, that is, “what, when, and how should we be messaging each other in the clinical realm?” Without a systematic way to analyze messaging data, any redesign or improvement work would be reliant on qualitative data or anecdote alone.

Future directions for this work are many and include more rigorous inclusion of metadata and time series analysis which might help examine how the communication network changes over time based on location, role, and other factors. Since overall correlation between messages sent and received is approximately 1:1, surveillance of the communication system for deviation in this correlation might reveal an emerging burden of automated messages from trigger-based systems like Vigilanz (Vigilanz Corp., Chicago, Illinois, United States). While these types of systems grant large medical centers the ability to monitor clinical information systems and alert providers in real time, they also carry high risks for centers that incorrectly implement or overimplement automated solutions that put frontline providers at risk for information/communication overload. There is also intriguing promise in utilizing these messaging data streams for anomaly detection whereby established messaging patterns are compared with current state, surveilling for signals that providers in one clinical area are interacting with the network in anomalous fashion perhaps indicating a cry for help in an escalating or emergent situation. Lastly, given the insights we are able to glean regarding the structure and hierarchy of the communication network, supply-chain optimization methods might be applicable to these systems to help users navigate complex communication networks to find the right recipient for their message.

Conclusion

Secure text-messaging systems represent an increasingly popular vehicle for communication in inpatient medical centers. The data generated by these systems represent a promising opportunity to characterize and understand the form and function of complex health care communication networks. Even basic network analysis reveals users vulnerable to alert fatigue and information overload and opportunities for building fault tolerance and role redundancy into these communication networks. Furthermore, this insight has the potential to inform the practice of frontline providers through identifying superusers and best practices or cohorts in need of remediation and education. This preliminary analysis is also a promising lens for systems-level oversight by operational actors and vehicle for innovative approaches to many other potential applications such as surveillance of automated notifications or situational awareness. More work is needed to combine messaging data with other data streams such as role and location data to fully leverage this potential.

Clinical Relevance Statement

Large and complex organizations are increasingly implementing messaging systems to facilitate Health Insurance Portability and Accountability Act compliant communication across a large network. In the absence of appropriate oversight and governance, these systems run the risk of inundating certain users and roles leading to alert fatigue and information overload.

Multiple Choice Questions

1. Applying network methods to messaging data, a node represents:

   • A single message between users.

   • A user in the communication system.

   • A connection between users.

   • An interconnected team of users.

   • Nodular message metadata.

   Correct Answer: The correct answer is option b. According to the methods section and the figure provided, nodes represent users within the messaging systems, edges are messages or a connection between users.

2. The definition of a “scale-free” network refers to a network that contains:

   • Evenly distributed nodes and edges.

   • Many nodes with few edges and few nodes with many edges.

   • A highly redundant structure of nodes and edges.

   • Many isolated “islands” of nodes and edges.

   • A state where the scale of the network is unknown.

   Correct Answer: The correct answer is option b. The manuscript refers to earlier work in graph theory and network analysis which states that scale free networks contain many nodes with few edges and few nodes with many edges.

• References

• 1 Institute of Medicine (U.S.). Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001
• 2 Leonard M, Graham S, Bonacum D. The human factor: the critical importance of effective teamwork and communication in providing safe care. Qual Saf Health Care 2004; 13 (Suppl. 01) i85-i90
• 3 McKnight L, Stetson PD, Bakken S, Curran C, Cimino JJ. Perceived information needs and communication difficulties of inpatient physicians and nurses. Proc AMIA Symp 2001; 453-457
• 4 Alvarez G, Coiera E. Interdisciplinary communication: an uncharted source of medical error?. J Crit Care 2006; 21 (03) 236-242
• 5 Artis KA, Dyer E, Mohan V, Gold JA. Accuracy of laboratory data communication on ICU daily rounds using an electronic health record. Crit Care Med 2017; 45 (02) 179-186
• 6 MACRA. MIPS & APMs - Centers for Medicare & Medicaid Services. Available at: https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Value-Based-Programs/MACRA-MIPS-and-APMs/MACRA-MIPS-and-APMs.html . Accessed August 31, 2018
• 7 Ernst AA, Weiss SJ, Reitsema JA. Does the addition of Vocera hands-free communication device improve interruptions in an academic emergency department?. South Med J 2013; 106 (03) 189-195
• 8 Ortega GR, Taksali S, Smart R, Baumgaertner MR. Direct cellular vs. indirect pager communication during orthopaedic surgical procedures: a prospective study. Technol Health Care 2009; 17 (02) 149-157
• 9 Health Information Policy. Available at: www.hhs.gov/hipaa/index.html . Accessed August 6, 2018
• 10 Hansen JE, Lazow M, Hagedorn PA. Reducing interdisciplinary communication failures through secure text messaging: a quality improvement project. Pediatr Qual Saf 2018; 3 (01) e053
• 11 van der Sijs H, van Gelder T, Vulto A, Berg M, Aarts J. Understanding handling of drug safety alerts: a simulation study. Int J Med Inform 2010; 79 (05) 361-369
• 12 Shah NR, Seger AC, Seger DL. , et al. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006; 13 (01) 5-11
• 13 van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (02) 138-147
• 14 The Joint Commission. Sentinel Event Alert Issue 50: Medical Device Alarm Safety in Hospitals. Oakbrook Terrace, IL: The Joint Commission; 2013
• 15 The Joint Commission. National Patient Safety Goal on Alarm Management. Oakbrook Terrace, IL: The Joint Commission; 2013
• 16 Roetzel PG. Information overload in the information age: a review of the literature from business administration, business psychology, and related disciplines with a bibliometric approach and framework development. Bus Res 2018. Available at: https://doi.org/10.1007/s40685-018-0069-z . Accessed June 3, 2019
• 17 Lee EK, Mejia AF, Senior T, Jose J. Improving patient safety through medical alert management: an automated decision tool to reduce alert fatigue. AMIA Annu Symp Proc 2010; 2010: 417-421
• 18 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. ; with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17 (01) 36
• 19 Baseman JG, Revere D, Painter I, Toyoji M, Thiede H, Duchin J. Public health communications and alert fatigue. BMC Health Serv Res 2013; 13: 295
• 20 Wu RC, Lo V, Morra D. , et al. The intended and unintended consequences of communication systems on general internal medicine inpatient care delivery: a prospective observational case study of five teaching hospitals. J Am Med Inform Assoc 2013; 20 (04) 766-777
• 21 Przybylo JA, Wang A, Loftus P, Evans KH, Chu I, Shieh L. Smarter hospital communication: secure smartphone text messaging improves provider satisfaction and perception of efficacy, workflow. J Hosp Med 2014; 9 (09) 573-578
• 22 Wu R, Lo V, Morra D. , et al. A smartphone-enabled communication system to improve hospital communication: usage and perceptions of medical trainees and nurses on general internal medicine wards. J Hosp Med 2015; 10 (02) 83-89
• 23 Barabási AL, Oltvai ZN. Network biology: understanding the cell's functional organization. Nat Rev Genet 2004; 5 (02) 101-113
• 24 Jeong H, Tombor B, Albert R, Oltvai ZN, Barabási AL. The large-scale organization of metabolic networks. Nature 2000; 407 (6804): 651-654
• 25 Newman ME. The structure of scientific collaboration networks. Proc Natl Acad Sci U S A 2001; 98 (02) 404-409
• 26 Albert R, Jeong H, Barabasi AL. Internet - diameter of the world-wide web. Nature 1999; 401 (6749): 130-131
• 27 Pandas Documentation. Available at: https://pandas.pydata.org/ . Accessed July 11, 2018
• 28 Python Software Foundation. Available at: https://www.python.org . Accessed July 11, 2018
• 29 Jupyter. 2018 . Available at: http://jupyter.org/ . Accessed August 31, 2018
• 30 Network X. . Documentation. Available at: https://networkx.github.io/documentation/networkx-1.10/index.html . Accessed July 11, 2018
• 31 Seaborn: statistical data visualization. Available at: https://seaborn.pydata.org/index.html . Accessed August 31, 2018
• 32 Matplotlib. Available at: https://matplotlib.org/ . Accessed August 31, 2018