Skip to main content
Fig. 8 | Journal of Internet Services and Applications

Fig. 8

From: Upgrading a high performance computing environment for massive data processing

Fig. 8

Representation of network traffic patterns in COMPSs executions. a) conventional file system; b) HDFS API; c) HDFS+Storage APIs. The legend to the right indicates the aggregate volume of traffic read from a vertex or transferred through an edge. In the Conventional file system, all data is transferred between the master and the workers; by using HDFS, the traffic is distributed among workers, since all I/O is performed by directly by HDFS and data moves from the datanodes holding the data to the worker nodes where each block will be processed; when COMPSs Storage API is used, tasks are assigned preferentially to workers executing at the same nodes where blocks are stored, and network traffic is limited to a few cases where locality is not achievable (e.g., when reading records at the block edges)

Back to article page