PortView: identifying port roles based on port fuzzy macroscopic behavior

Abstract

Port is a basic parameter in TCP/IP data communication. However, a port by its number has no clear association to the types of applications. Identifying such an association, which is referred to as a port role, provides important information to many network applications such as IDS and traffic shaper, as well as detecting new network services and security attacks. Traditional studies on port role identification are only based on port behavior shown on an individual host, other than jointly viewed macroscopic port behavior embodied by all relevant traffic flows among multiple hosts. Port role identification based on macroscopic behavior can reflect severs or clients to discover new services or attacks in the network. In this paper, we propose a novel port role identification approach called PortView, which is based on fuzzy macroscopic port behavior analysis. In our approach, we design a two-dimensional Macroscopic Port Classification Plane (MPCP) to classify port roles into six role zones using an EM fuzzy clustering algorithm. Comprehensive experiments using real Internet traces from the CERNET network validate that PortView can effectively and accurately identify port role based on its macroscopic port behavior.

1. Introduction

Most current Internet applications are based on TCP or UDP. Traditionally, network applications use fixed port numbers assigned by IANA [1]. Accordingly, TCP and UDP port numbers are directly associated to the types of various network application, e.g. TCP/80 for HTTP etc. TCP and UDP ports identified with 16-bit numbers can be partitioned into three types: (1) Well-known Ports, which are port numbers between 0 to 1023, and closely bound to system specific services and protocols; (2) Registered Ports, which are port numbers between 1024 to 49151, and used by processes of procedures and executed programs of common users; and (3) Private Ports, which are port numbers between 49152 to 65535, and dynamically or temporarily used by client communication processes or executed programs.

Each TCP or UDP port used in a TCP/IP network application implies a communication channel, which can also be exploited by an intruder. Port exploitation based network intrusion are mainly manifested in the following two styles: (1) exploiting common ports, and attacking hosts through known system bugs; (2) planting Trojan horses to victim hosts through backdoor opened by other network vulnerabilities. It is worth noting that port numbers used by various malicious software (e.g., Trojan programs) are widely distributed, which don’t only use registered ports. Port scanning is commonly used by an intruder to search for potential targets. Network administrators can monitor their networks by tracking abnormal port roles, such as a client host provides services outwards the public. Many other port related behaviors such as the changes of port in/out degreecan help check whether or not a host behaves normally. In general, port profile is a very important metric for application traffic classification. Correctly identifying server ports among network traffic has great significance for accurately classifying traffic application types.

At present, few studies focus on identifying port roles, i.e., a port belongs to a server or client port. Many metwork security and management applications, such as port scanning, worm detecting, traffic classification etc., are closely related to port role identification. In this paper, we define and analyze port roles based on their macroscopic traffic behavior. Macroscopic port roles first can be classified into active ports and non-active ports, and then active ports are classified into normal behavior roles and abnormal behavior roles. Abnormal behavior roles are classified into scan attacked ports and service failed ports, and normal behavior ports can be classified into client ports, server ports, or p2p ports etc. For this purpose, we use data mining to classify port roles from massive traffic data automatically. The contributions in this paper can be summarized as the following:

Define port role to distinguish different communication pattern.

Propose a two-dimensional Macroscopic Port Classification Plane (MPCP) that creates a new space for traffic behavior analysis.

Propose an EM fuzzy clustering algorithm to classify port traffic into six behavior zones in MPCP.

The rest of the paper is organized as follows. Firstly, we analyze Macroscopic Port Role in Section 2. We design a two dimensional Macroscopic Port Classification Plane and propose an EM clustering algorithm for fuzzy port behavior analysis in Section 3. Extensive experiments using real Internet traces to validate PortView are shown in Section 4. The related work is presented in Section 5. Finally Section 6 concludes this paper.

2. Macroscopic port role analysis

In this section, we will introduce the definition of Macroscopic Port Role as a new defining traffic characteristic, and use it re-profile several common network applications or behaviors.

2.1 Macroscopic port role

With the intent to define Macroscopic Port Role, we first define several port related parameters.

Let SF _i be a set of flows with source port i such that ∀ f ∈ SF _i , getSrcPort(f) = i. We denote n _i =|SF _i | as the size of SF _i . Here, getSrcPort(f) is the source port number of the flow f, retrieved via the function getSrcPort(). Similarly, getDstPort(), getSrcIP() and getDstIP() are the functions to retrieve destination port, source IP and destination IP from the flow f. Accordingly, a flow f is uniquely defined as f = {getSrcIP(f), getDstIP(f), getSrcPort(f), getDstPort(f)}.

All distinct source IP addresses retrieved from the flows in SF _i , which are related to the source port i, constitutes the source IP set denoted as SSIP _i . The number of all distinct source IP addresses with source port i is denoted as |SSIP _i .|. Similarly, all distinct destination IP addresses retrieved from the flows in SF _i constitute the destination IP set denoted as SDIP _i . The number of all distinct source IP addresses with destination port i is denoted as |SDIP _i .|.

Let DF _j be a set of flows such that ∀ f ∈ DF _j , getDstPort(f) = j. We denote m _j =|DF _j | as the size of DF _j . All distinct source IP addresses retrieved from the flows in DF _j , which are related to the destination port j, constitute the source IP set denoted as DSIP _j . The number of all distinct source IP addresses with destination port i is denoted as |DSIP _i |. .Similarly, all distinct destination IP addresses retrieved from the flows in DF _j constitutes the destination IP set denoted as DDIP _j . The number of all distinct destination IP addresses with destination port i is denoted as |DDIP _i |.

Definition 1. For a given port i, its Macroscopic Port Role (MPR) observed on a monitored network link during a measurement period T is defined by the 4-tuple port parameters < SSIP _i , SDIP _i , DDIP _i , DSIP _i >.

For a source (or destination) port i, if the corresponding SSIP _i and SDIP _i (or DSIP _i and DDIP _i ) are larger than a threshold, it is called active port. Otherwise, port i is called inactive port. The threshold will be computed by the Port Role Interval Classifying Algorithm in Section 3.4.

In the following, we broadly classify network traffic based on its role into two categories: abnormal and normal network roles. Then, for each category, we re-profile several selected common network applications or network scenarios using MPR analysis to illustrate PortView. It is worth noting that the classification criteria shown in this paper is not the only way to use PortView. The more granular classification criteria are, the more zones we may create in MPCP as discussed later in Section 3.

The different port role of MPR is described from Figures 1, 2, 3, 4, 5. In these figures, the left side of the dotted line means that the DIP hosts send some connection requests to an port i on the SIP hosts. The right side of the dotted line means that SIP hosts with port i send some connection requests to the SIP hosts. These figures show the Macroscopic Port Role.

Figure 1

MPR of port scanning. During a port scanning, a scanner attempts to connect to a specific port of a large number of hosts.

Figure 2

MPR of failed service. In a typical DDoS attack scenario, a large number of DIPs send requests connecting to a specific port i on a few SIPs with destination port i.

Figure 3

MPR of client application. The port on the client is usually chosen by the operating system dynamically. Different ports are used to communicate with multiple servers.

Figure 4

MPR of server application. A service port is the one used by a server listening to the requests from different clients, so the number of servers is usually less than the number of clients.

Figure 5

MPR in overlay application. Typically, an overlay node firstly obtains a list of neighbor nodes, and then sends request to each selected node. Hence, in an overlay network which uses a unified service port number to provide service, the communication pattern is that few clients send requests to lots of hosts.

2.2 MPR of abnormal traffic

In the traffic category of abnormal network role, we study the port role of (1) commonly observed port scanning traffic, and (2) a network service under DDoS attack.

• 1) MPR of port scanning: The MPR of port scanning can be illustrated in Figure 1, which clearly shows that on the left side of the dotted line, a few hosts denoted as DIP (Destination IP address) send too many connection requests to an interested port i possibly open on too many hosts denoted as SIP (Source IP address) with destination port i. Thus, we have |DSIP _i |<<|DDIP _i |. Usually only a few SIPs (Source IP address) with this port i open response to the connection requests with source port i, as shown on the right side of the dotted line in Figure 1, which can be described with: |SSIP _i | ≈ |SDIP _i |. Hence, we have the following MPR of port scanning:

  $$\begin{array}{l}\left|\mathit{DSI}{P}_i\right|<<\left|\mathit{DDI}{P}_i\right|\\ \left|\mathit{SSI}{P}_i\right|\approx \left|\mathit{SDI}{P}_i\right|\end{array}$$

  (1)

• 2) MPR of failed network service: In a typical DDoS attack scenario, a large number of DIPs send requests connecting to a specific port i on a few SIPs with destination port i, that implies |DSIP _i |>>|DDIP _i |. For those hosts being DDoS attacked severely, they cannot response interactively. Thus, only a few SIPs as shown on the right side of the dotted line in Figure 2 can response to the connection requests with source port i, which implies: |SSIP _i | ≈ |SDIP _i |. Thus, the MPR of failed service caused by successful DDoS attack can be described as:

  $$\begin{array}{l}\left|\mathit{DSI}{P}_i\right|>>\left|\mathit{DDI}{P}_i\right|\\ \left|\mathit{SSI}{P}_i\right|\approx \left|\mathit{SDI}{P}_i\right|\end{array}$$

  (2)

2.3 MPR of normal port behaviors

In the traffic category of normal network behavior, we study the port behavior of (1) client application, (2) server application, and (3) overlay application.

• 1) MPR of client application: In a typical client–server network application, for every monitored network flow sf _i = (ip ₁ , ip ₂ ,i ,j) destined to the host ip ₂ , a corresponding opposite directional flow df _i = (ip ₂ , ip ₁ , j, i) should appear. We denote such a pair of flows < sf _i , df _i > as <in-flow, outflow> pair. In this paper, we denote the MPR of client–server applications as regular port behavior, which can be defined as:

  $$\begin{array}{l}\left|\mathit{SSI}{P}_i|=|\mathit{DDI}{P}_i\right|\\ \left|\mathit{SDI}{P}_i|=|\mathit{DSI}{P}_i\right|\end{array}$$

  (3)

In the most common client–server paradigm (C-S mode), a client with IP address srcIP communicates with a server with IP address dstIP. The port on the client is usually chosen by the client’s operating system dynamically. Different ports are used if the same client needs to communicate with multiple servers. However, different clients may happen to choose the same port to communicate to different servers. We show the port behavior in C-S mode in Figure 3, where SIP and DIP represent client and server hosts respectively. As shown on the left of the dotted line in Figure 4, different SIP addresses with the same port communicate with DIP addresses, while DIP addresses response accordingly. We can see that a given port links a DIP to a SIP, and different SIP addresses may use the same port to communicate to their connecting DIP addresses. Clearly, the number of SIP addresses is equal to the number of DIP addresses. Thus, we have |DSIP _i |=|DDIP _i |. On the right side of the dotted line, we see that the number of SIP addresses is equal to the number of DIP addresses. Thus, we have |SSIP _i |=|SDIP _i |. It is possible that some connecting ports from source IP addresses have not received response from the destination IP addresses. However, even if in such a case, the number of SIP addresses is still equal to the number of DIP addresses since a source IP and a destination IP always emerge in pair in a network flow.

There are further two possible port behavior variations in C-S mode: (1) Different source IP addresses use the same port to communicate to a single destination IP. In this case, the number of source IP addresses is larger than the number of destination IP addresses, namely, |SSIP _i |≥|SDIP _i |. (2) A few source IP addresses send information frequently during the measurement interval. The source port used by a source IP reverses, at the same time, the other source IP uses the same port number to access a different destination IP, the destination IPs which the two source IPs access is random, From the probability theory standpoint, the number of source IPs and destination IPs in flow records in which the client port takes role as a source port are almost equal, namely, |DSIP _i | ≈ |DDIP _i |. Hence, we have the following observation on the port behavior of C-S mode:

$$\begin{array}{l}\left|\mathit{DSI}{P}_i\right|\approx \left|\mathit{DDI}{P}_i\right|\\ \left|\mathit{SSI}{P}_i\right|\approx \left|\mathit{SDI}{P}_i\right|\end{array}$$

(4)

• 2) MPR of server application: A service port is the one used by a server listening to the requests from different clients. Obviously, the number of servers is usually less than the number of clients. The port behavior of a service port displays that multiple clients marked by DIP may send requests to a service port of the servers SIP addresses as shown on the left side of the dotted line in Figure 4. The right side of the dotted line in Figure 4 shows the servers response accordingly from this service port of SIP addresses. Hence, for a service port, there are more srcIPs than dstIPs in its in-flow direction, i.e., |DSIP _i |>|DDIP _i |. Since each SIP should answer several clients’ requests, the number of srcIPs is less than that of dstIPs in their out-flow direction, i.e., |SSIP _i |<|SDIP _i |. Hence, we have the following observation on the port behavior of service port:

  $$\begin{array}{l}\left|\mathit{DSI}{P}_i\right|>\left|\mathit{DDI}{P}_i\right|\\ \left|\mathit{SSI}{P}_i\right|<\left|\mathit{SDI}{P}_i\right|\end{array}$$

  (5)

In traditional C-S model, a server may have multiple connections to clients. However, a P2P host is limited to provide only one connection to each P2P. Suppose the number of links a server provided is fixed, a P2P host can provide service for more P2P. Hence, we have the following observation on the port behavior of a P2P host:

$$\begin{array}{l}\left|\mathit{DSI}{P}_i\right|>>\left|\mathit{DDI}{P}_i\right|\\ \left|\mathit{SSI}{P}_i\right|<<\left|\mathit{SDI}{P}_i\right|\end{array}$$

(6)

• 3) MPR of overlay application: In an overlay network, a node uses a unified service port to provide service to other nodes. Typically, an overlay node firstly obtains a list of neighbor nodes, and then sends request to each selected node. Hence, in an overlay network which uses a unified service port number to provide service, the communication pattern is that few clients send requests to lots of hosts. Figure 5 shows the communication pattern based on a unified port of an overlay network. On the left side of the dotted line of Figure 5, few clients send requests to lots of hosts, i.e., |DSIP _i |<|DDIP _i |; On the right side of the dotted line, lots of hosts answer the requests to few clients, i.e., |SSIP _i |>|SDIP _i |. Hence, we have the following observation on the port behavior of a unified port in an overlay network:

  $$\begin{array}{l}\left|\mathit{DSI}{P}_i\right|<\left|\mathit{DDI}{P}_i\right|\\ \left|\mathit{SSI}{P}_i\right|>\left|\mathit{SDI}{P}_i\right|\end{array}$$

  (7)

3. Classifying port role algorithm

3.1 Port role metrics

As discussed in Sec. 2, the Macroscopic Port Role analysis can characterize communication patterns as shown in Eq. 1~Eq. 7. For facilitating MPR based traffic classification, we design a Macroscopic Port Classification Plane (MPCP) with X-Y axes representing two different behavior metrics. Then, we propose a fuzzy port behavior analysis method to divide MPCP into multiple zones with each zone representing different network behavior and corresponding network applications.

Definition 2. Port Exporting Network Activity X _i : for all network flows leaving from port i in a measurement time window, the ratio of the difference between the numbers of related source and destination hosts and the number of all the port i related hosts that send traffic is called port i’s Port Exporting Network Activity X _i .

$$X_i=\frac{\left|\mathit{SDI}{P}_i\right|-\left|\mathit{SSI}{P}_i\right|}{\left|\mathit{SDI}{P}_i\right|+\left|\mathit{SSI}{P}_i\right|}$$

(8)

Definition 3. Port Importing Network Activity Y _i : for all network flows destining to port i in a measurement time window, the ratio of the difference between the numbers of related source and destination hosts and the number of all the port i related hosts is called port i’s Port Importing Network Activity Y _i .

$$Y_i=\frac{\left|\mathit{DSI}{P}_i\right|-\left|\mathit{DDI}{P}_i\right|}{\left|\mathit{DSI}{P}_i\right|+\left|\mathit{DDI}{P}_i\right|}$$

(9)

From Definition 2 and 3, |SDIP _i |-|SSIP _i |<|SDIP _i |+|SSIP _i |, |DSIP _i |-|DDIP _i | < |DDIP _i |+|DDIP _i |. Thus, we have - 1 < X _i , Y _i  < 1, i ∈ [0, 65535].

Definition 4. Port Differential Network Activity B _i : for a given port i, the difference between its Port Exporting Network Activity X _i and Port Importing Network Activity Y _i .

$$B_i=X_i-Y_i=\frac{\left|\mathit{SDI}{P}_i\right|-\left|\mathit{SSI}{P}_i\right|}{\left|\mathit{SDI}{P}_i\right|+\left|\mathit{SSI}{P}_i\right|}-\frac{\left|\mathit{DSI}{P}_i\right|-\left|\mathit{DDI}{P}_i\right|}{\left|\mathit{DSI}{P}_i\right|+\left|\mathit{DDI}{P}_i\right|}$$

(10)

Definition 5. Port Accumulative Network Activity A _i : for a given port i, the accumulation of its Port Exporting Network Activity X _i and Port Importing Network Activity Y _i .

$$A_i=X_i+Y_i=\frac{\left|\mathit{SDI}{P}_i\right|-\left|\mathit{SSI}{P}_i\right|}{\left|\mathit{SDI}{P}_i\right|+\left|\mathit{SSI}{P}_i\right|}+\frac{\left|\mathit{DSI}{P}_i\right|-\left|\mathit{DDI}{P}_i\right|}{\left|\mathit{DSI}{P}_i\right|+\left|\mathit{DDI}{P}_i\right|}$$

(11)

Definition 6. Port Overall Network Activity D _i : for a given port i during a measurement time window, the whole numbers of port i’s related source and destination hosts. In other word, it is the all number of hosts related to source or destination port i.

$$D_i=\left|\mathit{SSI}{P}_i\right|+\left|\mathit{SDI}{P}_i\right|+\left|\mathit{DSI}{P}_i\right|+\left|\mathit{DDIP}i\right|$$

(12)

X _i and Y _i are the basic clustering metrics, which are used to classify port roles into six different zones. A and B are two derived parameters from X and Y, and used directly in the EM clustering algorithm to categorize active ports.

3.2 Fuzzy port behavior

The characteristics of port behavior can be summarized in Table 1. In the following, we introduce Macroscopic Port Classification Plane (MPCP). Then we propose fuzzy port behavior analysis to quantitatively map all earlier discussed network applications or scenarios into the corresponding MPCP zones.

Table 1 Port communication behavior

The Macroscopic Port Classification Plane is a 2-dimensional graph, as shown in Figure 6, where X axis represents Port Exporting Network Activity, and Y axis displays Port Importing Network Activity. In the case of normal network behavior, the corresponding Port Exporting and Importing Network Activities manifest X _i  ≈ Y _i (B _i is close to 0) based on Eq. 3. Thus, if the MPR analysis result shows the traffic is spanned along the diagonal area of MPCP (marked as zones 1, 2, 3 and 4 in Figure 7), the corresponding traffic belongs to normal network behavior; otherwise abnormal and the MPR analysis results should show the traffic is either located in zone 5 or 6 in Figure 6. If -δ≤B _i ≤β, δ > 0 and β < 1, it belongs to normal port behavior. As a failed service port, according to Eq. 1, its Port Exporting Network Activity is less than its Port Importing Network Activity, X _i <Y _i . It falls into zone 5. As Bi<-δ, port i falls into zone 5. According to the analysis of Sec. 3.2 (1), in the case of abnormal network behavior, the corresponding Port Exporting and Importing Network Activities are different: X _i  ≠ Y _i . For scanned port, based on communication relationship as Eq. 1, its Port Exporting Network Activity is more than its Port Importing Network Activity, X _i >Y _i . Thus, it is in zone 6, and B _i >β.

Figure 6

Macroscopic port classification plane. X axis represents Port Exporting Network Activity, and Y axis displays Port Importing Network Activity. The range of six zones can be showed in Table 2.

Figure 7

Distribution of flows. It shows the distribution of flow records with the changing of source port from 1 to 65535.

For zone 1, 2, 3 and 4 in the diagonal area, in addition to the condition -δ<B _i <β, according to the relationship of client port in Table 1, port i’s Port Accumulative Network Activity A _i  ≈ 0, - φ ≤ A _i  ≤ φ, and 0 < φ, φ < 1. Overlay service port is in zone 4, A _i  < - φ. Service port is in zone 2, A _i >φ. For the boundary between zone 2 and 3, according to the classification of service port and p2p port in Table 1, in zone 2, φ < A _i  ≤ 2 - γ, and in zone 3, 2-γ<A _i < 2. In conclusion, the range of six zones can be showed in Table 2.

Table 2 Port behavior zones

Table 2 shows that in order to classify the 6 roles, we need to determine five arguments δ, β, φ, φ, and γ. In the following, we will design a clustering method EM algorithm that can classify the two kinds of distributions.

3.3 EM algorithm for Two classes of distribution

In this section, we will discuss how to determine the boundary arguments for each zone. Observing that each port may play certain role (i.e., projected into certain zone in MPCP) with some probability, we use statistical clustering to determine zone boundaries in MPCP. The basis of statistical clustering is to build a finite mixed statistical model such that it can use k probability distributions to replace k clustering. Since a port may belong to different zones/roles with different probabilities, we define a port role as the MPCP zone it is most likely located (i.e., with the biggest probability distribution).

For two kinds of statistical clustering, if we need to do this on A and B classes, every class is supposed to be normal distribution. Clustering A’s expectation and variance are μ _A and σ _A . Clustering B’s expectation and variance are μ _B and σ _B . We take samples from the distributions of A and B. The probability of taking A or B is p _A or p _B respectively. Here, p _A +p _B =1. Here, the challenge is how to determine five arguments, μ _A , σ _A , μ _B , σ _B , and p _A (or p _B ), by only knowing the MPR analysis result.

Since the distribution that each port follows is unknown initially, we use the fuzzy clustering EM algorithm to classify the port behaviors. In every iterative step of classical k-average clustering algorithm, every port is considered exclusively belonging to a class. Thus, we adopt fuzzy clustering to tackle the challenge of multiple distributions a port may follow. Each port probabilistically belongs to a class. First, every port’s class is estimated. Then, compute every port’s clustering probability using initial estimated value. These probabilities are used to estimate the argument again. The above process is iteratively conducted for the second time. This method is called Expectation Maximization (EM). First, compute clustering probability, in another word, the expectation’s class value; Second, Maximum likelihood the given data’s distribution. The following test will show the EM algorithm in detail.

For n port’s metric: x ₁ , · · · , x _n , there are two classes A and B. z is a two-dimensional array. At the very beginning, let z _Aj be 1, meaning x _j is in A class; otherwise 0. Let z _Bj be 1, meaning x _j is in B class; otherwise 0. Then according to z’s value, we can compute k class’s argument using the following formula:

$$\begin{array}{l}{\mu }_A=\frac{{\displaystyle {\sum }_{j}E\left(z_{A_j}|x_j\right)·x_j}}{{\displaystyle {\sum }_{j}E\left(z_{A_j}|x_j\right)}},{\sigma }_A=\frac{{\displaystyle {\sum }_{j}E\left(z_{A_j}|x_j\right)·{\left(x_j-{\mu }_A\right)}^2}}{{\displaystyle {\sum }_{j}E\left(z_{A_j}|x_j\right)}}\\ {\mu }_B=\frac{{\displaystyle {\sum }_{j}E\left(z_{B_j}|x_j\right)·x_j}}{{\displaystyle {\sum }_{j}E\left(z_{B_j}|x_j\right)}},{\sigma }_B=\frac{{\displaystyle {\sum }_{j}E\left(z_{B_j}|x_j\right)·{\left(x_j-{\mu }_B\right)}^2}}{{\displaystyle {\sum }_{j}E\left(z_{B_j}|x_j\right)}}\end{array}$$

(13)

For a port i,

$$\begin{array}{l}{x}_{1,2}=\frac{{\mu }_A{\sigma }_B^2-{\mu }_B{\sigma }_A^2}{{\sigma }_A^2-{\sigma }_B^2}\pm \\ \frac{\sqrt{{\left({\mu }_A{\sigma }_B^2-{\mu }_B{\sigma }_A^2\right)}^2+\left({\mu }_A^2-{\mu }_B^2\right)\left({\mu }_A{\sigma }_B^2-{\mu }_B{\sigma }_A^2-2{\sigma }_A^2{\sigma }_B^{2}ln\left(p_B{\sigma }_A/p_A{\sigma }_B\right)\right)}}{{\sigma }_A^2-{\sigma }_B^2}\end{array}$$

(14)

The two class EM classifying method discussed in Eq.(14) can put data into 2 or 3 classes. As in Section 3.1, we provided several metrics that can classify port behaviors, and each time the method uses one dimension metric to do classification. The next subsection will give a detailed port role interval classifying algorithm based on the Eq. (14).

3.4 Port role interval classifying algorithm

In the measuring progress, classifying algorithm first records every port’s source IP numbers, source port’s destination IP numbers, destination port’s source IP numbers, destination port’s source IP numbers. After a measuring time period, the role style of the every port’s four metric is determined. In measuring data, if the number of a port’s related export and import hosts is small, then because of the tiny random difference between the numbers of source and destination hosts, some inactive port will be made with big random difference between port’s inward and outward communication metric. At the same time, we care about these active ports with large traffic of the host that influence the network a lot. And we remove these ports with small traffic and have little influence on the network. In order to determine a port is an active port or not, we compute every port’s activity metric D. According to the two-class EM algorithm in section 3.3, all the ports are classified into active and inactive ports. Because D is x ₁ , based on formula (19), we can compute active port’s threshold. For all the ports whose activity metric value D _i is bigger than or equal to x ₂ , they are defined as active ports. Otherwise, they are inactive ports.

For all the active ports, using port’s communication difference balance metric B, based on the two-class EM algorithm in section 3.3, we can compute the classification threshold of the normal and the abnormal port. The two-class EM algorithm’s two distribution’s interval can figure out normal port’s range is [x ₁ , x ₂ ], and abnormal port’s range is (-∞, x₁), (x₂, + ∞). Because port’s communication difference balance metric B’s range value is (-2, 2), abnormal port’s value range is (-2, x ₁ ), (x ₂ , 2). So we can compute the argument α, β, α = - x₁, β = x₂ in Table 2.

For all the normal port, using port communication difference accumulation metric A, based on the two-class EM algorithm in section 3.3, we can compute the classification threshold of client port and service port. The two-class EM algorithm’s two distribution’s interval can figure out client port’s range is [x ₁ , x ₂ ], and service port’s range is (-∞, x₁), (x₂, + ∞). Because A’s value range is (-2, x ₁ ), (x ₂ , 2), service port’s range is (-2, x ₁ ), (x ₂ , 2). So we can compute the argument φ, φ, φ = - x₁, φ = x₂ in Table 2.

Service ports are divided into there kinds: overlay service port, normal service port and P2P service port. Using communication difference accumulation metric A, besides classifying the styles of client port, it can also classify overlay service port. So if it is a overlay port, A’s value range is (-2,- φ). For the range (φ, 2), it contains normal and P2P service port. We again use the two-class EM algorithm to classify these two kinds of ports. The result is that service port’s range is (φ, x ₁ ), and P2P’s service port’s value range is (x ₂ , 2). So we get the argument γ = 2 - x₂. Based on the method above, we can compute the five arguments in Table 2. So we can classify all the active ports into the six kinds of port roles.

4. Experimental analysis

We will divide port roles as a whole. First, we introduce the experimental traces which are used for interval dividing of port roles. Second, we classify the ports into active ports and inactive ports according to the steps of algorithm in Section 3.4. For active ports, we classify it into normal ports and abnormal ports, and then we classify normal ports into four types of roles: client ports, server ports, P2P ports and overlay ports.

4.1 Experimental traces

Multiple real network traces were selected from the link between CERNET backbone and Jiangsu CERNET network to evaluate PortView. Specifically, we present two representational network traces in this section, namely Trace1 and Trace2. Trace1 was collected on April 10, 2010 with 175 million packets, 117 GB traffic traces, 495,000 flows, and its consecutive observation period is ten minutes. Trace2 was captured on May 4, 2012 with 2.82 million packets, 1.89 GB traffic traces, 17,000 flows, and its observation period is five second. Trace1 is selected to show how PortView performs given relatively rich correlation information among multiple hosts over a longer observation window (10 m). Trace2 is used to validate the agility of PortView given a short observation window (i.e., 5 s). Here, we define PortView agility as its ability to provide port role analysis result over a short time period (e.g., 5 s). Network flows are defined using 4-tuple, i.e. source address, destination address, source port, and destination port in this paper. Trace1 is analyzed from Subsection 4.1 to Subsection 4.3, and the analysis based on Trace2 will be reported in Subsection 4.4.

We can see clearly from Figure 7 that: (1) the number of flows with source port between 1 and 1023 is unevenly distributed; (2) the number of flows with port between 1024 and 5000 is distributed quite uniformly, every source port produces corresponding host amount, the number of host is between 20 and 100; and (3) the flows with port above 5000 is also unevenly distributed.

Figure 8 shows the two-dimensional distribution diagram of port communication difference. We can know from Figure 8 that ports distribute mainly near the origin. According to the Section 3, this type of ports are mainly client ports; In addition, there are a lot of ports which distribute near the diagonal of first quadrant, those ports are server ports, this indicates that most ports belong to normal port. Points in the negative diagonal of third quadrant indicate that the number of IP requesting to this port IP are less than requested port IP amount, this character is similar to scanning, but most of the scanning traffic has no response, that is to say the X-axis coordinate of scanning traffic is close to zero. Because of being in the inner of an overlay, most of requesting traffic has response from a server port of the overlay. There are few points in the third quadrant whose diagonal is near the X-axis, the character is similar to the points which are near the diagonal. Because the requesting traffic is less than the response traffic, points in the fourth quadrant indicate that parts of the traffic of this port are scanning attacks, such as 8088, 7155 and so on.

Figure 8

Port differential network activity. Ports distribute mainly near the origin, according to the Section 4, this type of ports are mainly client ports.

4.2 Port clustering

According to the activity metric value D of every port, two-class EM algorithm can cluster all 65536 ports into active and inactive ports. The corresponding five metrics (μ _A , σ _A , μ _B , σ _B , p _A ) = (162.426, 720.223, 9.297, 5.789, 0.1). Thus we can calculate activity metric value, and classify those ports whose activity value no more than 30 into inactive port, cluster the port whose activity value more than 30 into active port. The number of active ports in this experiment is 5313. In the following, we will analyze 5313 clustered active ports. Figure 9 shows the two-dimensional graph of metric of host difference of 5313 active ports.

Figure 9

Active ports. It shows the two-dimensional graph of metric of host difference of 5313 active ports.

We calculate communication difference balance value B of all 5313 active ports, and use two-class EM algorithm to calculate two distribution parameter values (μ _A , σ _A , μ _B , σ _B , p _A ) = (-0.2323, 0.3533, 0.0017, 0.084, 0.17). Hence, we can confirm the range of normal ports is [-0.0547, 0.0596], the range of abnormal ports is (-2,-0.0547) and (0.0596, 2). Correspondingly, we can calculate the parameters δ and β in Table 2 as δ = 0.0547 and β = 0.0596. The number of ports in normal range is 4512, and the number of abnormal ports is 801. Among these ports, there are 176 scanned ports and 625 server failure ports.

We will calculate communication difference cumulative value A of 4512 normal ports, and use two-class EM algorithm to calculate parameter values of client ports and server ports (μ _A , σ _A , μ _B , σ _B , p _A ) = (-0.0039, 0.0363, 1.1873, 0.7509, 0.85). Therefore, we can confirm the range of client ports is [-0.1325, 0.1186], the range of server ports is (-2,-0.1325) and (0.1186, 2). Correspondingly, we can calculate the parameters φ and φ in Table 2 as φ = 0.1325 and φ = 0.1186. Among the normal ports, there are 3875 client ports and 637 server ports.

We classify server ports into three types, overlay ports, server ports and P2P ports. We distinguish the type of client ports by communication difference cumulative value A, and classify overlay ports, i.e. the range of A as (-2,-0.1325) are overlay ports, and the amount is 25. The range of (0.1182, 2) include general server ports and P2P ports, and the amount is 612. In the next, we will use two-class EM algorithm to classify 612 ports with the distribution parameters as (μ _A , σ _A , μ _B , σ _B , p _A ) =(0.6966, 0.4845, 1.8111, 0.1091, 0.45). The result is that: the range of server ports is [0.1186, 1.585], the amount is 268; the range of P2P ports is (1.535, 2), and the amount is 344. Thus, we can have the parameter γ = 0.465 (2- γ = 1.535). We put the experiment result of measured parameter δ =0.0547, β = 0.1325, φ = 0.1186, γ = 0.465 into interval formula in Table 2. We can get Figure 10, which shows the interval distribution of four type real normal ports corresponding to Figure 6.

Figure 10

Normal ports. classify normal ports into four types, client ports, overlay ports, server ports and P2P ports.

In this test, we need use two-class EM algorithm three times to classify ports. The number of port used in the EM algorithm is 65536, 5313, and 4512 in the measured interval respectively. The maximal number of port is less than 65536. We test the cost of port clustering algorithm in the server equipped with CPU P4 Xeron2.4*2, memory 2 GB, hard disk 1.2 TB, NIC 1000 G, and its running time is 850 ms.

4.3 Experimental analysis

As discussed in Section 1, we know that well-known Ports, which range between 0 to 1023, and are closely bound to some specific services. Registered Ports, port numbers range from 1024 to 49151, they are used by processes of procedures and executed programs of common users. Ephemeral Ports, most implementations use port numbers between 1024 to 5000. Reserved Ports: port numbers larger than 5000 are reserved for other services. We can divided 65536 ports into four types: Well-known Ports, Ephemeral Ports, Registered Ports, Reserved Ports. The role of Well-known port is service port. The role of Ephemeral Port is client port. Registered Ports and Reserved Ports are linked to service port in a local host.

There are 5313 active ports in the experiment. Table 3 gives the relationship between the four port types and the six port roles defined in this paper.

Table 3 Relationship between the four port types and the six port roles

The number of active ports is only 2.14% of the total number of the Well-known ports, which shows that the vast majority of the Well-known port is closed. In Table 3, only seven of the Well-known port is as service port. We can’t find any port with client port role in the Well-known port, which also shows that our proposed method can give a correct classification to the client port role. All Ephemeral Ports are active ports, and nearly all Ephemeral Ports are client port role, which is consistent with the definition of Ephemeral Ports.

In the Registered Ports, Service Port and P2P Port account for approximately 50% of the port. Abnormal ports account for another 50%. The result shows that the Registered Ports, port numbers range from 5001 to 49151, is the most common user applications, but also the most vulnerable to attack. Reserved Ports are used nearly.

4.4 Agility validation on PortView

Trace2 was collected only over 5 seconds on May 4, 2012 with 2.82 million packets, 1.89 GB traffic traces, and 17,000 flows. Such a short observation window provides relatively less information to PortView such that we can validate the agility of PortView. We use the activity metric value D to cluster all ports into active and inactive ports. The corresponding five metrics (μ _A , σ _A , μ _B , σ _B , p _A ) = (62.34, 325.23, 5.45, 3.08, 0.16). The number of active ports in this experiment is 5215. We calculate communication difference balance value B of all 5215 active Ports. (μ _A , σ _A , μ _B , σ _B , p _A ) = (0, 0.527, 0.542, 0.671, 0.55). The number of ports in normal range is 4257, and the number of abnormal ports is 958.

We calculate communication difference cumulative value A of 4257 normal ports. The parameter values of client ports and server ports (μ _A , σ _A , μ _B , σ _B , p _A ) = (0, 0.0645, -0.0112, 0.1136, 0.85). Among the normal ports, there are 4151 client ports and 105 server ports. This experiment result shows that PortView can perform effectively even only provided a short observation window.

5. Related work

Port based traffic analysis can classify application layer protocols [2, 3] used port numbers to analyze the type and trend of traffic application. Wei Li et al. [4] made correlation analysis between metrics related to traffic and application types, and found that, correlation between server ports and application types is the strongest of all the traffic metrics, while that between client ports and application types is very weak, so the discovery of server’s port has great significance for improving the accuracy of traffic application classification.

McNutt and De Shon [5] analyzed correlation between ephemeral ports and malicious traffic patterns. Wang [6] analyzed the distribution of ports arger than 1024. Jeff Janies [7] studied the time series model of all the 65536 ports in a host, and described the model of server ports and the ports which were attacked by scanning through the visualization of time series graph. DongJin Lee [8] analyzed the distribution of the UDP to TCP ratio that corresponds to the ports. Allman [9] suggested using different methods to choose ephemeral ports to strengthen the security.

Kuai Xu [10] clustered source ports and destination ports, and calculated source IP entropy distribution, destination IP entropy distribution and source or destination port entropy distribution using information entropy, they classified entropy into low, medium, high three types of value, higher entropy indicated number of different corresponding IPs or ports are larger. We can check if a port is a server port or vulnerability by Kuai Xu’s Methodology. Thomas Karagiannis [11] believed that since client may use random ports to connect to server ports, if the port number of an IP address equals the number of related flows, the IP host is a client.

The problem in PortView is essentially similar to the challenges in superspreaders detection to maintain all the flows and the IP addresses existent in the memory during the measurement interval. Such as Snort [12] maintains a record for each active connection and a connection counter for each source IP. Because the number of flows and IP addresses are so huge in the network that these algorithms require a lot of memory to maintain the flows and the IP addresses records, or usually detect the superspreaders in low-speed links. In 2005, Venkataraman [13] proposed two algorithms, one-level filtering algorithm and two-level filtering algorithm, which maintain two hash tables to find superspreaders. These filtering algorithms proposed a t-superspreader IP detection algorithm based on sampling from the set of distinct source destination pairs. In this algorithm, t-superspreader IP is defined as more than tN connections to different destination IP addresses, where N is the number of source-destination IP pairs in the measurement interval.

Some data streaming algorithms have been applied to detect superspreaders. In 2007 Noriaki [14] also proposed a parametric algorithm based on the Bloom Filter to measure the flow numbers. However, they don’t compensate for the error brought by the hash collisions, and the memory consumption of the Bloom Filter (BF) is very large, so it is difficult to maintain the BF structure in the SRAM to adapt to the high speed links measurement. In 2003, Cohen [15] introduced a Spectral Bloom Filter (SBF), an extension of the original bloom filter to multi-sets, allowing the filtering of elements whose multiplicities are below a threshold. In 2005, Qi [16] proposed two algorithms for the detection of super sources and destinations. The first simple algorithm used a standard hash-based flow sampling algorithm, with a bitmap structure to maintain the flow records. The second algorithm uses a two-dimensional bit array to store the flow record and the flow number. In 2009, MyungKeun [17] created a virtual bit vector for each source by taking bits uniformly at random from a common one-dimension bit array instead of the two-dimensional bit array to store the flow record and the flow number.

Some interesting approach uses graphlet. One of the typical works has been proposed in [11] that can capture host behavior using empirically derived patterns. They represent these patterns using graphs, which they call graphlets. Having a library of these graphlets, they then seek for a match in the behavior of a host under examination. A similar work can also be found in [18], which proposed the use of a graph-based structure which is called a graphlet, to capture the interactions among the transport layer protocols, the destination IP addresses and the port numbers. Guillaume Dewaele [19] compared for two given hosts (with the anonymized IP provided in the trace) trace of the computed 9D features and of the associated graphlets.

The EM algorithm also groups the traffic flows into a small number of clusters and creates classification rules from the clusters in some previous works. Nguyen [20] looked at research into the application of Machine Learning techniques to IP traffic classification. In 2004 McGregor et al. [21] applied the EM algorithm to cluster traffic with similar observable properties into HTTP, FTP, SMTP, IMAP, NTP and DNS traffic. The EM algorithm proposed in AutoClass [22] determines the number of clusters and the parameters that govern the distinct probability distributions of each cluster. The EM algorithm in this paper is different from the previous work because we only classify port traffic into six behavior zones using the EM algorithm rather than some specific traffic applications.

6. Conclusion

Characterizing Macroscopic Port Roles from network traffic provides highly valuable information for various network management tasks. The main purpose of this paper is not to exhaustively list all possible port behaviors, but provides a definition and classification of port roles. In this paper, we firstly classify port roles into six categories, which are scanned ports, failed service ports, client ports, server ports, P2P ports, and overlay ports. Secondly, we propose a port role distribution diagram and define a port behavior measurement metrics to characterize network traffic. A two-class EM fuzzy clustering algorithm is designed to quantitatively determine the classification criteria in the classification distribution diagram. Finally we validate the proposed method based on network traffic captured from a link in the CERNET.

Our contributions include: (1) we research on ports of communication pattern and propose a method to evaluate ports role; (2) we define metrics which can distinguish different ports of communication pattern, and propose a two-class EM fuzzy clustering algorithm, which can achieve fast classifying multi-role of ports; (3) we introduce a new research direction of classifying port role from macroscopic points, which can be used to identify ports type in real-time, discover new applications, and detect new behavior of network attacks.