Catching modern botnets using active integrated evidential reasoning

Abstract

Botnets are now recognized as one of the major security threats to start various security attacks (e.g., spamming, DDoS). Although substantial research has been done towards botnet detection, it is becoming much more difficult today, especially for highly polymorphic, intelligent and stealthy modern botnets. Traditional botnet detection (e.g., signature, anomaly or flow based) approaches cannot effectively detect modern botnets. In this paper, we propose a novel active integrated evidential reasoning approach called SeeBot to detect modern botnets. SeeBot can seamlessly and incrementally combine host and network level evidences and incorporate active actions into passive evidential reasoning process to improve the efficiency and accuracy of botnet detection. Our experiments show that both performance and accuracy of botnet detection can be greatly improved by the active evidential reasoning, especially when the evidence is weak, hidden or lost.

1 Introduction

The total number of computers belonging to botnets increased from 3 millions in April-June 2009 to 6.5 millions during April-June 2010 [1]. Apparently, traditional botnet detection (e.g., signature, anomaly or flow based) approaches [2–6] cannot effectively detect and stop modern botnets. Based on a 2010 poll with chief information security officers and senior IT security directors at Fortune 500 corporations, all respondents stated that they considered malware and botnet to be a serious threat to their enterprise IT security.

Botnets are collections of infected computers that are controlled remotely by cyber-criminals. Originally botnets were created for a specific purpose such as sending spam, identity theft or DDoS (distributed denial-of-service) attacks. However, in 2010 bots that were designed to provide the cyber-criminal with the ability to build designer botnets (e.g., Zeus-based botnets) were rented out to other cyber-criminals for specific purposes (spam, identity theft, DDoS, etc). These criminal organizations invest significant building logical groupings of compromised systems that are organized around a sophisticated, resilient Command-and-Control (C&C) infrastructure or even through social networks [7]. Such criminal networks are exceptionally stealthy and easily evade signature or behavior-based defenses. They can mimic normal application and traffic patterns, and can change their core software far faster than traditional security solutions can update their signature-based systems. We refer to such professionally designed and cyber-criminal oriented botnets as modern botnets, which have the following features:

• Highly polymorphic: The characteristics of botnets are varying even faster (e.g., via polymorphism or code obfuscation) than the signature update from security vendors. The prevalence of improved do-it-yourself (DIY) botnet construction kits and associated exploit packs make this feature much more evident in 2010 [8].

• Highly intelligent: The bots, used to be called zombies, are powered with much more intelligence now, such as Honeypot-aware botnets [9]. Modern botnets can run multiple simultaneous infection mechanisms, update the malware installed on their victims systems regularly, and optimize their serial variant malware production systems to release “personalized” and one-of-a-kind malware with each new victim infection.

• Highly stealthy: Because of commerical motivation, modern botnets are designed to be more stealthy via many different mechanisms, such as randomly selected ports, traffic encryption and peer-to-peer based C&C.

Substantial research work has been done towards botnet detection. However, most botnet detection approaches attempted to follow and catch the trend of new botnet design, and then develop certain understanding or assumptions about the corresponding botnets. For example, (1) assuming certain botnet has a trackable payload pattern, payload signature based solutions [10] were developed; (2) assuming the existence of certain abnormal network activity or specific flow statistics, network anomaly detection based [6], flow feature [5] or communication pattern [4, 11, 12] detection based solutions were also developed; and (3) assuming the existence of similarities among bots, several behavior correlation based solutions have also been proposed [13, 14]. In general, the more assumptions we make on botnets, the more restriction the corresponding botnet detection solutions suffer from, and accordingly, the easier being bypassed by modern botnets.

No matter how a botnet may change its behavior or appear differently, the motivation of botnets stays the same, which is to conduct certain profitable activities in underground market. Among all changeable appearances (e.g., spreading and control methods) of botnets, we classify them into three levels:

• Polymorphic appearance: Some features or characteristics of botnets are highly variable as designed. For example, the malware signatures. Various new system and network vulnerabilities will keep being discovered and exploited. The large number of software vendors and service providers will continue to contribute to this trend.

• Changeable appearance: C&C provides the channel between a botmaster and bots. Once certain C&C mechanism being well-studied and effective detection methods becoming available, new C&C mechanisms will show up soon. However, this type of changing is not as fast as we can find in polymorphic appearance of botnets. On the other hand, common infection and spreading methods may be more effective. Thus, for a certain time period, C&C can be still regarded as reliably recognizable appearance.

• Stable appearance: the final goal of a botnet is to perform certain profitable attack activities, such as spamming and DDoS attack. Even such behavior has become well-understood, botnets will not change it. New profitable activities may be discovered later. However, comparing to another two types of appearance, Stable Appearance becomes the directly recognizable one.

In this paper, we advocate a rather different approach called SeeBot to show another niche in detecting modern botnets. SeeBot is built upon a new active evidential reasoning model, which integrates the advantage of both passive and active monitoring and detection into one framework. In this framework, SeeBot focuses on recognizing stable appearances, which are related to Infection and Attack actions (I&A), and Command and Control activities (C&C), as its initial detection targets (i.e., passive reasoning). In our approach, if the passive evidential reasoning is not sufficient to detect botnets, SeeBot automatically selects optimal verification actions to discover relevant symptoms that are important to collect the most critical evidences.

Our contribution in this work is twofold:

• We propose an active multi-layer causality model to seamlessly integrate active detection actions into passive evidential reasoning process, such that the robustness and resilience of a botnet detection system can be significantly increased, especially when initial symptoms are weak.

• We design an open and incremental evidential reasoning framework to be adaptive and extensible to a variety of different monitoring sources, such that the applicability and practicability of the system can be greatly improved, especially for detecting new botnets.

The rest of the paper is organized as the following. Section 2 discusses related work on botnet detection. Section 3 proposes an active evidential reasoning based botnet detection model. Section 4 evaluates SeeBot performance using simulations and controlled experiments with real traffic traces, and the conclusion is given in Section 5.

2 Related work

Sustantial research work has been done in botnet detection. In the following, we briefly discuss several related work.

Extensive studies have been conducted on understanding the characteristics and behaviors of various botnets. To collect and analyze bots, researchers widely utilize honeypot techniques [15–17]. Freiling et al. [16] used honeypots to track botnets in order to explore a root-cause methodology to prevent DoS attacks. Nepenthes [15] is a special honeypot tool for automatic malware sample collection. Rajab et al. [17] provided an in-depth measurement study of the current botnet activities by conducting a longitudinal multi-faceted approach to collect bots and track botnets. Cooke et al. [18] conducted several basic studies of botnet dynamics. In [19], Dagon et al. proposed to use DNS sinkholing technique for botnet study and pointed out the global diurnal behavior of botnets. Barford and Yegneswaran [20] provided a detailed study on the code base of several common bot families. Collins et al. [21] presented their observation of a relationship between botnets and scanning/spamming activities.

Several recent papers proposed different approaches to detect botnets. Ramachandran et al. [3] proposed using DNSBL (DNS blacklist) counter-intelligence to find botnet members that generate spams. This approach is useful for specific types of spam botnets. In [22], Reiter and Yen proposed a system TAMD to detect malware (including botnets) by aggregating traffic that shares the same external destination, similar payload, and that involves internal hosts with similar OS platforms. The corresponding aggregation method based on destination networks focuses on networks that experience an increase in traffic as compared to a historical baseline. Different from [14] that focuses on botnet detection. The scheme proposed in [22] aims to detect a broader range of malware.

Livadas et al. [4, 11] proposed a machine learning based approach for botnet detection using some general network-level traffic features of chat-like protocols such as IRC. Karasaridis et al. [5] studied network flow level detection of IRC botnet controllers for backbone networks by matching a known IRC traffic profile. Rishi [10] is a signature-based IRC botnet detection system by matching known IRC bot nickname patterns. Binkley and Singh [6] proposed combining IRC statistics and TCP work weight for the detection of IRC-based botnets. Gu et al., 2007 [23] described BotHunter, which is a passive bot detection system that uses dialog correlation to associate IDS events to a user-defined bot infection dialog model. Different from BotHunter’s dialog correlation or vertical correlation that mainly examines the behavior history associated with each distinct host, BotMiner utilizes a horizontal correlation approach that examines correlation across multiple hosts. BotSniffer [12] is an anomaly-based botnet C&C detection system that also utilizes horizontal correlation. However, it is used mainly for detecting centralized C&C activities (e.g., IRC and HTTP).

Many botnet detection solutions were designed based on certain assumptions on botnets with specific directly observable evidence (e.g., IRC botnet detection) or indirectly derivable evidence (e.g., correlation based botnet detection, a pattern of sequential observable network activities). Behavior similarity correlation based approach [13, 14] cannot adapt to modern multi-function botnets with polymorphic behaviors.

More recently, p2p has been exploited as a new C&C mechanism in many modern botnets, which brings to a botnet detection system new challenges mainly on two aspects: (1) how to detect p2p traffic from background traffic; (2) how to distinguish C&C p2p traffic from legitimate p2p applications. Several solutions [24, 25] have been proposed. Yen and Reiter, 2010 [24] showed that the different goals and circumstances, the features related to traffic volume, “churn” among peers, and differences between human-driven and machine-driven traffic make distinguishable behaviors in these p2p applications. Zhang et al., 2011 [25] proposed a novel botnet detection system based on statistical fingerprints to profile P2P traffic and identify stealthy P2P botnets.

In summary, we categorize those existing solutions designed based on certain pre-conditions (e.g., C&C mechanisms, sequential activities, behavior correlations) as condition-constrained botnet detection approach. All the solutions discussed here are in this category. To the best of our knowledge, SeeBot is the first active evidential reasoning framework for botnet detection that only assumes the intrinsic botnet activities. Many proposed solutions can work perfectly on detecting certain type of botnets as long as the expected pre-conditions are valid. The advantage of SeeBot lies in its high robustness, adaptability and applicability.

3 Active evidential reasoning

As discussed in Section 1, the three characteristics in modern botnet, namely highly polymorphic, intelligent and stealthy, make current botnet detection approaches miscellaneous and manifold. Botnet detection is essentially a process of detecting exposed botnet activities based on collected evidence.

An evidential reasoning approach usually uses a belief structure to model an assessment with uncertainty. In this section, we will formalize botnet detection as an evidential reasoning process. Accordingly, we will propose a new belief structure called Active Multi-layer Casuality Graph as shown in Figure 1, which seamlessly incorporate multiple components with different roles and levels, including Botnet, Evidence, Symptom and Actions, into the same active evidential reasoning framework.

Figure 1

Active multi-layer casuality graph.

In the following, we first introduce Active Multi-layer Casuality Graph, which is developed up the concept of casuality graph commonly used in evidential reasoning. Then we present the active evidential reasoning framework called SeeBot, and elaborate its functional modules.

3.1 Causality graph

A casualty graph [26] is a bipartite directed acylic graph to describe the Symptom-Cause correlation, which represents the causal relationship between each cause c _i and a set of observable symptoms $S_{c_i}$ that may be triggered by c _i . Symptom-Cause causality graph provides a vector of correlation likelihood measure called likelihood indicator I(s _j |c _i ), to bind a root cause c _i to its relevant observable symptoms $S_{c_i}$. In a causality graph between root causes C and symptoms S, if I(s _j |c _i )=0 or 1 for all (i,j), we call such causality model a deterministic model; otherwise, we call it a likelihood model.

However, a general casualty graph cannot satisfy the requirements in botnet detection, which can be mainly shown in the following two aspects.

First, the flat symptom structure in a casualty graph cannot completely represent the complicated relations among symptoms. A symptom may be the result from several other observed symptoms. For example, spamming, a symptom of common botnet attack, can be observed as the symptoms like (a) high volume of outbound TCP traffic with destination port TCP/25, and (b) multiple queries on different DNS MX records, etc. Accordingly, we break the original flat symptom structure into a two layer hierarchy between Symptom and Evidence to represent such complexity. Thus, a indirectly unobservable evidence can be jointly manifested by several directly observable symptoms. On the other hand, a symptom may contribute to believe the existence of different evidence.

Second, a general casualty graph can only represent a passive reasoning process. If any evidence is missing due to various reasons (e.g., packet loss), the passive reasoning result is commonly not satisfiable. Accordingly, in addition to the change made in the first step, we extend an action layer that is associated directly with the symptom layer to meet such requirement, which can selectively take actions to verify the most likely existed but lost evidence.

3.2 Active multi-layer causality graph

Active Multi-layer Causality Graph, denoted as AMCG and shown in Figure 1, consists of three bipartite directed acylic graphs hierarchically connected by different relationships. We use B={b₁,b₂,⋯,b _n } to denote the cause set representing different types of botnet (e.g., IRC or P2P botnet), E={e₁,e₂,⋯,e _m } to denote the evidence set that can be jointly used to detect the occurrence of botnet (e.g., P2P traffic, spamming), S={s₁,s₂,⋯,s _k } to denote the symptom set that can be directly observed, and used to determine the existence of evidences (e.g., high Max degree ratio [27], high volume of SMTP traffic), and A={a₁,a₂,⋯,a _q } to denote the action set used to check the symptoms.

There are two casualty correlations between B and E denoted as M_B×E, as well as between E and S denoted as M_E×S. M_B×E is used to define causal certainty between various botnet b _i (b _i ∈B) and evidence e _j (e _i ∈E). M_E×O is used to define causal certainty between evidence e _j (e _i ∈E) and symptom s _k (s _k ∈S). Evidence-Botnet causality graph provides a vector of correlation likelihood measure denoted as indication measure I(e _j |b _i ) to bind a type of botnet b _i to a set of its evidences $E_{b_i}$. Similarly, Symptom-Evidence causality graph provides a vector of correlation likelihood measure denoted as indication measure I(s _k |e _j ) to bind an evidence e _j to a set of its symptom $S_{e_j}$.

We also use A={a₁,a₂,⋯,a _q } to denote the list of actions that can be used to check symptoms. We describe the relation between actions and symptoms using Action Book represented as a bipartite graph as shown in Figure 1. For example, the symptom s₁ can be verified using action a₁ or a₂. The Action Book can be defined by network managers based on symptom type, the network topology, and the available symptom validation tools.

The active multi-layer hybrid causality graph Botnet-Evidence-Symptom-Action graph is viewed as a 6-tuple (B;E;S;A;C₁;C₂;C₃), where botnet set B, evidence set E, symptom set S, and action set A are four independent vertex sets. Every correlation edge in C₁ connects a vertex in E and a vertex in B to indicate causality relationship between evidences and botnets. Every correlation edge in C₂ connects a vertex in O and a vertex in E to indicate causality relationship between symptoms and evidences. Every correlation edge in C₃ connects a vertex in A and a vertex in S to indicate verifiable relationship between actions and symptoms, referred as the Action Book.

3.3 Active evidential reasoning framework

SeeBot consists of four modules as shown in Figure 2, which are Evidence Mining (EM), Evidential Reasoning (ER), Plausible Reasoning (PR) and Action Selection (AS) modules. Evidence Mining module processes received symptoms from a passive network monitoring system and generate the corresponding evidences for all hosts in a monitored network based on Symptom-Evidence casuality relationship specified in AMCG. Evidential Reasoning module passively analyzes evidences, shows botnet likelihood evaluation for all hosts identified in from Evidence Mining module, and dynamically constructs a plausible graph presenting a plausible relationship between those hosts and each botnet category. Plausible Reasoning module identifies the smallest set of botnets to explain observed evidences for all related hosts, verifies if the confidence level of the reasoning result is satisfactory.

Figure 2

Active evidential reasoning framework.

If the current related evidence is strong enough to explain the botnet hypothesis, then the reasoning process terminates. Otherwise, a list of most likely missing symptoms that can increase confidence on the botnet hypothesis is sent to Action Selection module. Selected actions are conducted to determine which unobserved symptoms have actually occurred and accordingly adjust hypothesis confidence level. If the new confidence level is satisfactory, then the reasoning process terminates; otherwise, the new symptom is fed into the fault reasoning module to create a new hypothesis. This process is recursively invoked until a highly credible hypothesis is found.

$$\begin{array}{l}\mathit{\text{CF}}\left(b_i\right)={\mathit{\text{CF}}}_1\left(b_i\right)\times {\mathit{\text{CF}}}_2\left(b_i\right)\\ =\prod _{t\in \{1,2\}}\left[1-\prod _{e_j\in E_t}\left(1-I\left(e_j\right)I\left(b_j\right|e_j)\right)\right]\end{array}$$

(1)

$$\begin{array}{l}C{F}^{n+1}\left(b_i\right|e)=\left[1-(1-{\mathit{\text{CF}}}_{T\left(e\right)})(1-I(e\left)I\right(b_i\left|e\right))\right]{\mathit{\text{CF}}}_{1-T\left(e\right)}\\ =I\left(e\right)I\left(b_i\right|e){\mathit{\text{CF}}}_{1-T\left(e\right)}+(1-I\left(e\right)I\left(b_i\right|e\left)\right)C{F}^n\end{array}$$

(2)

3.3.1 Evidence mining

In this paper, we characterize a botnet using two essential coexisted behavior or evidence categories, namely (1) Infection and Attack (I&A) evidence denoted as E₁ that reflect botnet motivation, and (2) coordinated command and control (C&C) evidence denoted as E₂ that show the fundamental difference of botnet from other malwares. For detecting a botnet, we should observe evidences from E₁AND E₂.

We use Symptom-Evidence bipartite casuality graph, as discussed in Section 3.1, to represent the complex relationships among various observations from different network and security monitoring systems. For I&A evidence category, we choose 10 E₁ evidences in the current version of SeeBot, denoted as E₁={e₁₁,⋯,e_1A}, including scanning activity, spamming, DDoS, portable executable (PE) binary downloading, etc. For C&C evidence category, we choose 4 E₂ evidences in the current version of SeeBot, denoted as E₂={e₂₁,⋯,e₂₄}, including P2P, IRC, HTTP, DNS. We use a parameter Evidence Degree (or ED) to indicate the number of associated symptoms. For example, as shown in Table 1, E D(e₁₁) = 2 and E D(e₂₁) = 3. More specifically, if we have observed {s₁,s₆,s₇}, we strongly believe there is P2P communication among certain investigated hosts [27]. Please note that Table 1 just shows an example of Evidence-Symptom-Action Casuality graph. Our current implementation of SeeBot has the average evidence degree 23 among all defined evidences.

Table 1 An example of evidence-symptom-action casuality

In our model as the example shown in Table 1, multiple symptoms are required to verify the existence of evidence, denoted as e₁₁⇐{s₁,s₂}; on the other hand, one symptom may also be used as supportive observation for different evidence, denoted as s₁→{e₁₁,e₂₁}. Furthermore, our Symptom-Evidence causality adopts a deterministic model, which implies I(s _k |e _j )=1 if e _j exists ($\forall s_k\in S_{e_j}$). We believe such simplification via the deterministic model is a necessary and effective step to release many typical botnet detection solutions from the burden in keeping track of the difference and details of various botnets. The input of this module is various tracking symptoms, and the output is all related evidences and their indication strength for each host relating to those symptoms.

3.3.2 Evidential reasoning

Evidence shows certain indication that leads to a decision with more or less uncertainty. The stronger indication an evidence shows, the less uncertainty a conclusion remains. Traditional uncertainty reasoning approaches based on Bayesian Network and Dempster-Shafer theory are inapplicable to intrusion detection due to lack of prior knowledge [28]. In our system, we adopt the scaling mechanism proposed in [28] to classify available evidence into three levels: strong (S), moderate (M), and weak (W) indications. Further, we evaluate a decision on botnet detection into three likelihood levels as well: strong, moderate and weak confidence. To facilitate the reasoning process and practical operation, we empirically quantify each evidence level with numerical values S=0.9,M=0.5 and W=0.1 for strong, moderate, and weak likelihood levels.

In evidential reasoning, the confidence on botnet detection (CF), as shown in Eq. 1, depends on the confidence on the detection of I&A (CF₁) and C&C (CF₂). Furthermore, CF₁ and CF₂ rely on the joint events and their likelihood levels (i.e., S,M,W) from E₁ and E₂ respectively. In Eq. 1, $I\left(e_j\right)=\frac{\left|S_{e_j}^o\right|}{\left|S_{e_j}\right|}$, is denoted as evidence indication strength. If one type of evidence is completely missing, we use a null evidence (e _u ) to represent, and I(e _u )=0. In such a case, C F=0.

Next, we present a method to incrementally update current confidence level denoted as C Fⁿ to a new level C Fⁿ⁺¹ when a new evidence e with I(e)=x is observed. Here, we use a variable T(e) (T(e)∈{0,1}) to represent the event category. If T(e)=0, e∈E₁; otherwise, e∈E₂. Since C Fⁿ=CF₁×CF₂=(1-(1-CF₁))(1-(1-CF₂)), we derive the method as shown in Eq. 2 to incrementally update current confidence.

3.3.3 Plausible reasoning

After the process of Belief Reasoning, a Plausible Bot Graph as shown in Figure 3 is constructed as the following to represent the dynamic relationship between the potential faulty components and the evidence.

• For each botnet in B, to associate a botnet vertex b _i .

• For each investigated host component, to associate a host vertex h _j . with its total belief metric Ψ _c .

• For each investigated host component h _j , to associate a link to all its related botnets with the weight I(b _i |h _j ) (here, I(b _i |h _j )=C F(b _i )).

Figure 3

Plausible Bot graph.

A plausible botnet reasoning problem is to find the minimal number of most likely botnets based on Plausible Bot Graph that explain all related evidences with investigated hosts.

Algorithm 1 Plausible Botnet Reasoning Algorithm

Theorem 1. A plausible botnet reasoning problem is NP-complete.

Proof. A plausible botnet reasoning problem can be reduced to a Weighted Set Cover (WSC) problem as follows. For a set of investigated hosts H={h₁,h₂,⋯,h _n }, a cover is defined as a subset of hosts $H_{b_i}$ that can be explained by one botnet b _i . Each cover $H_{b_i}$ is assigned a weight as $W\left(H_{b_i}\right)=1/\sum _{h_j\in H_{b_i}}I\left(b_i\right|h_j)$. Obviously, the less weight a cover is, it is more likely that the corresponding botnet is the cause. Thus, the plausible botnet reasoning is to find a collection of covers (i.e., botnets) such that $\bigcup _{b_i\in B}{H}_{b_i}=H$ with $\mathit{\text{min}}\left(\sum _{b_i\in B}W\right(H_{b_i}\left)\right)$. This is a Weighted Set Cover problem that is NP-complete.

We adopt greedy heuristic algorithm [26] for the plausible botnet reasoning problem as shown in Algorithm 1, where B is a set of potentially botnets; H is a set of investigated hosts; R is a set of inferred botnets.

3.3.4 Action selection

Figure 1 presents the verification relationship between evidence {e₁,e₂,⋯ } and actions {a₁,a₂,⋯ }. For example, Evidence e₁ can be verified by taking a combination of action a₁,a₂ and a₃, which can be denoted as a new virtual action vertex v₁ associated with a cost of the sum of C(a₁),C(a₂) and C(a₃). Action v₁ can verify all symptoms (s₁, s₂) that are verifiable by either a₁,a₂ or a₃. After converting joint actions to a virtual action, Symptom-Action correlation can be represented in a bipartite graph.

The goal of the Action Selection algorithm is to select the actions that cover all evidences E _UO with a minimal action cost. Based on Symptom-Action bipartite graph, we can model this problem as WSC problem to solve it [26].

4 Evaluation

In this section, we present our evaluation metrics, experiment methodology and experiment results.

4.1 Evaluation metrics

We evaluate SeeBot from the following three aspects: performance, accuracy and sensibility.

The performance of SeeBot is measured by botnet detection time τ, which is the time between receiving the first evidence (i.e., when malware becomes active) and identifying the actual botnets. The accuracy of SeeBot depends on two factors: (1) the detection ratio (μ), which is the ratio of the number of true detected botnets (B _d is the total detected fault set) to the number of actual occurred botnets B _h , formally $\mu =\frac{|B_d\cap B_h|}{B_h}$; and (2) false positive ratio (ν), which is the ratio of the number of false reported botnets to the total number of detected botnet, formally $\nu =\frac{|B_d-B_d\cap B_h|}{B_d}$. We have also analyzed system sensitivity showing the impact of the number of evidences and their evidence degrees on the system performance and accuracy.

4.2 Experiment methodology

One challenge in evaluating botnet detection solutions is the lack of group truth. The evaluation method adopted in our evaluation is called active evaluation, in which we execute real botnet binaries in a controlled network environment. By properly selecting different type of botnets, we can clearly demonstrate the applicability and robustness of SeeBot in detecting botnets.

Our controlled experiment environment has the following configuration: a Cisco ASA firewall with NetFlow enabled, a Cisco switch with SPAN port connecting a Snort IDS, and a event monitoring server with SeeBot installed for analyzing various symptoms, including Snort alerts, syslogs, DNS logs and host events (e.g., IRC activity log, HIDS events, AVG anti-virus application, malware alerts) from distributed systems. In our system, network related evidence is adaptively combined with host related evidence. More specifically, network evidence will be initially analyzed at the first stage. The confidence evaluation on the generated detection hypotheses will be consistently conducted and incrementally updated with new evidence either passively input to actively collected from the monitored network system. Host based evidence such as security logs will be collected only when the passively input network related evidence is insufficient.

The testbed consists of eight physical workstations and 20 virtual machines, with Ubuntu and Windows XP installed. The system has been directly connected to th1e Internet for several months to record all traffic traces that will be used later as the background traffic when we disconnect the testbed from public network and run malicious software. More than 500 different type of botnet binaries have been collected over one year, and we selected 24 binaries among them, including 9 IRC botnet, 8 HTTP botnet, and 7 P2P botnets. In each test, the number of infected machines M increased from 3 to 27 at a speed 3 each time. For each test run, we randomly copied 1, 2, 4, 8 botnet binaries to the selected infection machines and execute them. In order to evaluate the impact of SeeBot parameters, we set up SeeBot with 3 different E₁/E₂ configurations as: 4/4, 8/4, and 12/4. For each event configuration (e.g., 4/4, standing for 4 E₁ and 4 E₂ events), the average event degrees (ED) are set to 5, 10, 15, and 20.

4.3 Experiment results

Figure 4 shows the performance evaluation results. Figure 4-(a) indicates SeeBot can detect botnets effectively (average in 5s), especially when more hosts infected. This is one obvious feature in an evidential reasoning system. Since SeeBot is not designed for any specific type of botnets, the evidence from multiple mixed botnets can jointly increase the detection performance, which is also clearly shown in Figure 4-(a). Figure 4-(b) shows the detection rate is above 90% on average with false positive rate less than 5% on average as shown in Figure 4-(c). In our framework, we identify the I&A and C&C related behaviors represent intrinsic botnet activities. If the evidence related to one of these two behaviors is significantly missing, SeeBot will not perform well initially. However, active reaction feature in SeeBot can effectively improve its detection rate by searching for missing evidence.

Figure 4

Performance evaluation. (a) Detection time (s) (b) Detection rate (c) False Positive Rate.

Figure 5 implies that with the increase of evidences and the corresponding associated symptoms, the overall performance of SeeBot is evidently increased. However, the improvement due to increasing evidences and symptoms becomes much slow after certain thresholds as the total evidences |E₁|=8, |E₂|=4 and the average evidence degree 15. Such an observation shows SeeBot can perform equivalently good as long as certain amount of evidence available to the system.

Figure 5

Evidence degree impact. (a) The impact of evidence degree on detection rate. (b) The impact of evidence degree on false positive rate.

As shown in Figure 5 and the following experiment results, the false positive rate sometime can be high. The main reason of causing false positive results is due to the multi-association (i.e., one symptom may associate to multiple evidence) between the elements in the evidence and symptom sets, especially when some symptoms are not detected or lost.

SeeBot provides an open framework such that different symptoms, evidences and botnets can be selected and associated together. Apparently, the evidence indication strength (as defined in Sec.3.3.2) between associated evidences and botnets can directly affect the system performance. Apparently, the general rule is to associate “Strong” evidences with botnets. However practically, the feasibility of constructing such a framework may depend on the configuration of a real network system. To clearly characterize the impact of the evidence indication strength on the performance of SeeBot, we have conducted experiments in three cases with all weak, all mediate and all strong evidences to validate the SeeBot performance. In our experiments, the different infection rates show the similar results. Thus, we only show the experiment results with two botnets per infection. As shown in Figure 6, the different evidence indication strength does cause some differences (10%–20% difference) on both detection time and detection rate, and makes significant difference on false positive rate (50%–500% difference). The time difference is caused by the consumed time in active reasoning process when SeeBot tries to increase the confidence level on the detection results. Even with weak evidence, SeeBot can still localize relevant botnets the same way as its receiving strong evidence. Thus, the difference on detection rate is not obvious. Since weak evidence is typically associated suspicious botnets, the false positive rate can be really high due to such ambiguity.

Figure 6

The impact of evidence indication strength on (a) Detection time (s), (b) Detection rate, and (c) False Positive Rate.

Many existing botnet detection tools were designed based on various assumptions, e.g., the existence of dialog model in BotHunter. The advantage of SeeBot lies in its reasoning capability in botnet detection, especially when available evidence is incomplete, which does not rely on certain botnet behavior to work. Since various botnets are used in our evaluation, for a fair comparison to the related work, we have implemented the essential passive reasoning system based on the proposed solution from [2, 23, 27]. To illustrate the difference in the passive approach and SeeBot, we further created two scenarios: (1) with incomplete evidence; (2) with complete evidence. As shown in Figure 7, when the observable evidence is complete, the difference between the two approaches are not obvious. However, when we only provided incomplete evidence to the two systems, the advantage of SeeBot over the passive approach is very significant. For instance, the passive approach cannot even start working (we set two minutes as a timeout threshold) when the evidence availability (i.e., 80% of evidence missing in our experiment) is low. Even when evidence availability rate increase to 50% such that the passive system started working, SeeBot is clearly superior to the passive approach as shown in Figure 7.

Figure 7

The performance comparison between SeeBot and the passive approach on (a) Detection time (s), (b) Detection rate, and (c) False Positive Rate.

5 Conclusion and future work

In this paper, we advocate a rather different approach called SeeBot to show another niche in detecting modern botnets. SeeBot is built upon a new active evidential reasoning model, which integrates the advantage of both passive and active monitoring and detection into one framework. The experiments show that SeeBot can effectively detect modern botnets, especially when the initial evidence is not evident. Our future work includes the development of an uncertainty evaluation model to provide a confidence measurement on the detection results. We also feel it is important that SeeBot can detect hidden common characteristics embedded in collected evidences to characterize new botnets. The future version of SeeBot could be enhanced with certain effective learning mechanism such as reinforcement learning to achieve such a desirable feature.