Chapter 4: Machine Learning Pipeline¶

Understanding the Data¶

An important step towards using data for machine learning is understanding the nature and characteristics of the data. This step is important for a number of reasons. First, it helps us understand the quality of the data, including critical aspects such as the size of the dataset, where the data was gathered, and so forth. Second, a deeper exploration can help us understand the characteristics of the data that may ultimately lead us to defining features that improve classification accuracy.

Before exploring any dataset in detail, it is helpful to understand basic characteristics concerning its size, and the nature of the data. Let’s continue with the simple example from last chapter, where we loaded a packet trace into a Pandas dataframe and generated features.

Before we proceed with feature generation or machine learning, however, we should understand some basics from out dataset, including, for example, how many packets are in the dataset, the start and end times of the dataset, minimum and maximum packet lengths and so forth. We can use the describe method of the Pandas dataframe to get a summary of the dataset. The example below shows an example of the output of the describe method for the dataframe from the last chapter to enable some sanity checking of the data.

The output from this exercise is a helpful sanity check. The “shape” command shows us how many rows and columns that we have in our data (specifically, the number of rows, corresponding to the number of packets; and the number of columns, corresponding to the number of features). The describe command shows us some basic statistics about the data, including the minimum and maximum values for each feature, as well as the mean and standard deviation for each feature. The head command shows us the first few rows of the dataset; the tail command shows us the last few rows of the dataset; and the info command shows us the data types of each column, all of which can be helpful for sanity checking the data.

For example, we can see the minimum and maximum values for the length of each packet: 60 bytes and 1514 bytes, respectively. These values should make sense, given our domain knowledge about network traffic and packet maximum transmission unit (MTU). Similarly, we see that the minimum port number is 53 (a well-known service port for DNS), and the maximum port is less than 65536 (the maximum allowable port number, given that the field is only 16 bites).

from netml.pparser.parser import PCAP

pcap = PCAP('../examples/notebooks/data/http.pcap')
pcap.pcap2pandas()
pdf = pcap.df
pdf = pdf[['datetime', 'ip_src', 'port_src', 'ip_dst', 'port_dst', 'length']]

# Example sanity checking of data.
print("\n\nDimension of data: {}".format(pdf.shape))

# Print minimum and maximum times.
print("\n\nMin time: {}\nMax time: {}".format(min(pdf['datetime']), max(pdf['datetime'])))

# Describe the dataframe.
pdf.describe()

WARNING: No IPv4 address found on anpi0 !
WARNING: No IPv4 address found on anpi1 !
WARNING: more No IPv4 address found on en3 !

Dimension of data: (24798, 6)


Min time: 2022-09-29 20:24:42
Max time: 2022-09-29 20:25:30

Understanding the distribution of features can also help us understand which features may be useful for a particular classification problem. For example, in the example above, we might hypothesize that there is a relationship between the length of a packet and the type of application that generated the packet, or the direction of traffic. We can test this hypothesis by plotting the distribution of packet lengths for each port number, as shown below.

Dealing with Outliers¶

Inevitably, data gathered from real-world network measurements will often contain outliers. Outliers exist in many datasets, and there are many possible ways to deal with them. The first step in dealing with outliers is to identify them. One way to identify outliers is to use a box plot, which is a common way to visualize the distribution of data. A box plot is a graphical representation of the distribution of data, where the box represents the interquartile range of the data, and the whiskers represent the minimum and maximum values of the data. Outliers are typically represented as points outside of the whiskers. The example below shows a box plot of the minimum and maximum packet lengths for each flow in the dataset we explored with netml from the last chapter.

Exploring the distribution of packet lengths to identify outliers.

import matplotlib.pyplot as plt

# create a boxplot using matplotlib
fig, ax = plt.subplots()

# Explore the distribution of `netml` features
ax.boxplot([features[8], features[9]])
ax.set_xticklabels(['Minimum Lengths', 'Maximum Lengths'])

# set labels for the plot
ax.set_title('Packet Lengths for Each Flow')
ax.set_yscale('log')
ax.set_ylabel('Column Name')

# show the plot
plt.savefig('pipeline_length-boxplot.png')
plt.show()

Why Data Needs Labels¶

The goal of supervised machine learning is to use labeled data to train a model that can the predict correct labels (sometimes referred to as targets or target predictions) for data points that the model did not see when it was being trained.

As a human analogue, supervised machine learning is akin learning how to solve a general problem by seeing the “correct” answer to several specific instances and identifying more general patterns that are useful for solving any instance of the task. For example, when driving, you know that the correct action when seeing a stop light is to bring your car to a halt, even though you may have never seen that specific stop light at that instance ever before. In the context of networking, then, many such examples exist. A machine learning model might automatically recognize an attack or traffic from a specific application based on “signature” properties, even though the model has never before seen that specific traffic trace.

Any use of supervised learning to solve a problem must thus necessarily involve the collection of labeled training data. Training data typically has two aspects: examples (i.e., multi-dimensional data points) and labels (i.e., the “correct answers”). The training examples are observations or data points. The training labels consist of the “correct” categories or values that a trained machine learning algorithm should assign to each of the training examples.

In addition to training examples, creating a supervised machine learning model also requires an associated label for each set of feature. For single-label learning tasks, there will be a single label for every example in the data set. So if you have a matrix of training examples with 20 rows, you will need to have 20 labels (one for each example). The training labels are thus typically represented as a vector with the same number of elements as the number of training examples. For instance, each training label for a security task might be a 1 if the corresponding example corresponds to malicious traffic, or a 0 if the corresponding example is benign.

For multi-label learning tasks, the data may be associated with multiple label vectors or a combined label matrix. For example, you might want to predict users’ responses to a quality of experience questionnaire. For each question on the questionnaire, you would need a label corresponding to each user. For convenience, the vector (or matrix, for multi-label tasks) of training labels are usually given the variable name y. The first dimension of the training labels is the same cardinality as the first dimension of the training examples.

The example below shows an example of labeling data, based on two loaded packet captures. The first trace is a packet capture of benign traffic, and the second is a trace of HTTP scans for the log4j vulnerability. The example shows the use of netml to load the packet captures and extract features and the subsequent process of adding a label to each of the resulting data frames, before concatenating the two frames and separating into X and y, for eventual input to training a machine learning model.

hpcap = PCAP('../examples/notebooks/data/http.pcap', flow_ptks_thres=2, verbose=0)
lpcap = PCAP('../examples/notebooks/data/log4j.pcap', flow_ptks_thres=2, verbose=0)

# extract flows from pcap
hpcap.pcap2flows()
lpcap.pcap2flows()

# extract features from each flow via IAT
lpcap.flow2features('IAT', fft=False, header=False)
ld = pd.DataFrame(lpcap.features)

# extract features from each flow via IAT
hpcap.flow2features('IAT', fft=False, header=False)
hd = pd.DataFrame(hpcap.features)

hds = hd.loc[:,:4]
hds.shape

# label the normal and anomalous HTTP traffic traces
pd.set_option('mode.chained_assignment', None)
hds['label'] = 0
ld['label'] = 1

# concatenate the two data frames
data = pd.concat([ld,hds])

# separate into features and labels
X = data.loc[:,:4]
y = data['label']

Labeling for Prediction Problems¶

The format of the training labels distinguishes regression problems from classification problems. Regression problems involve labels that are continuous numbers. Classification problems involve discrete labels (e.g. integers or strings). Classification tasks can be further categorized into binary classification and multi-class classification by the number of distinct labels. While some supervised machine learning algorithms can be trained to perform either classification or regression tasks, others only work for one or the other. When choosing a machine learning algorithm, you should pay attention to whether you are attempting a classification or regression task. Our description of common machine learning models in the next chapter clearly states whether each algorithm is applicable to classification, regression, or both.

• Regression. Training labels for regression problems are simply encoded as real-value numbers, usually as a floating point type. Regression may be familiar if you have experience fitting lines or curves to existing data in order to interpolate or extrapolate missing values using statistics techniques. Regression problems can also be performed with integer labels, but be aware that most off-the-shelf regression algorithms will return predictions as floating point types. If you want the predictions as integers, you will need to round or truncate them after the fact.

• Binary Classification. Training labels for binary classification problems are encoded as 1s and 0s. The developer decides which class corresponds to 1 and which class corresponds to 0, making sure to keep this consistent throughout the ML training and deployment process.

• Multi-class Classification: Training labels for multi-class classification problems are encoded as discrete values, with at least three different values represented in the vector of training labels y. Note that multi-class classification (more than two discrete classes) is distinct from multi-label classification (multiple independent classification tasks, each with their own set of discrete classes). There are multiple options for encoding training labels in multi-class classification problems. Since most machine learning algorithms require training labels as numeric types (e.g. int, float, double), string class names usually need to be encoded before they can be used for ML training (decision trees are a notable exception to this requirement). For example, if you have labels indicating users’ quality of experience on a “poor”, “acceptable”, “excellent” scale, you will need to convert each response into a number in order to train an algorithm to predict the QoE of new users.

One way to perform encoding for multi-class classification is to assign each class to a unique integer (ordinal encoding). In the example above, this mapping could convert “poor” –> 0, “acceptable” –> 1, and “excellent” –> 2. The fully encoded training labels would then be a vector of integers with one element for each training example. Another way to encode multi-class training labels is to perform one-hot encoding, which represents each class as a binary vector. If there are N possible classes, the one-hot encoding of the ith class will be an N-element binary vector with a 1 in element N and a 0 in all other elements. In the example above, this mapping could convert “poor” –> [1,0,0], “acceptable” –> [0,1,0], and “excellent” –> [0,0,1].

Although it is useful to understand these subcategories of supervised learning, it is also wise not to assign excessive importance to their distinctions. Many machine learning algorithms are effective for either classification or regression tasks with minimal modifications. Additionally, it is often possible to translate a classification task into a regression task, or vice versa. Consider the example of predicting improvements in last-mile network throughput for different possible infrastructure upgrades. While it is reasonable to express throughput as a real-valued number in Mbps, you may not care about small distinctions of a few bits per second. Instead, you could reframe the problem as a classification task and categorize potential updates into a few quality of experience classes.

Training and Testing Sets¶

The goal of supervised learning is to train a model that takes observations (examples) and predicts labels for these examples that are as close as possible to the actual labels. For instance, a model might take traffic summary statistics (e.g., IPFIX records) as input features and predict quality of experience labels (or target predictions), with the goal that the predicted labels are as “close” as possible to the actual quality experienced by the customer.

However, if you don’t know the correct labels for new observations, how do you measure whether the model is succeeding? This is an important question and one that must be considered at the beginning of the ML process. Imagine that you’ve trained a machine learning algorithm with training examples and training labels, then you deploy it in the wild. The algorithm starts seeing new data and producing predictions for that data. How can you evaluate whether it is working?

The way to solve this problem is to test the performance of the trained algorithm on additional data that it has never seen, but for which you already know the correct labels. This requires that you train the algorithm using only a portion of the entire labeled dataset (the training set) and withhold the rest of the labeled data (the test set) for testing how well the model generalizes to new information.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

The examples that you reserve for the test set should be randomly selected to avoid bias in most cases. However, some types of data require more care when choosing test set examples. If you have data for which the order of the examples is particularly important, then you should select a random contiguous subset of that data as your test set. That maintains the internal structure that is relevant for understanding the test examples. This is particularly important for chronological data (e.g. packets from a network flow). If you do not select a contiguous subset, you may end up with a nonsensical test set with examples from one day followed by examples from a year later followed by examples from a few months after that.

This brings us to the golden rule of supervised machine learning: never train on the test set! Nothing in the final model or data pre-processing pipeline should be influenced by the examples in the test set. This includes model parameters and hyperparameters (more below). The test set is only for testing and reporting your final performance. This allows the performance evaluation from the test set to be as close of an approximation as possible to the generalization error of the model, or how well the model would perform on new data that is independent but identically distributed (i.i.d.) to the test set. For example, when you report that your model performed with 95% accuracy on a test set, this means that you also expect 95% accuracy on new i.i.d. data in the wild. This caveat about i.i.d. matters, because if your training data and/or test set is not representative of the real-world data seen by the model, the model is unlikely to perform as well on the real-world data (imagine a trained Chess player suddenly being asked to play in a Go tournament).

The best way to avoid breaking the golden rule is to program your training pipeline to ensure that once you separate the test set from the training set, the test set is never used until the final test of the final version of the model. This seems straightforward, but there are many ways to break this rule.

A common mistake involves performing a pre-processing transformation that is based on all labeled data you’ve collected, including the test set. A common culprit is standardization, which involves normalizing each feature (column) of your dataset such that variance matches a prespecified distribution with mean of zero. This is a very common pre-processing step, but if you compute standardization factors based on the combined training and test set, your test set accuracy will not be a true measure of the generalization error. The correct approach is to compute standardization factors based only on the training set and then use these factors to standardize the test set.

Another common mistake involves modifying some hyperparameter of the model after computing the test error. Imagine you train your algorithm on the training set, you test it on the test set, and then you think, “Wow, my test accuracy was terrible. Let me go back and tweak some of these parameters and see if I can improve it.” That’s an admirable goal, but poor machine learning practice, because then you have tuned your model to your particular test set, and its performance on the test set is no longer a good estimate of its error on new data.

Validation Sets¶

The correct way to test generalization error and continue to iterate, while still being able to test the final generalization error of the completed model, involves dividing your labeled data into three parts:

1. a training set, which you use to train the algorithm,

2. a test set, which is used to test the generalization error of the final model, and

3. a validation set, which is used to test the generalization error of intermediate versions of the model as you tune the hyperparameters.

This way you can check the generalization performance of the model while still reserving some completely new data for reporting the final performance on the test set.

If the final model performs well across the training, validation, and test sets, you can be quite confident that it will perform well on other new data as well. Alternatively, your final model might perform well on the training and validation sets but poorly on the test set. That would indicate that the model is overly tuned to the specificities of the training and validation sets, causing it to generalize poorly to new data. This phenomenon is called overfitting and is a pervasive problem for supervised machine learning. Overfitting has inspired many techniques to avoid and mitigate, several of which we describe below.

While dividing your labeled data into training, validation, and test sets works well conceptually, it has a practical drawback. The amount of data used to actually train the model (the training set) is significantly reduced. If you apply a 60% training, 20% validation, 20% test split, that leaves only 60% of your data for actual training. This can reduce performance, because supervised ML models generally perform better with more training data.

The solution to this is to use a process called cross validation, which allows you to combine the training and validation sets through a series of sequential model trainings called folds. In each fold, you pull out a subset of the training data for validation and train the model on the rest. You repeat this process with non-overlapping subsets for each fold, such that every training example has been in a validation fold once. In a 5-fold cross-validation, each fold would use 20% of the non-test data as a validation set and the remaining 80% for training. This would result in 5 different validation performance measurements (one for each folds) that you can average together for the average cross-validation performance.

Cross-validation provides a more robust measure of generalization performance than fixed training and validation sets. It is so common that supervised machine learning results are often reported in terms of the average performance on an N-fold cross-validation.

So how many folds is enough? Generally, the more folds, the better estimation of model performance. In the extreme case, called leave-one-out cross-validation, each fold uses a validation set with just one example, allowing the rest to be used for training. Each fold tests whether the model correctly predicts this one example when trained on all other examples. This provides the best estimate of generalization performance, because you’re training with the most data possible. If you have labeled 100 examples in your non-test set, leave-one-out cross-validation involves performing 100 folds and averaging the accuracy across across all 100. You can then repeat this process while tuning model hyperparameters to check the effects on generalization accuracy.

Unfortunately, leave-one-out cross-validation has its own drawbacks. Every cross-validation fold involves training a model, and if the training process is resource intensive, leave-one-out cross-validation can take a long time. Some models, especially neural network models, take a long time to train, so doing more than a few folds becomes impractical. For such models, a smaller number of folds (often 5 or 10) are used in practice.