Logistic Regression on Network Traffic

Logistic Regression is very similar to linear regression, except all of the points can only have $y$-values of $1$ or $0$. This is useful if we want to predict whether something is or isn't part of a particular class. Instead of fitting a line (as in linear regression), logistic regression involves fitting a probability curve.

Learning objectives

By the end of this hands-on you should be able to:

• State what changes when the target is a 0/1 label rather than a number, and why a line is replaced by a sigmoid.
• Fit a logistic regression with scikit-learn to predict a DNS packet's direction (request or response) from its length, and read the decision boundary off the curve.
• Build your own label for a classification problem (data packet versus ACK) from packet fields, and say what the choice of threshold does to the task.
• Evaluate a classifier on held-out data with accuracy and a confusion matrix, and relate the result to the two-population problem from the linear regression hands-on.

# Pandas, Numpy
import numpy as np
import pandas as pd

import logging
logging.getLogger("scapy.runtime").setLevel(logging.ERROR)

# Machine Learning
from sklearn.linear_model import LogisticRegression, LogisticRegressionCV

# Plotting
import matplotlib.pyplot as plt
%matplotlib inline

import sys
#sys.path.insert(1,"/Users/feamster/research/netml/src/")
import netml
from netml.pparser.parser import PCAP

hpcap = PCAP('data/http.pcap', flow_ptks_thres=2, verbose=10)

hpcap.pcap2pandas()
pcap = hpcap.df

Part 1: DNS Requests vs. Responses

Using our device traffic, let's see whether we can predict whether a DNS packet is a request or a response from its length.

First, import the data, extract only the DNS packets, and view the first few packets. pcap2pandas gives one row per packet; the is_dns column marks DNS packets and dns_resp is non-zero for responses.

# TODO: keep only the DNS packets and look at the first few rows
# dns_packets = pcap.loc[pcap['is_dns'] == True]
# dns_packets.head(5)

Each row in the printed data is a packet and each column is a feature of the packet.

Next let's divide the DNS packets into requests and responses, and convert them into points where the $x$-value is the length of the packet and $y$-value is $0$ for requests and $1$ for responses. This will allow us to fit the data to a logistic regression curve.

# TODO: keep just the two columns we need, packet length and the response flag
# dns_ft = dns_packets.loc[:, ['length', 'dns_resp']]
# dns_ft.head(10)

Let's see how many data points we have.

# TODO: how many (rows, columns)?
# dns_ft.shape

Next we will convert the DNS response column into a 0/1 value so that it is amenable to logistic regression.

# TODO: 0 for a request, 1 for a response
# dns_ft['response'] = dns_ft['dns_resp'].astype(bool).astype(int)
# dns_ft.head(10)

Plot the Points

Plot the packets as points: $x$ is the packet length, $y$ is 0 (request) or 1 (response). Label your axes. What do you notice about where the requests and the responses sit?

# TODO: scatter the points
# x = dns_ft['length'].values
# y = dns_ft['response'].values
# plt.plot(x, y, 'x', color='red')
# plt.ylabel("Request (0) or Response (1)")
# plt.xlabel("DNS Packet Length (Bytes)")
# plt.grid(linestyle='--', alpha=0.6)
# plt.show()

Train the Model

Fit a LogisticRegression from scikit-learn that predicts the response flag from the packet length. (fit wants a column vector of features, so reshape x with reshape(-1, 1).)

# TODO: fit a logistic regression
# regr = LogisticRegression(solver='lbfgs', C=1)
# regr.fit(x.reshape(-1, 1), y)

Visualize the Model

Plot the fitted probability curve (the sigmoid) over a range of packet lengths, with the points overlaid. Where is the decision boundary, the length at which the model's probability of "response" crosses 0.5?

# TODO: plot the sigmoid over a number line of packet lengths
# z = np.arange(1, 600, 0.5).reshape(-1, 1)
# plt.plot(z, regr.predict_proba(z)[:, 1], color='green')
# plt.plot(x, y, 'x', color='red')
# plt.ylabel("Request (0) or Response (1)")
# plt.xlabel("DNS Packet Length")
# plt.grid(linestyle='--', alpha=0.6)
# plt.show()

---

Model Selection/Cross Validation

How do we know we have a good model? How do we know that the machine learning algorithm is "good"?

We can perform a K-fold cross-validation on the data:

• holding out 1/K of the data for testing,
• training on the remaining data,
• repeating this K times, one for each fold
• averaging the resulting accuracy/score of the model

from sklearn.model_selection import KFold, cross_val_score

# TODO: five-fold cross-validation of the model above, scored by accuracy
# kf = KFold(n_splits=5)
# cv_results = cross_val_score(regr, x.reshape(-1, 1), y, cv=kf, scoring="accuracy")
# cv_results, cv_results.mean()

Hyper-Parameter Tuning: Grid Search

Grid search can allow for hyper-parameter tuning, with cross validation, such as the k-fold cross validation that we just performed. The search performs the evaluation above, automating the search through the set of all possible parameters defined in the grid.

The resulting model is returned as a result, and a model can be generated from the training data by calling fit on the results of the search.

from sklearn.model_selection import GridSearchCV

# TODO: search over the regularization strength C with five-fold cross-validation
# params = {'C': np.arange(1, 5, 1)}
# grid_model = GridSearchCV(estimator=regr, param_grid=params, cv=5,
#                           return_train_score=True, scoring='accuracy')
# grid_model_result = grid_model.fit(x.reshape(-1, 1), y)
# pd.DataFrame(grid_model.cv_results_)

Part 2: Data Packets vs. Acknowledgments

In the linear-regression hands-on, we saw that flows split into two populations: flows dominated by full-size data packets and flows dominated by small acknowledgments (ACKs). That suggests a natural classification problem, which came up as an extension in class: given a single TCP packet, is it a data packet or an ACK?

Reuse the per-packet data frame from netml (hpcap.df above, or generate flow features as in the linear-regression hands-on). Define the label yourself, for example by treating a TCP packet with no payload (or a packet length below some threshold) as an ACK, and everything else as a data packet. Then train a logistic regression that predicts the label from packet size, and optionally from the inter-arrival time as a second feature.

1. Build the feature matrix (packet size, optionally inter-arrival time) and the 0/1 label.
2. Split into training and test sets, and fit a LogisticRegression.
3. Report the accuracy on the test set. Is accuracy a reasonable metric here, given how the two classes are balanced?
4. Plot the fitted sigmoid over packet size, with the labeled points overlaid, as you did for the DNS example above. Where is the decision boundary, and does it match the threshold you used to define the labels? (If it matches too well, think about whether the problem is circular, and what other feature might make it interesting.)

from sklearn.model_selection import train_test_split

# TODO: select TCP packets from the per-packet data frame
# tcp = pcap[pcap['protocol'] == 'TCP'].copy()

# TODO: define the label: 1 for a data packet, 0 for an ACK
# tcp['is_data'] = (tcp['length'] > ACK_THRESHOLD).astype(int)

# TODO: build features. Start with packet size; optionally add inter-arrival time.
# X = tcp[['length']].values
# y = tcp['is_data'].values

# TODO: train/test split and fit
# X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
# clf = LogisticRegression().fit(X_train, y_train)
# print("Accuracy: {:.3f}".format(clf.score(X_test, y_test)))

# TODO: plot the sigmoid over packet size, with the labeled points overlaid
# sizes = np.linspace(X[:, 0].min(), X[:, 0].max(), 500).reshape(-1, 1)
# plt.scatter(X[:, 0], y, alpha=0.2, label='packets')
# plt.plot(sizes, clf.predict_proba(sizes)[:, 1], color='red', label='P(data packet)')
# plt.xlabel('Packet size (bytes)')
# plt.ylabel('P(data packet)')
# plt.legend()