Trees and Ensembles

Today we'll investigate whether a network eavesdropper can use device traffic to infer what people are doing inside their homes. We will pretend to be the eavesdropper and train a random forest classifier to perform this attack, using nothing but the timing and size of encrypted packets. We'll discuss what makes the attack work, why it constitutes a privacy risk, and how we can protect device owners. At the end, we compare the random forest against a single decision tree and against a k-nearest-neighbors classifier.

This hands-on reproduces an experiment from the following two papers:

• Noah Apthorpe, Dillon Reisman, and Nick Feamster. "A Smart Home is No Castle: Privacy Vulnerabilities of Encrypted IoT Traffic." 2017. The data we use is the Nest Cam trace behind Figure 2(b) of this paper: the camera alternates between live streaming and motion detection mode every two minutes, and the traffic rate alone reveals which mode it is in.
• Noah Apthorpe, Dillon Reisman, Srikanth Sundaresan, Arvind Narayanan, and Nick Feamster. "Closing the Blinds: Four Strategies for Protecting Smart Home Privacy from Network Observers." 2017. This paper evaluates defenses against the attack; we return to it in the discussion questions.

import numpy as np
import pandas as pd

import logging
logging.getLogger("scapy.runtime").setLevel(logging.ERROR)

import sklearn
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier

import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline

import warnings
warnings.filterwarnings('ignore')

plt.rcParams["figure.figsize"] = (8,6)

import netml
from netml.pparser.parser import PCAP

Application to IoT Privacy

In order to apply a classifier to our IoT device network data we need to take the following steps:

1. Convert the list of packets into points, with each point representing the device's network activity at a particular time.
2. Associate each point with a label (the activity the camera was performing at the time of the point).
3. Divide the points into a training set and a test set, and train a random forest classifier on the training set. A random forest is an ensemble of decision trees: each tree is trained on a bootstrap sample of the training data (and considers a random subset of the features at each split), and the forest predicts by majority vote over its trees.
4. Predict the labels of the test set using the classifier and calculate the accuracy of the predictions against the real labels.

1. Import data and convert to points

The data is currently stored as a list of packets, but we want it as points corresponding to time periods.

hpcap = PCAP('data/nestcam_live.pcap', flow_ptks_thres=2, verbose=10)

hpcap.pcap2pandas()
pcap = hpcap.df

pcap.head(4)

'_pcap2pandas()' starts at 2022-11-03 11:47:26
'_pcap2pandas()' ends at 2022-11-03 11:47:34 and takes 0.1279 mins.

	datetime	dns_query	dns_resp	ip_dst	ip_dst_int	ip_src	ip_src_int	is_dns	length	mac_dst	mac_dst_int	mac_src	mac_src_int	port_dst	port_src	protocol	time	time_normed
0	2016-07-29 15:10:03	None	None	172.24.1.84	2.887254e+09	52.87.161.133	8.781582e+08	False	66	18:b4:30:54:a5:db	27162184033755	b8:27:eb:ed:34:f0	202481601426672	46110.0	443.0	TCP	1469823003.220967	0.000000
1	2016-07-29 15:10:03	None	None	172.24.1.84	2.887254e+09	52.87.161.133	8.781582e+08	False	66	18:b4:30:54:a5:db	27162184033755	b8:27:eb:ed:34:f0	202481601426672	46110.0	443.0	TCP	1469823003.260909	0.039942
2	2016-07-29 15:10:03	None	None	52.87.161.133	8.781582e+08	172.24.1.84	2.887254e+09	False	1506	b8:27:eb:ed:34:f0	202481601426672	18:b4:30:54:a5:db	27162184033755	443.0	46110.0	TCP	1469823003.271401	0.050434
3	2016-07-29 15:10:03	None	None	52.87.161.133	8.781582e+08	172.24.1.84	2.887254e+09	False	1506	b8:27:eb:ed:34:f0	202481601426672	18:b4:30:54:a5:db	27162184033755	443.0	46110.0	TCP	1469823003.272394	0.051427

2. Data cleaning

The parsed data frame has one row per packet, with MAC addresses, IP addresses, ports, DNS fields, and so on. Clean it up as follows:

1. Keep only the camera's packets. The Nest Cam has IP address 172.24.1.84 (its MAC address is 18:b4:30:54:a5:db; the only other MAC address in the trace is the Raspberry Pi gateway). Keep only the rows whose ip_src or ip_dst is the camera. This also removes the handful of non-IP packets (ARP and the like), whose IP columns are empty.
2. Model the eavesdropper. The eavesdropper sits outside the home (for example, at the ISP), so it sees only IP-layer metadata, not MAC addresses. Because the traffic is encrypted, the only useful fields are when each packet was sent and how big it was.
3. Keep only time and length. Drop the MAC address columns, the IP address columns, and everything else. The result should be a data frame with exactly two columns: time and length.

camera_ip = '172.24.1.84'

# TODO: 1. keep only packets sent or received by the camera
#   packets = pcap[(pcap['ip_src'] == camera_ip) | (pcap['ip_dst'] == camera_ip)]
packets = None

# TODO: 2 and 3. keep only the 'time' and 'length' columns; drop everything else
#   packets = packets[['time', 'length']].copy()

# packets.head(5)   # uncomment once packets is filled in above

3. Normalize timestamps

The time column holds seconds since the Unix epoch (January 1, 1970, 00:00:00 UTC). For plotting and binning we care about when during the capture each packet was sent, so convert the timestamps to seconds since the first packet and the capture starts at t = 0.

Keep the capture's start time, too. The activity labels we join in step 6 were written down as local clock times (16:10:00, 16:12:20, ...) in the lab where the capture was made. To put them on the same clock as the packets we need to know what the lab's clock read when the capture started. datetime.fromtimestamp with no time zone uses the zone of whatever machine runs this notebook, which would make the answer depend on where you sit, so name the capture's zone explicitly. (The capture is from the Princeton study linked at the top, so US Eastern time.)

from datetime import datetime
from zoneinfo import ZoneInfo

# netml stores timestamps as Decimal objects; convert them to floats first
packets['time'] = packets['time'].astype(float)

# the lab's clock: the zone in which the label file was written
CAPTURE_TZ = ZoneInfo('America/New_York')

# wall-clock time of the first packet, in the lab's zone (used in step 6 to align the labels)
t0 = packets['time'].min()
capture_start = datetime.fromtimestamp(t0, CAPTURE_TZ)
print('capture starts at', capture_start.strftime('%Y-%m-%d %H:%M:%S %Z'))

# normalize so that the first packet is at t = 0
packets['time'] = packets['time'] - t0

packets.head(3)

Now let's convert the list of packets into send rates by binning the packets into equal-length time windows and computing the total amount of data sent (the sum of the packet lengths) in each window. The send_rates() function below does this; it returns one rate per window along with the start time of each window.

def send_rates(data, window_len_sec):
    '''Calculates send rates from packet DataFrames
    Arguments:
      data: pandas DataFrame with 'time' (seconds since the start of the capture)
            and 'length' columns, like that returned from pcap2pandas()
      window_len_sec: interval for calculating rates
    Returns:
       rates: array of send rates (bytes per second), one per window
       times: array of window start times (seconds), same length as rates
    '''
    # which window does each packet fall into? window k covers [k*T, (k+1)*T)
    window = np.floor(data['time'].values / window_len_sec).astype(int)

    # total bytes per window, including the windows with no packets at all
    n_windows = window.max() + 1
    bytes_per_window = np.bincount(window, weights=data['length'].values, minlength=n_windows)

    rates = bytes_per_window / float(window_len_sec)
    times = np.arange(n_windows) * float(window_len_sec)
    return rates, times

# Bin size (seconds) over which to compute rates. Start with 1-second windows.
T = 1

# TODO: compute the send rates and the start time of each bin with send_rates()
#   rates, rate_times = send_rates(packets, T)
rates, rate_times = None, None

# TODO: plot the rates over time. Compare your plot with Figure 2(b) of the paper.
#   plt.plot(rate_times, rates)
plt.ylabel("Send rate (bytes/sec)")
plt.xlabel("Time since start of capture (sec)")
plt.show()

4. Explore data representations

Often, the choice of data representation is at least as important as the choice of model. Try choosing different values for T (the bin size) and see how it affects the plot.

Questions:

• What may be some benefits/drawbacks of having a small bin size?
• What may be benefits/drawbacks of having a large bin size?

5. Represent rates as d-dimensional points

Next let's represent the rate timeseries as a set of d-dimensional points, each covering d consecutive rate samples. Ultimately, we will associate each point with a specific activity. The d variable sets the number of dimensions. We have set this to two right now so that visualization is easy; later you can increase it.

The rates_to_points function below generates $m$ d-dimensional points from the rate timeseries above.

# Turn the rate timeseries into points for training a classifier
def rates_to_points(rates, times, sampling_period):
    '''Group consecutive rate samples into points of dimension sampling_period.
    Each point is the rate in sampling_period consecutive windows; its time is the
    start of its first window. Any leftover samples at the end (fewer than a full
    point) are dropped.'''
    n_points = rates.size // sampling_period
    points = rates[:n_points * sampling_period].reshape(n_points, sampling_period)
    point_times = times[:n_points * sampling_period:sampling_period]
    return points, point_times

# number of rate samples to include in each point.
# How many total seconds will each point represent?
# This is the number of features each point has (the *dimension* of the feature space).
d = 2

# get d-dimensional points and the time for each point.
# we need the times because we're going to label each point based on the activity at that time.
points, point_times = rates_to_points(rates,
                                      rate_times,
                                      d)

Let's take a quick look at the result. We started off with one rate per bin. Then we grouped consecutive bins into d-dimensional points, each with a start time. We thus have about $N/d$ points if our original rate timeseries had $N$ samples.

# TODO: print the total number of points and the first point
#   print(len(points)); print(points[0])

# TODO: print the total number of point_times and the first one
#   print(len(point_times)); print(point_times[0])

Now we have points and associated times. If you choose d = 2, then each point is two-dimensional, which lets us plot the points. Let's try that first and plot the result.

# TODO: scatter plot of the points (rate in the first bin vs. rate in the second bin)
#   plt.scatter(*zip(*points))
plt.grid(linestyle='--', alpha=0.6)
plt.xlabel('Rate at t1 (bytes/sec)')
plt.ylabel('Rate at t2 (bytes/sec)')
plt.show()

6. Associate d-dimensional points with activity labels

First, read the labels from the text file. Each line gives the (local) clock time at which the camera switched activity, and the new activity. There are two labels, which are the classes we want to predict:

• livestream, which indicates that the camera is streaming video to the cloud (a user is watching the feed); and
• motion, which indicates that the camera is in motion-detection mode: it monitors locally and only uploads when it detects movement.

In traffic terms these are closer to "busy" and "idle" than to two activities: while livestreaming the camera uploads continuously, and in motion-detection mode it sends almost nothing until something moves, then a short burst. That contrast is what the classifier will pick up.

labels = pd.read_csv('data/nestcam_live_labels.txt', header=None,
                     names=["time", "activity"], skipinitialspace=True)
labels.head(10)

The label times are clock times (HH:MM:SS) on the lab's clock, while rate_times is seconds since the start of the capture. In step 3 we worked out what the lab's clock read when the capture started (capture_start), so put each label on the capture clock by subtracting that start time. The first label comes out at about -3 seconds: it was written down three seconds before the first packet arrived, which is negligible against the two-minute activity periods.

(Do not assume that the first label coincides with the first packet; it happens to be close here, but in other traces, including the ones in Assignment 3, the labels start well into the capture.)

def to_seconds(hms):
    '''Convert an "HH:MM:SS" string to seconds since midnight.'''
    h, m, s = (int(x) for x in hms.strip().split(':'))
    return 3600 * h + 60 * m + s

# what the lab's clock read when the capture started, in seconds since midnight
capture_start_s = to_seconds(capture_start.strftime('%H:%M:%S'))

# TODO: express each label's start time in seconds since the start of the capture
#   labels['t'] = labels['time'].apply(to_seconds) - capture_start_s
labels['t'] = None

# Convert the activity names into class numbers (livestream = 0, motion = 1)
label_encoder = LabelEncoder()
labels['class'] = label_encoder.fit_transform(labels['activity'])

labels

Finally, map the points to labels: the label of a point is the activity that was in progress at the point's start time.

def label_points(labels, point_times):
    '''Return the class of the activity in progress at each time in point_times.
    Arguments:
      labels: DataFrame with a 't' column (activity start time, seconds) and a
              'class' column, sorted by 't'
      point_times: array of times (seconds, on the same clock as labels['t'])
    Returns:
      numpy array with one class number per entry of point_times
    '''
    # TODO: for each time t in point_times, find the last label whose start
    # time is <= t and return its class.
    # Hint: np.searchsorted(labels['t'].values, point_times, side='right') - 1
    #       gives the index of that label (clip it to be at least 0);
    #       or walk through the labels in a loop as t increases.
    point_labels = None
    return point_labels

point_labels = label_points(labels, point_times)
print(point_labels)

Now that we've labeled the data, we can associate each point with a label (class) and re-plot the scatterplot above, coloring each point by its class.

# Assign colors to each class (helps with visualization)
colors = ['red', 'green', 'blue']

# TODO: scatter plot of the points, colored by point_labels
#   plt.scatter(*zip(*points), c=point_labels, cmap=mpl.colors.ListedColormap(colors))
plt.grid(linestyle='--', alpha=0.6)
plt.xlabel('Rate at t1 (bytes/sec)')
plt.ylabel('Rate at t2 (bytes/sec)')
plt.show()

7. Split into training and test sets

Let's divide the points into a training set and a test set.

# TODO: split the points and labels into training and test sets.
# The 'test_size' parameter determines what fraction of the data is reserved for testing.
#   points_train, points_test, labels_train, labels_test = \
#       train_test_split(points, point_labels, test_size=0.2, random_state=1)
points_train, points_test, labels_train, labels_test = None, None, None, None

Random Forest

Now we will train a random forest classifier on the training set.

Note: Your choice of data representation makes a huge difference as far as accuracy is concerned! Try this exercise for different values of T (bin size) and d (point dimension) and see how it affects accuracy.

Train a Random Forest Classifier

Train a RandomForestClassifier on your labeled training points.

# TODO: create a random forest and train it on the training points
#   rf = RandomForestClassifier(n_estimators=100, random_state=0)
#   rf.fit(points_train, labels_train)
rf = None

Perform prediction on the test set, and use accuracy_score to report accuracy.

# TODO: predict the labels of the test points
#   labels_pred = rf.predict(points_test)
labels_pred = None

# TODO: compute the accuracy of the predictions against the true test labels
#   accuracy = accuracy_score(labels_test, labels_pred)
accuracy = None

print("Random forest accuracy:", accuracy)

---

Discussion Questions

1. Why is this attack a privacy risk?

2. How could we (IoT device programmers, network operators, etc.) protect people from this attack?

The "Closing the Blinds" paper linked at the top evaluates four strategies (blocking traffic, encrypting DNS, tunneling through a VPN, and traffic shaping). Which of them would defeat this classifier, and at what cost?

Additional Exercises

1. Adjust parameters to improve accuracy.

Now that we have a baseline accuracy, we can tweak the data preprocessing and classifier parameters to improve the accuracy. Look back through the code we've run so far. Which values have we set arbitrarily that could affect the results? Try changing these parameters and re-running the code to see how the classification accuracy is affected. Remember to re-run all of the cells below each change (or just restart the kernel and re-run all cells).

# TODO: try a few (T, d) combinations, e.g. T in {1, 2, 5} and d in {1, 2, 5, 10},
# and record the random forest's test accuracy for each.

2. Compare with a single decision tree.

A random forest is an ensemble of decision trees. Train a single DecisionTreeClassifier on the same training set and compare its test accuracy with the forest's. Then increase d (try 5 or 10) and see whether the gap between one tree and the forest changes. Use rf.feature_importances_ to see which of the d rate samples in each point the forest relies on most.

# TODO: train a DecisionTreeClassifier on (points_train, labels_train) and report its test accuracy
#   tree = DecisionTreeClassifier(random_state=0)
tree = None

# TODO: print rf.feature_importances_

3. Compare with k-nearest neighbors.

This hands-on used to be built around a k-nearest-neighbors (k-NN) classifier, which involves no training at all: it stores the training points and labels a test point by majority vote among its k closest training points. Train a KNeighborsClassifier (try n_neighbors of 1, 3, 5, and 15) on the same split and compare its accuracy with the random forest's. Which classifier is more sensitive to T and d? Which would you expect to be more robust to noisy labels?

# TODO: for k in [1, 3, 5, 15], train a KNeighborsClassifier(n_neighbors=k) and report its test accuracy
#   for k in [1, 3, 5, 15]:
#       knn = KNeighborsClassifier(n_neighbors=k)
#       knn.fit(points_train, labels_train)
#       print(k, accuracy_score(labels_test, knn.predict(points_test)))