Trees and Ensembles

Today we'll investigate whether a network eavesdropper can use device traffic to infer what people are doing inside their homes. We will pretend to be the eavesdropper and train a random forest classifier to perform this attack, using nothing but the timing and size of encrypted packets. We'll discuss what makes the attack work, why it constitutes a privacy risk, and how we can protect device owners. At the end, we compare the random forest against a single decision tree and against a k-nearest-neighbors classifier.

This hands-on reproduces an experiment from the following two papers:

• Noah Apthorpe, Dillon Reisman, and Nick Feamster. "A Smart Home is No Castle: Privacy Vulnerabilities of Encrypted IoT Traffic." 2017. The data we use is the Nest Cam trace behind Figure 2(b) of this paper: the camera alternates between live streaming and motion detection mode every two minutes, and the traffic rate alone reveals which mode it is in.
• Noah Apthorpe, Dillon Reisman, Srikanth Sundaresan, Arvind Narayanan, and Nick Feamster. "Closing the Blinds: Four Strategies for Protecting Smart Home Privacy from Network Observers." 2017. This paper evaluates defenses against the attack; we return to it in the discussion questions.

import numpy as np
import pandas as pd

import logging
logging.getLogger("scapy.runtime").setLevel(logging.ERROR)

import sklearn
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier

import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline

import warnings
warnings.filterwarnings('ignore')

plt.rcParams["figure.figsize"] = (8,6)

import netml
from netml.pparser.parser import PCAP

Application to IoT Privacy

In order to apply a classifier to our IoT device network data we need to take the following steps:

1. Convert the list of packets into points, with each point representing the device's network activity at a particular time.
2. Associate each point with a label (the activity the camera was performing at the time of the point).
3. Divide the points into a training set and a test set, and train a random forest classifier on the training set. A random forest is an ensemble of decision trees: each tree is trained on a bootstrap sample of the training data (and considers a random subset of the features at each split), and the forest predicts by majority vote over its trees.
4. Predict the labels of the test set using the classifier and calculate the accuracy of the predictions against the real labels.

1. Import data and convert to points

The data is currently stored as a list of packets, but we want it as points corresponding to time periods.

hpcap = PCAP('data/nestcam_live.pcap', flow_ptks_thres=2, verbose=10)

hpcap.pcap2pandas()
pcap = hpcap.df

pcap.head(4)

'_pcap2pandas()' starts at 2022-11-03 11:47:26
'_pcap2pandas()' ends at 2022-11-03 11:47:34 and takes 0.1279 mins.

	datetime	dns_query	dns_resp	ip_dst	ip_dst_int	ip_src	ip_src_int	is_dns	length	mac_dst	mac_dst_int	mac_src	mac_src_int	port_dst	port_src	protocol	time	time_normed
0	2016-07-29 15:10:03	None	None	172.24.1.84	2.887254e+09	52.87.161.133	8.781582e+08	False	66	18:b4:30:54:a5:db	27162184033755	b8:27:eb:ed:34:f0	202481601426672	46110.0	443.0	TCP	1469823003.220967	0.000000
1	2016-07-29 15:10:03	None	None	172.24.1.84	2.887254e+09	52.87.161.133	8.781582e+08	False	66	18:b4:30:54:a5:db	27162184033755	b8:27:eb:ed:34:f0	202481601426672	46110.0	443.0	TCP	1469823003.260909	0.039942
2	2016-07-29 15:10:03	None	None	52.87.161.133	8.781582e+08	172.24.1.84	2.887254e+09	False	1506	b8:27:eb:ed:34:f0	202481601426672	18:b4:30:54:a5:db	27162184033755	443.0	46110.0	TCP	1469823003.271401	0.050434
3	2016-07-29 15:10:03	None	None	52.87.161.133	8.781582e+08	172.24.1.84	2.887254e+09	False	1506	b8:27:eb:ed:34:f0	202481601426672	18:b4:30:54:a5:db	27162184033755	443.0	46110.0	TCP	1469823003.272394	0.051427

2. Data cleaning

The parsed data frame has one row per packet, with MAC addresses, IP addresses, ports, DNS fields, and so on. Clean it up as follows:

1. Keep only the camera's packets. The Nest Cam has IP address 172.24.1.84 (its MAC address is 18:b4:30:54:a5:db; the only other MAC address in the trace is the Raspberry Pi gateway). Keep only the rows whose ip_src or ip_dst is the camera. This also removes the handful of non-IP packets (ARP and the like), whose IP columns are empty.
2. Model the eavesdropper. The eavesdropper sits outside the home (for example, at the ISP), so it sees only IP-layer metadata, not MAC addresses. Because the traffic is encrypted, the only useful fields are when each packet was sent and how big it was.
3. Keep only time and length. Drop the MAC address columns, the IP address columns, and everything else. The result should be a data frame with exactly two columns: time and length.

camera_ip = '172.24.1.84'

# TODO: 1. keep only packets sent or received by the camera
#   packets = pcap[(pcap['ip_src'] == camera_ip) | (pcap['ip_dst'] == camera_ip)]
packets = None

# TODO: 2 and 3. keep only the 'time' and 'length' columns; drop everything else
#   packets = packets[['time', 'length']].copy()

# packets.head(5)   # uncomment once packets is filled in above

3. Normalize timestamps

The time column holds seconds since the Unix epoch (January 1, 1970, 00:00:00 UTC). We do not care about absolute time; we care about when during the capture each packet was sent. So convert the timestamps to seconds since the first packet, so that the capture starts at t = 0.

Why relative time? The activity labels we join in step 6 were written down as local clock times (16:10:00, 16:12:20, ...) in the lab where the capture was made, whereas converting an epoch timestamp with datetime.fromtimestamp uses the timezone of whatever machine runs this notebook. Matching the two on absolute time therefore depends on where you run the notebook (and on daylight saving time). Working in relative time sidesteps the timezone offset entirely: the capture and the label file both start at the same moment, so we put both on a "seconds since start" clock.

# netml stores timestamps as Decimal objects; convert them to floats first
packets['time'] = packets['time'].astype(float)

# normalize so that the first packet is at t = 0
t0 = packets['time'].min()
packets['time'] = packets['time'] - t0

packets.head(3)

Now let's convert the list of packets into send rates by binning the packets into equal-length time windows and computing the total amount of data sent (the sum of the packet lengths) in each window. The send_rates() function below does this; it returns one rate per window along with the start time of each window.

def send_rates(data, window_len_sec):
    '''Calculates send rates from packet DataFrames
    Arguments:
      data: pandas DataFrame with 'time' and 'length' columns 
              like that returned from pcap_to_pandas()
      window_len_sec: interval for calculating rates
    Returns:
       rates: array of send rates
       times: array of times corresponding to each window in rates
    '''
    data = data.sort_values(by=["time"])
    windows = []
    times = []
    curr_time = data.iloc[0]["time"]
    end_time = curr_time + window_len_sec
    i = 0
    while curr_time < data.iloc[-1]["time"]:
        windows.append(0)
        times.append(curr_time)
        while i < len(data) and data.iloc[i]["time"] < end_time:
            windows[-1] += data.iloc[i]["length"]
            i += 1
        curr_time = end_time
        end_time = curr_time + window_len_sec
    rates = np.array(windows) / float(window_len_sec)
    times = np.array(times)
    return rates, times

# Bin size (seconds) over which to compute rates. Start with 1-second windows.
T = 1

# TODO: compute the send rates and the start time of each bin with send_rates()
#   rates, rate_times = send_rates(packets, T)
rates, rate_times = None, None

# TODO: plot the rates over time. Compare your plot with Figure 2(b) of the paper.
#   plt.plot(rate_times, rates)
plt.ylabel("Send rate (bytes/sec)")
plt.xlabel("Time since start of capture (sec)")
plt.show()

4. Explore data representations

Often, the choice of data representation is at least as important as the choice of model. Try choosing different values for T (the bin size) and see how it affects the plot.

Questions:

• What may be some benefits/drawbacks of having a small bin size?
• What may be benefits/drawbacks of having a large bin size?

5. Represent rates as d-dimensional points

Next let's represent the rate timeseries as a set of d-dimensional points, each covering d consecutive rate samples. Ultimately, we will associate each point with a specific activity. The d variable sets the number of dimensions. We have set this to two right now so that visualization is easy; later you can increase it.

The rates_to_points function below generates $m$ d-dimensional points from the rate timeseries above.

# Turn the rate timeseries into points for training a classifier
def rates_to_points(rates, times, sampling_period):

    # generate points. each point holds sampling_period consecutive rate samples
    points = [rates[i:min(i+sampling_period, rates.size-1)] for i in range(0, rates.size, sampling_period)]
    times = [times[i] for i in range(0, times.size, sampling_period)]
    return np.array(points[:-1]), np.array(times[:-1])

# number of rate samples to include in each point.
# How many total seconds will each point represent?
# This is the number of features each point has (the *dimension* of the feature space).
d = 2

# get d-dimensional points and the time for each point.
# we need the times because we're going to label each point based on the activity at that time.
points, point_times = rates_to_points(rates,
                                      rate_times,
                                      d)

Let's take a quick look at the result. We started off with one rate per bin. Then we grouped consecutive bins into d-dimensional points, each with a start time. We thus have about $N/d$ points if our original rate timeseries had $N$ samples.

# TODO: print the total number of points and the first point
#   print(len(points)); print(points[0])

# TODO: print the total number of point_times and the first one
#   print(len(point_times)); print(point_times[0])

Now we have points and associated times. If you choose d = 2, then each point is two-dimensional, which lets us plot the points. Let's try that first and plot the result.

# TODO: scatter plot of the points (rate in the first bin vs. rate in the second bin)
#   plt.scatter(*zip(*points))
plt.grid(linestyle='--', alpha=0.6)
plt.xlabel('Rate at t1 (bytes/sec)')
plt.ylabel('Rate at t2 (bytes/sec)')
plt.show()

6. Associate d-dimensional points with activity labels

First, read the labels from the text file. Each line gives the (local) clock time at which the camera switched activity, and the new activity. There are two labels, which are the classes we want to predict:

• livestream, which indicates that the camera is streaming video to the cloud (a user is watching the feed); and
• motion, which indicates that the camera is in motion-detection mode: it monitors locally and only uploads when it detects movement.

labels = pd.read_csv('data/nestcam_live_labels.txt', header=None,
                     names=["time", "activity"], skipinitialspace=True)
labels.head(10)

The label times are clock times (HH:MM:SS), while rate_times is seconds since the start of the capture. Put the labels on the same relative clock: the first label was written down when the capture started, so express each label's start time as seconds since the first label. (The first packet actually arrives about three seconds after the first label was recorded; this is negligible compared with the two-minute activity periods.)

def to_seconds(hms):
    '''Convert an "HH:MM:SS" string to seconds since midnight.'''
    h, m, s = (int(x) for x in hms.strip().split(':'))
    return 3600 * h + 60 * m + s

# TODO: express each label's start time in seconds since the first label
#   seconds = labels['time'].apply(to_seconds)
#   labels['t'] = seconds - seconds.iloc[0]
labels['t'] = None

# Convert the activity names into class numbers (livestream = 0, motion = 1)
label_encoder = LabelEncoder()
labels['class'] = label_encoder.fit_transform(labels['activity'])

labels

Finally, map the points to labels: the label of a point is the activity that was in progress at the point's start time.

def label_points(labels, point_times):
    '''Return the class of the activity in progress at each time in point_times.
    Arguments:
      labels: DataFrame with a 't' column (activity start time, seconds) and a
              'class' column, sorted by 't'
      point_times: array of times (seconds, on the same clock as labels['t'])
    Returns:
      numpy array with one class number per entry of point_times
    '''
    # TODO: for each time t in point_times, find the last label whose start
    # time is <= t and return its class.
    # Hint: np.searchsorted(labels['t'].values, point_times, side='right') - 1
    #       gives the index of that label (clip it to be at least 0);
    #       or walk through the labels in a loop as t increases.
    point_labels = None
    return point_labels

point_labels = label_points(labels, point_times)
print(point_labels)

Now that we've labeled the data, we can associate each point with a label (class) and re-plot the scatterplot above, coloring each point by its class.

# Assign colors to each class (helps with visualization)
colors = ['red', 'green', 'blue']

# TODO: scatter plot of the points, colored by point_labels
#   plt.scatter(*zip(*points), c=point_labels, cmap=mpl.colors.ListedColormap(colors))
plt.grid(linestyle='--', alpha=0.6)
plt.xlabel('Rate at t1 (bytes/sec)')
plt.ylabel('Rate at t2 (bytes/sec)')
plt.show()

7. Split into training and test sets

Let's divide the points into a training set and a test set.

# TODO: split the points and labels into training and test sets.
# The 'test_size' parameter determines what fraction of the data is reserved for testing.
#   points_train, points_test, labels_train, labels_test = \
#       train_test_split(points, point_labels, test_size=0.2, random_state=1)
points_train, points_test, labels_train, labels_test = None, None, None, None

Random Forest

Now we will train a random forest classifier on the training set.

Note: Your choice of data representation makes a huge difference as far as accuracy is concerned! Try this exercise for different values of T (bin size) and d (point dimension) and see how it affects accuracy.

Train a Random Forest Classifier

Train a RandomForestClassifier on your labeled training points.

# TODO: create a random forest and train it on the training points
#   rf = RandomForestClassifier(n_estimators=100, random_state=0)
#   rf.fit(points_train, labels_train)
rf = None

Perform prediction on the test set, and use accuracy_score to report accuracy.

# TODO: predict the labels of the test points
#   labels_pred = rf.predict(points_test)
labels_pred = None

# TODO: compute the accuracy of the predictions against the true test labels
#   accuracy = accuracy_score(labels_test, labels_pred)
accuracy = None

print("Random forest accuracy:", accuracy)

---

Discussion Questions

1. Why is this attack a privacy risk?

2. How could we (IoT device programmers, network operators, etc.) protect people from this attack?

The "Closing the Blinds" paper linked at the top evaluates four strategies (blocking traffic, encrypting DNS, tunneling through a VPN, and traffic shaping). Which of them would defeat this classifier, and at what cost?

Additional Exercises

1. Adjust parameters to improve accuracy.

Now that we have a baseline accuracy, we can tweak the data preprocessing and classifier parameters to improve the accuracy. Look back through the code we've run so far. Which values have we set arbitrarily that could affect the results? Try changing these parameters and re-running the code to see how the classification accuracy is affected. Remember to re-run all of the cells below each change (or just restart the kernel and re-run all cells).

# TODO: try a few (T, d) combinations, e.g. T in {1, 2, 5} and d in {1, 2, 5, 10},
# and record the random forest's test accuracy for each.

2. Compare with a single decision tree.

A random forest is an ensemble of decision trees. Train a single DecisionTreeClassifier on the same training set and compare its test accuracy with the forest's. Then increase d (try 5 or 10) and see whether the gap between one tree and the forest changes. Use rf.feature_importances_ to see which of the d rate samples in each point the forest relies on most.

# TODO: train a DecisionTreeClassifier on (points_train, labels_train) and report its test accuracy
#   tree = DecisionTreeClassifier(random_state=0)
tree = None

# TODO: print rf.feature_importances_

3. Compare with k-nearest neighbors.

This hands-on used to be built around a k-nearest-neighbors (k-NN) classifier, which involves no training at all: it stores the training points and labels a test point by majority vote among its k closest training points. Train a KNeighborsClassifier (try n_neighbors of 1, 3, 5, and 15) on the same split and compare its accuracy with the random forest's. Which classifier is more sensitive to T and d? Which would you expect to be more robust to noisy labels?

# TODO: for k in [1, 3, 5, 15], train a KNeighborsClassifier(n_neighbors=k) and report its test accuracy
#   for k in [1, 3, 5, 15]:
#       knn = KNeighborsClassifier(n_neighbors=k)
#       knn.fit(points_train, labels_train)
#       print(k, accuracy_score(labels_test, knn.predict(points_test)))