Agenda — Autumn 2026 (Chicago)

Nine-week term, meetings Mondays and Fridays, 80 minutes each. Notes are reconstructed from the class recordings after each meeting; where a recording was partly inaudible the entry says so rather than guessing.

Meeting 1 (Mon Sep 28)

• Course mechanics (syllabus walk-through)
  • Everything hangs off the course web page and the GitHub repo: slides (now Quarto decks, kept current), hands-on notebooks, past midterms and finals (every exam question ever asked, including the Summer 2026 Paris final)
  • The book (book.pdf) is on Canvas Files, not on the web; a fresher copy will be uploaded. Canvas is otherwise unused
  • Agenda files: one per term, built from the class transcript after each meeting; an Autumn 2026 file (this one) is the answer to “what did we cover / what is on the exam”
  • Exam prompts (docs/prompts/) are public: past exams + agenda + topic list → generate your own practice midterms the same way the real one is drafted
  • Exams are in class but not timed; designed for a short sitting, you have the whole period and beyond. Accommodations page: sds.html on the course site (not linked publicly). Everyone gets untimed exams and deadline flexibility
  • Midterm: one 8.5×11 handwritten sheet, both sides. Final: two sheets (or reuse the midterm sheet plus one)
  • To do tonight: join Slack, fill out the intake Google Form with GitHub username and private repo URL, create the private repo. Roster will be checked against submissions; nudges, scores, and late-hour balances arrive by Slack DM; written feedback as GitHub issues
  • Repo setup clarified in Q&A: clone the course repo (notebooks, slides, web page; git pull occasionally for fixes) and make your own private repo for assignments. Ignore the template instructions for now, they are being simplified. The #github Slack channel shows every push to the course repo
  • Assignments: four, released Mondays, due the following Friday-ish (one to two weeks each), starting Oct 5. “Assignment 0” on the schedule is a placeholder. Topics subject to change (one will likely be generative models, after the popular September assignment). Each has a rubric; do the work and you get credit
  • Midterm has an in-class part plus a take-home part done alone (coding agents probably allowed; to be decided)
  • Project: groups of 3–4 (four encouraged given class size), proposal due week 5 (Oct 26), presentations in week 9, deliverable due the Sunday of finals week. Examples from the Paris term to come
  • Participation: no reading responses, no quizzes; show up, ask questions, engage
  • Late policy: 96 late hours for the term, tracked from Git commit timestamps, no need to ask; no additional extensions except for genuine extenuating circumstances (health, family), for which you should DM
  • Communication: post questions in public Slack first (classmates answering counts as participation); DMs for personal matters; email is the most reliable fallback. TAs Tajvir and Anagha; office hours by Calendly sign-up, posted in Slack
  • Academic honesty: collaborate freely, use coding agents freely, but acknowledge collaborators and sources; likely to be asked for prompts. Understand what you turned in: the midterm and final ask about the assignments. Copying without acknowledgment is the one thing that has gotten people in trouble
  • A dates sheet (assignment out/due dates, proposal, project) will be posted and pinned in Slack
• Why this course exists
  • Built five or six years ago as “everything I wish I knew as a practitioner in one place”: how to apply ML to systems problems, hands-on, without a math-course treatment
  • Most students have seen linear regression already (mathematical foundations, adversarial ML) so the course spends its time on application, not derivations
  • Scope narrowed from “computer systems” to computer networks: “the internet is my favorite system.” Few students have taken networking, so treat this as a networking course that happens to be applied ML
• Motivation: why ML for networks (Introduction, Lecture 1, first half)
  • The internet is critical infrastructure, prone to failure, misconfiguration, and attack, and it does not self-heal or self-manage (yet)
  • Troubleshooting example: a Zoom call or Netflix stream degrades. Detecting there is a problem before a trouble ticket, then localizing it (building Wi-Fi, upstream/CDN, the application, an old device) are all inference problems, hence ML
  • Attacks are continuous (“we are being attacked right now”); detection and mitigation are inference problems too
  • Capacity provisioning and forecasting (three devices per student now; how much more next year?)
  • Why not rules and closed-form equations: they used to work (compute loss rate and RTT, predict page load time); a modern page load opens many parallel connections to tens of servers, so no single equation applies. ML plus abundant data replaces the pencil-and-paper model
  • Pervasive encryption (past ~10 years): good for privacy, but resolution, bitrate, rebuffering can no longer be read off the traffic and must be inferred
  • Attacks evolve; conditions drift. Model drift anecdote: a Verizon model predicting call drops and data-rate degradation at cell towers stopped working when summer arrived (humidity and leaves on trees change radio propagation). Even models need retraining; covered around week 8–9
• The ML pipeline picture
  • A foundations course covers the “modeling” box; this course covers everything around it: getting data out of the system, representing it, presenting it to the model, and the efficiency consequences (training time, time to prediction)
  • First few weeks are mostly about getting network data into a form a model can consume
• Three application areas (preview of Lectures 2–4)
  • Security: detecting and mitigating attacks (Friday)
  • Performance: inferring video quality (resolution, rebuffering) from encrypted traffic; this is Assignment 1
  • Resource management: short- and long-term capacity decisions
• Brief history of ML in networking
  • ~30 years ago: email spam filtering with a naive Bayes classifier (asked in class; will be covered)
  • Mid-2000s: ML for botnet detection from network traffic (the instructor’s startup era)
  • Mid-2000s to recent: network programmability. The vision: measure, infer (attack, performance degradation), then act through control software. Still the vision; increasing use of generative/agentic AI in production, but not there yet
• Learning objectives
  • Know when ML is the right tool at all; when a simple rule suffices; when regression, a tree ensemble, or a deep model fits. Deep learning gets compared against simpler models throughout; for many problems it is unnecessary
  • Apply an end-to-end pipeline: real data is dirty (errors, missing values, outliers). Look at the data before modeling
  • Understand data-representation trade-offs: the same phenomenon can be represented many ways
  • Practical pitfalls: overfitting, label scarcity, concept drift, privacy
• Homework for Friday: clone the course repo and make sure the Hands-On 1 (packet capture) notebook loads locally. No hands-on today; Friday will start with Hands-On 1 (possibly two hands-ons) and Lecture 2, Security

Meeting 2 (Fri Oct 2)

• Hands-On 1: Packet Capture Basics (instructor walk-through, then about 25 minutes in pairs)
  • Wireshark: capture on the Wi-Fi interface, with a capture filter restricting the trace to one host (the CS department web server). Find the server’s address with dig +short, start the capture, reload the page, stop, and save
  • The first hands-ons use CSV exports; soon everything will use the raw packet-capture format (pcap)
  • The habit being built: do not hand a data set to a model before looking at it. Garbage in, garbage out. Ask whether the capture makes sense
  • Reading the trace
    • Each row is a packet. Length is in bytes. Sorting by length shows the large packets all flow from server to client, as expected for a download
    • Large packets top out near 1,400 to 1,500 bytes because packets have a maximum size. A negative length or a 10,000-byte packet would mean something is wrong with the data. (The exact numbers are not exam material; the sanity-checking habit is)
    • Opening sequence: the TCP three-way handshake, then the TLS client hello and server hello, then the data transfer, visible where the packets get large. TCP and TLS details are background, not exam material
    • A packet is layered. Link-layer headers (Wi-Fi, Ethernet) are not used in this course. The IP header carries source and destination addresses and length. The TCP header carries more fields
  • What becomes a feature: timing, packet length, direction, and header fields
    • Header fields are built differently by different operating systems, so they can identify the operating system or device (the idea behind nmap). Past exams ask when using such fields as model input is a good or a bad idea
  • The payload is encrypted, so the application data cannot be read from the trace. Good for privacy, and the reason inference is needed
    • This is Assignment 1: before encryption, video resolution, rebuffering, and startup delay could be read directly from traffic. Now a network operator who wants to know whether users are having a bad streaming experience has to infer it from data like this
  • Notebook steps 3 and 4: compute summary statistics (mean and median packet size) and visualize (histogram, cumulative distribution)
  • Using coding agents for syntax is encouraged. Hands-ons are not submitted, but exams can ask about them; solutions will be posted
  • Questions that came up during the hands-on
    • Estimating round-trip time: match a data packet’s sequence number to the acknowledgment for it and subtract the timestamps. Assuming adjacent packets pair up is the sloppy version
    • Sanity-checking the result: about 41 ms was observed, high for a nearby server. A cross-country round trip is roughly 80 ms; a ping to a data center across the street was 3 to 4 ms
    • Latency varies with congestion, Wi-Fi, and server response time. Working out which is itself an inference problem. Plot the distribution rather than trusting one number
    • Two students doing the same task captured very different numbers of packets. Possible causes include retransmissions and different network conditions; left as an open question
  • Where this sits: the start of the pipeline, acquiring data, checking it, and cleaning it. Feature construction comes next
• Security (Lecture 2), first part only
  • The course framework applied to security: measure, model, then act (block, rate-limit)
  • Email spam was one of the first real-world applications of machine learning anywhere: naive Bayes on the words of the message (naive Bayes is covered around week 3)
  • Content-based filters can be evaded. Attackers hid innocuous text in the message (white-on-white text in HTML email) to confuse the classifier. A classic adversarial machine learning problem
  • Behavioral, network-level signals persist regardless of content: timing, sizes, and where the traffic goes
  • Predicting an attack before it happens: attacks need setup, and setup is observable. Example: many similar domain names registered at once is a feature that an attack is coming
  • The remaining slides in the deck were not covered
• Preview of Hands-On 2 (Monday)
  • Two traces: an ordinary page load and an attacker scanning a web server for a known vulnerability. Compare timing and size characteristics, as a step toward features and classification
• Logistics
  • Hands-On 2 was not reached today. All three use cases (security, performance, resource management) will be done by the end of next week
  • Assignment 1 goes out next week

Meeting 3 (Mon Oct 5)

• Housekeeping
  • Assignment 1 (Video Quality Inference) is officially out: the notebook and a rubric are in the public template; copy them into your private repo
  • Office hours: a TA holds them Wednesdays at 3 pm on Zoom; the instructor’s are Tuesday evenings by sign-up, details to be announced
  • Intake form and private repo: most are set; anyone with problems should say so
• Security (Lecture 2), ML applied to network security
  • Two directions: ML for security, and security of ML systems. This course does the first. The second has its own course (adversarial machine learning) and is not covered here, except that a detector’s designer must expect attackers to try to evade it
  • Detecting attacks is a classic machine learning problem. Other examples: detecting malware infections from a device’s abnormal behavior; attacks built on the domain name system (malicious domain registrations, phishing); anomaly detection in traffic in general (unusual volume, unusual time of day, unexpected destinations, devices that should not be on the network). Anomalies can be failures or misconfigurations as well as attacks
  • Why security is an especially hard ML problem
    • The things to detect are rare and often new, so there is little or no training data
    • Attacks are often one of a kind; the next one looks different
    • Class imbalance: unlimited normal traffic, very few examples of the attack. Generative models can help by synthesizing more attack examples (covered around week 8)
    • Concept drift: a deployed model stops working, because conditions change (the seasonal cellular example) or because adversaries adapt
    • Many data formats in practice; this term uses a small number on purpose
    • Real-time constraints: time to detection matters for closed-loop operation, not only accuracy on a test set
  • Spam, revisited
    • Email spam filtering was one of the first practical applications of machine learning; naive Bayes on message words (naive Bayes is covered in a couple of weeks)
    • Content filters are cheap to evade (stuffing innocuous text into the message). Behavioral and network-level signals are costlier to evade: sending volume (hundreds of messages in a short time), unusual hours, many recipients per message, connection rate, recipients per session, connection duration
    • The detailed slides on that work are not exam material; the idea that sending behavior is a signal independent of content is
  • Predicting attacks before they happen
    • A spam campaign needs a website, which needs a registered domain. With access to registration data, bursts of related registrations are a signal before any spam is sent
    • The same idea was applied to botnets and to disinformation campaigns: newly registered domains that imitate news sites, hosted somewhere that makes no sense for the intended audience
  • Midterm flag: given a kind of attack or unusual behavior (a botnet, a phishing campaign, a disinformation campaign), what features would you look for and what data would you need to detect it? Feature design and data collection, not a specific paper
• Hands-On 2: Security (scanning)
  • Two traces supplied in the course data: an ordinary web page fetch and a scan of a web server for the Log4j vulnerability. Same questions as Hands-On 1: packet count, duration, packet-length distribution, protocol types, inter-arrival times, number of unique destinations
  • What the traces show: scanning sessions are much shorter, there is little large server-to-client content, and the unique-destination counts and inter-arrival times differ. Charts to be posted
  • Clarified in class: you will not be asked to write Python functions; work cell by cell and use coding help for syntax. Only a few students have trained a model with scikit-learn before, which is fine
• Performance (Lecture 3): inferring quality of experience
  • Network operators have limited visibility into the user’s experience on an application they do not control, yet their decisions affect it, and they need to know when a bad experience is their failure to fix
  • Video is the dominant traffic
  • Speed is not experience. A speed-test number does not map to what a user feels. A 2013 study for a regulator plotted page load time against throughput: beyond a point, more throughput does not load pages faster
    • Why: latency, not throughput, is the bottleneck. Server processing time, propagation delay (speed of light), queueing
    • Road analogy: lanes are throughput (cars per minute); trip time is latency. Adding lanes does not shorten the trip
  • Video QoE metrics (the subject of Assignment 1): startup delay, resolution, resolution switches, rebuffering
    • The same question, asked by journalists a few years later: does more speed mean better video? Same flattening. The finding ran on the front page of a national newspaper in August 2019
  • The encryption wall. As video took off around 2015 to 2016, traffic became encrypted. Operators cannot see resolution, startup, or rebuffering from the traffic. That turns QoE into an inference problem on timings, sizes, inter-arrival times, and the number of parallel connections (one player opens more connections under poor conditions, itself a possible signal)
  • The pipeline used in that study, broader than the assignment
    • Service identification: find the video traffic among everything else in a home, using DNS lookups for the video servers’ domain names
    • Features: bytes per second, packets per second, packet-size statistics, retransmissions, latency
    • Segments: adaptive streaming (DASH) fetches video in segments whose size reflects the chosen resolution; segment boundaries are visible as gaps in the download, and segment sizes become features. Assignment 1 does exactly this
• Hands-On 3: QoE inference (started, about eight minutes)
  • Step 1: from the supplied capture, identify the video traffic by DNS lookups to the video server domains. Next: find segment downloads and their sizes from gaps in the traffic. Finish at the start of Friday’s class
  • Demo: the browser’s developer tools (Inspect, Network tab) show the same video server domains and the byte-range request for each segment. The session can be saved as an HTTP archive (HAR) file, another useful data source; not exam material
• For Friday
  • Finish Hands-On 3, then the third use case: resource optimization (how players and networks adapt)

Meeting 4 (Fri Oct 9)

• Housekeeping
  • Project proposal due Monday October 26. Talk to the TAs as you cook up ideas; they have announced office hours and are a good resource
  • In-class midterm Friday October 30, here
  • Monday’s class is on Zoom (the instructor is in New York); the link is in Slack. Anyone who wants to sit in the classroom and join from there is welcome to
  • Office hours are by sign-up, Tuesday evenings, link in Slack. Sign up for two contiguous slots if you need more time. If the time does not work for many people, another evening will be added
  • Completed versions of the hands-ons will be released at some point
• Hands-On 3, continued (about 20 minutes): identify the video traffic from DNS, then find segment boundaries
  • Two tricks: use the DNS answers to pick out the video servers (the assignment will not need this step), and find segments, which are larger than packets, by the zero-byte packets that separate them. Segments per second then becomes a useful feature. Assignment 1 is the inference step that follows from this
  • A student noticed that the player opens several connections to Netflix but uses one far more than the others, and asked why the data is not simply split evenly across them. Nothing is sent without a request from the player: the player asks for segments one at a time, choosing a bitrate for each, so the connections are not a uniform split of one stream. The smaller connections are probably audio, and others may carry different resolutions or standby streams. Because the traffic is encrypted this is partly a guess, which is the whole point of the inference problem
• Lecture 4: Resource allocation (chapter 2.3)
  • How much of a finite resource, usually capacity (bitrate, throughput), to give to an application, flow, user, or device. Two timescales
  • Short timescale: packet-by-packet or segment-by-segment decisions: how fast to send, which video resolution to request, when and along which path to send inside a data center. Congestion control used to be heuristics in the operating system (watch loss and latency, speed up or slow down); increasingly these decisions are made by learned models in real time, a shift that began about ten years ago
  • Long timescale: forecasting. Campus traffic and device counts grow every year; IT has to decide on more access points, more internal capacity, more transit from commercial providers. Content delivery networks (a global network of web caches; students named Cloudflare, Google, AWS, Azure, Akamai, Fastly, and Netflix’s own) face the same questions: where to put servers, what draining a data center for maintenance does to users
  • What-if (counterfactual) evaluation: is the system resilient to a Chicago power outage, and how bad would it be? You cannot decide whether a mitigation is worth its cost without predicting what happens. Twenty years ago these were closed-form equations (loss rate to TCP throughput); with parallel streams and adaptive players that math says nothing about user experience, which is where ML took over. Still a wide-open research area, including A/B testing and, in the instructor’s group, whether generative models can produce traffic “in the style of” another player or another network type for counterfactuals. Good project territory
  • A student asked how resource allocation differs from the performance use case. The instructor’s framing: performance (Lecture 3, Assignment 1) is inference from data; resource allocation is deciding and acting. The student connected this to the control loop from Lecture 1, inference on one side and action on the other, which the instructor adopted
  • Adaptive bitrate streaming. The video is encoded at several bitrates in segments, each with its own URL; the byte-range requests seen in the browser’s developer tools on Monday were segment requests. The player continually estimates network conditions and the state of its playback buffer and requests the next segment at a chosen quality. Switching quality is just requesting a different object (DASH, dynamic adaptive streaming over HTTP). A Netflix paper used buffer occupancy as the main signal
    • Objectives in tension: high average quality; no quality switches (more annoying than a steady lower resolution); no rebuffering; fairness to other applications. Capacity estimates fluctuate because nobody has a reservation: run two speed tests back to back and you get two numbers
    • Fairness is broken as a concept. It used to be reasoned about per connection; now Netflix versus a browser is apples and oranges. Pie analogy: equal thirds is not fair if the kids only want a quarter each. A student suggested fairness in terms of each application’s quality of experience rather than throughput; the instructor agreed nobody knows how to do that yet. Open research, good project territory
    • Rule-based heuristics do not generalize, so reinforcement learning has been applied: features (past segment throughput, download time, next-segment sizes per bitrate, buffer occupancy, segments remaining, last bitrate), an action (bitrate for the next segment), and a reward. Many students have seen RL before, so the RL chapter may get class time later in the term. Today was motivation only
  • Learned congestion control. One layer down, the operating system’s transport protocol decides how fast to push packets, using signals such as acknowledgment timing and latency. Classic schemes (TCP Vegas, Reno) were pencil-and-paper heuristics; just over ten years ago a student of the instructor’s own PhD advisor showed that a model could take those features and output the sending rate. Will not be asked: the names or workings of the classic congestion-control schemes
  • WISE, the what-if scenario evaluator (Google, about 2008; brought to the instructor by a former student). Search response time is a distribution and the tail matters; milliseconds are worth large sums. Web pages are served with a front-end/back-end split: the nearby front end immediately ships static content (logos, style sheets) while the search itself goes to a back end and comes back as structured data to be merged in. Question: if the India front end goes down for maintenance and Indian users are served from Taiwan, what is the new response-time distribution? Inputs: round-trip times browser-to-front-end and front-end-to-back-end, loss rate, and similar; target: response-time distribution. A regression with a basis expansion, trained on Google’s abundant data, deployed at Google. The system also did causal discovery, because changing one feature (round-trip time) changes others (loss), so the change has to be propagated before the regression is applied. Midterm flag: you will not be asked how WISE works, but you will need to know linear regression and basis expansion, which are coming in the methods part of the course
  • Provisioning and capacity forecasting. How much capacity an ISP should buy toward its peers, decided months or years ahead because it means lighting or trenching fiber. COVID-19 as the case: traffic volumes jumped and the upload/download ratio inverted as homes became mini campuses; ISPs had to augment capacity faster than their models predicted, using time-series forecasting. The plots were shown but not walked through
  • Model drift / concept drift, seeded for later (around week 8 or 9): a deployed model works until conditions change, as with seasonality in the cellular example and with COVID
  • Not covered: the Remy and Pensieve details, the “reality check” slide on learned congestion control, and the drift-mitigation list
• For Monday (on Zoom): a new module begins, the ML pipeline, starting with data acquisition (chapter 3.2 to 3.3). Class ended a few minutes early

Meeting 5 (Mon Oct 12, on Zoom)

• Housekeeping
  • Class met on Zoom; back in person on Friday.
  • Assignment 1 due Friday October 23. Project proposal due Monday October 26. In-class midterm Friday October 30.
  • Office hours are by sign-up on Tuesday evenings; the first session is Tuesday October 13.
• Hands-On 3 wrap-up
  • Recap of what the hands-on did: take a traffic capture, use DNS to identify the service, then use properties of the traffic to find segment boundaries, so that features such as segments per second or segment size become possible.
  • Today’s hands-on is probably the last one that does not use the netml library; later hands-ons build on it.
• Lecture 5: Data acquisition (chapter 3.2 to 3.3)
  • Data acquisition is the first stage of the ML pipeline: get data out of the network, turn it into features, feed those to models. In networking this is called network measurement or Internet measurement; it is a research area of its own (the Internet Measurement Conference) and the course spends one day on it.
  • Why measure, by use case:
    • Performance and operations: detect and diagnose outages and performance degradations (full outage versus intermittent degraded bandwidth or quality of experience); generate reports for customers and operators, including service level agreements (SLAs). An SLA such as “five nines” means 99.999% availability, about 5 minutes of downtime per year (correction: “maybe 30 seconds” was said in class). An enterprise like the University buying upstream service would expect an SLA from its provider.
    • Tune configuration to actual traffic: after a failure or an unexpected traffic increase, operators may want to adjust; the first step in any what-if evaluation (Meeting 4) is measuring that the scenario exists. Capacity planning, as discussed earlier.
    • Security: attack detection from logs or the traffic itself; characterizing anomalous behavior so it can be detected later; fingerprinting devices and services (example: finding legacy devices older than ten years that need upgrading); gathering normal and attack traffic to feed intrusion detection.
  • Active versus passive measurement
    • Active: inject synthetic traffic (probes) into the network and observe how endpoints respond. A speed test was run live in class: it picks a server, downloads as much as it can, then uploads, and reports download and upload throughput, idle latency, latency under download and upload, packet loss, and jitter. Ping to a well-known site and a DNS lookup (with its query time) were also shown as active measurements; the specific commands will not be asked.
    • Passive: capture existing traffic as it traverses the network and infer properties from it. Wireshark and Assignment 1 are both passive measurement.
    • Midterm flag: given a description of a measurement you have not seen before, say whether it is active or passive. Also a common question: the design trade-offs between the two.
    • Trade-offs. Active is easy to deploy (you need a server to measure against, though it can be someone else’s), has no real privacy concern because the traffic is yours, but may not reflect real user experience and adds load to the network. Passive reflects actual user behavior at the time of the problem (if a user complained about Zoom half an hour ago, the log has it), but requires capture infrastructure and raises privacy concerns because all traffic is being captured. An active test run after the fact may not reproduce a transient problem.
    • A student asked what “capture infrastructure” means and whether active measurement does not also need infrastructure. Passive capture infrastructure is anything that captures bytes on the wire (tcpdump, Wireshark, libpcap, eBPF) plus whatever aggregates packets into flows; capturing someone else’s traffic requires significant access. Active measurement needs a server to measure against, but ping, a DNS lookup, or a speed test need nothing deployed by the person measuring.
    • A student asked how, given full access to the network, you tell a network problem from a bad user device. Answer: with active measurement you often do not know what you are measuring: the access link, the home Wi-Fi, the user’s device, or something else. A plot was shown in which circle size is the downstream speed measured by a speed test: with high ISP speeds the home Wi-Fi is usually the bottleneck, with low ISP speeds the access link usually is. Passive end-to-end measurement has the same limitation: you can see that a download was slow, not why.
    • Pitfall anecdote: speed-test data from phones was used to argue that an ISP never delivered more than 100 Mbps, but the phones’ radios at the time could not exceed 100 Mbps, so the tests were measuring the phone, not the ISP. Lesson: be clear about the conditions under which data was gathered and what is actually being measured.
  • Dimensions of an active measurement: upload and download throughput (“speed”); idle latency (nothing else going on, as with ping); latency under load, also called responsiveness (latency while an upload or download is running); packet loss; jitter. Road analogy: idle latency is the drive downtown at 3 a.m., latency under load is rush hour, and road width only matters once there is load. Traceroute was skipped.
  • Passive measurement: flow-level records
    • Instead of a packet-by-packet log, summarize traffic per flow: packet and byte counts, start and end times (duration), whether any TCP flags were seen (a binary did-you-see-it), per direction, often in time windows.
    • A flow is the set of packets sharing a five-tuple key: source IP, destination IP, source port, destination port, protocol (commonly TCP). Packets are hashed into buckets by that key. Whether the reverse direction goes in the same bucket or a separate flow is a design choice. Conventionally a router ends a flow after a period of silence, about 15 to 30 seconds (30 is the usual default), and later packets start a new flow; post-process if that does not suit the problem.
    • Midterm flag: the precise definition of a flow is an easy question to ask and check.
    • Flow export is standardized (IPFIX, an IETF RFC), so any network hardware can export flow records; it is usually implemented on routers or line cards in the middle of the network where full packet capture is impractical. Standard records also carry the type-of-service byte, the input interface, and the next hop (not used in this course).
    • Aggregates are much lower volume than packet captures but lose most timing information; the netml library developed for the course keeps some timing and size information (for example inter-arrival times), which the course’s own research found useful for security-motivated anomaly detection.
    • On very high-speed links flow records may be computed from sampled packets (for example every tenth packet); not something that arises with the end-host captures used in this class.
    • eBPF (extended Berkeley Packet Filter): custom, verified packet filters pushed safely into the kernel so feature-like statistics can be computed at high speed without passing every packet to user space. It replaced the old trade-off between flexible-but-slow software and fast-but-expensive capture hardware, and is standard in large industry networks. Will not be asked: how eBPF works or is implemented, or the capture hardware.
  • Preview: the two ways to represent a pcap for learning are aggregating into flow records (the netml approach) or feeding the whole capture, as bits, to a neural network. More on feature representation Friday.
  • A netml enhancement is listed as a possible course project.
• Hands-On 5: Data acquisition (notebook)
  • Load a packet trace and generate per-flow statistics by hand: bucket packets by source IP, destination IP, source port, and destination port (protocol can be omitted here because the trace is all UDP on port 53), then compute bytes, packets, and duration per flow, and bytes and packets per second.
  • This is the last time packets are manipulated without a library; netml does this automatically from Hands-On 6 and in Assignment 1. The point is to see what decisions aggregation involves.
  • Students worked in self-selected Zoom breakout rooms for about 15 minutes; the breakout work was not captured.
  • On return a student asked about flows with zero duration. These are one-packet flows; whether their duration is zero, the round-trip time, or something else is a design decision, and when someone hands you a flow dataset you do not know which decision they made.
• Key takeaways named at the end: the active versus passive trade-offs, what a flow is, and the design decisions involved in producing and consuming flow data.
• For Friday (in person): turning the extracted flow data into features (chapter 3.4); Hands-On 6 uses netml.

Meeting 6 (Fri Oct 16)

The recording begins a few minutes into housekeeping; the opening announcements are reconstructed from what was repeated on the recording and in Slack.

• Housekeeping
  • Assignment 1 due Friday October 23. Project proposal due Monday October 26. In-class midterm Friday October 30; it covers everything through the end of week four, that is, through next week’s classes.
  • TA office hours are posted in Slack; the instructor’s office hours are Tuesday evenings by sign-up.
  • A student asked what a good project looks like. The instructor will post topics and examples from past terms (the summer-quarter projects were especially good); the project page on the site already lists possible directions.
  • Reminder: add the TAs as collaborators on your private repo along with the instructor; they will read assignments and proposals.
• Lecture 6: Feature extraction and representation (chapter 3.4)
  • Back to the pipeline: collect raw traffic, represent it as a feature matrix, train, deploy. Today is the second step, which is tricky and often matters as much as or more than the choice of model.
  • Options for a packet trace: throw the raw trace at a model (nPrint does this; trade-offs below) or compute features that separate the classes you care about. Example from the hands-ons: scan versus normal web traffic differ in typical packet sizes and flow lengths. Even one feature has many encodings: packet size as an integer, or binned as small, medium, large.
  • Why it is hard: raw captures are large and unstructured, many transformations are possible, and the best one depends on the task. Even packets per second has a hidden choice, the time bin (one second, ten, thirty, a minute). Encryption hides content, which is why Assignment 1 and the QoE work infer from metadata.
  • Building-block metrics: throughput, latency (the road analogy from Monday), jitter (variation in inter-packet delay), packet loss (itself representable many ways: count, rate, window, percentiles), and statistics over the five-tuple flow.
  • The common path from packets to features: parse into five-tuple flows, then aggregate per flow and, going beyond NetFlow and IPFIX, per time window within a flow to get a time series. Libraries: scapy for parsing, eBPF for in-kernel capture, netml for the course.
  • netml (links posted in Slack): one call turns a pcap into flows, another turns flows into a feature matrix that goes straight into scikit-learn. The weeks of hand-mangling captures in the hands-ons can now be two calls. The library is due for a refresh; improving it is a suggested project.
  • A student asked about packet length and payload. Captures commonly record only headers (the Wireshark columns), not the 1500-byte payload, which is encrypted anyway. netml can emit the per-packet size sequence (the SIZE representation).
  • STATS features per flow: duration, packets per second and bytes per second (over the flow, or per configurable time bin), and packet-size statistics (mean, standard deviation, interquartile range, min, max). Duration is a plausible attack-versus-legitimate discriminator. A student asked whether this is the only feature vector to know; no, it is a useful default, not gospel, and anything that helps training is fair game.
  • A student asked whether features should be normalized. It depends on the model: a linear model mixing 50-byte and 1,500-byte packets behaves badly without scaling, while tree-based models such as random forests split on thresholds and can usually skip it. netml does not normalize; another project opening.
  • A student asked about measuring how noisy or dirty data is. netml has no sanity checks (negative round-trip times, malformed addresses); adding them is a good project. Always look at the distributions of packet sizes and inter-arrival times before trusting a feature.
  • Other representations netml supports: SIZE (time series of packet sizes), IAT (inter-arrival times), and sampled versions that count packets or sum bytes per time bin, with the bin width configurable; windows can be equal time or equal bytes.
  • Survey of published anomaly-detection systems: no two papers represent traffic the same way, so there is no single right representation. Header fields such as TTL, TCP flags, and window size carry useful information. Frequency-domain (FFT) features, which netml can compute, did not help on the problems the group studied.
  • Representation learning. Deep learning is the main form: feed raw data with labels and let the model learn the features, as image classifiers work from pixels rather than hand-counted edges. The appeal is one representation applied once and reused across tasks.
  • Midterm flag: explain the trade-offs between manual feature engineering (netml-style) and representation learning (nPrint, deep learning).
  • Costs of manual engineering: it needs domain knowledge (you must know what a packet is to compute packets per second); reproducibility suffers (the DARPA 1998 intrusion dataset is described differently, down to packet and flow counts, across hundreds of papers); and features built for one task (intrusion detection, spam, scans) may not transfer, so the engineering is redone each time. A library like netml aims to cover most needs.
  • nPrint: if images classify from pixels, classify packets from bits. Each packet becomes a fixed-width bitmap aligned to protocol field boundaries; each bit is 0, 1, or -1 for a field that is absent. Alignment matters because the model is only seeing bit positions: if the source address sat at a different offset in different packets, position would mean nothing. Costs: about twice the storage, and no notion of time, so rates and timing are better computed directly. A link and the nPrint datasets and benchmarks were posted; running one of those benchmarks through your own pipeline is an acceptable if modest project.
  • Spurious correlations. The husky-versus-wolf classifier that turned out to detect snow in the background. In traffic: a model may latch onto the source IP address of the attack in the training set, or more subtly onto TTL, which encodes the number of hops and therefore the topology; the instructor’s group once scrubbed addresses and still found the model had overfit to TTL. So representation learning does not remove the need for domain knowledge; you need it to check what the model learned. This is the same explainability concern that appears in fairness contexts (proxy features such as race in lending).
  • A student asked whether nPrint loses information. Not bit for bit, but it drops timing and the relationships between flows and directions. Whether the huge bitmap can be compressed without hurting accuracy is an open question and a good project.
  • A student asked whether nPrint lets you skip feature engineering. Not really: it lets a deep model find which header bits matter, but it cannot express rates, and the spurious-correlation trap remains. A student suggested using a deep model on raw data to discover representations, then encoding those features with netml for a lighter model; the instructor endorsed that as a general technique. Deep models also need more data.
  • Granularity choices: per packet, per flow, per time bin, per N packets; whether burstiness matters, since flow-level aggregation hides sudden rate changes unless bursts are computed first.
  • Privacy and availability constraints: full payloads carry user data even when they are not encrypted; DNS lookups are useful for application identification (the Netflix CDN example) but increasingly hidden by DNS over HTTPS and DNS over TLS, and the server name in the TLS handshake is being encrypted too; IP addresses are both an overfitting risk and, in some jurisdictions, personally identifiable information; TCP sequence numbers are privacy-safe but rarely useful.
  • Bottom line: representation is empirical. Set up the sweep (time bins, netml versus nPrint, dozens of variants) and let coding agents run it overnight; the instructor would like to see projects that automate representation search, since AutoML tools tune models but not representations.
  • Not covered: the encrypted-traffic classifier vignette, the feature-engineering comparison slide (discussed in Q&A instead), and the summary slide.
• Hands-On 6: Feature extraction with netml (notebook)
  • About 20 minutes with a neighbor: install netml, load a capture, pcap to flows, flows to features, inspect. Most of it can be copied from the netml documentation. Group work was not captured. A quick check at the end found students with netml installed, flows extracted, and features computed; the room stayed for the full period to finish.
• Next (Mon Oct 19): data preparation and representation (chapter 4.1), Hands-On 7.

Meeting 7 (Mon Oct 19)

The recording begins at the end of the housekeeping block.

• Housekeeping
  • Assignment 1 is due Friday October 23; the project proposal Monday October 26; the in-class midterm Friday October 30. Add both TAs as collaborators on assignment repositories.
  • This is week four. Friday’s meeting, on building the ML pipeline, ends the first half of the course; the second half is about models and their applications. The midterm covers the first half: data acquisition, representation, preparation, and the pipeline. The agenda file is the complete list of what it can draw on. The final is “cumulative only so far as life is cumulative” and will be mostly about the use of models.
  • Slides are in the course GitHub repository as Quarto source and are being revised continually; a legacy PowerPoint version with a few more figures is there too. The instructor will render the decks to PDF and post a zip in Slack rather than committing rendered PDFs.
  • Hands-On 6 was not revisited; nobody asked for more time on it.
• Where we are in the pipeline
  • Packets in, a transformation step, a representation step (netml, nPrint), an optional compression step, then model training. Today’s first model is trained in the hands-on. Efficient representations and the compression step were flagged again as a project area.
  • Why representation matters: traffic is huge, so representations have to scale; sequential dependencies carry signal (a handshake followed by nothing may be a scan); inter-arrival times are one thing netml captures.
  • Cross-flow dependencies are an open research problem: a page load opens eight or more parallel connections, the TCP connection cannot happen before the DNS lookup, and object downloads cannot start before the index page arrives. Neither netml nor nPrint captures dependencies between flows. Students interested in this were invited to office hours.
  • Representation and model choice are coupled. A student described using an LSTM, which needs a representation with a time index; another reported that a time-series representation did not improve prediction in their application. The instructor’s point: if the data has no real temporal structure, forcing it into a time-series representation and model may not help, a theme that returns in the deep-learning lectures.
  • Whether to normalize depends on the model: linear models need it, tree-based models such as random forests generally do not. The choice of model shapes how you represent the data and vice versa.
• Features as domain knowledge
  • The best engineered features encode domain knowledge a model on raw data might not discover. For the log4j scan: short connections, small packets, higher IP-address entropy. Verify the intuition in the data, then encode it; this tends to give simpler, more interpretable models.
  • Preview of the deep-learning reading: in its DDoS-detection comparison, nearest-neighbor and tree models perform as well as or better than deep models. Why is a second-half question.
  • Deciding what to encode is science; deciding how is more art. Example from the instructor’s spam-detection research: hundreds of adjacent IP addresses all sending mail from one network is a sign of a bot farm, since a real network has one or a handful of mail servers. Encoding “density of mail senders” took five to ten attempts; what worked was the distance to the k nearest other senders. Exploring many encodings of one idea is a good fit for agentic coding tools.
• Feature operations
  • One-hot encoding turns a categorical feature into one binary column per value. Ordinal encoding (red, green, blue as 1, 2, 3) is fine for some models but wrong for linear regression, where it implies an order that does not exist.
  • Worked example: protocol as TCP, UDP, ICMP. Coding them 0, 1, 2 is fine for a random forest or other tree model, but a linear model would treat ICMP as twice UDP, so use one-hot. Neural networks also often do better with one-hot inputs.
  • Discretization: bin packets per second in tens, so 15 and 17 fall in the same bin. With domain knowledge, choose the bin edges: a 100-byte threshold separates acknowledgments from data packets; port numbers split into privileged server ports and unprivileged client ports, a natural step function. Binning also adds robustness: the slightly short last packet of a transfer is still “a 1500-byte packet” to the model.
  • Missing values: scikit-learn will impute missing values by default (for example with the column average), which you may not want. Options are to drop the row, impute, or encode “missing” as its own value, which is what nPrint does. A class discussion followed.
    • Students suggested cases: sequence numbers, where a missing value may mean a corrupted packet you want to keep; timestamps, where a missing ACK time is bounded by its neighbors and a negative or zero time is clearly wrong and safely corrected; packet sizes, probably not; using other correlated fields in the same row to fill the gap.
    • The networking twist: unlike a missing pixel in the middle of an ocean, a missing or strange value in traffic may be the anomaly you are trying to detect, so imputing it away can destroy the signal.
    • Midterm flag: the instructor said this may become a midterm question: give a case where imputing an average for a missing value is appropriate and a case where it is not.
  • Normalization: min-max scaling maps a feature to [0, 1] by subtracting the minimum and dividing by the range; standardization subtracts the mean and divides by the standard deviation. Needed for linear models for the same reason as one-hot (“blue is not twice green”); the bytes-to-packets regression in next week’s linear-models lecture will show it.
  • Transformations: a log transform tames long-tailed network features, such as flow sizes where most flows are short and a few have ten thousand packets. Basis expansion comes with linear regression next week. None of these is a recipe; they are tools to try.
• Data pitfalls (covered in about ten minutes)
  • Five failure modes: insufficient data, non-representative data, poor-quality data, spurious correlations, overfitting.
  • Insufficient data: more data generally means better generalization, especially for deep learning (this answers last week’s question about whether raw-bitmap representations need a lot of data: yes). In security the problem is rare attack classes; you may have to hunt for or generate examples of a specific scan or exploit. For Assignment 1, the dataset contains no 4K video, so no classifier trained on it can recognize 4K.
  • Non-representative data: the Finnish translation example. Finnish has no gendered pronouns; translating English to Finnish and back produced gender-stereotyped pronouns learned from the training text. Students observed that non-representative data is a special case of insufficient data, which the instructor accepted (either answer would get credit on a check-all-that-apply), and that a dataset can become non-representative as the world changes.
  • Spurious correlations: the husky-and-wolf slide, already covered twice, was not repeated.
  • Cleaning: this is why the hands-ons keep asking for max, min, and mean of every column. Not all outliers are errors; be intentional about removing or keeping them. If timestamps are definitely wrong you must decide whether to drop, fix, or keep the data, and when the data was expensive to collect, “no data, bad data, or patched-up data” is a real trade-off. Paxson’s Strategies for Sound Internet Measurement was recommended as optional reading; it will not be examined. The instructor plans to give a deliberately messy dataset to clean in a future year.
• Data leakage and splits (covered briefly; more on Friday)
  • The golden rule: never train on the test set. It happens by accident: normalizing features before the train/test split leaks test-set statistics into training. The instructor will check an upcoming hands-on on linear models for exactly this mistake. Do the split first, before any other operation on the data.
  • Temporal leakage: shuffling all flows and then splitting puts future traffic in the training set, so the model predicts the past from the future.
  • Arvind Narayanan and colleagues at Princeton (the AI Snake Oil authors) went back through published research papers and found these mistakes to be common.
  • Validation set: the training set is further split to hold out a validation set used to tune model parameters. This will matter once models are being trained; details Friday.
  • Overfitting and the bias-variance trade-off, introduced only: a model can be made to fit its training set almost perfectly. A reported 99% accuracy should set off alarm bells. The warning sign is training error that keeps improving while validation error gets worse. The figure from Hastie et al. (test error forms a U against model complexity while training error only falls) was shown at the end of class and has been added to the slides. Much more on this in the coming weeks.
  • Not covered: the planned breakout on auditing a pipeline, the labeling slide, the data-engineering checklist, and the takeaways. The slides on training/validation/test sets and on overfitting were covered only as previews.
• Hands-On 7: Data preparation (notebook)
  • About fifteen minutes: take the netml features from last time, normalize them, and make a first prediction with scikit-learn. If students need more time, it will be given at the start of Friday’s class.
  • A student asked what to do when feature vectors come out with different lengths: truncate to the shorter or pad the longer. There is no single right answer; try both.
  • A student asked whether the assignments follow the hands-on philosophy of using an AI coding assistant for the code. Yes: the point is understanding the data and the models, not syntax, though the instructor noted that dimension-mismatch problems are the kind of thing it was once useful to struggle with.
  • Assignment 1 questions: some features are fingerprints of the particular household where a capture was taken rather than of video quality; a student found accuracy dropped from 70–80% to about 30% when one household’s data was held out of training. Removing such proxy features, balancing the classes, and choosing how many examples of each class to train on all affect the errors a model makes, and some models tolerate class imbalance worse than others. The instructor flagged this as an extension to the assignment in a future year and encouraged exploring sampling strategies now.
  • Group work was not captured by the recording, and there was no report-back.
• Friday: the ML pipeline, training and evaluation (chapter 4.2–4.3), Hands-On 8. A student will post a question from the problem set in Slack for the instructor to address at the start of class.

Meeting 8 (Fri Oct 23)

Reconstructed from the class recording, which starts a few minutes in, during questions about the project proposal, and stops when Hands-On 8 began. The hands-on work itself was not captured.

• Project proposal (due Monday October 26)
  • One page, PDF or Markdown, pushed to your private course repository; for a group, one member’s repository is enough and the instructor will deduplicate
  • Contents as listed on the project page: title, summary, data, models you expect to try, evaluation, learning objective, plus the group members and what each one will do
  • Projects that extend an existing library or tool: say what you will build and what the deliverables are; you are graded on your own work and tests, not on whether an upstream maintainer accepts a pull request
  • The project page was updated after class to say Monday of Week 5, to accept Markdown, and to ask for member roles
  • Assignment 1 is due tonight at 11:59 p.m.
• Hands-On 7 follow-up
  • The notebook’s “normalize” wording in the shape-alignment step was changed to “align”, because it read as if feature scaling happened before the train/test split; aligning column counts is not scaling. Any scaler is fit on the training set only, after the split
• Lecture 8: Model training and evaluation
  • Labels, briefly: continuous targets (throughput, latency, page load time, video startup time) make a regression problem; binary labels (spam or not, attack or benign) and multi-class labels (DNS, unencrypted web, encrypted web, video, gaming) make classification; ordered tiers of quality of experience are also discrete labels
  • The pipeline: split into training and test sets (80/20 is a common proportion); fit on the training features and labels gives a trained model; predict on the test features gives predicted targets; the gap between predictions and the true test labels is what every evaluation metric summarizes, and the right metric depends on whether the target is continuous, binary, or multi-class
  • Why split: evaluating on data the model was trained on is cheating on the test. Hold data back and touch it only for the final measurement
  • Three pieces, not two: the training set fits the model parameters; the validation set, carved out of what would otherwise be training data, is for tuning hyperparameters (the lambda of ridge and lasso regression, a polynomial degree, a tree depth); the test set is never touched until training and tuning are done
  • Random versus temporal splits: a random 80/20 coin flip per row assumes the rows are independent and identically distributed with no temporal relationship. When there is one, train on the earlier months and test on the later ones. That also simulates the real job, such as forecasting next month’s load to decide whether to provision another server
  • Bias and variance: bias is error on the training set, variance is how much the fitted model changes with changes in the training data. A more complex model lowers bias and raises variance. Whiteboard example in one feature: a straight line with some error on every point versus a high-degree polynomial that passes through every training point with zero error and then does worse on new points. That is overfitting
  • Midterm flag: the bias–variance trade-off figure, or something like it, could appear on the midterm
  • Detecting overfitting: validation error that starts climbing while training error keeps falling says the model is too complex; the validation set is a peek at how the test set will behave without using the test set
  • Remedies: more training data (often hard); early stopping, as in choosing how many epochs to train a neural network; regularization, penalizing nonzero coefficients in a linear model; limiting the polynomial degree or tree depth; ensembles of independently trained models, which come with tree-based models later in the course
  • Overfitting the hyperparameters to one validation slice is the next risk. The remedy is k-fold cross-validation: partition the training data into k folds (five is common), train on k minus 1 and validate on the remaining fold, rotate, and average the performance. scikit-learn does this with one function call. Leave-one-out cross-validation does the same with a single held-out point per round, for small datasets where carving out another fifth leaves too little to train on; it is slow for large data or expensive models
  • A student described the common practice of skipping a separate validation set and tuning hyperparameters by hand across cross-validation folds. The instructor said it is done, including by them, and is quick and dirty rather than orthodox; whether it finds the best hyperparameters is less clear
  • Confusion matrix for a binary classifier (attack is the positive class): rows are the true labels, columns the predictions, the diagonal is correct and everything off it is wrong. The same layout generalizes to multi-class (three by three, ten by ten). scikit-learn produces it, and Hands-On 8 asks for one
  • False positive is a false alarm (benign traffic flagged as an attack); false negative is a missed detection. Both have operational costs: an operator flooded with false alarms starts ignoring the tool, and a missed attack is a missed attack. You cannot have both at zero, but a threshold lets you trade one against the other
  • Midterm flag: given the numbers in a confusion matrix, compute accuracy, precision, recall, and the rest. The formulas can go on the one-page cheat sheet, but understand them intuitively
  • Accuracy is correct predictions over everything, and it misleads under class imbalance: when attacks are rare, a classifier that always says benign is 99.9 percent accurate and misses every attack. This is the base-rate fallacy from medical screening, where a test with high accuracy and a low base rate still produces many false positives
  • Precision (positive predictive value): of the alarms raised, what fraction were real. Recall (true positive rate, sensitivity, detection rate): of the real positives, what fraction were caught. Specificity: of the real negatives, what fraction were correctly passed over. Precision and recall are the pair used most
  • F1 is the harmonic mean of precision and recall, between zero and one, higher is better, and either low precision or low recall drags it down; a common single number in papers
  • Thresholds and the ROC curve (true positive rate against false positive rate): always-alarm sits at the top right with full detection and a full false-positive rate, never-alarm at the bottom left, and a tunable classifier traces a curve between them. The diagonal is random guessing with an area under the curve of 0.5; the best possible area is 1; higher is better, and the libraries compute it
  • ROC works well with balanced classes. Under imbalance, which is the usual case in network security papers, use the precision–recall curve: as the threshold loosens and recall rises, a bad classifier’s precision collapses under false alarms while a good one keeps precision high across the range; the area under that curve is the summary. The instructor drew this on the board because the deck lacks the slide, and will add it
  • Regression metrics such as root mean squared error were skipped and will be covered Monday with linear regression
  • Midterm flag: linear regression will not be on the midterm
• Where the course is
  • This closes the first part of the course: motivation, getting data out of network systems, turning data into features (manual engineering versus representation learning), setting up the pipeline, and evaluation metrics
  • Part two is families of supervised models, starting Monday with a linear regression review, then clustering, then, time permitting, generative AI, diffusion, and reinforcement learning in the last week or two; weeks eight and nine may be rearranged to get there
  • A student asked about training two models on different random samples of the data and combining them; the instructor said that is ensemble learning, which arrives with tree-based models around the end of week six
• Hands-On 8: the ML pipeline (notebook)
  • About twenty minutes in class, working together: the same feature pipeline as Hands-On 7, then a train/test split, training, and evaluation, including plotting the ROC and precision–recall curves just drawn in lecture
  • Worked solutions to the hands-ons will be zipped and posted to Slack over the weekend
• Not covered
  • The whole-class audit of the intrusion-detection pipeline carried from Meeting 7 was not reached
  • Data leakage was not treated as its own topic beyond the rule that test data stays untouched; see Meeting 7
• Next: Lecture 9, probabilistic models (naive Bayes), plus the regression metrics and linear regression review deferred to Monday October 26; Hands-On 9; in-class midterm Friday October 30

Meeting 9 (Mon Oct 26)

Reconstructed from the class recording, which starts a few minutes in, during housekeeping, and ends when the hands-on began.

• Housekeeping
  • Every due time is now 11:59 p.m. Chicago; anything due on a date is fine any time that day
  • Project proposal due tonight. Push it into your private course repository; in a group, one copy or one per member is fine, as long as the proposal names every member. The instructor’s scripts scan every private repo and match proposals against the roster; anyone unaccounted for will get a Slack message
  • Assignment 2 is the take-home midterm. The only difference from the other assignments is that it is individual work: no working with other students. Coding agents, the TAs, and the instructor are all fine to consult. Due date as listed; the late-hour grace policy applies to it like any other assignment
  • Grading: the take-home counts in the midterm bucket together with the in-class exam, historically about half and half, exact split to be confirmed
  • In-class midterm Friday October 30, covering Meetings 1 through 8; the agenda page is the full log of what was covered. Conceptual questions: no code and no pseudocode, since nothing in this course is tested that way. Expect some recall and basic-understanding questions, and a few that give a scenario and ask you to apply a concept. Not designed to trick anyone. The full 80 minutes is available; some people finish in 15 to 20 minutes, others use the whole period
  • The instructor’s view, for the record: in-class exams are an unrealistic format, but pencil and paper is the one assessment that reliably shows who has not been following; hence both an in-class exam and a take-home
• Not covered: regression metrics. Friday’s evaluation material (false positives and negatives, precision, recall, ROC and AUC, the pitfalls of accuracy) is all in scope for the midterm. Metrics for regression (root mean squared error and the like) were not covered on Friday or today; they are deferred to the linear regression lecture a week from today, and will not be on the midterm
• Lecture 9: Probabilistic and non-parametric models (chapter 5; the start of the supervised-learning unit)
  • Where we are: real machine learning models from here on. Two simple families today, which recur through the course as baselines: probabilistic models, which estimate a distribution per class and ask which class most likely produced the observed features (a maximum-likelihood estimator, as in the math-for-ML course), and non-parametric models, whose size grows with the data
  • Theme for the whole course: on structured data whose features separate the classes, simple models often do the job
  • Naive Bayes, the setup: given features x, predict the class y that maximizes the posterior P(y given x); with spam as the running example, compare P(spam given the words) and P(ham given the words) and pick the larger. The posterior is hard to estimate directly and would need enormous data
  • Bayes’ rule rewrites the posterior as P(x given y) times P(y), divided by P(x). The denominator does not depend on y, so it is dropped: the classifier only compares values across classes, it never computes a true probability. The prior P(y) is either declared from outside knowledge (say, 90 percent of email is spam) or estimated from the training set, which is dangerous when the training classes are imbalanced. The likelihood P(x given y) of an entire feature vector is essentially zero for any specific email, so it is not usable as is
  • The naive assumption: treat the features as independent given the class, so P(x given y) becomes the product over features of P(x_i given y), each of which is a simple count from the training set (how many legitimate emails contain “learning”, divided by the number of legitimate emails). The assumption is wrong (“machine” and “learning” are not independent), which is why the method is called naive, and it is what makes the computation practical at all, since the joint grows exponentially with the number of features
  • Distribution families: for presence or counts of words use Bernoulli or multinomial; for continuous features scikit-learn offers Gaussian naive Bayes, which the instructor rarely uses
  • The zero-probability problem: a word absent from the training set for a class makes the whole product zero. The fix is a small additive constant so no likelihood is ever zero. Student questions: an unseen word gets the small constant for both classes, so it scales both scores equally and does not change the comparison; the constant must not be so small that the product underflows, and scikit-learn’s default is a modest value, not a tiny one
  • Class imbalance: Naive Bayes is known to do poorly with imbalanced training data, since the prior and the conditionals both get mis-estimated. Remedies are rebalancing the training set or setting the prior by hand
  • Case study: the white-on-white Shakespeare text that spammers appended to messages to inflate the legitimate-class score, one of the earliest examples of adversarial machine learning: attackers manipulating the feature distribution at test time, dating to the 1990s
  • Advantages: trivial to compute (just counting), scales to very large feature sets because of the independence assumption, works well on small datasets with separable classes, interpretable (each per-feature likelihood can be inspected), no high-dimensional distributions to estimate, and the gap between the class scores gives a rough confidence. Disadvantages: cannot capture feature interactions by design, poor probability estimates, sensitive to the training set
  • Why it works despite the violated assumption: the numbers are wrong, but the rank ordering across classes usually still holds, and a classifier only needs the ordering. The meta-lesson for the course: models often work despite violated assumptions
  • Midterm flag: the instructor called “why does Naive Bayes work even though the independence assumption is violated” a good exam question (the midterm scope was stated as through Meeting 8)
  • k-nearest neighbors: the one non-parametric model in the course. No training beyond storing the data (a k-d tree is a typical structure), so the model grows with the dataset; to classify a new point, compute its distance to the stored points, take the k nearest, and vote. The parameters are k and the distance function, commonly Euclidean
  • Curse of dimensionality: as features are added, every point ends up roughly equally far from every other, so the k nearest neighbors stop meaning anything unless the data grows exponentially with the dimension. Feasible on flow-level features; hopeless on an nPrint representation
  • Uses: geolocation and recommender systems (a student had used it for one); website fingerprinting, the privacy attack that identifies which page an encrypted stream is loading from the vector of object sizes, where k-NN has been used for over a decade; attack detection and flow classification, including the Log4j example. Its real role is as a baseline: simple, nothing to tune, easy to beat; project groups doing classification should compare against it
  • Pitfalls: prediction is slow because every stored point is scanned; the stored training set can be large; features must be standardized, since one feature on a scale of 1 to 1000 next to one on 1 to 10 stretches the distance; sensitive to noisy points; use k greater than 1; stored data goes stale over time (summer versus winter traffic), a drift problem that is not specific to k-NN
  • Pitfalls specific to Naive Bayes on network data: correlated features (an assumption violation to watch, not a reason to abandon the model); imbalance letting the prior dominate; and non-Gaussian continuous features, such as packet sizes, which spike at 64 and 1500 bytes, so Gaussian naive Bayes on packet or flow sizes is not recommended, though it may still work
• Hands-On 9: Naive Bayes spam classifier (notebook)
  • About twelve minutes at the end of class on the SMS spam dataset (messages labelled ham or spam). Instruction for today: load the data, then do Part 2 only, the scikit-learn multinomial Naive Bayes classifier
  • The from-scratch parts (building the vocabulary and counts by hand, Part 3) are likely to be dropped; the instructor will reorder the notebook for future terms and decide next Monday
• Next: Friday October 30 is the in-class midterm, the whole period. Monday November 2: go over the midterm, more time on Hands-On 9, then linear regression with the deferred regression metrics

Meeting 10 (Fri Oct 30)

• In-class midterm, the whole period, covering Meetings 1 through 8. No lecture content
• Graded exams were returned on November 1 to each student’s private repository (a midterm/ folder with the scanned pages, a transcript of the answers, and the score on every part) and to the Gradebook issue there
• Everything used to grade is public in the course repository under docs/midterm/2026/: the answer key, the rubric applied to every exam, the grading notes (every grading decision and why), and a class-wide analysis checking for questions where the class split from the key (none were flagged)
• Class statistics, 57 exams out of 50 points: mean 42.2, median 43.5, middle half of the class 39.4 to 46.0
• Regrade requests go as a comment on the student’s Gradebook issue, naming the question part; not in Slack

Meeting 11 (Mon Nov 2)

Reconstructed from the class recording, which begins partway into the discussion of the midterm feedback survey and runs to the end of class. Student remarks during the discussion and the hands-on report-back were mostly off-microphone and are summarized where they could be made out.

• Feedback survey and what will change (the instructor went through the survey responses)
  • Hands-on solutions: they exist and will be posted; the instructor is deciding how (a zip in Slack is the fallback). Every hands-on will get explicit learning objectives written into the notebook, starting with the ones still to come, and a short wrap-up at the end of each hands-on in class
  • More time for the hands-ons: agreed; they have been rushed
  • “Hard to find things”: everything lives on the course website, with GitHub for work and Slack for announcements. This is the first full nine-week term run entirely on GitHub (GitHub Classroom was discontinued over the summer and the instructor rebuilt the workflow); the site will be made more explicit about where things are
  • Exams: answer boxes will be about twice as large on the final and the exam correspondingly longer (eight pages or so), now that grading is no longer limited by a human reading handwriting. Select-all-that-apply questions are fine to keep. The multiple-choice options on the midterm should have been randomized (the correct options all fell in the left column) and will be
  • A separate announcements channel versus questions in #general: undecided; the instructor has kept everything in #general so nobody misses it, and asked for opinions
  • Due dates: students keep asking; the dates will be put in more places now that keeping copies consistent can be automated
  • Requested topics for the remaining weeks: reinforcement learning (a classic networking application) will be worked in; generative models are already planned; other requests welcome now
• How the midterm was graded (described in response to the survey)
  • Scans were OCR’d locally (Apple Vision and small local models; the exams were not sent to a cloud service for OCR), rendered to a web page the instructor could read, and auto-graded against a point-by-point rubric derived from the LaTeX solutions. Low-confidence parts went to three additional independent graders; disagreements (about a hundred question instances across the class) were read and resolved by hand, always to the higher grade. Token cost was on the order of four cents per exam
  • Everything used to grade is public in the course repository: the exam and solutions, the rubric, the grading notes (including instructions to give partial credit for any reasonable answer, to accept an X or an arrow where a filled bubble was asked for, and to give near-full credit on a question the instructor judged bad), and a class-wide analysis of where answers diverged from the key. Each student’s repository received their scanned exam and an HTML page with the OCR transcript beside each answer, so a misread can be spotted
  • One student had not received the scan in their repository; try git pull first, and tell the instructor if it is still missing. Checks will be added so this does not recur
  • The TAs are reviewing all auto-grades as well
• Midterm review: the questions most people missed
  • A student reports Zoom was choppy half an hour ago; a speed test from their laptop now shows high throughput and low latency. This does not show the network was the cause: the test ran later and conditions are transient; a speed test measures a different path than the one to the Zoom service; and even on the same path other bottlenecks (the laptop, the application) are not measured
  • Splitting data for a forecasting problem: a random split trains on samples that occur after the ones being tested, which is leakage for a temporal task
  • Time of day (or month, or year) as a feature: the traces in the assignment were captured in different years, so a classifier could separate them on the date alone. The feature is an artifact of how the data was collected, the same lesson as the IP-address and TTL examples
  • The flow-timeout question: the instructor called it too much minutiae and not conceptual, and will regrade it more generously
  • Finding segment boundaries in a video stream (gaps marked by zero-byte packets, as in Assignment 1) and why a player pulls segments at different bitrates (adaptive streaming)
  • When not to impute the mean: a missing or odd value may be the signal (a malformed packet can be an attack), so be intentional rather than taking the library default; a garbled timestamp in an otherwise ordered series is a case where correction is reasonable
  • The perils of accuracy as a metric, as discussed in Lecture 8
  • A student pointed out that one rubric item expected an alternative the question had not asked for. The instructor agreed it was an auto-grader error, will fix the rubric and the solutions, and will run a full regrade over everyone with instructions that scores can only go up. Another student noted that reasonable answers absent from the solutions were marked down; such cases should be submitted as regrade requests, and the solutions will be extended with answers the instructor had not anticipated
• Regrades and timeliness (policy stated in class)
  • Regrade requests go as a comment on the Gradebook issue in your private repository, which is also the record of everything turned in and the place for any grading discussion. Add a note if you want the instructor personally to look at it; those get a human read of the scanned exam
  • Grades are a first cut, not set in stone; regrading is now cheap, so the plan is to return work fast and fix what needs fixing. For Assignment 2 (due Friday November 6 at 11:59 pm) the target is feedback within about 24 hours, by Sunday morning
  • Project proposals: 47 were submitted and the instructor and TAs are reading all of them; feedback soon, and the Gradebook will at least record receipt
  • With grading automated, TA and instructor office hours are open for concepts, research, and anything else, not debugging notebooks
• Where we are: week six, behind the syllabus (which would have the class through trees and ensembles). Most of the class has seen linear regression, so the lecture was cut to the metrics and the hands-on
• Regression metrics (new slides, added to the Lecture 8 pipeline deck beside the classification metrics)
  • A regression’s error is the residual per sample, the gap between the true value and the line’s prediction, drawn as the red segments on the scatter of bytes against packets. Aggregates: mean squared error, root mean squared error, mean absolute error; always computed on held-out data
  • Squared error weights big misses much more than absolute error. On a trace with large data flows and many small acknowledgment flows, minimizing squared error pulls the line toward the large flows because their errors are larger numbers; minimizing absolute error does not. The two lines were close on the example, so which to use is a matter of trying and seeing
  • The deeper problem: the data has two modes (data flows and ACK flows), and one line fit to both serves both badly, the ACK flows most of all
  • Final flag: the ACK-packet problem and the ways to address it, flagged by the instructor after class as a possible exam question. Given a two-mode regression dataset like this one, explain why a single least-squares line fails and name the remedies: transform or scale the target (for example a log transform), change the loss to absolute error, or classify first (data versus ACK) and fit a separate model to each
• Hands-On 10, Part 1: linear regression on traffic (notebook)
  • About fifteen minutes: extract flow features with netml as in earlier hands-ons, fit a linear model predicting bytes from packets, and see the two modes in the result. The notebook then asks what to do about it: log-transform the target, fit two models after classifying data versus ACK, or change the loss function. There is no single right answer
  • Report-back by show of hands: some had tried scaling or normalizing the target, some a different error, some a staged classifier. The instructor singled out the staged classifier as a technique that transfers to many real data problems. A student’s suggestion about using additional packet-level fields from the capture was taken as homework by the instructor; the details were not captured
  • The instructor will work through the notebook, post solutions or approaches, and look for a less contrived example of the same problem (page size against page load time was suggested as a candidate)
• Not covered: the rest of Lecture 10 (basis expansion and polynomial regression, the bias-variance picture, ridge and lasso regularization) and Hands-On 10 Part 2. Friday opens with polynomial basis expansion and more time on Part 2; logistic regression will be quick and support vector machines may be skipped to catch up; trees and random forests come after Monday
• Next: Friday November 6. Assignment 2, the take-home midterm, is due that night at 11:59 pm

Meeting 12 (Fri Nov 6)

Reconstructed from the class recording, which begins during the opening housekeeping. Student questions were mostly off-microphone and are paraphrased from the instructor’s restatements; the two hands-on work periods were not captured.

• Housekeeping
  • Assignment 2 (the take-home midterm) is due tonight at 11:59 pm. Late hours are now counted automatically from the repository history and are sometimes wrong (moving files around can confuse the timestamps); if your late hours look off, say so, since more often than not the error is the instructor’s. Grades and feedback are targeted for this weekend; regrades as before, by asking
  • Assignment 3, anomaly detection, is officially released Monday November 9 with its template repository, due the Friday before Thanksgiving (November 20)
  • Project proposals: nobody has had feedback yet (show of hands), and the instructor took that as an action item for the weekend. Every student is now accounted for on a project; the titles and abstracts have been read and look promising. Two or three groups were nudged today for not naming a dataset. Tuesday-evening office hours are the place to talk projects
  • The midterm has been regraded twice in batches. The published grading notes in the repository record the class-wide decisions, including the questions judged ambiguous and given full credit
  • Hands-on solutions: by request, worked solutions now go into each student’s private repository under a hands-on-solutions folder rather than a zip in Slack; solutions through Hands-On 10 are there. Cadence, decided by show of hands (more students wanted to sit with a hands-on for a day or two than wanted the answer immediately): the solution to each hands-on is pushed before the next class meeting, so Hands-On 11 arrives before Monday
  • Where we are: end of week six, behind the syllabus (which would have trees and ensembles done). Today: polynomial basis expansion, logistic regression, the start of trees. Support vector machines are skipped entirely
• Basis expansion (the end of Lecture 10)
  • Many relationships between features and targets are not linear, and a linear model can still capture them. The instructor’s example: round-trip latency against the number of packets queued in a buffer, which rises non-linearly as the queue fills
  • Final flag: the instructor called this “a great exam question”, yes or no: if the relationship between a feature and the target is non-linear, can you still use a linear model for prediction? Yes, by transforming the features (basis expansion) or by a kernel function, which may come up next week with other models
  • Why bother: linear models have the properties proved in the math courses, and they are simple and fast
  • The idea: transform the raw inputs with basis functions and predict with a weighted sum of the transformed features; the model is still linear in its parameters because the transform is applied to the features. Polynomial expansion: beside a feature x, add columns for x squared, x cubed, and so on. The fit was a line, m times x plus b; now the model has the square of the feature as another knob and can pick up a quadratic relationship
  • The cost is model complexity: a higher degree can fit quadratic and cubic relationships and can also overfit, the bias-variance trade-off again. The degree is a hyperparameter to tune
• Hands-On 10, Part 2: polynomial expansion (notebook)
  • About twelve minutes, Part 2 only; Part 3 and the bonus were not done. Instruction: use scikit-learn’s polynomial-features transform rather than building the columns by hand, then look at the expanded feature matrix, since seeing the columns is what makes the idea click; train and test on the transformed set
  • Walk-through of the solution: a degree-3 expansion of the packet count, fit and predict on the expanded features, and the green curve through the scatter is now a cubic rather than a line. The ACK-flow problem from Monday is still there; the instructor will improve the example and add a look at the expanded matrix and a better plot to the solution
  • Questions: should each feature get its own degree? Yes, worth trying, though it expands the search space a lot. Why only the first netml feature when so many are available? Because the example is a toy; the notebook could do much more
• Logistic regression (Lecture 11, compressed)
  • Much of networking is binary classification: malicious or benign, DNS query or response, bot or human, spam or not. Linear regression is the wrong tool: it predicts a continuous value, its output is unbounded, and binary labels are not Gaussian, so the line-plus-noise picture behind least squares does not hold
  • Logistic regression is linear regression followed by a squashing function, the sigmoid: the weighted sum of the features is mapped into the interval from 0 to 1, with minus infinity going to 0 and plus infinity going to 1 and an S-curve in between. Two benefits: the prediction is bounded whatever the input, and it has a probabilistic reading, how confident the model is that the label is 1, growing as the feature moves further from the boundary
  • The sigmoid is also a building block of neural networks, one of the choices of activation applied between a neuron’s input and output; the course reaches that in a week or two
  • When it works: a decision boundary, in one dimension a threshold on x and in more dimensions a line or hyperplane, separates the classes. When the classes are not linearly separable it is the wrong choice without a further mapping; that is where deeper models come in
  • Networking examples: DNS packets by size (queries small, responses large), predicting which kind a packet is; and the remedy to the ACK problem, classify each packet or flow as data or acknowledgment first, then fit separate models, the staged approach from Monday
  • Case study, by request for more of them: logistic regression is still used in research papers for denial-of-service detection. Plausible features: request rate in a window and the number of unique source addresses, since a server that usually sees a hundred requests per second and a few hundred addresses suddenly seeing ten thousand requests per second or a million addresses maps naturally onto a two-class model
  • Final flag: a different logistic-regression case study, flagged by the instructor after class. Given a networking problem that is not the DNS or ACK example (denial-of-service detection, bot versus human, spam), say what the two classes are, propose two or three features, and explain what the sigmoid output gives you that a plain threshold on one feature does not (a confidence, a decision boundary in more than one dimension, a building block for other models)
• Hands-On 11: logistic regression (notebook)
  • About fifteen minutes. The learning objectives at the top say why: understand what the sigmoid does to the prediction, and practice fitting one. The only trick is that the data has no 0/1 column: build the label (as in the Log4j hands-on), use packet length as the feature, declare a solver, reshape the feature into a column vector, fit, and call the probability prediction, which returns a value between 0 and 1 rather than a hard label. Evaluation (confusion matrix and the metrics derived from it) was not expected in the time given
  • The instructor showed the solution and found a mismatch: the solution classifies DNS queries against responses, while the notebook the class has classifies data packets against ACKs. Same idea; the notebook and solution will be reconciled over the weekend so Part 1 is DNS and Part 2 is data versus ACK. The DNS data is not clean and the fit is not great; the data-versus-ACK version separates better
  • Question: why not just a threshold on packet size? For this toy problem a threshold would do. Logistic regression earns its place when a confidence is wanted (and a threshold still has to be chosen on it), or as a component whose 0-to-1 output feeds something else, such as a neuron. For this problem it is too simple to be interesting, which is a segue to trees
• Decision trees (the start of Lecture 12, about ten minutes; random forests and boosting are saved for Monday so they are not rushed)
  • A few students had used decision trees before, one in a managerial decision-making class, one as part of an ensemble
  • Core idea: partition the feature space into regions by a sequence of binary splits. Board example on flows: packets per second greater than 50? If yes, destination port 443? If yes, benign; if no, attack. If packets per second was not above 50, split on flow duration, and so on. A node with no further decision is a leaf, and the leaf is the classification. The instructor will add the drawing to the slide
  • The same tree drawn as a partition of a two-dimensional feature space (packets per second against duration): each split is a line, each leaf a rectangle. The leaf’s label for classification is the majority class of the training points that fall in its rectangle; or keep splitting
  • The hyperparameter is depth. In the limit the rectangles shrink until each holds one training point: 100 percent training accuracy and wildly overfit. In practice a tree is overgrown and then pruned; how, next time
  • Advantages (from the class and the instructor): you can see exactly how a decision was reached, so the model is interpretable and auditable and can be adjusted by hand, which is why it appears in a management course; simple and fast; no feature normalization needed, so a feature running from 1 to 10,000 can sit beside a categorical one such as a color; easy to retrain. A student asked whether a region can be split further while keeping the earlier structure: yes
  • The cliffhanger for Monday: a tree is sensitive to exactly which points are in the training set (why split at one second?), and overfits easily. Hyperparameter tuning helps; ensembles of trees are the better fix
  • Final flag: advantages and disadvantages of a decision tree, flagged by the instructor after class. Name the advantages (interpretable and auditable, simple and fast, no feature normalization, continuous and categorical features side by side, easy to retrain) and the disadvantages (sensitive to exactly which points are in the training set, overfits easily as depth grows), and say what fixes the disadvantages: limiting or pruning depth, and ensembles of trees
• Not covered: support vector machines, skipped for the term; the rest of Lecture 10 (the bias-variance curve in detail, ridge and lasso); Hands-On 10 Part 3 and the bonus; the rest of Lecture 11 (softmax, the noise-model view); in Lecture 12, how a split is chosen, regression trees, and pruning
• Next: Monday November 9, trees continued and ensembles (bagging, random forests, boosting) with Hands-On 12. Traffic representations and nPrint are already done, so the class catches up to the syllabus next week; unsupervised learning starts by the end of it

Meeting 13 (Mon Nov 9)

Reconstructed from the class recording, which runs through the introduction to the hands-on; the hands-on work period was not captured. Student questions were mostly off-microphone and are paraphrased from the instructor’s restatements.

• Housekeeping
  • Hands-On 11 solutions (logistic regression) went into each private repository before class, as voted Friday; anyone who did not get them should say so. The hand-drawn board figures from Friday are now redrawn on the trees slides
  • Assignment 2 (the take-home midterm) has been graded and returned through the Gradebook issues. Questions and regrade requests go in a comment on that issue; the instructor planned to pick them up right after class
  • Late hours: the automatic accounting is still buggy. The rule it uses: small changes to an already-committed submission (under roughly 5 to 10 percent of the file, including moving it to another folder) keep the date of the first commit, so a Friday 11 pm submission moved two hours later is not late. Several students saw wrong late hours after reorganizing their repositories; comment on the Gradebook issue and it will be fixed. Late hours exist to keep the term on schedule, not to punish; genuine emergencies never count against them
  • Project proposal feedback is still owed, probably Tuesday
  • The campus account lockout over the weekend does not affect this course’s infrastructure (GitHub and Slack)
  • A survey on the automated grading workflow (rubrics public, grades and regrades through GitHub) is coming this week; students are invited to suggest questions. Over the winter break the instructor plans to run the automated grader on past, human-graded exams to test whether it is more consistent than human graders
• Decision trees, finished (Lecture 12, slides 3 to 10)
  • Recap: a tree is a sequence of rules on feature values; features can be continuous or categorical, no scaling is needed, and the result is interpretable. The redrawn slide shows the same tree over flows (packets per second, then destination port or duration) and the rectangles it carves out of a two-feature space: attack or benign is predicted by the majority class of the training points in the leaf’s rectangle, and the misclassifications are the points sitting in the wrong rectangle
  • Questions from after Friday’s class: the picture works in any number of dimensions (splits become hyperplanes), and a single split cannot be a diagonal line, because each split tests one feature against one threshold, so the boundaries are axis-aligned
  • Final flag: given the tree, draw the partition of the feature space; or given the partition, draw the tree; or given a set of labeled points, divide the space and give the tree. (The instructor calls this the “second midterm” question.)
  • Regression vs. classification trees: scikit-learn has a classifier and a regressor. In a classification tree the leaf predicts the majority class; in a regression tree the points in a region carry continuous values and the leaf predicts their mean (or median). The textbook example is predicting a baseball player’s salary from years played and hits
  • Choosing a split has two parts: which feature, and which threshold. For regression, try each threshold, predict each side by its mean, and keep the cut with the smallest residual sum of squares. For classification, the two common criteria are Gini impurity and entropy. Few students had seen entropy, so the instructor derived it on the board: the entropy of a region is minus the sum over classes of the class proportion times its logarithm, so a region split evenly between two classes has entropy 1 bit and a pure region has entropy 0 (correction to the board: the instructor hesitated over the leading minus sign and the arithmetic; the formula does have the minus sign out front, and the even two-class case works out to 1). Intuition is what matters: training picks the split whose regions are as pure as possible, or, for regression, whose mean squared error is smallest. The slide on the weighted Gini computation will be redone from scratch
  • Final flag: given labeled points in a region and a proposed split (for example, flow duration at 10 seconds), compute the entropy on each side; a simple computation, and the most detailed arithmetic that would be asked
  • Overfitting: every split is one level deeper, and a tree can keep splitting until every training point has its own rectangle, giving perfect training accuracy, zero entropy, and a model that has memorized the training set. Depth is a hyperparameter found by validation: watch where validation error turns up on the bias-variance curve (the one on the midterm) and prune there
  • Two ways to control depth. Early stopping (stop when a split no longer reduces impurity, or set a maximum depth) is a straw man: impurity is not monotone in depth, so a split that looks useless can be followed by one that helps a lot, and the search stalls in a local minimum; the instructor will add a worked example. The preferred approach grows the full tree and prunes back, trading leaf error against a penalty on the number of leaves (the trade-off is between purity per region and the number of regions, not the number of points in a region). The equation is in the slides; the math of cost-complexity pruning will not be asked
  • CART (Classification and Regression Trees) is greedy: at each node it tries every feature and every threshold, takes the locally best split, recurses, and never backtracks. That bounds the search, and it also makes the tree brittle: a small change in the training set changes what is locally best at some node, and everything below it comes out different
  • A student asked why not train several trees on different training sets and combine them; that is essentially the idea of ensembles, and the segue to the rest of the lecture
  • Final flag: the advantages and limitations of decision trees (slide 10), including why a tree is brittle, with an example, or, given a training set and its first split, a small change to the training set that would change that first cut. Study guides and practice questions on this are promised
• Ensembles: bagging and random forests
  • Two families of tree ensembles: bootstrap aggregation (bagging) and boosting. The instructor’s practical advice: faced with unfamiliar structured data such as network features, run a random forest first. It is fast, needs no feature scaling or class rebalancing, handles mixed feature types, gives feature importance, and on tabular data it often matches or beats deep learning because the features already carry domain knowledge
  • Bagging: a single tree has high variance, so train many trees, each on slightly different data. With only one training set of size n, draw n samples with replacement (the jar-of-balls sampling from discrete math); each bootstrap sample contains about two thirds of the unique points. Train a tree on each sample
  • The extra trick that makes a random forest: if one feature dominates, every tree splits on it first and the trees are correlated. So at each node of each tree, consider only a random subset of the features and pick the best split among those. That decorrelates the trees. Prediction is a majority vote for classification or the average for regression (a student asked whether thresholds are combined; no, each tree predicts and the predictions are combined)
  • Final flag: the two things that turn a set of decision trees into a random forest (bootstrap aggregation, and a random subset of features at each split); or, what goes wrong with bagging alone and how a random forest fixes it. The instructor will put together an example where the correlation problem actually appears
  • Out-of-bag error estimation (slide 13): skip the details; it will not be asked. The practical point: a random forest reports feature importance, the average reduction in impurity or residual error each time a feature is used across all the trees. How it is computed will not be asked, but it is very useful: unlike a single tree’s, a forest’s feature importance is stable, and even when the final model is something else (a deep network, say), a quick random forest tells you which features matter for the task
  • Why it works: a training point that would move a split in one tree may be absent from a given bootstrap sample or not among that node’s candidate features, so no single point controls the aggregate. The trees are independent and train in parallel
• Boosting (two slides)
  • Start with many short, stubby trees (often one level deep, “weak learners”), each with plenty of error; train the next tree to predict the residual of the current model and subtract it; repeat. AdaBoost example (loan approval): a one-level tree on credit score misclassifies four points, so the next stump is trained on those and splits on income; successive weak learners correct the previous ones’ mistakes. Parameters: how many trees, how shallow. Unlike a random forest, boosting trains sequentially on residuals and does not parallelize the same way. A networking example for this slide is owed
  • A 2026 paper applies gradient boosting to intrusion detection on IoT traffic, close to what the hands-on does
  • Summary: trees partition the feature space and are interpretable but brittle; depth is tuned by overgrowing and pruning; random forests train trees in parallel on bootstrap samples with random feature subsets; boosting trains stubby trees in sequence on the residuals
• Hands-On 12: IoT activity inference with a random forest (notebook)
  • The class plays network eavesdropper: from a Nest camera’s encrypted traffic, build a time series of send rates, turn it into labeled points, and train a random forest to tell livestreaming from a motion event. The setup is from the instructor’s group’s papers. The classifier is simple because the device is idle most of the time and bursts when something happens; the same reasoning applies to a smart speaker that goes quiet until spoken to. About 20 minutes were given; the instructor planned to show the solution before the end of class and will poll the class on whether to give hands-ons more time. The work period and report-back were not captured
• Not covered: the out-of-bag error slide in detail; the weighted Gini worked example (slide to be redone); the interpretability and SHAP slide
• Next: Friday November 13 opens with ten minutes on the new worked examples in the trees deck (choosing a split by Gini and by entropy, why early stopping can quit too soon) and ten minutes to finish Hands-On 12, then deep learning, and why tree ensembles compete with it on tabular data. Assignment 3 (anomaly detection) is out, due Friday November 20, and uses the same smart-home traces without labels

Meeting 14 (Fri Nov 13)

• Trees, the worked examples promised on Monday (the four new slides in the trees deck, also posted to Slack)
  • A regression tree: divide the feature space (packet loss, round-trip time) the same way a decision tree does, and predict a continuous target (throughput) as the average of the training points in each region.
  • Choosing a split by Gini impurity, on ten points: compute the impurity before the split, then the weighted impurity of each candidate split. The split on packets per second at 50 leaves mostly attack on one side and mostly benign on the other and scores better than the candidate that leaves three-and-two on both sides. The impurity measure puts a number on what the picture already shows; entropy does the same job (the entropy slide is in the deck but was not walked through).
  • Why stopping as soon as impurity stops improving can quit too soon: on the example, no single split on either feature reduces the impurity (both sides stay roughly half and half), but one more level gives pure regions. Grow deep, then decide whether to prune back.
  • The slide on why bagging alone is not enough was not covered; see Monday’s notes.
• Logistics
  • Project proposal feedback is in each team’s Gradebook issue, with a general note on Slack. Of the thirty projects, about seventeen work against a leaderboard (nPrint, streaming video, early application identification), four address limitations of the netML library, and nine are more open-ended research, many of them individual.
  • Nobody needs to resubmit. The suggested deliverables are suggestions, and not taking one will not cost anything. Tweaks can go in the Gradebook issue or come to office hours (Tuesday evenings, by sign-up).
  • Unless someone objects, the list of project titles and team members will be posted to the channel so that teams working on related things can find each other. Everyone should do a lit review, including checking whether the netML alternatives and the nPrint benchmarks are still current.
  • The course cannot provide compute; plan on a laptop plus Colab, and ask if a project really needs more.
  • A short survey on how the course management is working will go out next week. The write-up of how the course is run behind the scenes is linked in Slack.
• Hands-On 12 was not walked through; the solution is in every private repo.
• Lecture 13: deep learning
  • Not new: the first neural nets date to the 1970s. What changed is compute and data. Neural nets need a lot of both.
  • Where it sits: machine learning with hand-designed features, then representation learning, where the algorithm finds the representation, then deep learning, where the learned representations are highly complex (the figure from Goodfellow’s Deep Learning book, recommended reading). The pitch is that you throw the raw data at the model instead of engineering features, as in image recognition (ImageNet, the husky-versus-wolf example).
  • Why consider it: linear models need a linear decision boundary, and the workarounds the course has covered (polynomial basis expansion, ensembles of trees) have limits. Raw network data is high-dimensional and unstructured, and choosing features takes domain expertise. nPrint grew out of trying to beat nmap, a rule-based system for fingerprinting an operating system from how a host answers a scan, with a neural net on raw packet bits. (Do not run nmap on the campus network; it looks like the first step of an attack.)
  • A student asked whether you can feed hand-engineered features to a neural net instead of raw data. Yes, and it can save a lot of training; the net may still capture nonlinear interactions a linear model would miss. The hands-on does exactly this.
  • The neuron: inputs, one learned weight per input, a bias, a weighted sum, and an activation function that supplies the nonlinearity. A single neuron with a sigmoid activation is exactly logistic regression, which is why the course taught it. ReLU is the default activation in the hidden layers now; the output activation is chosen by what works.
  • The network: stack neurons into layers, each layer fully connected to the next (dropping an edge is setting its weight to zero). Depth and width are hyperparameters; three layers is common, VGG-16 has sixteen. Simpler networks often work fine and train faster.
  • Training, one epoch: a forward pass turns inputs into predictions; the loss (squared error for regression, cross-entropy for classification) compares them with the labels; the backward pass (backpropagation) adjusts every weight to reduce the loss. For each weight the derivative of the loss with respect to that weight says whether to move it up or down; the chain rule gets the derivatives for all the weights at once (the math was skipped). The learning rate sets the step size: too large oscillates or overshoots the minimum, and what you find is a minimum, not necessarily the global one.
  • A student asked when to use leaky ReLU rather than plain ReLU. Answered after class on Slack, and now on its own slide: leaky ReLU keeps a small slope for negative inputs so a neuron whose input is always negative can still learn (the dying-ReLU problem), but in practice it is never worse and only occasionally slightly better than ReLU, dead neurons are rare with good initialization and Adam, and when practitioners did move off ReLU the fix that won was a smooth curve (GELU, SiLU), not a second straight line. Hands-On 13 has an optional to-do to try it.
  • Final flag: there will not be a question on the final about leaky ReLU or choosing among activation functions. That was a follow-up question, and optimizing activation functions is outside the scope of the planned material; it is enrichment.
  • Deep learning in networking research: papers use elaborate hybrids (CNNs, LSTMs, self-attention) for intrusion detection and the like. The instructor has not yet seen a classification or regression problem on this kind of data where deep learning is obviously the answer. nPrint plus a neural net does beat rule-based nmap, but feature importance over the learned model is a heat map over a few packet fields, and engineering those fields and handing them to a random forest has worked about as well. Healthy skepticism is warranted.
  • When it is worth it: lots of data (it is data-hungry and expensive to train), raw inputs (a random forest on a packet bit field will not do anything useful), or a problem a linear model cannot separate (XOR). When it is not: small data (naive Bayes and even linear regression do better; trees are brittle too), interpretability (trees and forests give feature importances; a net is harder to read, though that is changing), and constrained energy or memory. Start with a random forest; if that fails, a multi-layer perceptron; a CNN if a packet is a picture (the nPrint paper’s original title was “packets as pictures”); sequence models for temporal context (not covered this term).
  • A student made the case against deep learning for this data: the class knows the domain and can build the representation itself, and a net throws that domain knowledge away unless you want an embedding. Agreed, with the exception that unsupervised settings (next week’s autoencoders) are a good fit, because there the data is a mess and the goal is to find structure.
  • The spoiler for the hands-on: the IoT DDoS paper (Doshi, Apthorpe, Feamster), from the deep learning and security workshop and an undergraduate project, compared k-nearest neighbors, a linear SVM, a decision tree, a random forest and a neural net on stateless features of consumer IoT traffic. All of them work well, k-NN about the best. Simple stateless features plus ensembles of trees are usually enough.
  • Not covered: the convolution and CNN slides, the RNN slide, the nPrint slides (already done in an earlier meeting), and the training-challenges material (vanishing gradients, why sigmoid and tanh fell out of use in hidden layers, batch normalization, residual connections). Those slides are in the deck for reference and were not discussed in class.
  • Final flag: vanishing gradients and the comparison of activation functions were not taught and will not be on the final. The deck has since been updated with cleaner figures and extra slides (backpropagation pictured and by the numbers, ReLU vs. leaky ReLU, GELU and SiLU); the backpropagation slides restate what was said in class, the activation-function slides are enrichment.
• Hands-On 13: DDoS detection for consumer IoT
  • Normal and attack (flooding) traffic from IoT devices, already in CSV form in the course repo. Steps 2 and 3 (features and labels) just need to run; step 4, training and testing the neural net, is the point. Set the epochs to about 10 if training is slow (the paper used 100). The last step compares against k-NN and the rest.
  • The result: every model performs about the same. Two reasons given: the notebook hand-engineers the features, which is the part deep learning is supposed to be good at, so there is nothing left for the net to add; and network traffic is structured enough that engineered features separate the classes with linear boundaries.
  • A student noted the notebook uses an older deep-learning library (Keras); the instructor may add a PyTorch version. The notebook is otherwise kept as in the paper.
  • Rohan Doshi was an undergraduate when this was done; class projects can turn into papers.
• Where the course is headed
  • Lecture 14 (nPrint) is skipped because it was already covered, so the schedule is back on track. Next week is unsupervised learning: dimensionality reduction (PCA) and clustering, with autoencoders folded in. The instructor said he may compress unsupervised learning to give generative AI more than one lecture in the remaining three meetings; the exact plan for the last three meetings will be posted on Slack.
  • The final is on the last Friday. It is really a second midterm: it covers the second half of the course, the models and their applications, not the first four weeks (the midterm already covered those). “Cumulative only in the sense that life is cumulative” was a joke, as in week four.