Practice Activity, Local Notebook: Visualize on Your Laptop

GSB 5544 · Computing and Machine Learning for Business Analytics

This is the second notebook of the practice activity. Open it in Positron, on your laptop. Do not upload it to SageMaker.

Before you start, your week_7 folder must contain this notebook and the three files you downloaded in step 6:

ca_coop_tmax_monthly_2015_2024.csv
timing_summary_remote.csv
remote_measurements.json

In Positron: File → Open Folder and choose your course folder, then open this notebook from week_7. If the notebook asks for a kernel (top right), choose the Python you use for this course.

Every code cell starts with # RUNS ON: laptop (Positron). That comment is a label. It does not move the code. The setup cell checks where this kernel really is. Cells with ____ have a blank to fill before you run them. If a cell stops with NameError: name '____' is not defined, you ran it before filling in its blank; fill the blank and run it again.

The workflow, with the part you have finished

Step	What you do	Where you are	Which computer runs code	Where the data is
1	Sign in to your own AWS account	Browser, AWS website	none	S3 (NOAA’s public bucket)
2	Open and start your SageMaker space	Browser, SageMaker Studio	none yet	S3
3	Read NOAA data from S3	Remote notebook (JupyterLab tab)	SageMaker	S3 → SageMaker’s RAM
4	Filter, transform, aggregate	Remote notebook	SageMaker	SageMaker’s RAM
5	Save a small summary CSV	Remote notebook	SageMaker	SageMaker’s disk
6	Download the summary	Browser, JupyterLab file browser	none (a file copy)	SageMaker’s disk → your laptop’s disk
7	Open it and make the chart	Local notebook (Positron)	your laptop	laptop’s disk → laptop’s RAM
8	Save your work and stop the space	Positron, then browser	none	your laptop’s disk; SageMaker’s disk

Steps 1 to 6 are done. You are at step 7.

# RUNS ON: laptop (Positron)
%pip install -q pandas pyarrow s3fs psutil matplotlib      # harmless if already present

import os
import json
import time
import socket
import psutil
import pandas as pd
import matplotlib.pyplot as plt

BUCKET = "noaa-ghcn-pds"


def human(n, base=1000):
    """Turn a byte count into a readable string. base=1000 gives KB/MB/GB; base=1024 gives KiB/MiB/GiB."""
    units = ["B", "KB", "MB", "GB", "TB", "PB"] if base == 1000 else ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]
    for unit in units:
        if n < base:
            return f"{n:,.1f} {unit}"
        n /= base
    return f"{n:,.1f} (huge)"


def where_am_i():
    """Report which computer this kernel is really on. A RUNS ON comment cannot do that."""
    on_sagemaker = (os.path.exists("/opt/ml/metadata/resource-metadata.json")
                    or os.path.expanduser("~") == "/home/sagemaker-user")
    place = "SageMaker (remote)" if on_sagemaker else "not SageMaker (probably your laptop)"
    print("Computer name   :", socket.gethostname())
    print("This looks like :", place)
    return place


def machine_report(work_dir="."):
    """Measure the computer running this cell. Disk numbers are for the filesystem that holds work_dir."""
    vm = psutil.virtual_memory()
    du = psutil.disk_usage(work_dir)
    return {
        "computer name":                    socket.gethostname(),
        "physical CPU cores":               psutil.cpu_count(logical=False),   # may be None on some virtual machines
        "logical CPUs (vCPUs in the cloud)": psutil.cpu_count(logical=True),
        "total RAM (GB)":                   round(vm.total / 1e9, 1),
        "available RAM right now (GB)":     round(vm.available / 1e9, 1),
        "work disk, total (GB)":            round(du.total / 1e9, 1),
        "work disk, free (GB)":             round(du.free / 1e9, 1),
    }


if where_am_i() == "SageMaker (remote)":
    print()
    print("NOTE: this notebook is written to run on your laptop, in Positron, and this kernel is on SageMaker.")
    print("      Download this file and open it in Positron instead.")

Check that the three downloaded files are where this notebook can see them.

# RUNS ON: laptop (Positron)
NEEDED = ["ca_coop_tmax_monthly_2015_2024.csv", "timing_summary_remote.csv", "remote_measurements.json"]
missing = [name for name in NEEDED if not os.path.exists(name)]

print("This notebook is running in:", os.getcwd())
if missing:
    print("MISSING:", missing)
    print("Move them from your Downloads folder into the folder above, then run this cell again.")
else:
    print("All three downloaded files are here.")
    with open("remote_measurements.json") as f:
        REMOTE = json.load(f)

Part A · The workflow, continued

Step 7. Open the summary in Positron and make the chart

You are in: the local notebook, in Positron. Code runs: on your laptop. Data moves: laptop’s disk → laptop’s RAM.

The file you are about to read was computed on another computer from data you never downloaded. Fill in the function that reads a CSV.

# RUNS ON: laptop (Positron)
small = pd.____("ca_coop_tmax_monthly_2015_2024.csv")
DOWNLOADED_BYTES = os.path.getsize("ca_coop_tmax_monthly_2015_2024.csv")

print(small.shape, "read from a file of", human(DOWNLOADED_BYTES))
small.head()

Draw one line per year: month on the x-axis, mean daily high on the y-axis. Label the chart for what it is: the mean of daily highs across reporting California cooperative-network stations, not a statewide temperature.

Fill in the two column names.

# RUNS ON: laptop (Positron)
fig, ax = plt.subplots(figsize=(9, 4.5))
for year, rows in small.groupby("year"):
    ax.plot(rows[____], rows[____], marker="o", markersize=3, linewidth=1, label=str(year))

ax.set_xticks(range(1, 13))
ax.set_xlabel("Month")
ax.set_ylabel("Mean daily high (°C)")
ax.set_title("California cooperative-network stations: mean daily high by month, 2015–2024")
ax.legend(ncol=2, fontsize=8, loc="upper left", bbox_to_anchor=(1.01, 1))
fig.savefig("ca_coop_tmax_monthly.png", dpi=150, bbox_inches="tight")
plt.show()

Two machines, side by side

machine_report() measures the computer running this cell: your laptop. The remote machine’s numbers come from the file you downloaded. Fill in the function call.

# RUNS ON: laptop (Positron)
LOCAL_MACHINE = ____()

resources = pd.DataFrame({"laptop": LOCAL_MACHINE, "SageMaker (remote)": REMOTE["machine"]})
resources

Read the table carefully. “Physical CPU cores” and “logical CPUs” are different things, and a cloud vCPU is a logical CPU. The work-disk rows describe each computer’s own disk; neither is S3.

Four sizes of the same data

# RUNS ON: laptop (Positron)
w = REMOTE["workflow"]
sizes = pd.DataFrame(
    {
        "where": ["S3 (NOAA's bucket)", "SageMaker's RAM", "SageMaker's disk", "your laptop's disk"],
        "size":  [human(w["source bytes in S3"]), human(w["largest DataFrame bytes"]),
                  human(w["summary CSV bytes"]), human(DOWNLOADED_BYTES)],
    },
    index=["source files, ten years", "largest one-year DataFrame", "exported summary CSV", "downloaded summary CSV"],
)
print(f'Rows read remotely: {w["rows read"]:,}   Rows in the summary: {w["summary rows"]}')
sizes

Part B · A fair timing test, local half

This is the same timing_job() as in the remote notebook, character for character: one year, the same columns, the same filter, the same transformation, the same aggregation. Do not edit it.

For this test only, your laptop reads the 2024 slice straight from S3. That is deliberate: both machines must do the same job for the comparison to mean anything. It is a small read. It is not the workflow; in the workflow the remote machine did the reading.

# RUNS ON: laptop (Positron)
TIMING_YEAR = 2024
TIMING_COLUMNS = ["ID", "DATE", "DATA_VALUE", "Q_FLAG"]


def timing_job():
    """Read one slice from S3, then summarize it. Returns the summary and the two measured times."""
    t0 = time.perf_counter()
    raw = pd.read_parquet(
        f"s3://{BUCKET}/parquet/by_year/YEAR={TIMING_YEAR}/ELEMENT=TMAX/",
        columns=TIMING_COLUMNS,
        storage_options={"anon": True},
    )
    t1 = time.perf_counter()                                    # reading ends here

    keep = raw[raw["ID"].str.startswith("USC0004") & raw["Q_FLAG"].isna()]
    daily = pd.DataFrame({
        "date":   pd.to_datetime(keep["DATE"].astype(str), format="%Y%m%d"),
        "tmax_c": keep["DATA_VALUE"] / 10,
    })
    summary = (daily.groupby("date")["tmax_c"]
                    .agg(stations="count", mean_tmax_c="mean")
                    .round({"mean_tmax_c": 2})
                    .reset_index())
    t2 = time.perf_counter()                                    # processing ends here

    times = {
        "read seconds":    round(t1 - t0, 2),
        "process seconds": round(t2 - t1, 2),
        "rows read":       len(raw),
        "summary rows":    len(summary),
    }
    return summary, times

# RUNS ON: laptop (Positron)
LOCAL_TIMING = []
for run in (1, 2):
    local_summary, times = timing_job()
    LOCAL_TIMING.append({"machine": "laptop", "run": run, **times})
    print(f"run {run}: {times}")

First, check that both machines computed the same thing. A timing comparison between two different answers is meaningless. Fill in the file that holds the remote machine’s result.

# RUNS ON: laptop (Positron)
remote_summary = pd.read_csv(____, parse_dates=["date"])

same_shape  = local_summary.shape == remote_summary.shape
same_days   = bool((local_summary["date"].values == remote_summary["date"].values).all()) if same_shape else False
same_counts = bool((local_summary["stations"].values == remote_summary["stations"].values).all()) if same_shape else False
max_gap     = float((local_summary["mean_tmax_c"] - remote_summary["mean_tmax_c"]).abs().max()) if same_shape else float("nan")

print("Same shape          :", same_shape, local_summary.shape, remote_summary.shape)
print("Same dates          :", same_days)
print("Same station counts :", same_counts)
print("Largest difference in mean_tmax_c:", max_gap, "°C")
print("EQUIVALENT" if (same_shape and same_days and same_counts and max_gap < 0.011) else "NOT EQUIVALENT: do not interpret the timings until you know why")

If the result is not equivalent, the usual cause is that NOAA updated the archive between your two runs, or that one notebook’s timing_job() was edited. Rerun both halves close together.

Then compare the measured times.

# RUNS ON: laptop (Positron)
timing = pd.DataFrame(LOCAL_TIMING + REMOTE["timing"]).set_index(["machine", "run"])
timing["total seconds"] = (timing["read seconds"] + timing["process seconds"]).round(2)
timing

Interpret your own results

Answer from the numbers in your tables, not from what you expected. Two or three sentences each.

T1. On which machine was reading faster, and by how much? Reading includes the network trip from S3 and decoding the files. Which of those do you think explains the difference you saw, given where each computer is?

Your answer:

T2. On which machine was processing faster? Look at the resource table: does the difference in CPUs and RAM account for it?

Your answer:

T3. Compare run 1 and run 2 on each machine. What changed, and what does that tell you about trusting a single timing?

Your answer:

T4. For this small job, was the remote machine worth using for speed alone? What would have to be different about the job for your answer to change?

Your answer:

The five questions

Answer each in one or two sentences, with specifics from this activity.

Where is the original data stored?

Your answer:
Which computer performed the analysis in Part A?

Your answer:
What did I download, and how big was it compared with what the remote machine read?

Your answer:
Which computer created the visualization?

Your answer:
How did the two machines compare, in resources and in measured time?

Your answer:

Step 8. Save your work and stop the remote machine

You are in: Positron, then your browser. Code runs: nowhere.

Remote notebook: File → Save Notebook. Right-click it in the JupyterLab file browser and Download a copy.
Studio tab: on the gsb5544 space page click Stop space, confirm in the dialog, and wait until the status reads Stopped. Closing the browser tab does not stop it.
Console Home (open console.aws.amazon.com in a new tab): read Credits remaining and note it. The panel can lag by several hours.
Laptop: save your local notebook in Positron (Cmd+S, or Ctrl+S on Windows).

A stopped space keeps its disk, so the remote notebook and CSV files are there next time. Its RAM is cleared. You created nothing in S3, so there is nothing to delete there.

What to submit

Both halves, because the work happened in two places:

03_practice_activity_remote.ipynb, run in SageMaker, with its output showing.
03_practice_activity_local.ipynb (this notebook), run in Positron, with its output and your written answers.
ca_coop_tmax_monthly.png, the chart made on your laptop.
A screenshot of the gsb5544 space page showing status Stopped.

The computer names printed by the two setup cells should be different. If they are the same, one of the notebooks was run in the wrong place.