Practice Activity, Remote Notebook: Analyze in the Cloud

GSB 5544 · Computing and Machine Learning for Business Analytics

This activity has two notebooks. This is the first.

Notebook Where you open it Steps
03_practice_activity_remote.ipynb (this one) JupyterLab, inside your SageMaker space 1 to 6, and the remote half of Part B
03_practice_activity_local.ipynb Positron, on your laptop 7 and 8, and the local half of Part B

The workflow

Step What you do Where you are Which computer runs code Where the data is
1 Sign in to your own AWS account Browser, AWS website none S3 (NOAA’s public bucket)
2 Open and start your SageMaker space Browser, SageMaker Studio none yet S3
3 Read NOAA data from S3 Remote notebook (JupyterLab tab) SageMaker S3 → SageMaker’s RAM
4 Filter, transform, aggregate Remote notebook SageMaker SageMaker’s RAM
5 Save a small summary CSV Remote notebook SageMaker SageMaker’s disk
6 Download the summary Browser, JupyterLab file browser none (a file copy) SageMaker’s disk → your laptop’s disk
7 Open it and make the chart Local notebook (Positron) your laptop laptop’s disk → laptop’s RAM
8 Save your work and stop the space Positron, then browser none your laptop’s disk; SageMaker’s disk

Part A (steps 1 to 8) is the real workflow: a remote machine reads ten years of daily high temperatures, about 150 MB of source files and tens of millions of rows, and reduces them to a file small enough to email.

Part B is a separate, controlled timing test: the same small job run once here and once on your laptop, so the two machines can be compared fairly.

Every code cell starts with a # RUNS ON: comment. It names the intended computer. It does not move the code there. The setup cell checks where this kernel really is. Cells with ____ have a blank to fill before you run them. If a cell stops with NameError: name '____' is not defined, you ran it before filling in its blank; fill the blank and run it again.

By the end you should be able to answer five questions:

  1. Where is the original data stored?
  2. Which computer performed the analysis?
  3. What did I download?
  4. Which computer created the visualization?
  5. How did the two machines compare?

Step 1. Sign in to your own AWS account

You are in: your browser. Code runs: nowhere. Data is: in S3.

  1. Go to console.aws.amazon.com and sign in as Root user with the email and password of the account you created in the reading, section 5, part A.
  2. On Console Home, read Credits remaining in the Cost and usage panel and write it down.

Step 2. Open and start your SageMaker space

You are in: your browser. Code runs: nowhere yet. Data is: in S3.

  1. In the search bar type SageMaker AI and open it. Check the address bar contains us-east-1; if not, choose US East (N. Virginia) from the region menu at the top right.
  2. In the left menu choose SageMaker Studio, then click Open Studio.
  3. In Studio click JupyterLab, then click gsb5544 in the spaces table.
  4. Click Run space and wait until the status reads Running. The meter is now on.
  5. Click Open JupyterLab. It opens in a new browser tab. This tab is a window onto the remote machine.
  6. In JupyterLab’s file browser (left side), click the upload arrow and upload this notebook, 03_practice_activity_remote.ipynb. Double-click it to open it there.

If you are reading this in Positron, stop: you are in the wrong place for steps 3 to 5. Continue only in the JupyterLab tab.

Run the setup cell. Its last two lines tell you which computer this is.

# RUNS ON: SageMaker (remote)
%pip install -q s3fs pyarrow psutil      # harmless if already present

import os
import json
import time
import socket
import psutil
import boto3
import pandas as pd
from botocore import UNSIGNED
from botocore.config import Config

BUCKET = "noaa-ghcn-pds"         # NOAA's public bucket
REGION = "us-east-1"             # the region the bucket is in; your SageMaker space should be in it too

# unsigned = anonymous: this client attaches no credentials, because the bucket is public
s3 = boto3.client("s3", region_name=REGION, config=Config(signature_version=UNSIGNED))


def human(n, base=1000):
    """Turn a byte count into a readable string. base=1000 gives KB/MB/GB; base=1024 gives KiB/MiB/GiB."""
    units = ["B", "KB", "MB", "GB", "TB", "PB"] if base == 1000 else ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]
    for unit in units:
        if n < base:
            return f"{n:,.1f} {unit}"
        n /= base
    return f"{n:,.1f} (huge)"


def where_am_i():
    """Report which computer this kernel is really on. A RUNS ON comment cannot do that."""
    on_sagemaker = (os.path.exists("/opt/ml/metadata/resource-metadata.json")
                    or os.path.expanduser("~") == "/home/sagemaker-user")
    place = "SageMaker (remote)" if on_sagemaker else "not SageMaker (probably your laptop)"
    print("Computer name   :", socket.gethostname())
    print("This looks like :", place)
    return place


def machine_report(work_dir="."):
    """Measure the computer running this cell. Disk numbers are for the filesystem that holds work_dir."""
    vm = psutil.virtual_memory()
    du = psutil.disk_usage(work_dir)
    return {
        "computer name":                    socket.gethostname(),
        "physical CPU cores":               psutil.cpu_count(logical=False),   # may be None on some virtual machines
        "logical CPUs (vCPUs in the cloud)": psutil.cpu_count(logical=True),
        "total RAM (GB)":                   round(vm.total / 1e9, 1),
        "available RAM right now (GB)":     round(vm.available / 1e9, 1),
        "work disk, total (GB)":            round(du.total / 1e9, 1),
        "work disk, free (GB)":             round(du.free / 1e9, 1),
    }


def slice_prefix(year, element):
    """The key prefix of one year-and-element slice of the Parquet copy of GHCN-Daily."""
    return f"parquet/by_year/YEAR={year}/ELEMENT={element}/"


def prefix_size(prefix):
    """Count the objects under a prefix and add up their sizes, without reading any of them."""
    objects, total = 0, 0
    for page in s3.get_paginator("list_objects_v2").paginate(Bucket=BUCKET, Prefix=prefix):
        for obj in page.get("Contents", []):
            objects += 1
            total += obj["Size"]
    return objects, total


if where_am_i() != "SageMaker (remote)":
    print()
    print("NOTE: this notebook is written to run in your SageMaker space, and this kernel is somewhere else.")
    print("      The RUNS ON comments describe the plan. They do not move the code. Open this file in JupyterLab on SageMaker.")

Record the remote machine’s resources. The local notebook will put your laptop’s numbers beside these.

# RUNS ON: SageMaker (remote)
REMOTE_MACHINE = machine_report()
REMOTE_MACHINE

Part A · The workflow

Step 3. Read NOAA data from S3 into the remote machine

You are in: the remote notebook. Code runs: on SageMaker. Data moves: S3 → SageMaker’s RAM.

The data. NOAA’s GHCN-Daily archive, Parquet copy, daily maximum temperature (TMAX), one slice per year, 2015 through 2024.

The subset. Stations whose id begins USC0004: the California stations of the U.S. Cooperative Observer network. This is a few hundred volunteer-run stations. It is not every weather station in California, the stations are not evenly spread, and an unweighted average over them is not a statewide temperature estimate. It is a consistent, well-defined subset that is the right size for practice.

First, look before you read. Fill in the helper that measures a prefix.

# RUNS ON: SageMaker (remote)
YEARS = list(range(2015, 2025))

SOURCE_BYTES = 0
for year in YEARS:
    objects, size = prefix_size(slice_prefix(year, "TMAX"))
    SOURCE_BYTES += size
    print(year, f"{objects:>3} objects", human(size))

print()
print("Total source size in S3:", human(SOURCE_BYTES))
print("Available RAM here     :", REMOTE_MACHINE["available RAM right now (GB)"], "GB")

Ten slices together are much larger in RAM than on disk, and you do not know by how much until you load one. So the plan is: read one year, summarize it, keep only the summary, then read the next. Only one year is ever in RAM.

Step 4. Filter, transform, and aggregate, on the remote machine

You are in: the remote notebook. Code runs: on SageMaker. Data is: in SageMaker’s RAM.

For each year the function below must:

  1. keep the California cooperative stations (ID starts with USC0004),
  2. drop values that failed NOAA’s quality checks (Q_FLAG is not empty). A dropped or absent value is missing, never zero,
  3. convert DATE to a date and DATA_VALUE from tenths of a degree to degrees Celsius,
  4. return one row per month: how many stations reported, how many station-days, and the mean daily high.

Fill in the four blanks.

# RUNS ON: SageMaker (remote)
def summarize_year(raw, year):
    keep = raw[raw["ID"].str.startswith("USC0004") & raw["Q_FLAG"].isna()].copy()
    keep["date"]   = pd.to_datetime(keep["DATE"].astype(str), format="%Y%m%d")
    keep["tmax_c"] = keep["DATA_VALUE"] / 10
    keep["month"]  = keep["date"].dt.month

    out = (keep.groupby("month")
               .agg(stations=("ID", "nunique"),
                    station_days=("tmax_c", "size"),
                    mean_tmax_c=("tmax_c", "mean"))
               .round({"mean_tmax_c": 2})
               .reset_index())
    out.insert(0, "year", year)
    return out

Now run it for all ten years. This is the heavy part, and the remote machine does all of it. Expect it to take from several seconds to a minute or two; the cell prints one line per year so you can see it working.

# RUNS ON: SageMaker (remote)
pieces = []
ROWS_READ = 0
LARGEST_DF_BYTES = 0

t0 = time.perf_counter()
for year in YEARS:
    raw = pd.read_parquet(
        f"s3://{BUCKET}/{slice_prefix(year, 'TMAX')}",
        columns=["ID", "DATE", "DATA_VALUE", "Q_FLAG"],
        storage_options={"anon": True},
    )
    ROWS_READ += len(raw)
    LARGEST_DF_BYTES = max(LARGEST_DF_BYTES, int(raw.memory_usage(deep=True).sum()))
    pieces.append(summarize_year(raw, year))
    print(f"{year}: read {len(raw):,} rows")
    del raw                                           # free this year before reading the next
WORKFLOW_SECONDS = time.perf_counter() - t0

summary = pd.concat(pieces, ignore_index=True)
print()
print(f"Read {ROWS_READ:,} rows in {WORKFLOW_SECONDS:.0f} s. Summary has {len(summary)} rows.")
summary.head()

Step 5. Save a small summary CSV on the remote machine

You are in: the remote notebook. Code runs: on SageMaker. Data moves: SageMaker’s RAM → SageMaker’s disk.

The file is written next to this notebook, on the space’s own disk. It is not in S3. Fill in the method that writes a CSV.

# RUNS ON: SageMaker (remote)
SUMMARY_FILE = "ca_coop_tmax_monthly_2015_2024.csv"
summary.to_csv(SUMMARY_FILE, index=False)
SUMMARY_BYTES = os.path.getsize(SUMMARY_FILE)

print("Source files in S3      :", human(SOURCE_BYTES))
print("Largest one-year table  :", human(LARGEST_DF_BYTES), "in this machine's RAM")
print("Exported summary CSV    :", human(SUMMARY_BYTES), "on this machine's disk")
print("Free on this work disk  :", machine_report()["work disk, free (GB)"], "GB")
print(f"The summary is about 1/{SOURCE_BYTES // SUMMARY_BYTES:,} of the source size.")

Part B · A fair timing test, remote half

Part A cannot be compared with your laptop fairly, because you are not going to run it there. This part can. It is a deliberately small job: one year (2024), the same four columns, the same filter, the same transformation, the same aggregation, defined once in timing_job() below. The identical function is in the local notebook. Do not edit it in either place.

What it measures, and what it does not:

  • Read seconds covers fetching the bytes from S3 over the network and decoding the Parquet files into a table. On your laptop the network is your home or campus internet; here it is AWS’s internal network.
  • Process seconds covers the filter, the conversions, and the group-by. No network is involved.
  • A second run is often faster than the first because parts of the software and the connection are already warm. That is why you run it twice and keep both.
  • These are measurements from one moment on two particular machines. Nothing here promises that the cloud is faster.
# RUNS ON: SageMaker (remote)
TIMING_YEAR = 2024
TIMING_COLUMNS = ["ID", "DATE", "DATA_VALUE", "Q_FLAG"]


def timing_job():
    """Read one slice from S3, then summarize it. Returns the summary and the two measured times."""
    t0 = time.perf_counter()
    raw = pd.read_parquet(
        f"s3://{BUCKET}/parquet/by_year/YEAR={TIMING_YEAR}/ELEMENT=TMAX/",
        columns=TIMING_COLUMNS,
        storage_options={"anon": True},
    )
    t1 = time.perf_counter()                                    # reading ends here

    keep = raw[raw["ID"].str.startswith("USC0004") & raw["Q_FLAG"].isna()]
    daily = pd.DataFrame({
        "date":   pd.to_datetime(keep["DATE"].astype(str), format="%Y%m%d"),
        "tmax_c": keep["DATA_VALUE"] / 10,
    })
    summary = (daily.groupby("date")["tmax_c"]
                    .agg(stations="count", mean_tmax_c="mean")
                    .round({"mean_tmax_c": 2})
                    .reset_index())
    t2 = time.perf_counter()                                    # processing ends here

    times = {
        "read seconds":    round(t1 - t0, 2),
        "process seconds": round(t2 - t1, 2),
        "rows read":       len(raw),
        "summary rows":    len(summary),
    }
    return summary, times
# RUNS ON: SageMaker (remote)
REMOTE_TIMING = []
for run in (1, 2):
    timing_summary, times = timing_job()
    REMOTE_TIMING.append({"machine": "SageMaker (remote)", "run": run, **times})
    print(f"run {run}: {times}")

timing_summary.to_csv("timing_summary_remote.csv", index=False)

Now save every measurement from this notebook into one small file, so the local notebook can read them instead of you copying numbers by hand.

# RUNS ON: SageMaker (remote)
measurements = {
    "machine": REMOTE_MACHINE,
    "workflow": {
        "years":                   [YEARS[0], YEARS[-1]],
        "source bytes in S3":      SOURCE_BYTES,
        "rows read":               ROWS_READ,
        "largest DataFrame bytes": LARGEST_DF_BYTES,
        "summary CSV bytes":       SUMMARY_BYTES,
        "summary rows":            len(summary),
        "seconds":                 round(WORKFLOW_SECONDS, 1),
    },
    "timing": REMOTE_TIMING,
}
with open("remote_measurements.json", "w") as f:
    json.dump(measurements, f, indent=2)

for name in [SUMMARY_FILE, "timing_summary_remote.csv", "remote_measurements.json"]:
    print(f"{name:<40} {human(os.path.getsize(name))}")

Step 6. Download the summary to your laptop

You are in: the JupyterLab tab’s file browser. Code runs: nowhere; your browser copies files. Data moves: SageMaker’s disk → your laptop’s disk.

  1. In JupyterLab’s file browser, click the refresh arrow. Three new files appear next to this notebook.
  2. Right-click each one and choose Download:
    • ca_coop_tmax_monthly_2015_2024.csv
    • timing_summary_remote.csv
    • remote_measurements.json
  3. Choose File → Save Notebook, then right-click this notebook and Download it too. You submit it.
  4. On your laptop, move all four files from your Downloads folder into your week_7 folder, the same folder that will hold the local notebook.

Now switch computers

Leave this browser tab. Open Positron on your laptop, open your course folder, and open 03_practice_activity_local.ipynb from the week_7 folder. Steps 7 and 8 are there.

Nothing after this point needs the remote machine, so you may stop the space now (step 8 in the local notebook has the checklist). If you leave it running while you work in Positron, it keeps billing.