Lab, Remote Notebook: Weather as a Business Covariate

GSB 5544 · Computing and Machine Learning for Business Analytics

You manage inventory planning for a beverage distributor with two markets: San Luis Obispo, CA and Phoenix, AZ. Before anyone builds a demand model, the team needs each market’s recent weather history in a form small enough to share. The raw history sits in NOAA’s public S3 bucket, mixed in with every other weather station on Earth.

You will repeat the workflow from the practice activity, with less guidance:

Step	What you do	Where you are	Which computer runs code	Where the data is
1	Sign in to your own AWS account	Browser, AWS website	none	S3 (NOAA’s public bucket)
2	Open and start your SageMaker space	Browser, SageMaker Studio	none yet	S3
3	Read NOAA data from S3	Remote notebook (JupyterLab tab)	SageMaker	S3 → SageMaker’s RAM
4	Filter, transform, aggregate	Remote notebook	SageMaker	SageMaker’s RAM
5	Save a small summary CSV	Remote notebook	SageMaker	SageMaker’s disk
6	Download the summary	Browser, JupyterLab file browser	none (a file copy)	SageMaker’s disk → your laptop’s disk
7	Open it and make the chart	Local notebook (Positron)	your laptop	laptop’s disk → laptop’s RAM
8	Save your work and stop the space	Positron, then browser	none	your laptop’s disk; SageMaker’s disk

This lab has two notebooks. This one runs in JupyterLab inside your SageMaker space and covers steps 1 to 6. 04_lab_local.ipynb runs in Positron on your laptop and covers steps 7 and 8, the charts, and the business interpretation.

Market	Station id	Station name
San Luis Obispo	`USC00047851`	SAN LUIS OBISPO POLY, CA
Phoenix	`USW00023183`	PHOENIX AIRPORT, AZ

Rules for this lab

All reading from S3 and all summarizing happens here, on the remote machine. Downloading raw NOAA files to your laptop and analyzing them there does not satisfy the lab.
All charts are made locally, in Positron. Do not plot in this notebook.
A missing or quality-flagged observation is missing. Never count it as zero.
No additional AWS services. SageMaker and the public bucket are all you need.

Sign in, start your gsb5544 space, open JupyterLab, upload this notebook, and open it there (practice activity, steps 1 and 2). Then run the setup cell.

# RUNS ON: SageMaker (remote)
%pip install -q s3fs pyarrow psutil      # harmless if already present

import os
import json
import time
import socket
import psutil
import boto3
import pandas as pd
from botocore import UNSIGNED
from botocore.config import Config

BUCKET = "noaa-ghcn-pds"         # NOAA's public bucket
REGION = "us-east-1"             # the region the bucket is in; your SageMaker space should be in it too

# unsigned = anonymous: this client attaches no credentials, because the bucket is public
s3 = boto3.client("s3", region_name=REGION, config=Config(signature_version=UNSIGNED))


def human(n, base=1000):
    """Turn a byte count into a readable string. base=1000 gives KB/MB/GB; base=1024 gives KiB/MiB/GiB."""
    units = ["B", "KB", "MB", "GB", "TB", "PB"] if base == 1000 else ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]
    for unit in units:
        if n < base:
            return f"{n:,.1f} {unit}"
        n /= base
    return f"{n:,.1f} (huge)"


def where_am_i():
    """Report which computer this kernel is really on. A RUNS ON comment cannot do that."""
    on_sagemaker = (os.path.exists("/opt/ml/metadata/resource-metadata.json")
                    or os.path.expanduser("~") == "/home/sagemaker-user")
    place = "SageMaker (remote)" if on_sagemaker else "not SageMaker (probably your laptop)"
    print("Computer name   :", socket.gethostname())
    print("This looks like :", place)
    return place


def machine_report(work_dir="."):
    """Measure the computer running this cell. Disk numbers are for the filesystem that holds work_dir."""
    vm = psutil.virtual_memory()
    du = psutil.disk_usage(work_dir)
    return {
        "computer name":                    socket.gethostname(),
        "physical CPU cores":               psutil.cpu_count(logical=False),   # may be None on some virtual machines
        "logical CPUs (vCPUs in the cloud)": psutil.cpu_count(logical=True),
        "total RAM (GB)":                   round(vm.total / 1e9, 1),
        "available RAM right now (GB)":     round(vm.available / 1e9, 1),
        "work disk, total (GB)":            round(du.total / 1e9, 1),
        "work disk, free (GB)":             round(du.free / 1e9, 1),
    }


def slice_prefix(year, element):
    """The key prefix of one year-and-element slice of the Parquet copy of GHCN-Daily."""
    return f"parquet/by_year/YEAR={year}/ELEMENT={element}/"


def prefix_size(prefix):
    """Count the objects under a prefix and add up their sizes, without reading any of them."""
    objects, total = 0, 0
    for page in s3.get_paginator("list_objects_v2").paginate(Bucket=BUCKET, Prefix=prefix):
        for obj in page.get("Contents", []):
            objects += 1
            total += obj["Size"]
    return objects, total


if where_am_i() != "SageMaker (remote)":
    print()
    print("NOTE: this notebook is written to run in your SageMaker space, and this kernel is somewhere else.")
    print("      The RUNS ON comments describe the plan. They do not move the code. Open this file in JupyterLab on SageMaker.")

# RUNS ON: SageMaker (remote)
STATIONS = {"San Luis Obispo": "USC00047851", "Phoenix": "USW00023183"}
YEARS    = list(range(2015, 2025))
ELEMENTS = ["TMAX", "PRCP"]        # daily high (tenths of °C) and daily precipitation (tenths of mm)

L1. The machine you are renting

Store machine_report() for this machine as REMOTE_MACHINE and display it.

# RUNS ON: SageMaker (remote)
REMOTE_MACHINE = machine_report()
REMOTE_MACHINE

L2. Size up the job before reading anything

Use prefix_size() and slice_prefix() to find the size of every slice you need: both elements, all ten years. Print one line per slice and store the total in SOURCE_BYTES. Print the total in readable units next to this machine’s available RAM.

Then, in the markdown cell: can you tell from these numbers alone whether all of it would fit in RAM at once? What will you do instead?

# RUNS ON: SageMaker (remote)
SOURCE_BYTES = 0
for element in ELEMENTS:
    for year in YEARS:
        objects, size = prefix_size(slice_prefix(year, element))
        SOURCE_BYTES += size
        print(f"{element} {year}: {objects:>3} objects, {human(size)}")

print()
print("Total source size :", human(SOURCE_BYTES))
print("Available RAM here:", REMOTE_MACHINE["available RAM right now (GB)"], "GB")

Your answer:

L3. Extract both markets

Write load_market_weather(stations, years, elements). For every year and element it reads that one slice from S3, keeps only the stations you asked for, and returns a single tidy DataFrame wx with exactly these columns:

Column	Contents
`market`	the market name from `stations`
`station_id`	the station id
`date`	a real date
`element`	`"TMAX"` or `"PRCP"`
`value`	the measurement in real units: °C for `TMAX`, mm for `PRCP` (the file stores tenths)

Requirements:

Drop rows whose Q_FLAG is not empty before anything else.
Read one slice at a time, and let Parquet do the station filter during the read, so that only matching rows reach this machine’s RAM. This starter call shows how; ids is a list of station ids:

part = pd.read_parquet(
    f"s3://{BUCKET}/{slice_prefix(year, element)}",
    columns=["ID", "DATE", "DATA_VALUE", "Q_FLAG"],
    filters=[("ID", "in", ids)],
    storage_options={"anon": True},
)

Call your function, time it, and print how many rows wx has for each market and element. Store the elapsed seconds in EXTRACT_SECONDS and the DataFrame’s memory use in WX_BYTES.

# RUNS ON: SageMaker (remote)
def load_market_weather(stations, years, elements):
    id_to_market = {station_id: market for market, station_id in stations.items()}
    ids = list(id_to_market)
    frames = []
    for year in years:
        for element in elements:
            part = pd.read_parquet(
                f"s3://{BUCKET}/{slice_prefix(year, element)}",
                columns=["ID", "DATE", "DATA_VALUE", "Q_FLAG"],
                filters=[("ID", "in", ids)],
                storage_options={"anon": True},
            )
            part = part[part["Q_FLAG"].isna()]
            frames.append(pd.DataFrame({
                "market":     part["ID"].map(id_to_market),
                "station_id": part["ID"],
                "date":       pd.to_datetime(part["DATE"].astype(str), format="%Y%m%d"),
                "element":    element,
                "value":      part["DATA_VALUE"] / 10,
            }))
        print("finished", year)
    return pd.concat(frames, ignore_index=True)


t0 = time.perf_counter()
wx = load_market_weather(STATIONS, YEARS, ELEMENTS)
EXTRACT_SECONDS = time.perf_counter() - t0
WX_BYTES = int(wx.memory_usage(deep=True).sum())

print(f"{len(wx):,} rows in {EXTRACT_SECONDS:.0f} s, {human(WX_BYTES)} in RAM")
wx.groupby(["market", "element"]).size()

L4. Hot days per year

For each market and year, count the days with a daily high of 35 °C or higher (95 °F). Build hot_days with one row per market and year and these columns: market, year, days_reported, hot_days.

days_reported is the number of days that year with a valid TMAX. It is there so that a reader can tell a year with few hot days from a year with few observations.

# RUNS ON: SageMaker (remote)
tmax = wx[wx["element"] == "TMAX"].copy()
tmax["year"] = tmax["date"].dt.year
tmax["hot"]  = tmax["value"] >= 35

hot_days = (tmax.groupby(["market", "year"])
                .agg(days_reported=("value", "size"), hot_days=("hot", "sum"))
                .reset_index())
hot_days["hot_days"] = hot_days["hot_days"].astype(int)
hot_days

L5. Monthly rainfall

For each market, find the typical rainfall of each calendar month:

Total the precipitation for every market, year, and month, and count the days that reported.
A monthly total is only trustworthy if nearly every day reported. Keep a month only if at least 90% of its days have a valid PRCP value. (date.dt.days_in_month gives the length of a month.) Discarding an incomplete month is different from treating its missing days as dry.
Average the kept monthly totals over the years.

Build monthly_rain with twelve rows per market and these columns: market, month, years_used, avg_monthly_prcp_mm (rounded to 1 decimal).

# RUNS ON: SageMaker (remote)
prcp = wx[wx["element"] == "PRCP"].copy()
prcp["year"]  = prcp["date"].dt.year
prcp["month"] = prcp["date"].dt.month
prcp["days_in_month"] = prcp["date"].dt.days_in_month

by_month = (prcp.groupby(["market", "year", "month"])
                .agg(total_mm=("value", "sum"),
                     days_reported=("value", "size"),
                     days_in_month=("days_in_month", "first"))
                .reset_index())
complete = by_month[by_month["days_reported"] >= 0.9 * by_month["days_in_month"]]

monthly_rain = (complete.groupby(["market", "month"])
                        .agg(years_used=("year", "nunique"), avg_monthly_prcp_mm=("total_mm", "mean"))
                        .round({"avg_monthly_prcp_mm": 1})
                        .reset_index())
print("Months dropped as incomplete:", len(by_month) - len(complete), "of", len(by_month))
monthly_rain

L6. Export, and record what happened here

Save the two summaries on this machine’s disk as lab_hot_days.csv and lab_monthly_rain.csv. Then write lab_remote_measurements.json containing a dictionary with:

"machine": REMOTE_MACHINE
"source bytes in S3": SOURCE_BYTES
"rows kept": the number of rows in wx
"DataFrame bytes": WX_BYTES
"extract seconds": EXTRACT_SECONDS, rounded
"exported bytes": the combined size on disk of the two CSV files

Print the size of each of the three files.

# RUNS ON: SageMaker (remote)
hot_days.to_csv("lab_hot_days.csv", index=False)
monthly_rain.to_csv("lab_monthly_rain.csv", index=False)

measurements = {
    "machine":            REMOTE_MACHINE,
    "source bytes in S3": SOURCE_BYTES,
    "rows kept":          len(wx),
    "DataFrame bytes":    WX_BYTES,
    "extract seconds":    round(EXTRACT_SECONDS, 1),
    "exported bytes":     os.path.getsize("lab_hot_days.csv") + os.path.getsize("lab_monthly_rain.csv"),
}
with open("lab_remote_measurements.json", "w") as f:
    json.dump(measurements, f, indent=2)

for name in ["lab_hot_days.csv", "lab_monthly_rain.csv", "lab_remote_measurements.json"]:
    print(f"{name:<32} {human(os.path.getsize(name))}")

L7. What did this session cost?

Estimate the compute cost of this session: hours the space has been running, times the hourly price of its instance type. Use about $0.05 per hour for ml.t3.medium in us-east-1, and check the SageMaker pricing page if you chose a different instance. Reading NOAA’s public bucket from inside the same region adds no data-transfer charge, and you created nothing in S3.

The starter below tries to read the space’s start time. If that lookup fails, set hours by hand from the time you clicked Run space.

# RUNS ON: SageMaker (remote)
import datetime as dt

PRICE_PER_HOUR = 0.05            # ml.t3.medium, us-east-1, approximate; change it if your instance type differs

try:
    meta = json.load(open("/opt/ml/metadata/resource-metadata.json"))
    sm = boto3.client("sagemaker", region_name=REGION)
    app = sm.describe_app(DomainId=meta["DomainId"], SpaceName=meta["SpaceName"],
                          AppType=meta["AppType"], AppName=meta["ResourceName"])
    hours = (dt.datetime.now(dt.timezone.utc) - app["CreationTime"]).total_seconds() / 3600
except Exception as e:
    print("Could not read the space's start time:", type(e).__name__)
    hours = 1.0                  # <- set by hand: hours since you clicked Run space

print(f"Running for about {hours:.2f} h  ×  ${PRICE_PER_HOUR}/h  =  ${hours * PRICE_PER_HOUR:.3f}")

Download, then switch computers

In JupyterLab’s file browser click the refresh arrow, then right-click and Download each of: lab_hot_days.csv, lab_monthly_rain.csv, lab_remote_measurements.json.
File → Save Notebook, then download this notebook too.
Move all four files into your lab folder on your laptop, next to 04_lab_local.ipynb.
Open Positron and continue in 04_lab_local.ipynb.

Nothing after this point needs the remote machine. Stop the space now, or at the end of the local notebook, but do stop it.