Lab, Local Notebook: Visualize and Interpret

GSB 5544 · Computing and Machine Learning for Business Analytics

Open this notebook in Positron, on your laptop. It needs the three files you downloaded from your SageMaker space, in the same folder as this notebook:

Everything here runs on your laptop. No cell in this notebook reads from S3.

# RUNS ON: laptop (Positron)
%pip install -q pandas pyarrow s3fs psutil matplotlib      # harmless if already present

import os
import json
import time
import socket
import psutil
import pandas as pd
import matplotlib.pyplot as plt

BUCKET = "noaa-ghcn-pds"


def human(n, base=1000):
    """Turn a byte count into a readable string. base=1000 gives KB/MB/GB; base=1024 gives KiB/MiB/GiB."""
    units = ["B", "KB", "MB", "GB", "TB", "PB"] if base == 1000 else ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]
    for unit in units:
        if n < base:
            return f"{n:,.1f} {unit}"
        n /= base
    return f"{n:,.1f} (huge)"


def where_am_i():
    """Report which computer this kernel is really on. A RUNS ON comment cannot do that."""
    on_sagemaker = (os.path.exists("/opt/ml/metadata/resource-metadata.json")
                    or os.path.expanduser("~") == "/home/sagemaker-user")
    place = "SageMaker (remote)" if on_sagemaker else "not SageMaker (probably your laptop)"
    print("Computer name   :", socket.gethostname())
    print("This looks like :", place)
    return place


def machine_report(work_dir="."):
    """Measure the computer running this cell. Disk numbers are for the filesystem that holds work_dir."""
    vm = psutil.virtual_memory()
    du = psutil.disk_usage(work_dir)
    return {
        "computer name":                    socket.gethostname(),
        "physical CPU cores":               psutil.cpu_count(logical=False),   # may be None on some virtual machines
        "logical CPUs (vCPUs in the cloud)": psutil.cpu_count(logical=True),
        "total RAM (GB)":                   round(vm.total / 1e9, 1),
        "available RAM right now (GB)":     round(vm.available / 1e9, 1),
        "work disk, total (GB)":            round(du.total / 1e9, 1),
        "work disk, free (GB)":             round(du.free / 1e9, 1),
    }


if where_am_i() == "SageMaker (remote)":
    print()
    print("NOTE: this notebook is written to run on your laptop, in Positron, and this kernel is on SageMaker.")
    print("      Download this file and open it in Positron instead.")
# RUNS ON: laptop (Positron)
NEEDED = ["lab_hot_days.csv", "lab_monthly_rain.csv", "lab_remote_measurements.json"]
missing = [name for name in NEEDED if not os.path.exists(name)]
print("This notebook is running in:", os.getcwd())
print("MISSING: " + ", ".join(missing) if missing else "All three downloaded files are here.")

L8. Read what you downloaded

Read the two CSV files into hot_days and monthly_rain, and the JSON file into REMOTE. Print each table’s shape and the size on disk of each downloaded file.

# RUNS ON: laptop (Positron)
hot_days     = pd.read_csv("lab_hot_days.csv")
monthly_rain = pd.read_csv("lab_monthly_rain.csv")
with open("lab_remote_measurements.json") as f:
    REMOTE = json.load(f)

print("hot_days    :", hot_days.shape)
print("monthly_rain:", monthly_rain.shape)
for name in NEEDED:
    print(f"{name:<32} {human(os.path.getsize(name))}")

L9. Chart 1: hot days per year

One line per market: year on the x-axis, number of days at or above 35 °C on the y-axis. Label both axes with units, give the chart a title that says what it shows, and save it as lab_hot_days.png.

Before you plot, look at days_reported. If any market-year has far fewer than 365 reported days, say so in your interpretation in L11.

# RUNS ON: laptop (Positron)
print(hot_days.pivot(index="year", columns="market", values="days_reported"))

fig, ax = plt.subplots(figsize=(9, 4))
for market, rows in hot_days.groupby("market"):
    ax.plot(rows["year"], rows["hot_days"], marker="o", linewidth=1.3, label=market)
ax.set_xticks(sorted(hot_days["year"].unique()))
ax.set_xlabel("Year")
ax.set_ylabel("Days with a high of 35 °C or more")
ax.set_title("Hot days per year, 2015–2024")
ax.grid(alpha=0.3)
ax.legend()
fig.savefig("lab_hot_days.png", dpi=150, bbox_inches="tight")
plt.show()

L10. Chart 2: typical rainfall by month

Show the average monthly precipitation for both markets across the twelve calendar months, in millimeters, in a form that makes the two markets easy to compare. Save it as lab_monthly_rain.png. Then print the wettest month of each market.

# RUNS ON: laptop (Positron)
wide = monthly_rain.pivot(index="month", columns="market", values="avg_monthly_prcp_mm")

ax = wide.plot(kind="bar", figsize=(9, 4), width=0.8)
ax.set_xlabel("Calendar month")
ax.set_ylabel("Average monthly precipitation (mm)")
ax.set_title("Typical rainfall by month, complete months of 2015–2024")
ax.grid(axis="y", alpha=0.3)
ax.figure.savefig("lab_monthly_rain.png", dpi=150, bbox_inches="tight")
plt.show()

print("Wettest month:", wide.idxmax().to_dict())

L11. The business read

Answer as the inventory planner, in six to eight sentences, using numbers from your two charts.

  • How differently should the two markets be stocked across the year? Refer to both heat and rain.
  • Is there anything in days_reported or years_used that should make the team cautious about a particular year or month?
  • Name two other datasets you would want to join to this weather history before trusting a demand model, and say where each one probably lives: on a laptop, in a company database, or in cloud storage.

Your answer:

L12. Two machines and four sizes

Build one table that puts your laptop’s machine_report() beside the remote machine’s (from REMOTE["machine"]). Then print, in readable units: the source size in S3, the size of the remote DataFrame, the combined size of the exported CSVs, and the combined size of the files as downloaded on this laptop.

# RUNS ON: laptop (Positron)
resources = pd.DataFrame({"laptop": machine_report(), "SageMaker (remote)": REMOTE["machine"]})
print(resources)
print()
downloaded = os.path.getsize("lab_hot_days.csv") + os.path.getsize("lab_monthly_rain.csv")
print("Source files in S3       :", human(REMOTE["source bytes in S3"]))
print("Remote DataFrame in RAM  :", human(REMOTE["DataFrame bytes"]), f'({REMOTE["rows kept"]:,} rows)')
print("Exported CSVs, remote    :", human(REMOTE["exported bytes"]))
print("Downloaded CSVs, laptop  :", human(downloaded))
print("Remote extract took      :", REMOTE["extract seconds"], "s")

L13. Where the work happened

(a) Answer each of the five questions in one or two sentences, with specifics from this lab.

  1. Where is the original data stored?
  2. Which computer performed the analysis?
  3. What did I download?
  4. Which computer created the visualization?
  5. How did the two machines compare?

Your answers:

(b) This weather summary will be refreshed every week once the demand model is live. Would you run the weekly refresh on your laptop or on a SageMaker space? Give two reasons that use what you measured: the source size, the size of what comes home, the resources of each machine, and the cost from L7.

Your answer:

L14. Save and stop

Paste a screenshot of the gsb5544 space page showing status Stopped into the cell below, or submit it as a separate image.

Evidence:

What to submit

  1. 04_lab_remote.ipynb, run in SageMaker, with output showing.
  2. 04_lab_local.ipynb (this notebook), run in Positron, with output and written answers.
  3. lab_hot_days.csv and lab_monthly_rain.csv.
  4. lab_hot_days.png and lab_monthly_rain.png.
  5. The screenshot showing the space Stopped.

A submission is graded on both halves. A lab whose charts were made on SageMaker, or whose summaries were computed on a laptop, has not done the workflow.