Topics of Practice: Remote Computing on Data You Do Not Download

GSB 5544 · Computing and Machine Learning for Business Analytics

This notebook is short on purpose. It practices the middle of the workflow, the part that happens on a computer that is not yours:

measure the remote machine → look at how big the data is → read a manageable piece → summarize it → save a small file

Run this notebook in JupyterLab inside your SageMaker space (reading, section 5, parts D and E), not in Positron. Replace each ____ and run the cell with Shift+Enter. If a cell stops with NameError: name '____' is not defined, you ran it before filling in its blank; fill the blank and run it again.

Every code cell starts with a # RUNS ON: comment. That comment is a label for you. It does not move the code anywhere. A cell runs on whichever computer the notebook’s kernel is on. The setup cell checks where that really is.

Bring the laptop numbers you recorded in the reading, section 1.

# RUNS ON: SageMaker (remote)
%pip install -q s3fs pyarrow psutil      # harmless if already present

import os
import json
import time
import socket
import psutil
import boto3
import pandas as pd
from botocore import UNSIGNED
from botocore.config import Config

BUCKET = "noaa-ghcn-pds"         # NOAA's public bucket
REGION = "us-east-1"             # the region the bucket is in; your SageMaker space should be in it too

# unsigned = anonymous: this client attaches no credentials, because the bucket is public
s3 = boto3.client("s3", region_name=REGION, config=Config(signature_version=UNSIGNED))


def human(n, base=1000):
    """Turn a byte count into a readable string. base=1000 gives KB/MB/GB; base=1024 gives KiB/MiB/GiB."""
    units = ["B", "KB", "MB", "GB", "TB", "PB"] if base == 1000 else ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]
    for unit in units:
        if n < base:
            return f"{n:,.1f} {unit}"
        n /= base
    return f"{n:,.1f} (huge)"


def where_am_i():
    """Report which computer this kernel is really on. A RUNS ON comment cannot do that."""
    on_sagemaker = (os.path.exists("/opt/ml/metadata/resource-metadata.json")
                    or os.path.expanduser("~") == "/home/sagemaker-user")
    place = "SageMaker (remote)" if on_sagemaker else "not SageMaker (probably your laptop)"
    print("Computer name   :", socket.gethostname())
    print("This looks like :", place)
    return place


def machine_report(work_dir="."):
    """Measure the computer running this cell. Disk numbers are for the filesystem that holds work_dir."""
    vm = psutil.virtual_memory()
    du = psutil.disk_usage(work_dir)
    return {
        "computer name":                    socket.gethostname(),
        "physical CPU cores":               psutil.cpu_count(logical=False),   # may be None on some virtual machines
        "logical CPUs (vCPUs in the cloud)": psutil.cpu_count(logical=True),
        "total RAM (GB)":                   round(vm.total / 1e9, 1),
        "available RAM right now (GB)":     round(vm.available / 1e9, 1),
        "work disk, total (GB)":            round(du.total / 1e9, 1),
        "work disk, free (GB)":             round(du.free / 1e9, 1),
    }


def slice_prefix(year, element):
    """The key prefix of one year-and-element slice of the Parquet copy of GHCN-Daily."""
    return f"parquet/by_year/YEAR={year}/ELEMENT={element}/"


def prefix_size(prefix):
    """Count the objects under a prefix and add up their sizes, without reading any of them."""
    objects, total = 0, 0
    for page in s3.get_paginator("list_objects_v2").paginate(Bucket=BUCKET, Prefix=prefix):
        for obj in page.get("Contents", []):
            objects += 1
            total += obj["Size"]
    return objects, total


if where_am_i() != "SageMaker (remote)":
    print()
    print("NOTE: this notebook is written to run in your SageMaker space, and this kernel is somewhere else.")
    print("      The RUNS ON comments describe the plan. They do not move the code. Open this file in JupyterLab on SageMaker.")

The setup cell supplied the routine pieces so you do not have to rebuild them: human() for readable sizes, where_am_i(), machine_report(), slice_prefix(), and prefix_size(). Read their docstrings above; you will call them below.

Start here: what stores, what computes

	What it is	Stores data?	Runs your code?	What you pay for
Your laptop	The computer in front of you	Yes, on its disk	Yes, with its own CPU and RAM	Nothing extra
SageMaker space	A computer you rent in an AWS data center, with its own CPU, RAM, and a small disk	A little: notebooks and small files	Yes, with its CPU and RAM, not yours	Each hour its status is Running
S3 bucket	Storage only. Files kept in AWS, readable from your laptop or from SageMaker	Yes	No. It can only hand over bytes	Each GB stored per month (NOAA’s bucket is paid for by AWS’s open-data program, not by you)

Storing is not processing. Data in S3 does nothing until some computer reads it. The bytes travel to wherever the code runs, and what that code loads must fit in that computer’s RAM.

Disk is not RAM. Disk keeps files when the machine is off. RAM is the working space of a running program. S3, the remote machine’s disk, and the remote machine’s RAM are three different places.

Fill in each blank with laptop, SageMaker, or S3.

# RUNS ON: SageMaker (remote)
where = {
    "Stores the NOAA weather files we read today":             "S3",
    "Runs the cells of this notebook":                         "SageMaker",
    "Shows you this notebook in a browser tab":                "laptop",
    "Can store data but cannot run any code":                  "S3",
    "Bills by the hour while Running, even when you are idle":  "SageMaker",
    "Will run Positron and draw the final chart":              "laptop",
}

expected = ["s3", "sagemaker", "laptop", "s3", "sagemaker", "laptop"]
given = [v.strip().lower() for v in where.values()]
wrong = [q for q, g, e in zip(where, given, expected) if g != e]
print("All correct" if not wrong else "Check these: " + "; ".join(wrong))

A. Measure the remote machine and compare it with your laptop

Reading, section 1. machine_report() measures the computer running this cell. Its disk numbers describe the filesystem that holds this notebook, which on SageMaker is the space’s own small disk, not S3.

• Physical cores are real processor cores. Logical CPUs count what the operating system can schedule on; with hyper-threading one physical core appears as two. A cloud vCPU is one logical CPU, so “2 vCPUs” does not mean two physical cores.
• Total RAM is what is installed. Available RAM is what a new program could use right now. Available is the smaller number and the one that matters.

Fill in the function call, then paste in your laptop’s numbers from the reading.

# RUNS ON: SageMaker (remote)
remote = machine_report()
remote

# RUNS ON: SageMaker (remote)
laptop = {                                   # <- replace every value with what you recorded in the reading, section 1
    "computer name":                     "my laptop",
    "physical CPU cores":                0,
    "logical CPUs (vCPUs in the cloud)": 0,
    "total RAM (GB)":                    0.0,
    "available RAM right now (GB)":      0.0,
    "work disk, total (GB)":             0.0,
    "work disk, free (GB)":              0.0,
}

compare = pd.DataFrame({"laptop": laptop, "SageMaker (remote)": remote})
compare

Save this machine’s details as a markdown file. The cell below writes the remote report as a small .md table next to this notebook. Download it with the rest of your files; it is the record of which machine did today’s work, and the instructor’s copy of it is the Remote machine reference page on the course site.

# RUNS ON: SageMaker (remote)
import datetime as dt

try:                                                         # the space's own description of itself, if available
    meta = json.load(open("/opt/ml/metadata/resource-metadata.json"))
    space_info = {"space name": meta.get("SpaceName"), "domain id": meta.get("DomainId"),
                  "app type": meta.get("AppType"), "region (from ARN)": meta.get("ResourceArn", "::::")[9:].split(":")[2]}
except Exception:
    space_info = {"space name": "not reported by this image"}

lines = [f"# Remote machine report", "",
         f"Measured on {dt.date.today().isoformat()} by `machine_report()` inside SageMaker.", "",
         "| Measure | Value |", "|---|---|"]
lines += [f"| {k} | {v} |" for k, v in {**space_info, **remote}.items()]
report_text = "\n".join(lines) + "\n"

with open("remote_machine_report.md", "w") as f:
    f.write(report_text)
print(report_text)

Check yourself. Do not assume the rented machine is bigger or faster. Look at your table. Which machine has more available RAM? More logical CPUs? Answer in one comment, using your own numbers.

# RUNS ON: SageMaker (remote)
# Example: my laptop has 16 GB total and 5.1 GB available with 8 logical CPUs; the remote machine has about 4 GB total with 2 vCPUs. The remote machine is smaller. What it offers is location (next to the data) and that it can be resized, not size.

B. Bucket, object, key, and why this read is anonymous

Reading, sections 3 and 4. A bucket is a named container. An object is one stored file plus its metadata. A key is the object’s full name inside the bucket. A prefix is the beginning of a key, which is how S3 imitates folders.

The s3 client in the setup cell is unsigned: it sends no credentials. NOAA’s bucket is public, so it answers anyone. You still needed your own AWS account today, but for a different reason: the account is what lets you rent this computer. The data is free to read; the computing is yours to pay for.

Fill in the delimiter that makes S3 group keys like folders, and the metadata field that holds an object’s size.

# RUNS ON: SageMaker (remote)
resp = s3.list_objects_v2(Bucket=BUCKET, Delimiter="/")
print("Top-level prefixes:", [p["Prefix"] for p in resp.get("CommonPrefixes", [])])

key = "csv/by_year/2024.csv"                          # one object: a whole year, every station, as text
meta = s3.head_object(Bucket=BUCKET, Key=key)         # metadata only; the file is not downloaded
print("Bucket:", BUCKET)
print("Key   :", key)
print("Size  :", human(meta["ContentLength"]))

C. Look at the size before you read

Reading, section 2. The same year exists in the bucket twice: as one large CSV object, and as Parquet files split by year and element. Today we read the Parquet slice for 2024 daily highs (TMAX).

Fill in the helper that builds the slice’s prefix, and the base that gives binary units.

# RUNS ON: SageMaker (remote)
prefix = slice_prefix(2024, "TMAX")
objects, SOURCE_BYTES = prefix_size(prefix)

print("Prefix          :", prefix)
print("Objects         :", objects)
print("Size, decimal   :", human(SOURCE_BYTES))               # 1 MB = 1,000,000 bytes
print("Size, binary    :", human(SOURCE_BYTES, base=1024))  # 1 MiB = 1,048,576 bytes
print("The CSV year was:", human(meta["ContentLength"]))

The two size lines describe the same bytes in two unit systems. Storage and cloud consoles mostly use decimal units; operating systems often use binary ones.

Source-file size is not how much RAM the data will need. Text CSVs often grow when loaded and compressed Parquet often grows much more. There is no multiplier you can trust in advance, so the honest method is: start with a slice you know is small, load it, and measure.

D. Read a manageable dataset into the remote machine’s RAM

The bytes now travel from S3 to this SageMaker machine. Nothing reaches your laptop except the text printed below.

Fill in the setting that makes the read anonymous, and the argument that makes pandas count text columns honestly.

# RUNS ON: SageMaker (remote)
t0 = time.perf_counter()
raw = pd.read_parquet(
    f"s3://{BUCKET}/{prefix}",
    columns=["ID", "DATE", "DATA_VALUE", "Q_FLAG"],       # only the columns we need
    storage_options={"anon": True},
)
READ_SECONDS = time.perf_counter() - t0

ROWS_READ = len(raw)
DF_BYTES = int(raw.memory_usage(deep=True).sum())

print(f"Rows read          : {ROWS_READ:,} in {READ_SECONDS:.1f} s")
print(f"Source files in S3 : {human(SOURCE_BYTES)}")
print(f"DataFrame in RAM   : {human(DF_BYTES)}   ({DF_BYTES / SOURCE_BYTES:.0f} times the source size, measured, not predicted)")
raw.head(3)

E. Filter, transform, and aggregate, on the remote machine

GHCN-Daily has three conventions you must handle:

• DATE is text like 20240131.
• DATA_VALUE for TMAX is in tenths of a degree Celsius: 183 means 18.3 °C.
• Q_FLAG is empty when a value passed NOAA’s quality checks and holds a letter when it failed. Failed values are dropped. A day with no valid value is missing; it is never zero.

Summarize one station, the Cal Poly station in San Luis Obispo (USC00047851), by month.

Fill in the quality filter, the unit conversion, and the grouping column.

# RUNS ON: SageMaker (remote)
STATION = "USC00047851"                        # SAN LUIS OBISPO POLY, CA

one = raw[(raw["ID"] == STATION) & raw["Q_FLAG"].isna()].copy()
one["date"]   = pd.to_datetime(one["DATE"].astype(str), format="%Y%m%d")
one["tmax_c"] = one["DATA_VALUE"] / 10
one["month"]  = one["date"].dt.month

summary = (one.groupby("month")["tmax_c"]
              .agg(days_reported="count", mean_tmax_c="mean", hottest_c="max")
              .round(1)
              .reset_index())

print(f"{ROWS_READ:,} rows read, {len(one):,} kept for this station, {len(summary)} rows in the summary")
summary

F. Save a small summary on the remote disk, and compare four sizes

The summary is written to this machine’s disk, next to this notebook. It is not in S3 and not on your laptop yet. In the practice activity you will download a file like this and chart it in Positron; there is no chart here because the charting belongs on your laptop.

Fill in the method that writes a CSV and the function that reads a file’s size from disk.

# RUNS ON: SageMaker (remote)
OUT = "slo_tmax_2024_monthly.csv"
summary.to_csv(OUT, index=False)
CSV_BYTES = os.path.getsize(OUT)

sizes = pd.DataFrame(
    {
        "where it lives": ["S3 (NOAA's bucket)", "this machine's RAM", "this machine's disk", "this machine's disk"],
        "size":           [human(SOURCE_BYTES), human(DF_BYTES), human(CSV_BYTES), human(machine_report()["work disk, free (GB)"] * 1e9)],
    },
    index=["source files", "loaded DataFrame", "exported summary CSV", "free space on the work disk"],
)
sizes

Check yourself. In one comment: which of those four numbers would also be the size of the file you download to your laptop, and which one disappears when the space stops?

# RUNS ON: SageMaker (remote)
# The exported summary CSV is what I would download; it is the same few hundred bytes on either disk. The loaded DataFrame lives only in RAM and disappears when the space stops. The source files stay in S3 untouched.

G. Stop the meter

A space bills for every hour its status is Running, whether or not you are typing and whether or not this tab is open. A cell cannot stop the machine it is running on, so this happens in the Studio tab.

• Remote notebook: File → Save Notebook. Right-click it in the JupyterLab file browser and Download a copy.
• Studio tab: on the gsb5544 space page click Stop space, confirm in the dialog, and wait until the status reads Stopped. Closing the browser tab does not stop it.
• Console Home (open console.aws.amazon.com in a new tab): read Credits remaining and note it. The panel can lag by several hours.
• Laptop: save your local notebook in Positron (Cmd+S, or Ctrl+S on Windows).

If you are going straight on to the practice activity, leave the space running and do this checklist at the end of that activity instead.