Module 11

JSON & Nested Data

API data comes back as dicts inside dicts, lists inside dicts. This module is about reaching into that shape safely, and reshaping it into flat rows a report can actually use.

Concept

Why Nested Data Needs Flattening

A spreadsheet, a CSV, a log line — they all want one flat row per record. But API data comes back nested: a user dict contains a profile dict inside it, and a groups list inside that. Before you can report on it cleanly, you often have to reshape it into flat rows.

Lab 1

Look at one USERS record. Name one dict-inside-a-dict in it.

profile

Lab 2

Name one list-inside-a-dict in a USERS record.

groups

Lab 3

If you wanted one spreadsheet row per user, how many "email" columns would that row have — one, or one per group?

one — it's per-user, not per-group

Lab 4

If you wanted one spreadsheet row per (user, group) MEMBERSHIP instead, would a user with 3 groups need 1 row or 3 rows?

3 rows — one per group they belong to

Lab 5

Name one real IT/SecOps report that would need the "one row per group membership" shape instead of "one row per user."

e.g. an access-review report showing who's in which group

Concept

Serialize / Deserialize

These are the formal names for tools you already use. Serialize means turning a Python object into JSON text (json.dumps/json.dump). Deserialize means turning JSON text back into a Python object (json.loads/json.load).

import json

data = {"id": "00u001", "status": "ACTIVE"}
text = json.dumps(data)      # serialize
back = json.loads(text)      # deserialize
print(back == data)

Lab 1

Run the example above. Confirm it prints True.

→ True

Lab 2

Which of these four calls SERIALIZE, and which DESERIALIZE: json.dumps, json.loads, json.dump, json.load?

serialize: dumps, dump; deserialize: loads, load

Lab 3

Serialize a Python list of 3 numbers, then deserialize it back. Print the type at each step.

→ <class 'str'> then <class 'list'>

Lab 4

Serialize a dict to a file with json.dump(), then deserialize it back with json.load(). Confirm the round-trip is equal to the original.

→ True

Lab 5

Why does an API send serialized (text) data instead of a live Python object directly?

text can travel over a network; a live Python object can't

Now go practice

Open problems/02_serialize_deserialize.py and do all 5 problems locally.

Concept

Schema

A schema is the expected shape of a payload — which keys exist, and what type each one is. Real data doesn't always match the schema you expect: a key might be missing, or a value might be None when you expected a string. Guarding with .get()/in is how you handle a schema that isn't perfectly followed.

def matches_schema(user):
    has_department = "department" in user["profile"]
    has_login = user.get("lastLogin") is not None
    return has_department and has_login

Lab 1

Write matches_schema exactly as shown. Call it on USERS[7] (Layla). Print the result.

→ False

Lab 2

Call it on USERS[3] (Dana). Print the result.

→ False

Lab 3

Call it on USERS[0] (Julio). Print the result.

→ True

Lab 4

Loop all 12 USERS, calling matches_schema on each. Print how many DON'T match.

→ 2 (Dana and Layla)

Lab 5

Write a function that prints each user's firstName and whether they matched the schema, looping all 12.

12 lines, 2 say False

Now go practice

Open problems/03_schema.py and do all 5 problems locally.

Concept

Nested Object Access

Reaching into a dict-inside-a-dict or a list-inside-a-dict, beyond simple one-level lookups: pulling out a specific item from a nested list, or checking whether something specific is present in it.

def has_group(user, group_name):
    names = [g["name"] for g in user["groups"]]
    return group_name in names

Lab 1

Write has_group exactly as shown. Call it on USERS[0] (Julio) with "VPN-Users". Print the result.

→ True

Lab 2

Call it on USERS[0] with "Admins". Print the result.

→ False

Lab 3

Call it on USERS[3] (Dana, empty groups) with any group name. Confirm no crash — just False.

→ False

Lab 4

Write a function that returns just the FIRST group name a user belongs to, or None if they have none. Test it on Julio and on Dana.

→ Engineering then None

Lab 5

Write a function that counts how many users belong to a given group name, looping all of USERS. Call it with "VPN-Users".

→ 5

Now go practice

Open problems/04_nested_access.py and do all 5 problems locally.

Concept

Checkpoint: A Schema-Guarded Extractor

Combine schema-checking and nested access: pull a small, safe, flat dict out of each full user record.

def extract_summary(user):
    return {
        "email": user["profile"]["email"],
        "department": user["profile"].get("department", "Unknown"),
        "group_count": len(user["groups"]),
    }

Lab 1

Write extract_summary exactly as shown. Call it on USERS[0] (Julio). Print the result.

→ {'email': 'jrivera@example.com', 'department': 'IT', 'group_count': 2}

Lab 2

Call it on USERS[7] (Layla, missing department). Confirm no crash.

→ department shows Unknown

Lab 3

Call it on USERS[3] (Dana, zero groups). Print the result.

→ group_count is 0

Lab 4

Loop all 12 USERS, calling extract_summary on each, building a list of the results. Print len() of that list.

→ 12

Lab 5 · Checkpoint

Extend extract_summary to also include "has_never_logged_in": user.get("lastLogin") is None. Run it over all 12 users, printing just the ones where that's True.

→ 1 result (Dana)

Now go practice

Open problems/05_extractor_checkpoint.py — 5 problems, #5 is a checkpoint combining sub-skills 2-4.

Concept

Comprehensions, Applied to Nested Data

You already know list comprehensions from Module 6. This is the same tool, reaching further in — pulling values out of a NESTED structure, not just filtering a flat list.

# every group name Julio belongs to
julio_groups = [g["name"] for g in USERS[0]["groups"]]

# every group name across ALL users (flat list, duplicates allowed)
all_group_names = [g["name"] for u in USERS for g in u["groups"]]

Lab 1

Run the first comprehension above. Print the result.

→ ['Engineering', 'VPN-Users']

Lab 2

Run the second comprehension. Print len() of the result.

→ 15

Lab 3

Write a comprehension of every user's email who has AT LEAST ONE group.

→ 9 emails (Dana, Tri, and Pat all have zero groups)

Lab 4

Write a comprehension of every user's firstName who belongs to "VPN-Users" (use a helper or inline the check).

5 names

Lab 5

Write a comprehension building a list of (firstName, department) tuples for every user, guarding the missing department with .get().

12 tuples, no crash

Now go practice

Open problems/06_comprehensions_nested.py and do all 5 problems locally.

Concept

Loop ↔ Comprehension, Both Directions

Fluency means going both ways: turn a loop into a comprehension, AND expand a comprehension back into a loop. Being able to read either form is what makes someone else's code (or an interview whiteboard) easy to follow.

# The loop way:
result = []
for u in USERS:
    if u["status"] == "ACTIVE" and len(u["groups"]) > 1:
        result.append(u["profile"]["email"])

# The comprehension way -- same result:
result = [
    u["profile"]["email"]
    for u in USERS
    if u["status"] == "ACTIVE" and len(u["groups"]) > 1
]

Lab 1

Run both versions above. Confirm they produce the identical list.

→ ['jrivera@example.com', 'achen@example.com', 'sokafor@example.com', 'mfoster@example.com'], both ways

Lab 2

Take this comprehension and expand it into an equivalent loop: [u["id"] for u in USERS if u["status"] == "SUSPENDED"]

a 3-line loop, same result

Lab 3

Take this loop and convert it into a comprehension:

out = []
for u in USERS:
    out.append(u["profile"]["lastName"])

one-line comprehension, same 12 last names

Lab 4

Convert this nested-loop into a single comprehension:

out = []
for u in USERS:
    for g in u["groups"]:
        out.append(g["name"])

[g["name"] for u in USERS for g in u["groups"]], 15 items

Lab 5

Write your own comprehension with BOTH a filter and a computed expression (not just a plain field lookup) — e.g. combine two fields into one string, only for ACTIVE users.

Now go practice

Open problems/07_loop_comprehension_conversion.py and do all 5 problems locally.

Concept

Flattening Group Membership: the Accumulator, Again

You've built accumulator dicts before (Module 6, Module 8). Here's the same nested-loop pattern applied to flattening membership: how many users are in EACH group, across the whole fixture.

def count_group_members(users):
    counts = {}
    for u in users:
        for g in u["groups"]:
            name = g["name"]
            counts[name] = counts.get(name, 0) + 1
    return counts

Lab 1

Write count_group_members exactly as shown. Call it on USERS. Print the result.

→ {'Engineering': 4, 'VPN-Users': 5, 'SecOps': 2, 'Admins': 2, 'Sales-Team': 1, 'Contractors': 1}

Lab 2

Print how many distinct groups exist (the number of keys in the result).

→ 6

Lab 3

Find the single group with the MOST members. Print its name and count.

→ VPN-Users 5

Lab 4

Rewrite the accumulator's inner loop using a comprehension to first build the flat list of all group names, THEN accumulate over that flat list instead of the nested structure directly. Confirm the same result.

same 6-entry dict

Lab 5

Sum all the values in the result dict. Does it equal the total number of (user, group) memberships?

→ 15, yes

Now go practice

Open problems/08_accumulator_flatten.py and do all 5 problems locally.

Concept

Checkpoint: Combining Everything So Far

Schema-guarding, nested access, comprehensions, and the accumulator — all together in one script.

Lab 1

Build the group-member-count accumulator from sub-skill 8. Print it.

Lab 2

Using a comprehension, build a list of every group name JULIO belongs to.

→ ['Engineering', 'VPN-Users']

Lab 3

Using extract_summary from sub-skill 5, build a list of summaries for every user, guarding the two bad records.

12 summaries, no crash

Lab 4

From that list of summaries, use a comprehension to pull just the emails of users with group_count == 0.

→ 3 emails (Dana, Tri, and Pat)

Lab 5 · Checkpoint

One script combining all four: print the accumulator, then the summaries list length, then the zero-group emails — three outputs, zero crashes.

Now go practice

Open problems/09_combined_checkpoint.py — 5 problems, #5 is a checkpoint combining sub-skills 3-8.

Concept

Flatten to Rows

The mastery payoff: turn nested USERS records into a FLAT list of dicts, one row per (user, group) pair — the shape a report tool or spreadsheet actually wants.

def flatten_memberships(users):
    rows = []
    for u in users:
        for g in u["groups"]:
            rows.append({
                "email": u["profile"]["email"],
                "lastName": u["profile"]["lastName"],
                "department": u["profile"].get("department", "Unknown"),
                "group": g["name"],
            })
    return rows

Lab 1

Write flatten_memberships exactly as shown. Call it on USERS. Print len() of the result.

→ 15

Lab 2

Print the first 3 rows.

→ Julio/Engineering, Julio/VPN-Users, Amy/VPN-Users

Lab 3

Find Layla's row(s). Confirm department shows "Unknown", not a crash.

→ 1 row, department: 'Unknown'

Lab 4

Confirm Dana has ZERO rows in the flattened output (she has no groups to flatten).

→ 0 rows for Dana

Lab 5

Each ORIGINAL user with N groups produces N rows. Verify: sum the group counts across all 12 users, confirm it equals len(flatten_memberships(USERS)).

→ both equal 15

Lab 6

Rewrite flatten_memberships using a nested comprehension instead of nested loops. Confirm identical output.

same 15 rows

Now go practice

Open problems/10_flatten_to_rows.py and do all 6 problems locally.

Concept

Final Capstone: Flatten, Sort, Number, Write

Combine flattening with Module 10's file tools: fetch the nested fixture, flatten it to rows, sort those rows, number them, and write a report to a file.

def main():
    rows = flatten_memberships(USERS)
    sorted_rows = sorted(rows, key=lambda r: r["lastName"])
    with open("membership_report.txt", "w") as f:
        for i, r in enumerate(sorted_rows, start=1):
            f.write(f"{i}. {r['lastName']} -- {r['group']} ({r['department']})\n")

Lab 1

Write main exactly as shown (reuse your flatten_memberships). Call it.

writes membership_report.txt, no crash

Lab 2

Read the report file back. Print how many lines it has.

→ 15

Lab 3

Print the first 3 lines of the report.

sorted by last name — Chen, then...

Lab 4

Confirm the report contains NO line for Dana (she has zero group rows).

confirmed absent

Lab 5 · Final checkpoint

Extend main to also print a one-line summary before writing: total rows, and how many distinct groups appear. Run the whole thing end-to-end.

→ 15 rows, 6 groups (or your own wording)

Now go practice

Open problems/11_final_capstone.py — 5 problems, #5 is the final checkpoint for all of Module 11.

Complete

Module 11 done

why flattening matters → serialize/deserialize → schema → nested object access → comprehensions applied to nested data → loop ↔ comprehension conversion → the accumulator, reframed as flattening → flatten to rows → the capstone: flatten, sort, number, write.

Run python check.py in problems/ to grade everything.