What a Data Engineer Owns

By the end of this session you will be able to:

• Write a table you are responsible for as a contract: grain, freshness target, and the checks that decide whether today's load is correct
• Trace a number on a dashboard back through the trusted, staged, and raw layers to the file or event that produced it
• Name the three ways one real table can break, and the person who notices each one

The unit of work is a table, not a script

Ask a new data engineer what they own and you get a list of scripts. Ask the finance analyst who calls them at 9am and you get a list of tables. She does not know your script exists. She knows orders had 4,812 rows yesterday, has 9,624 today, and revenue appears to have doubled.

That gap is the job. You are paid to make a small number of tables true, on time, with a definition that does not quietly change under people who already built on it.

Every table you ship comes with four claims:

1. Grain. What exactly one row means. "One row per order" and "one row per order status change" are different tables and produce different revenue.
2. Freshness. By what time, containing data up to when. "Daily" is not a freshness target. "Yesterday's orders by 07:00" is.
3. Correctness checks. Assertions that run on every load and fail loudly, not "it looked fine".
4. Owner. A human name. Yours, until you hand it over on purpose.

Get the grain wrong and every downstream join fans out: someone reports a number three times too big and nobody can say when it started. Get freshness wrong and the number is correct but arrives after the meeting it was for.

Data arrives through three doors

Nearly everything you ingest arrives in one of three shapes, each failing in its own way.

Files. A system drops CSV, JSON, or Parquet on a schedule. Failure modes: the drop is late, never happens, happens twice, or the header changes and column seven is now total_amount instead of amount. Files are the easiest to reason about: the evidence is still on disk. That is why phase one is built on them.

Events. A producer writes messages to a topic as things happen. Failure modes: an event arrives forty minutes after the event it logically follows, a producer replays yesterday to recover from its own outage, or you receive the same message twice because the delivery guarantee is at-least-once. When the thing happened and when you received it are two different columns, and you need both.

Databases. You pull from somebody else's operational database. These are the quiet failures. A row was hard deleted at the source and your copy still has it. A migration ran on Tuesday and a column changed type. The updated_at column your load depends on is not touched by every code path that writes the row.

None of these throw an exception. A job that exits 0 having loaded half a file is the standard bad day here, which is why "did it crash" is a useless health check.

Raw, staged, trusted

Between the door and the table sit three layers. They exist because you cannot re-download the past.

Raw is what arrived, kept exactly as it arrived, never edited. If the source sent garbage, raw keeps the garbage. It is your only defense when someone asks in November what August actually contained.

Staged is raw parsed into typed columns with names you chose. Still one row per source record. No joins, no business logic, no dropping bad rows: just types, names, and a column recording which file or offset each row came from.

Trusted is the table people query. Deduplicated, at a stated grain, with the checks attached.

One day of an orders drop:

order_id,customer,amount,status,updated_at
1001,ACME,240.00,pending,2026-08-03T09:12:04Z
1001,ACME,240.00,paid,2026-08-03T14:41:55Z
1002,Globex,,paid,2026-08-03T10:02:11Z
1003,Initech,88.50,paid,2026-08-02T23:58:40Z
1003,Initech,88.50,paid,2026-08-02T23:58:40Z

Five rows, three orders. Order 1001 appears twice legitimately: the exporter emits one row per status change. Order 1003 appears twice illegitimately: the exporter retried and wrote the line again. Order 1002 is paid with no amount, either a source bug or a free order.

Raw keeps all five rows. Staged keeps all five, typed, with a source_file column. Trusted keeps three, one per order_id, carrying the latest status by updated_at, and refuses the load when a paid row has a null amount.

You can see that before writing any pipeline code:

tail -n +2 orders_2026-08-03.csv | wc -l
tail -n +2 orders_2026-08-03.csv | cut -d, -f1 | sort -u | wc -l

Five and three. That gap is the entire content of the trusted layer's dedup rule. One caveat: cut splits on every comma, so this holds only while no field is quoted. That fragility is why "Files Are the Substrate" replaces it with a real reader.

Who complains, and how fast

Every trusted table has consumers, and they detect failures at very different speeds.

• A human on a dashboard notices missing data instantly and wrong numbers fast, but only during working hours, and only for the numbers they look at.
• A scheduled report notices lateness, because it runs at a fixed time and reads whatever is there. It will publish a half-loaded table without hesitating.
• A model or feature job notices almost nothing. It consumes a null-heavy column for six weeks, and the damage surfaces as a slow accuracy decline nobody connects back to you.

That ordering is your alerting priority. Your checks have to catch what your slowest consumer cannot: row counts, freshness, null rates on key columns, and distinct-key equality catch most real incidents, and all four are cheap.

Write the consumers down by name. A table with no named consumer is one you can delete, and deleting it is a real option.

Try it

Pick one table you actually depend on. If you do not have one, use the orders sample above. Fill in this template and save it as a text file beside the data:

table: orders
grain: one row per order_id, current state
source: daily CSV drop from the billing exporter, one file per date
freshness: yesterday's orders queryable by 07:00 local, every day
owner: you
consumers:
  - finance daily revenue sheet, read at 09:00
  - churn model feature job, read at 03:00
correct when:
  - count(*) equals count(distinct order_id)
  - zero rows where status = 'paid' and amount is null
  - max(updated_at) is within the last 26 hours
breaks when:
  - the exporter retries and the file contains a duplicated line
  - a new status value appears that the finance filter does not match
  - the drop lands at 09:30 instead of 07:00

Success condition: you can point at one line under correct when that would have caught the duplicated 1003 row, and you can write a real person or team beside every consumer. If either is blank, you do not own that table yet. You run a script that touches it.

Then run the two commands above against a file of your own. Rows and distinct keys are almost never the same number, and knowing which one your consumers expect is most of this job.

Common mistakes

• Owning the script instead of the table. You debug the run, declare it green, and never check what landed. End every run with an assertion about the table, not the process.
• Cleaning data in the raw layer. It feels tidy and destroys your only evidence. Fix in staged or trusted, and keep raw immutable so you can replay from it.
• Treating exit code 0 as success. Half a file, an empty file, and a full file all exit 0. Compare row counts against the previous load and the source.
• Guessing the grain. People assume one row per order because the table is called orders. Prove it with a distinct-key count before joining to it.
• Changing a column without telling anyone. Renaming amount to total_amount is a five-second edit that silently breaks a finance sheet you have never seen. Hence the consumer list.

Where this goes next

Next session is "An Environment That Starts": terminal, Git, uv, and Docker Compose assembled into a lab you can tear down and rebuild in one command. Ownership means little until you can rebuild the table from raw on demand.