How we collect and check the data.
59 public datasets stored, as of the Oct 29 catalog. We date every change, store each capture as the source gave it, and have a separate reader check our figures.
The Marble Fleet Count has its own page: method, version 1.0.
How we collect
59 public datasets stored, as of the Oct 29 catalog. Every dataset points at a public source: an agency file, a public API, or a public web page. See every dataset.
Each dataset has its own schedule, shown in the data directory. The main registry run starts at 5:55 am Mountain. Later runs try again for anything that failed.
Each row shows how often we check it. Some sources are stored only when the publisher changes them; the row says which. On a quiet day we record that we checked and that nothing changed.
We ask politely. Before the first request to a site, we read its rules. We stay under a set number of requests per second. If a host says no, we stop and flag it. We do not work around a refusal.
We store the file as the source gave it, except where we remove information about private individuals, as in the FMCSA company census. Our tables are built from that stored copy, never in place of it.
How we date every change
Every change carries two dates.
- The day the source recorded it. This is the date inside the source’s own file.
- The day we first saw it. This is the date of the first capture that held it.
The two dates can differ. A source can post a change late. We never move the first date back to look better.
This is what point in time means here. You can ask what a record looked like on any day we hold. The answer is what the source said on that day, not what it says now.
How a capture is stored
Each capture is a dated folder. It holds the files, a manifest that lists every file with its size and, for tables, its row count, and a checksum file with one SHA-256 hash per file. The manifest is written last. A folder with no manifest is not a capture.
A stored capture is never overwritten. If a second copy differs by a single byte, the system stops and tells a person.
What the Auditor checks
Automatic checks run every day to see that each day’s capture landed, and a separate Auditor, who did not build the capture, checks our published figures. Figures in our reports and case studies are audited before release.
- Records and completeness. An automated check confirms that each dataset due that day left a record, and that each stored capture is complete: its manifest is there and lists every file.
- Figure check, each day. For the FAA registry, a separate script, written apart from the capture code, rebuilds the headline figures from the raw file and compares them with what we publish. The two must match exactly.
The Capture checked mark on our pages refers to the capture job’s check of the registry file, not to an audit.
How we handle a miss
A miss is a dataset that was due and was not collected.
We flag it to ourselves the same day. We name the feed, note the reason from the capture record, and record when it was fixed. If it is not fixed yet, it stays open.
A day with a miss stays marked as a miss, even after the fix. We never delete a miss or fill a gap with a guess. Some publishers drop old days at each update. For those feeds, a missed day cannot be recovered, and we say that.
What we cannot do
We count what a source publishes. We do not see what it leaves out.
History before our first capture is short. The date each feed began is in the data directory.
The FAA registry, in detail
The FAA aircraft registry is the flagship. It is published five days a week, and each file replaces the last. This is how we keep it.
Capture
- OvernightFAA posts
- 5:55Capture
- 6:30Retry
- 7:05Watchdog
- VerifiedKept
- The FAA posts a new file Tuesday to Saturday. No file is dated a Sunday or a Monday.
- A cloud job starts at 5:55 am Mountain and tries again at 6:30 and 7:30. A separate check at 7:05 flags the day if its capture is missing.
- Each capture keeps all seven FAA data files, with a SHA-256 checksum for each and a manifest.
- A capture is dated by the FAA file’s own date, never by our clock.
- Nothing stored is ever overwritten.
Snapshots
- Every row carries the date of its snapshot and the checksum of its file.
- Every change carries two dates: the FAA’s date, and the day we first saw it.
- History starts March 19, 2026. Anything earlier is reconstructed, and labeled as reconstructed.
Integrity gates
A file must pass every gate. A failure stops the load. Counts below are as of the capture of November 18, 2026.
- Complete. The FAA zip passes its integrity test. All seven data files are present. None is empty.
- Full size. At capture, the main file must hold at least 90% of the previous capture’s rows. At load, at least 90% of the median of the prior seven snapshots. No date is hard-coded.
- Checksum. A file whose hash differs from its manifest is refused.
- Field count. A line with the wrong number of fields is set aside, not loaded. 81 lines so far.
- Fill rate. Non-empty values per column must match between the source file and the table. 13,757 checks, 0 mismatches.
- Layout. A header that differs from the known layout is refused.
- Reconciliation. Across 146 periods between snapshots, registrations minus deregistrations must equal the net change in aircraft. 11,335 minus 3,661 is 7,674. It matches exactly.
“Verified” on the last capture line means the capture passed the first two gates before anything was stored.
Cloud and manual parity
- Two separate pulls run: the cloud job, and a manual pull on a separate machine.
- All seven data files are compared by SHA-256.
- The cloud capture counts as ready only after seven consecutive FAA publications with every file identical.
- The manual pull stays until then.
What we hold
The row of marks shows the FAA registry only. For each FAA publication in the last 30 days it shows whether we hold a stored copy: held, missed or refused, with a short dash on Sundays and Mondays, when the FAA posts no file. Held means stored. It does not mean every check passed that day. On September 30 the cloud download did not finish, and we hold a partial same-day copy from our second pipeline: four of the seven data files. Other feeds are not shown here; a missed day on any feed stays marked as missed.
In the last 30 days, through November 18, 2026, the ledger shows 21 files held, 1 missed. See every publication in the capture ledger.
When the FAA ships a broken file
It has happened two times. Each file was refused. A broken file is never repaired and never filled in. A day with no good capture stays missing, and the gaps are published.
Three years of registry history. 840 of 851 publications since August 16, 2023, through November 18, 2026. Year-end files back to 2010. Our own capture since March 19, 2026. Our own capture holds 117 of the 175 publications since it began. Since November 14, 2026, none has been missed and no files were refused. See the capture ledger
Limits
- Registration is not flight. The registry shows who holds an aircraft, not whether it flies.
- Drones registered under the FAA’s small drone rule (14 CFR part 48) are not in this file.
- Tying a registrant to a company is an inference. Every match carries its method, its confidence and its evidence.
Data directory · Capture ledger · The data · Compliance · Sources