This content originally appeared on HackerNoon and was authored by Mandar
\ \ In August 2025, the Gifford Fire burned more than 131,000 acres of the Los Padres National Forest, the largest wildfire in California that year. At the height of it, nine air tankers were working the fire on a single day, and it was only fifteen percent contained. InciWeb, the federal incident clearinghouse run by the Forest Service, recorded the Gifford Fire's acreage, containment percentage, and personnel count in daily updates. FlightAware tracked every one of those tankers in real time, by tail number, as they flew retardant runs out of air tanker bases across the state. Both datasets were public. Neither talked to the other.
That gap matters because aerial firefighting resources are finite and the allocation problem is real. A tanker committed to a fire at 85 percent containment is a tanker not available to the ignition two hundred miles away that is still at zero. You cannot reason about that allocation problem without a system that correlates aircraft activity with incident state, and that system does not exist as a standard government data product. I built one. This article covers the three engineering decisions that shaped it: why I used GitHub as a time-series data store, how the geospatial flight-to-fire matching works, and what the cascading cron pipeline looks like when two scrapers share a data dependency and you want neither to race the other. The live dashboard is running here, built on the architecture described below.
The Data Gap That Costs Response Time
InciWeb and FlightAware are both public, both authoritative, and never designed to talk to each other. InciWeb is managed by the USDA Forest Service and updated by incident management teams in the field. Its data reflects ground truth: how many acres are burning, what percentage is contained, how many personnel are assigned. FlightAware is a commercial flight tracking service with ADS-B coverage across the continental United States. It knows where every registered aircraft is, at what altitude, and with what tail number.
The operational consequence of that gap is that no one outside a specific incident command has a real-time view of which aircraft are working which fires and whether that allocation matches current containment state. Researchers studying aerial firefighting effectiveness, and policy analysts evaluating tanker fleet sizing, are working from after-action reports and NIFC seasonal statistics rather than day-of operational data. This pipeline is an attempt to close that gap at the research level.
Pipeline Architecture in Three Minutes
The pipeline has four stages that run on a daily automated schedule.
The first stage is an InciWeb scraper that pulls active wildfire incident data, including acreage, containment percentage, coordinates, and personnel count, and appends it to per-incident CSV files stored in a GitHub repository.
The second stage is a FlightAware scraper that pulls current flight data for aircraft tail numbers associated with the U.S. aerial firefighting fleet, specifically the Very Large Air Tankers and Large Air Tankers tracked in NIFC records.
The third stage is a flight matcher that runs after both scrapers complete. It takes the current flight positions and matches each aircraft to the nearest active incident within a 30-kilometer radius, filtering out incidents above 80% containment on the assumption that those fires are no longer drawing active aerial support.
The fourth stage writes the matched output and updates a pre-aggregated summary file that the dashboard reads on load. The dashboard itself runs client-side on Vercel and pulls data directly from the GitHub repository via the Contents API.
\
\
GitHub as a Time-Series Database
The first architectural question is where to store the data. The obvious answer is a database. Postgres with PostGIS handles time-series geospatial data well. I use it in production at Land IQ for exactly this class of wildfire and agricultural parcel data. But for this project I chose GitHub, and the reasons are specific enough to be worth stating plainly.
The use case has three properties that make GitHub surprisingly appropriate.
First, the data is append-only. Each daily scrape of InciWeb adds one new row to a per-incident CSV file. There are no updates, no deletes, and no random-access writes. The access pattern is sequential reads and sequential appends, which is the pattern GitHub's object storage handles without complaint.
Second, the data needs to be accessible from a browser without a server-side query layer. The dashboard runs client-side on Vercel. Introducing a server just to proxy CSV reads against Postgres or S3 adds infrastructure to maintain, a connection pool to tune, and an IAM policy to rotate. GitHub's Contents API already does authenticated file retrieval with a simple REST call, and Vercel's serverless functions can wrap that call to keep the token out of the browser.
Third, historical versions are intrinsically valuable in a way that most storage backends do not give you for free. Git's immutable commit history means every daily snapshot is automatically versioned. When InciWeb quietly changed how it reports containment percentages during a mid-season data migration, I caught it by diffing the CSVs in the commit history. A database would have overwritten the prior values.
The implementation stores per-incident CSVs in an inciweb_data directory. Before each append, the scraper reads the last row of the existing file and diffs 12 specific fields: acreage, containment percent, personnel count, updated timestamp, incident commander, and others. If none of those fields changed, the scraper skips the write. This is not primarily a bandwidth optimization. It prevents no-op commits from polluting the git log. In practice, the commit history becomes a meaningful signal: a burst of daily commits on a given incident indicates that fire is actively evolving. A week of silence on a file means that incident has not changed, and the dashboard can treat the most recent row as current without fetching intermediate snapshots.
The read path uses a Vercel serverless function that wraps the GitHub Contents API. The function handles two modes: list mode returns a directory listing used to populate the incident dropdown, and raw mode fetches the full CSV for a selected incident. Path traversal is blocked before any API call. The GitHub API allows 5,000 authenticated requests per hour per token, which is well above single-user dashboard usage. It becomes a constraint if concurrent users start batching all incident files on load, which is why the dashboard only fetches data for the selected incident rather than eagerly prefetching all CSVs at once.
The primary limitation of this architecture is query performance. There is no query engine in the loop. To compute aggregate metrics across all incidents, I cannot run a SQL GROUP BY against the stored files. Instead, a post-processing script runs after each scrape, reads all per-incident CSVs sequentially, and writes a pre-aggregated summary file. The dashboard reads that file for its summary cards. This is the materialized view pattern implemented manually, without the convenience of a database trigger. If this pipeline needed to support ad-hoc queries or serve enough concurrent users to hit the GitHub API rate limit, I would move to a proper data store. The current architecture trades query flexibility for operational simplicity, and that is the right tradeoff for a tool that one person maintains and that primarily serves research rather than production incident management.
Geospatial Correlation: Matching Aircraft to Active Incidents
The flight matcher takes a list of current aircraft positions and a list of active incidents with coordinates and runs a nearest-neighbor match using the Haversine formula.
Haversine gives the great-circle distance between two points on a sphere given their latitude and longitude. For distances under a few hundred kilometers it is accurate enough that the approximation error is irrelevant compared to the imprecision in the underlying data. The 30-kilometer radius threshold came from looking at typical tanker operating patterns around incidents in the NIFC data: aircraft working a fire rarely stray more than 25 to 30 kilometers from the perimeter during active operations, and beyond that distance the assignment becomes ambiguous.
The 80% containment filter exists because incidents above that threshold are statistically unlikely to be receiving active aerial support. Keeping them in the match pool produces false positives where aircraft transiting between assignments get incorrectly attributed to a nearly contained fire nearby. Filtering them out reduces noise at the cost of occasionally missing a late-stage aerial operation, which is an acceptable tradeoff for a research tool.
De-duplication across scraper runs required a composite key rather than a timestamp alone. The InciWeb scraper runs daily but InciWeb itself does not always update every incident every day. Using a timestamp key would create duplicate rows whenever the scraper ran without new incident data. The composite key combines incident ID, date, and a hash of the 12 tracked fields. If the hash matches the prior row, the write is skipped. This keeps the CSV history clean and makes the commit log meaningful as a signal of actual incident activity rather than scraper activity.
Cascading Cron Design: Why Timing Is a Data Dependency
The pipeline has two scrapers and one matcher. The flight matcher depends on fresh incident data from the InciWeb scraper. If the matcher runs before the InciWeb scraper finishes, it operates on stale incident coordinates and produces incorrect correlations.
The naive solution is to run all three jobs sequentially in a single cron. The problem is that the InciWeb scraper and FlightAware scraper are independent and can run in parallel. Forcing them to run sequentially adds unnecessary wall-clock time to a pipeline that is already time-sensitive because flight data ages quickly.
The solution is a cascading cron with a 2.5-hour gap between the scrapers and the matcher. Both scrapers run in parallel at the same start time. The matcher runs 2.5 hours later. That gap is not arbitrary: it is based on the observed worst-case completion time for both scrapers across six months of daily runs, with a buffer for network latency and GitHub API rate limit backoff.
The alternative would be an event-driven trigger where the matcher starts only after both scrapers emit a completion signal. That is the correct architecture for a production system. For a one-person research pipeline where operational complexity is itself a maintenance cost, a time-based gap with observed headroom is the pragmatic choice.
What Six Months of Daily Runs Actually Shows
Daily output across a season surfaces patterns that after-action reports don't, and the clearest is that aerial activity tracks the uncontained phase of a fire, not a mid-containment band. Across more than 900 matched aircraft-day observations, nearly half occurred while the fire was at zero percent containment and about two-thirds below twenty percent; only about one percent occurred above eighty percent, which is exactly why the 80% filter discards almost nothing. The planes show up for the fight, not the mop-up.
The Gifford Fire from the opening is the clean illustration. Eleven air tankers were matched to it by tail number, peaking at nine on a single day on August 8, at fifteen percent containment, with the fire still growing past 99,000 acres. As containment climbed through the twenties and thirties the matched count fell, nine, eight, seven, six, one, and after it crossed sixty percent on August 15, not one aircraft was matched to the fire again, even as it sat on the active list at 95-to-97 percent containment for months.
Two limits are worth stating. The 30-kilometer radius breaks down in complex terrain, where a canyon fire can put an aircraft's GPS track outside the radius because the straight-line path to the incident coordinate crosses a ridge the plane has to fly around, a known weakness of point-based Haversine matching that fire-perimeter polygons would fix. And the pipeline captures position, not sorties: an aircraft that made six drops and returned to base before the scrape never appears. It records who is where, not how many times.
What This Is and What It Isn't
This tool is a research instrument, not an operational one. It is designed to surface patterns across fire seasons at a level of granularity that NIFC seasonal reports do not provide. It is not designed to inform real-time tanker dispatch decisions, and it should not be used for that purpose. The data latency, the snapshot-based flight matching, and the absence of sortie counts all make it unsuitable for operational use.
What it is suitable for is the kind of question that motivated it: across a fire season, which incidents drew the most sustained aerial attention, and does that attention correlate with containment outcomes? That question requires day-level resolution across dozens of concurrent incidents, which is exactly what the pipeline produces.
The next version, if there is one, would replace point-coordinate incident matching with fire perimeter polygon matching using NIFC's published GeoJSON boundaries. That change alone would substantially improve the accuracy of the flight-to-fire correlation, particularly for large fires in complex terrain where the incident command post coordinate and the active fire perimeter can be 15 or more kilometers apart.
Wildfire operations generate more data than most incident management teams can consume. InciWeb, NIFC, FAMWEB, and public flight tracking together produce a continuous stream of incident state, resource allocation, and aerial activity. My research at George Mason University, and the modeling work published in MDPI Fire in 2025, kept pointing to the same constraint: it is not the availability of data that limits response quality, it is the latency of cross-domain correlation. An aircraft assigned to a fire at 90% containment is unavailable to the ignition two hundred miles away. This pipeline does not solve that allocation problem. It makes the inputs visible in a form where the problem can finally be stated precisely. That is where the work starts. As I write this in June 2026, the pipeline is tracking the Dellenbaugh Fire in Utah, 766 acres, zero percent contained, its first air tanker matched two days ago. The system is running today.
The pipeline and dashboard are live here for anyone who wants to explore the data directly.
\
This content originally appeared on HackerNoon and was authored by Mandar
Mandar | Sciencx (2026-06-24T06:14:52+00:00) Using GitHub as a Wildfire Time-Series Database: A Geospatial Aviation Pipeline. Retrieved from https://www.scien.cx/2026/06/24/using-github-as-a-wildfire-time-series-database-a-geospatial-aviation-pipeline/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.