Texas construction intelligence, assembled from twenty public data sources
zabalist.comEverything about who is building what in Texas is technically public and practically unusable. Permits live in a city Socrata portal, statewide project registrations in a TDLR feed, site plans behind a scraper-hostile municipal portal, procurement in a state bid board, parcel and tax data in three separate county appraisal districts with three different undocumented APIs. Zabalist is the system that pulls all of it together and makes it queryable.
A contractor who wants to know which developers are starting projects in their trade and their county has no single place to look. The information exists — it is just scattered across a city permit portal, a state licensing feed, a procurement bid board, three county appraisal districts, and a handful of trade-association member directories, none of which share an identifier, a schema, or a spelling convention.
Names are the core difficulty. The same firm appears as 'Kimley-Horn', 'KIMLEY-HORN AND ASSOCIATES INC', and a dozen other variants across sources. Any naive join produces either thousands of duplicate companies or silently merged distinct ones, and both failure modes destroy the product.
Then there is the shape problem: subdivisions, site plans, building permits and statewide licensing filings are four different data models describing overlapping real-world work, with different lifecycles and different field names for the same concept.
Austin Socrata permits and health inspections, Texas Comptroller receipts, statewide TDLR project registrations, the Austin Build + Connect site-plan portal, Travis, Williamson and Hays appraisal districts, ProdigyCAD enrichment, the state ESBD procurement board, Procore's contractor network, two homebuilder association directories, SBA and SAM.gov certifications, FRED economics, Google Places, and a geocoding fallback chain. Fifteen-plus numbered pipeline stages run as scheduled jobs with per-stage failure isolation, so one bad upstream feed degrades a slice rather than the nightly run.
Raw filing text becomes canonical companies through escalating tiers: Postgres trigram fuzzy matching handles the easy majority, then Claude or Gemini disambiguates the genuinely ambiguous pairs, and automatic entity creation is gated behind a people-count threshold so a typo cannot mint a phantom firm. pgvector is installed and embeddings back the site-plan company cohort; the broader directory is resolved by the fuzzy-plus-LLM path.
Subdivisions, site plans, permits and TDLR filings are unified into a single project entity with a work-items join table — 413,350 projects over 517,271 work items — plus viewport and heatmap RPCs so a map can render a bounded region without pulling the corpus into the browser.
Rather than a full audit log, Postgres triggers on four tables fire only when a tracked field actually moves — cost, status, contractor, owner, lot count — writing a previous-state JSONB snapshot behind a GIN index. The alert emailer diffs the snapshot to produce human-readable lines like a cost moving from $10M to $12.5M, which is what a subscriber actually wants to read.
The parts that were genuinely hard — and how they were solved.
Travis, Williamson and Hays each expose a different undocumented search backend. Hays uses a structured keyword query grammar that had to be derived from observed traffic; Williamson runs through a Tyler proxy API. Each district's request shape, and the wrong shapes that fail silently, is documented in the repository so the next person doesn't rediscover it.
Finding comparables widens in three stages — same subdivision and block, then the whole subdivision, then a fuzzy match across renamed and re-platted phases — with an early exit once twenty comparables are found. On a real Georgetown property that chain went 17, then 53, then 169 candidates, which is exactly the behaviour you want: precise when precision is available, broad when it isn't.
The appraisal engine derives dollar-per-square-foot and dollar-per-acre adjustment rates from the actual comparable pairs in each dataset rather than applying a constant, enforces a reliability threshold on total adjustment size, and reconciles across methods — following paired-sales analysis as appraisers practise it.
Headless Chrome with stealth patching and a serverless Chromium build, Playwright in CI, and a proxy API as fallback, running inside a six-hour scheduled job. Municipal portals actively resist automation; the pipeline is built to degrade and retry rather than to assume cooperation.
Ingestion runs on scheduled CI, but the expensive rollups — directory statistics, entity page stats, data-source freshness — run as pg_cron jobs inside Postgres on their own cadence. Read paths hit precomputed caches instead of aggregating 885,569 work-history rows on request.
Vector embeddings cover a specific site-plan cohort rather than the full 143,389-entity directory, and several built surfaces have little production usage yet. That distinction is tracked internally rather than smoothed over, because a data product that overstates its own coverage is a data product nobody can trust.