Methodology
What our data actually says, what we chose to compute, and what we chose not to compute. Nothing here is presented without a source; nothing here is presented without a caveat.
Data as of 2026-08-13 15:35
Source inventory
Every source, its dataset ID, its license, its rate limit, its last successful ingest, and its coverage window. If a row is red, that source is failing. Alerts and portfolio counts continue to reflect the last-known-good snapshot for that source.
| Source | Dataset / endpoint | Access / cadence | Rows landed | Last success | Status |
|---|---|---|---|---|---|
| acris_master | bnx9-e6tj | NYC Open Data Socrata JSON Weekly | 15,980 | 2026-08-13 15:35 | Success |
| acris_legals | 8h5j-fqxa | NYC Open Data Socrata JSON Weekly | 59,749 | 2026-08-13 15:36 | Success |
| acris_parties | 636b-3b5g | NYC Open Data Socrata JSON Weekly | 16,333 | 2026-08-13 15:34 | Success |
| acris_pages | — | ACRIS GetImage (undocumented endpoint; DOF partnership sanctioned) Weekly (new docs only) | — | 2026-07-15 22:37 | Success |
| ll44_projects | ucdy-byxd | NYC Open Data Socrata JSON Weekly | 1,014 | 2026-08-13 15:39 | Success |
| housing_production | hg8x-zxpr | NYC Open Data Socrata JSON Weekly | 8,313 | 2026-08-13 15:40 | Success |
| j51 | y7az-s7wc | NYC Open Data Socrata JSON Monthly | 13,236 | 2026-08-13 15:40 | Success |
| property_exemptions | muvi-b6kx | NYC Open Data Socrata JSON Monthly | 99,774 | 2026-08-13 15:42 | Success |
| pluto_bbl_enrichment | geosearch_v2 | NYC Open Data Socrata JSON Quarterly | 108 | 2026-08-13 15:39 | Success |
| coredata_snapshot | — | Manually-exported dated snapshot (offline) Manual | 0 | 2026-08-13 15:42 | Success |
| nhpd_snapshot | — | Manually-exported dated snapshot (offline) Manual | 0 | 2026-08-13 15:42 | Success |
| extraction_worker | — | Anthropic Claude (Sonnet vision) Weekly (only new docs) | 932 | 2026-08-13 15:42 | Success |
| crosscheck_builder | — | Internal join (ACRIS extraction × LL44 × HP) Weekly | 82 | 2026-08-13 15:42 | Success |
| alert_builder | — | Internal (SQL rules) Weekly | 455 | 2026-08-13 15:43 | Success |
| document_link_resolver | — | Internal (heuristic on extracted references) Weekly | 10,535 | 2026-08-13 15:42 | Success |
| orchestrator | — | Pipeline orchestration Weekly | 3 | 2026-08-13 15:39 | Success |
| replay_backfill | bnx9-e6tj | Internal replay test Ad-hoc | 6 | 2026-07-13 20:41 | Success |
| acris_references | pwkr-dpni | Unknown Unknown | 13,710 | 2026-08-13 15:37 | Success |
| acris_remarks | 9p4w-7npp | Unknown Unknown | 1,591 | 2026-08-13 15:37 | Success |
Ingest run history
Every ingest run, including failures, with duration, rows landed, and errors. This product does not paper over a failed refresh.
| Started | Finished | Source | Kind | Rows landed | Duration | Status | Error |
|---|---|---|---|---|---|---|---|
| 2026-08-13 15:42 | 2026-08-13 15:43 | alert_builder | — | 455 | 39s | Success | |
| 2026-08-13 15:42 | 2026-08-13 15:42 | crosscheck_builder | — | 82 | 2s | Success | |
| 2026-08-13 15:42 | 2026-08-13 15:42 | document_link_resolver | — | 10,535 | 18s | Success | |
| 2026-08-13 15:42 | 2026-08-13 15:42 | extraction_worker | — | 932 | 2s | Success | |
| 2026-08-13 15:42 | 2026-08-13 15:42 | nhpd_snapshot | — | 0 | 0s | Success | |
| 2026-08-13 15:42 | 2026-08-13 15:42 | coredata_snapshot | — | 0 | 0s | Success | |
| 2026-08-13 15:40 | 2026-08-13 15:42 | property_exemptions | muvi-b6kx | 99,774 | 1m 48s | Success | |
| 2026-08-13 15:40 | 2026-08-13 15:40 | j51 | y7az-s7wc | 13,236 | 13s | Success | |
| 2026-08-13 15:39 | 2026-08-13 15:40 | housing_production | hg8x-zxpr | 8,313 | 5s | Success | |
| 2026-08-13 15:39 | 2026-08-13 15:39 | ll44_projects | ucdy-byxd | 1,014 | 2s | Success | |
| 2026-08-13 15:39 | 2026-08-13 15:39 | orchestrator | — | 3 | 1s | Success | |
| 2026-08-13 15:37 | 2026-08-13 15:39 | pluto_bbl_enrichment | geosearch_v2 | 108 | 1m 55s | Success | |
| 2026-08-13 15:37 | 2026-08-13 15:37 | pluto_bbl_enrichment | 64uk-42ks | 7,109 | 21s | Success | |
| 2026-08-13 15:37 | 2026-08-13 15:37 | acris_remarks | 9p4w-7npp | 1,591 | 25s | Success | |
| 2026-08-13 15:36 | 2026-08-13 15:37 | acris_references | pwkr-dpni | 13,710 | 55s | Success | |
| 2026-08-13 15:35 | 2026-08-13 15:36 | acris_legals | 8h5j-fqxa | 59,749 | 59s | Success | |
| 2026-08-13 15:34 | 2026-08-13 15:35 | acris_master | bnx9-e6tj | 15,980 | 55s | Success | |
| 2026-08-13 15:28 | — | acris_pages | — | — | — | Running | |
| 2026-08-13 15:28 | 2026-08-13 15:34 | acris_parties | 636b-3b5g | 16,333 | 5m 32s | Success | |
| 2026-08-13 15:20 | — | acris_parties | 636b-3b5g | — | — | Running | |
| 2026-08-13 15:20 | 2026-08-13 15:20 | acris_pages | — | — | 6s | Paused (rate policy) | ACRIS bandwidth-policy redirect on doc=FT_2270005959027 page=1; aborting entire run immediately. |
| 2026-08-13 15:07 | — | acris_parties | 636b-3b5g | — | — | Running | |
| 2026-07-28 03:30 | — | acris_pages | — | — | — | Running | |
| 2026-07-28 01:46 | — | acris_pages | — | — | — | Running | |
| 2026-07-20 13:20 | 2026-07-20 13:20 | acris_pages | — | — | 5s | Paused (rate policy) | ACRIS bandwidth-policy redirect on doc=2018122000415001 page=1; aborting entire run immediately. |
| 2026-07-19 10:08 | 2026-07-19 12:37 | acris_pages | — | 1,900 | 149m 42s | Partial | |
| 2026-07-18 16:18 | 2026-07-18 18:45 | acris_pages | — | 1,944 | 147m 26s | Partial | |
| 2026-07-17 12:34 | 2026-07-17 12:34 | alert_builder | — | 455 | 38s | Success | |
| 2026-07-17 12:34 | 2026-07-17 12:34 | crosscheck_builder | — | 82 | 2s | Success | |
| 2026-07-17 12:33 | 2026-07-17 12:34 | document_link_resolver | — | 10,309 | 18s | Success |
Extraction pipeline
- ACRIS Master identifies HPD / HDC agreements by document type (AGMT, TERA) and party name normalization.
- ACRIS Legals resolves the recording to one or more BBLs (borough-block-lot).
- ACRIS Parties normalizes to canonical HPD/HDC entity variants (case, punctuation, LLC suffixes).
- GetImage fetches each page as a TIFF. Placeholder pages are flagged and skipped.
- Vision LLM (claude-opus-4-6 by default) reads the scan and returns field-level extractions with self-reported confidence, page number, source quote, and bounding box in normalized [0,1] page coordinates.
- Cross-check joins the extracted horizon fields to LL44 program-level publications and Housing Production building-level records.
- Horizon adjudication decides which value the header displays when sources disagree; the losing values remain visible on the building page.
Every field the model returns is stored with a source_run_id. Reruns append; they do not overwrite. Historical runs remain viewable.
Horizon methodology
A term is expressed one of four ways:
- Absolute date:“expires December 31, 2039.” Bucketed directly against today.
- Relative to observed event:“25 years after project completion (June 3, 2010).” Resolved into an absolute date.
- Relative to unobserved event:“25 years after project completion (project not yet observed).” Bucketed as needs base date and never silently invented.
- Perpetual / permanent: displayed as such, bucketed 10+ years.
Non-terminated terms only are considered for the header. Terminated terms remain in the timeline for auditability.
Cross-check
Three sources measure related but different things:
- ACRIS is the text of the recorded regulatory agreement. It is the legal document.
- LL44 (HPD) is an HPD publication of program-level data. Its unit of analysis is the project, not the building.
- Housing Production is a building-level record of completed affordable units.
Discrepancies are surfaced as discrepancy_flag on the field. Neither value is silently discarded. The comparability matrix below documents which fields are and are not comparable across sources.
Adjudication policy
When ACRIS and LL44 disagree on a horizon-relevant field, the header shows the ACRIS extraction, because it's the legal document. The cross-check panel on the building page shows both values, side-by-side.
When ACRIS confidence is below 0.6 and LL44 has a matching value, we still show the ACRIS value in the header, but visibly de-emphasized with a lower-confidence pill. Both remain in cross-check.
Comparability matrix
| Field | ACRIS | LL44 | Housing Prod. | Comparable? |
|---|---|---|---|---|
| Unit count | Legal units at recording | Project total | Building-level | Yes when BBL is 1:1 with project |
| Expiration date | Absolute or relative | Publication-year plus term | Not available | Approximate; expect ±90 days |
| Program name | Free-text | Enumerated | Not present | Fuzzy match only |
| Affordable unit mix | AMI + bedroom counts | Not always published | Not available | Partial |
Scan corpus scope
We do not have the disk budget or the render throughput to hold every ACRIS scan for every HPD / HDC record ever recorded. We hold: (a) every HPD/HDC recording in the selected full-scan neighborhoods and (b) every alert-triggered recording. Everything else is metadata-only until requested, and a Scan not yet in corpus badge makes that visible.
ACRIS GetImage caveat
ACRIS GetImage returns a placeholder TIFF for a small number of officially-recorded instruments (misfiles, redactions, or scans held under privacy holds). The extractor flags these pages and the document's scan status becomes undocumented. The metadata (party, doc type, date) still contributes to portfolio counts, but no horizon or unit-count field is emitted.
Unreadable scans
A scan is marked unreadablewhen OCR quality confidence falls below the retry threshold across at least three consecutive pages. Unreadable scans surface as a portfolio alert; the corresponding building card offers a “Rerun extraction” action but does not silently guess.
Needs base date
“35 years after project completion” is not a date until we observe the completion date. Records with unobserved base events are bucketed needs base date and cross-checked against Housing Production and DOB completions. If a match is found on later ingest, the record moves out of this bucket automatically; the original relative expression is retained in the audit log.
HPD internal firewall
Some HPD program-level fields (recapture triggers, subordination priorities, exact AMI mix by bedroom) live behind HPD's internal firewall and are not published to open data. This product does not scrape internal systems. When a field is known to exist internally but not on open data, we mark it hpd-internal rather than guess or blank it.
Offline snapshots
When an upstream source is unavailable at ingest time we retain the last-known-good snapshot rather than dropping rows. Every table view stamps the snapshot date; every stale banner names the source that failed. The ingest run history above is the audit log for what fell over and when.
Pre-1966 records
ACRIS coverage begins in 1966; older recordings live in reel-and-page microfilm at the City Register. We display a “chain runs off the edge of ACRIS coverage” marker rather than pretend the chain started with the earliest ACRIS record.
Staten Island
Staten Island (Richmond County) records live at the Richmond County Clerk, not at ACRIS. This product does not attempt to combine those sources. Staten Island BBLs return an in-scope-not-in-scope empty state.
Alerts
Alerts fire only on ingest deltas. If a BBL matches your rule but was already matching before the last ingest, no alert. This means alerts are cheap on a static snapshot and loud when the world actually changes.
Suppression: per-alert, either one ingest cycle or 30 days. Suppression is per-user, and per-rule.
Confidence & ranges
Confidence is self-reported by the extraction model, bucketed into low (<0.6), medium (0.6-0.85), and high (0.85+). Confidence is a signal about the model, not about the document. A high-confidence extraction of a garbled scan is still a high-confidence extraction of a garbled scan.
Party-name dictionary
HPD and HDC entities appear in ACRIS as free-text party names. We normalize variants (case, punctuation, LLC suffixes, historical name changes) against a published dictionary. If a variant is unknown, the record is stored but excluded from portfolio counts until an operator adds the mapping.
Adjudication history
The table below shows the last 200 adjudicated cross-check decisions with the extracted and reference values side-by-side and the operator's reason. Every decision persists; nothing is silently discarded.
LLM security
The Ask endpoint is scoped to a single document: model context is populated only with page text and metadata for the document you are viewing. External URLs are not followed; user prompts cannot exfiltrate other documents. All Ask traffic is rate limited per session and the endpoint responds 401 when the environment is token-gated. Model output is stripped of HTML before rendering.
Notification channels
Alert deliveries currently support in-app queue only. Email and Slack channels are planned but disabled in this build. When they land, delivery routing and per-user digest windows will appear on the alert rule editor; nothing is emailed today regardless of what a rule is configured to do.
Legal disclaimer
This product surfaces the public data record for informational purposes. Nothing on this site is legal advice, and nothing here supersedes the recorded instrument or the applicable regulatory agreement. Users are responsible for confirming a horizon, unit count, or program status against the primary source before making a decision that relies on it.
Changelog
Substantive changes to the methodology are logged in the git history of this application. Because this product is a public interface to a public dataset, the methodology is subject to change without notice as source datasets change; the ingest history above is your best guide to what actually changed and when.