Methodology

How this product is built, what public data it uses, how it stitches identifiers, how it trains and scores, and what it deliberately does not do. Every choice is intended to be recoverable from `capital_risk_source_config`, `capital_risk_ingest_runs`, and `capital_risk_model_runs`.

Concourse research artifact. Not an official DDC risk score or vendor rating. Model outputs are correlational, not causal.

1. Problem and scope

NYC DDC manages over 1,600 active capital projects (schools, plazas, streets, sewer work). Every project is a coordination between design, ULURP, agency review, weather, permits, and construction. Delays and overruns are frequent. Historically, the only visible signal has been quarterly Capital Projects Snapshot updates plus one frozen 2024-01 dataset of labeled reasons.

The Capital Risk product answers, for every active DDC project, three questions from public data alone: how likely is a delay in the next 12 months, how likely is a cost overrun of >10%, and what does the historical distribution of completion time look like for similar projects. It also names the top drivers per project, based on historical fits.

2. Sources and access modes

Seven external public sources, all NYC Open Data or Checkbook NYC:

fb86-vt7u — Capital Projects Snapshot (city). FMS IDs, project names, agencies, current budgets, spend-to-date, reporting periods.
qj5n-h5qp — Capital Projects Snapshot (non-city). Complementary budgets, plans, community boards.
s7yh-frbm — Capital Projects Milestones. Per-task sequence, original vs actual start/end, borough, managing agency. Frozen 2024-01.
wa2y-rh4b — Capital Projects Delays (Reasons). Labeled `delay_desc`, category, overrun percent, scope text. Frozen 2024-01.
fi59-268w — Capital Projects Detail (CPDB). Rich per-project descriptions used for stitch signals.
rukc-mmqu — DDC-managed infrastructure roster.
checkbook_ddc_contracts — Checkbook NYC contract spend for DDC-managed projects. Vendor name, original + current amount, mod percent.

Access modes and freshness live in `capital_risk_source_config`. Every fetch appends a row to `capital_risk_ingest_runs` with started/finished timestamps, row counts, and any warn/error notes. If a source is frozen, the source_config row carries a `frozen_disclosure_text` that is displayed on every page that consumes that source.

3. How records are stitched

The FMS ID is the current-day primary key. The 2024-01 legacy dataset uses PROJECT_ID. Every stitch decision is recorded in `capital_risk_projects.stitch_status`, `stitch_confidence`, `stitch_method`, and `stitch_notes`, so anyone can see how a project was joined and rebuild the stitch offline.

Six stitch statuses:

exact_id_stitched — FMS ID appears in both current and legacy datasets.
exact_id_and_contract_linked — Above plus Checkbook contract linkage.
secondary_stitched — Matched on a secondary key (project name + agency + borough tuple with high similarity).
secondary_and_contract_linked — Above plus contract linkage.
unstitched_no_legacy_data — Active FMS project that has no historical row.
unstitched_no_contract — Active FMS project with no Checkbook link.

4. Targets, features, and lagging

Three modeled targets:

Delay probability — likelihood the project experiences a slip >90 days over the next 12 months.
Overrun probability — likelihood the project's current budget will exceed 110% of its original within its remaining phase.
Survival — full distribution over predicted completion date, from a survival origin milestone.

Features are grouped: project attributes (budget bucket, borough, agency, phase, plan category), historical stitched signals (past slippage, past overrun percent, past modification percent), and Checkbook contractor proxies (contract count, mod percent max, current-to-original ratio).

Every feature that is not knowable at prediction time is excluded or lagged. The excluded/lagged list is persisted in capital_risk_model_runs.leakage_exclusion_json and rendered verbatim on the model card.

5. Model choice and validation

Delay and overrun are gradient-boosted binary classifiers with time-based cross validation (folded by reporting period so a training fold never sees a future reporting_period). AUC and precision at recall 0.5 are the primary metrics; MAE is reported for the regression heads. Survival uses a discrete-time hazard formulation per milestone. All persistence is under a single model_run_id.

Baselines are logistic regression on borough + phase + budget bucket alone. Both baseline AUC and baseline MAE are published in the same metrics_json so the gain over baseline is immediate on the model card.

6. Risk thresholds and combined score

The risk pill (low / moderate / high / critical) is a per-target thresholding of the raw probability. Bin edges live in capital_risk_model_runs.metrics_json.threshold_policy and are the same across the portfolio.

The combined risk score is w_delay · delay_probability + w_overrun · overrun_probability. Weights live in capital_risk_model_runs.combined_risk_weights and currently default to 0.5 / 0.5. Weights are stored, not baked in, so a future run can change them without a code change.

7. Unscored policy and reason codes

Projects that cannot be scored fairly are marked unscored. The row still exists in capital_risk_predictions with scoring_status set to one of unscored_insufficient_history, unscored_incomplete_stitch, or unscored_model_unavailable. A plain-language reason is rendered on the project page and the model card.

When a comparable historical cohort is below capital_risk_model_runs.cohort_policy.min_cohort_size, the top-factor explanation switches to the persisted fallback text rather than a per-project narrative. This is intentional to avoid overclaiming.

8. Coverage honesty

Three coverage metrics are pinned to the top of every portfolio view:

FMS-to-legacy stitch coverage (PASS threshold ≥60%).
FMS-to-Checkbook contract coverage (soft target ≥50%).
Portfolio scoring coverage (target ≥70% of active DDC projects with a scored prediction).

If any threshold fails, the portfolio filter bar shows a below-target chip and the model card records the shortfall. Coverage gaps are not hidden behind a friendlier number.

9. Leakage controls

Any feature that would let the model see the future is either excluded or lagged. Both lists are persisted in capital_risk_model_runs.leakage_exclusion_json and rendered verbatim on the model card.

current_phase — excluded. Phase is updated together with the outcome we are trying to predict; including it leaks the label.
projected_construction_completion — excluded. This is the human-forecasted completion date and is exactly the label the model is trying to compete with; using it as a feature would trivialize the survival head.
latest_spend_to_date and any forecast-completion-derived ratios — lagged by at least one reporting period, so training folds cannot see the same period as their label.

The training pass runs an assertion at fit time: if any feature marked excluded is present in the training frame, the run fails with error='leakage assertion'and does not publish. See the model card for the current run's policy.

10. Refresh cadence and freshness

Capital Projects Snapshot publishes ~3x/year. Milestones and Delay Reasons are frozen 2024-01. Checkbook is pulled at a lower cadence and marked stale after 30 days without a fresh pull. The freshness banner at the top of every page names the source that is stalest for that view.

Last observed publish for headline sources:

Capital Projects Snapshot (city): 2026-07-15 · reporting_period 202605
Checkbook DDC contracts: 2026-07-15
Milestones + Delay Reasons: Legacy data frozen 2024-01

Refreshes are triggered by re-running the ingest scripts under scripts/capital-risk/. Every re-run appends to capital_risk_ingest_runs.

11. Limits (what public data cannot support)

The public build cannot answer several questions truthfully. It states them here rather than approximate:

PASSPort / VENDEX contractor performance: not available on NYC Open Data. Vendor reliability is proxied by Checkbook modification percent, which is a coarse substitute. Connects when inside the firewall.
Change-order line items: Checkbook exposes current vs original amount only; the reason for a change (scope growth, differing site conditions, price escalation) is not in the public feed. Connects when inside the firewall.
3x/yr snapshot cadence: Capital Projects Snapshot publishes roughly three times per year. Between publishes, delay probability updates depend only on Checkbook and any secondary source that refreshed. Continuous phase progress connects when inside the firewall.
Frozen 2024-01 legacy: the delay-reason taxonomy has not been re-published since January 2024. No new labeled reasons are added to the historical corpus. Every page that consumes wa2y-rh4b or s7yh-frbm renders a frozen tag.
Partial stitching: some active FMS IDs never appear in the frozen legacy dataset. Those projects show as unstitched_no_legacy_data and remain unscored rather than getting a fabricated prediction.
Non-DDC managing agencies: the delay-reason page limits scope to DDC-managed rows in v1. Citywide labels are available in the public source but are not ingested until the next iteration.

12. Inside the firewall

Public data is intentionally lagged and coarse. Inside the DDC firewall, three substitutions immediately upgrade quality: continuous internal phase milestones replace the quarterly Capital Projects Snapshot; contract amendment reasons and dollar deltas replace the raw Checkbook total; an internal delay taxonomy replaces the frozen 2024-01 reasons. See inside-firewall factors on the model card ›

14. Change log

No documented changes yet.