FireNet-2Hz: the design as it stands, and why
FireNet-2Hz is a neural network that learns to reproduce a production fire simulation (FDS) around a house, one 0.5 s step at a time. The 29 October draft described a plan. This page describes the model now in training: what it reads and predicts, how it is trained and scored, what was tried and dropped with the measured result, and what is still open. Items marked changed differ from the draft. Items marked open still need a decision.
- The core is the same U-Net. Native FDS blocks use halo convolutions, 32 channels, 2 layers per side. The 4, 16 and 64 m levels are dense grids. The U-Net now runs in bf16; the carried state, the heads and the state update stay fp32.
- Faces are trained on FDS's own wall-cell values, recovered by inverting FDS's corner rule. The face set now includes faces across mesh-block boundaries and the inert ground strip next to the house. With these, FDS's own values run through our FER path land within 1.6% of production FER on all 9 test runs (the bar was 3%).
- Training uses 216 runs with 2-step windows. From step 30k, half the windows are pushforward samples (up to 20 steps without gradient). The pushforward HRR term made the model diverge twice. Its weight is now 0 while a replacement is chosen.
- On 4 H100s, 4 windows per GPU, balanced by size, with bf16 and a Triton CSR kernel, the rate with pushforward went from 11.4 to 16.2 windows/s. The bottleneck is now the 66–81 s scene switch.
- The draft's wind rotation, built-in PMMA burning curve, 1 → 20-step curriculum and one-scene milestone (M1) are not part of the current design.
One step: state at t → state at t + 0.5 s
Each input is one of three things: fixed geometry, forcing from the scenario file, or something the model produced one step earlier. Anything else would be unavailable during a rollout.
The level depths follow measured transport. Hot gas moves 3–7 m per 0.5 s step at the median and 6–25 m at the 99th percentile, so one step can cross many native cells. Native blocks keep FDS's cell sizes. The coarse levels are plain dense grids anchored on 4 m cells, so they run as ordinary cuDNN convolutions.
Why bf16 only inside the U-Net changed. The draft fixed fp32 (TF32) everywhere. A scale check on 40 runs found that bf16's range is fine for every field. Its precision is fine inside the network, but not for the carried state or the running sums. Running the state in bf16 too was tested: fuel mass drifted −0.16% on every run, and FER got worse on 14 of 20 runs. So only the U-Net's activations are bf16. See throughput.
The grid: FDS's blocks, corner values, and a one-cell halo
Click or hover a coloured halo corner to see which neighbour values fill it.
FDS splits space into blocks of cubic cells, with small cells near the house. Production output stores corner values, so the model works on corners. A 16³-cell block is a 17³ corner grid, which the halo pads to 19³.
Inside a block every cell has the same size, so the 3×3×3 kernel runs as usual. Before each native layer, each block's halo is filled from its neighbours, converted to this block's spacing:
- Same size or coarser neighbour: trilinear interpolation in the neighbour. Where the points line up, this is an exact copy.
- Finer neighbour: φV-weighted average of the fine corners inside this corner's box.
- Outside the domain: the first layer gets free-stream values (ambient gas, the wind, and the nearest corner's statics). Deeper layers get a learned fill, one for open sides and one for the ground.
How it runs changed. The halo and the interior gather are one sparse matrix product per layer, P = A x, whose backward is Aᵀ g. This avoids the atomics and the slow sort-based scatter that the first code spent much of its time in. Short-row products go through a small Triton kernel.
Pick a halo corner.
Alternatives (OctNet, mixed depth, graphs) are compared in convolution on cells of different sizes. Mixed depth is a separate follow-up (SCI-6238). Refreshing the halo only every second layer was tested and dropped: it was ×1.12–1.18 slower and changed the output by 10%.
What is predicted, derived, or dropped
Gas (on every corner)
Δ vs absolute. Slow, transported fields are predicted as a change. Fast or on/off fields (heat release, radiation, fluxes, burning rate) are predicted outright.
Units. Each field is put through asinh or log before the network sees it, for example asinh((T − 20 °C)/50 K) and asinh(HRRPUV / 100 kW/m³). Each output is in units of that field's spread on the training runs.
Dropped. Pressure is a low-Mach diagnostic. Radiation emission is mostly a function of T, and G carries the radiation target. The draft listed this as open; it is now settled.
Floors, no ceilings. Straight-through clamps keep T, Z, HRRPUV, q_inc and burning rate physical while still passing gradient. G is positive by construction through the log.
Where the combustion heat comes from
Vegetation particles carry almost no heat, so they are skipped. The model predicts the two real fuels, the ground surface fuel and the PMMA faces. The target is production FDS as it runs, so the model does not correct the near-massless trees. Fireline and ignitors are prescribed by the scenario and enter as forcing.
Faces changed
| Field | Main house (inert) | Ground strip ≤ 2 m from the house (inert) new | PMMA buildings | Ground fuel |
|---|---|---|---|---|
| Wall temperature | held at ambient | held at ambient | predicted Δ · weight 1 | predicted Δ · weight 1 |
| Convective heat flux | predicted · 0.5 | predicted · 0.5 | predicted · 0 (via net) | predicted · 0.25 |
| Incident heat flux | predicted · 0.5 | predicted · 0.5 | predicted · 0 (via net) | predicted · 0.25 |
| Radiative heat flux | derived | derived | derived | derived |
| Total (net) heat flux | derived · 1: the FER quantity | derived · 1 | derived · 1 | derived · 0.25 |
| Burning rate | – | – | predicted ≥ 0 · 1, gated by ignition | predicted ≥ 0 · 1 |
| Fuel mass | – | – | integrated, never limits the burn | integrated, never increases · 1 |
The numbers after each dot are the per-channel weights in the face loss. Teal: predicted and fed back. Blue: derived from predictions. Gray: fixed. Targets are FDS's native per-face wall-cell values (see scoring), no longer the mean of a face's 4 stored corners.
- Why the ground strip. Production's FER point set includes the ground cells inside the main-building footprint. Those hold 0.2–17.8% of production FER on the 9 test runs, and they sit within 1.2 m of a house face. Inert ground within 2 m (in plan) of a house face is now a model face, held at ambient like the house.
- Why PMMA mass never limits the burn. FDS burns PMMA by its heat-release ramp, with no burn-away. One run burns 11.2 kg/m² against a density × thickness of 7.7. So the model's PMMA mass starts at a value it cannot exhaust, as FDS's does. A deck with burn-away fails loudly instead.
- Emissivity. House, inert ground and PMMA use 0.9, as FDS does, so the derived net flux equals FDS's TOTAL HEAT FLUX on the house (max error 1.5·10⁻⁴ kW/m²). Ground fuel keeps 1.0 for now; its FDS fluxes imply 0.44–0.56. open
Faces: a list with a per-face head settled
FDS solves each wall cell as its own 1-D heat-conduction problem, coupled to its neighbours only through the gas. So the head is pointwise, with no face-to-face convolutions.
It reads the same decoder features that produce the gas at t + Δt, so the walls see the model's own next-step gas.
Fluxes use FDS's own identity, so net flux always agrees with its parts:
q_net = q_conv + q_rad ← FER
m ← max(m − Δt · ṁ″, 0)
PMMA burning: learned, behind a hard ignition gate changed. The draft leaned toward building in the scenario's burning curve. The code instead learns the burning rate from the face's features and memory. A PMMA face may burn only once its predicted wall temperature has reached 371 °C at this or an earlier step, and it then stays ignited. Ignition causality therefore holds by construction, and the curve does not have to be hard-coded per deck.
Faces across block boundaries new. A solid cell in one FDS mesh block next to a gas cell in the neighbouring block now makes a face, kept with its gas cell as FDS keeps it. On one run this added 2,905 faces, every one on a face centre FDS stores. Without them, house walls that start on a block boundary were missing.
Forcing and time: read forcing at the step being predicted
Forcing (fireline, ignitors, wind, Δt) is read at the frame that ends the step. Earlier modular models read their boundary inputs one step late and needed a post-processing shift to fix it.
Δt is an input because FDS writes at the first step past each 0.5 s mark (for example 1.01 s).
No absolute time. FDS is time-invariant given state and forcing. A clock input would invite shortcuts like "the structure burns at 60 s" that don't transfer to new scenes. The fireline channel is the deck's ramp times the fireline's density, which is forcing, not a clock. What FDS keeps but doesn't write lives in face memory: the ignited flag, incident energy since ignition, and cumulative net heat (the dose).
Wind rotation was not built not in the design. The draft proposed turning each scene by one of the 8 grid symmetries so that the wind points within 0–45° of +x. The current code does not do this. Wind enters as u, v in the scene's own frame, from the deck's speed and direction ramps at every frame.
Loss: fire where FDS has fire, and no false fire
Every term is divided by a running average of its own size, so the weights below are relative importances. Under data parallel, the running average uses the mean over ranks, so every GPU keeps the same scale and a resumed run restores it exactly.
Plain windows (H = 2 steps from an FDS state)
- Gas: Huber per corner on T, U, V, W, Z and G, weighted by √(φV). Corners where either FDS or the model is hot (T > ambient + 50 K) get extra weight, so the fire set carries about half the total (the boost is capped at 20×). Missing fire and false fire both cost. HRRPUV has no per-corner term: it is scored only through the totals.
- Faces: the same rule, with the channel weights in the face table.
- Overlap (4 m, 16 m): a soft Dice score on the hot fraction of each coarse cell. A plume one cell off then costs less than a missing plume.
- House exceedance: per-step cross-entropy on q_net > 40 kW/m² for each house face, plus the error in soft time above 40 kW/m².
- No asymmetric weights: in an earlier model, a 3:1 asymmetry alone once caused a heat runaway.
Pushforward windows (up to 20 steps without gradient, then H steps)
- Past about 10 s, fine-scale gas decorrelates even between two FDS runs, so there are no per-corner terms. They would train toward a blurred plume.
- Instead the loss scores house dose and time above 40 kW/m², house flux quantiles (0.5, 0.9, 0.95, 0.99), 16 m fields, and total heat release. Each term has a dead zone at FDS's own run-to-run spread: dose 2%, time above 2.3%, quantiles 5%, HRR 10%, and 16 m per-field floors.
- The HRR term is at weight 0 in the current run. See pushforward.
Schedule changed
The draft planned a 1 → 20-step curriculum on one scene, then long rollouts up to 400 steps. That plan is replaced:
- H = 2 prefix for 150k optimizer steps on 4 GPUs. Windows are 2 steps long and start anywhere in 0–200 s.
- Pushforward from step 30k, on half the windows: n ≤ 20 steps without gradient from an FDS state, then an H-step window with gradient. n is capped at 20 because pushforward rollouts vary in length, and on 4 GPUs every step waits for the longest one.
- Fork A continues the prefix at H = 3, then H = 4, for 40k steps each. Fork B (longer unrolls) and fork C (input noise instead of pushforward, the cheap baseline that pushforward must beat) are deferred to SCI-6250. Both fork from the saved H = 2 checkpoint.
- Learning rate: linear warmup, then cosine decay to a floor. Gradients are clipped at norm 1. The learning rate is scaled down during pushforward. Steps with non-finite gradients are skipped; 100 skips mark the run as diverged.
- Why short windows: with no activation checkpointing, a 2-step window costs about 0.16 s and 4 GPUs see many scenes. The single-scene path keeps the old curriculum for overfit checks.
Pushforward: it helps, but its heat-release term blew up twice
Pushforward trains on states the model reached by itself, so it learns to correct its own drift. In the 4-GPU run it helped at first, and then one term took over the gradient. This is the sequence, with the measured results.
Candidate replacements for the HRR term: a low weight, a per-term gradient cap or PCGrad, HRR scored locally per 16 m block instead of over the domain, or clipping. With the HRR term off, the flux-quantile term carries 60–70% of the pushforward gradient. Dose and time above 40 kW/m² contribute almost nothing, because they sit inside their dead zones.
Conservation: a soft penalty with a dead zone
The shape is the code's: Huber with δ = 1 applied to max(0, |r| − τ). Shown for |r| ≤ 0.5.
About 98% of the heat leaves through the open boundaries, so an exact energy budget can't close from 0.5 s snapshots. Instead the model's totals are compared with FDS's, on 16 m blocks and over the whole domain:
E_s = Σ φV · ρ₀c_pT∞(1 − T∞/T)
M_Z = Σ φV · ρ(T) · Z
MLR = Σ ṁ″ · A per vapour (wood, MMA)
L = huber(max(0, |r| − τ))
- The penalty compares both the level of each total and its change over the step.
- τ = 0 in plain windows: from a true state FDS is bit-repeatable, so any mismatch is error. In pushforward, τ is FDS's own spread: 0.24 for 16 m blocks and 0.10 for the domain.
- Per-vapour fuel release is now a term, as the draft proposed. Each vapour has one source, so this match is exact.
- The penalty is switched on by a flag. Once the gas term improves by less than 1% per 2k windows, its weight ramps 0 → 0.1 over 2k windows.
Data and scoring: FER computed the way production computes it
At this stage the goal is to match production's evaluation and physics exactly, not to improve scores. The model predicts what FDS computes, on the same surfaces, and production's FER code runs on its output unchanged. The test: FDS's own values, put through our path, must reproduce production's number.
FDS's own face values through our FER path, compared with production FER on the 9 test runs (range over runs). The dashed line is the agreed 3% bar.
How FER is computed
- Production's point set. FER is the mean point FER over every boundary point of the cells that meet the main-building footprint. That set includes faces buried between two solid house voxels, which carry about 0 flux: 55–75% of the cells, so production FER is roughly half of the exterior-face FER. We keep that dilution instead of correcting it, so our numbers compare like with like. PLF = min(FER, 1.5) / 1.5.
- FDS's corner rule, both ways. FDS writes each boundary corner as the mean of the exposed wall cells around it. Prep inverts that rule to recover FDS's native per-face values, which become the training targets. The earlier target, the mean of a face's 4 corners, was a smoothed version: its 90th percentile was 5–17% off on hot faces. Corners rebuilt from the recovered values match the stored ones to ≤ 1.5·10⁻⁴. Eval then pushes the model's face values through the same rule. An alternative vertex-id rule was tested and was worse (−7 to −36%).
- Closing the gap. Adding the cross-block faces and the ground strip brought FDS through our path from 0.4–24% under production to within 0.0–1.6% on the 9 runs, and within 1.4% on 4 unseen runs.
Data and split changed
The dataset holds 500 production runs as FDS wrote them, at 2 Hz in fp32, one object per frame. Prep succeeds on 494 of them. The draft's 5-fold plan is replaced: train on the 216 prepped "extra" runs, and split the scored 280 into validation 80 / holdout 198 (seeded, stratified on burned). The sets share no scenario, house or group. Validation picks checkpoints; the holdout is read once, on the pick.
What each checkpoint is scored on
Every validation run is rolled out freely from frame 0 to 200 s, with no help from FDS. A person reads the table and picks the checkpoint; no single number decides.
Training throughput: bf16, Triton CSR and size-balanced batches new
4 × H100, windows per second
1 × H100, ms per model step, median deck (fp32, 20-step window)
Adopted
- bf16 U-Net: ×1.44–1.56 faster per H = 2 window and ×1.2 per no-grad rollout step, measured on 20 validation runs. FER and field error match fp32 to within the noise of turning TF32 off.
- Triton CSR kernel for the short-row sparse products (halo, its transpose, unpool): exact; 3–5% faster per training window and 10% per no-grad step.
- 4 windows per GPU, balanced by size (
--slots 4), for an effective batch of 16 on 4 GPUs. Each scene visit draws 16 runs and sorts them by corner count into 4 size tiers. Each GPU gets one run per tier, and runs are swapped within tiers until the largest per-GPU total stops falling. Ranks then wait 0.1% of the time (4–6% before), with an imbalance of 1.006. The largest gradient norm fell from 42.9k to 2.1k, and no step was non-finite. Since rank waiting was small, most of the speed-up is bf16 and Triton.
Rejected, with the measurement
- bf16 state: fuel mass drifted −0.16% on every run; FER worse on 14 of 20 runs.
- Pre-padded layout (blocks stored with their halo): exact but ×1.38–1.54 slower. Convolutions, not gathers, dominate the native stack (weight gradient 28%, input gradient 17%, forward 10%, CSR halo 9%).
- Halo every second layer: ×1.12–1.18 slower, and the output differs by 10%.
- Linearity splits (first layer with statics precomputed; decoder without the concat): exact, but slower in eager mode.
- Earlier (single GPU): convolution as 27 shifted matmuls (slower than cuDNN), a Triton gather-GEMM halo conv, selective activation checkpointing, and CUDA-graph capture of a window (−1.5%; the step was GPU-bound).
Where the time goes now
dynamic=True), and cuDNN autotunes the new shapes. On 1 GPU a switch costs about 10 s; why 4 GPUs are so much worse is not yet known. Next fix: stop the per-scene recompiles. openMemory. The draft estimated a fully checkpointed fp32 20-step window at about 19 / 37 GiB on the median / largest scene. Measured peaks were 12.9 / 25.2 GiB, so it fits one H100 as claimed. Step time was the problem: the first code ran 4× over its estimate. Training now uses H = 2 windows with no activation checkpointing.
What changed since the 29 October draft
| Draft (29 Oct) | Now | Why, with the evidence |
|---|---|---|
| fp32 (TF32) throughout | bf16 U-Net; fp32 state, heads, update | ×1.44–1.56 per window at matching FER; a bf16 state drifted fuel mass −0.16% |
| Face targets from the boundary output | FDS's native wall-cell values (corner rule inverted) | 4-corner means were 5–17% off (p90) on hot faces |
| House, PMMA, ground-fuel faces | + cross-block faces, + inert ground ≤ 2 m from the house | FDS through our path: 0.4–24% under production → within 1.6% |
| FER on house faces | FER on production's point set, buried faces included | Same number as production; buried faces halve FER |
| PMMA burning built in (leaning) | learned, behind a hard 371 °C ignition gate | Causality by construction; no per-deck curve |
| PMMA fuel mass integrated | never limits the burn | FDS burns PMMA by ramp, with no burn-away (11.2 vs 7.7 kg/m² on one run) |
| Wind snapped to 0–45° by grid symmetries | not built | Wind enters as u, v in the scene frame |
| Curriculum K = 1 → 20 on one scene, then rollouts to 400 steps | 216 runs, H = 2 prefix, pushforward n ≤ 20 from 30k, fork A at H = 3, 4 | Cheap short windows across many scenes; long n makes the GPUs wait on each other |
| Pushforward HRR term, weight 0.3 | weight 0 in the current run | Owned 94–99% of pushforward updates; two diverged restarts |
| M1: overfit one scene first | dropped as a milestone; kept as a smoke check | Went straight to multi-run training (one-scene face loss 2.09 → 0.29 in the smoke test) |
| 5-fold split | train 216 / val 80 / holdout 198 | More held-out runs from one training run |
| Open: fuel release per vapour; drop pressure and emission; faces as a list | all settled as proposed | In the code |
Open questions
| # | Question | Where it stands |
|---|---|---|
| 1 | What replaces the pushforward HRR term: a low weight, a per-term gradient cap or PCGrad, local per-16 m HRR, or clipping? | HRR at weight 0 in the current run; the 60k evaluation is pending |
| 2 | Stop the per-scene recompiles at a scene switch; find out why 4 GPUs pay 66–81 s against about 10 s on one GPU. | Next speed fix |
| 3 | Sign-off on the faster recipe for fork A and SCI-6250: 4 slots, bf16, Triton, scene visits of 100 steps, learning rate scaled for batch 16. | Proposed |
| 4 | Fork B (longer unroll) and fork C (input noise instead of pushforward). | Deferred to SCI-6250 |
| 5 | Ground-fuel emissivity: 1.0 now; FDS fluxes imply 0.44–0.56. | Fix if ground-fuel net flux matters |
| 6 | Vegetation static channels are zeros (particles skipped; production trees are near massless). | Fill if production decks change |
| 7 | Wind rotation by grid symmetries. | Not built; revisit if wind direction fails to generalise |
| 8 | Mixed-time-step training (0.5, 1, 2 s). | Follow-up, SCI-6262 |