What a real pretraining run costs

1

Concept

A pretraining cost estimate should be reproducible, not a dramatic dollar figure detached from assumptions. Start with algorithmic work, translate it into measured hardware time, then add the rest of the program. Each boundary answers a different question: “What did the final run consume?” is not “What did it cost to develop the model?”

For a dense decoder, a rough training estimate is

where $N$ is non-embedding parameters and $D$ is training tokens. The factor approximates forward and backward matrix work; attention, embeddings, sparsity, recomputation, and architecture alter it. A more credible estimator sums operations from actual layer shapes and sequence length. Label every approximation.

Convert FLOPs to device-hours using realized throughput:

where $P_{peak}$ is applicable peak operations per second and $u$ is measured utilization. Do not insert a vendor peak from one dtype if the run uses another recipe. Utilization includes kernel efficiency and idle time but may be reported under different definitions. Use pilot telemetry from the intended topology.

Compute price needs a date and purchasing model. Cloud on-demand, reserved capacity, a negotiated cluster, and owned hardware produce different rates. For ownership, amortize hardware, networking, facilities, maintenance, financing, and expected utilization. An accelerator sitting idle still has capital cost; a rental price includes provider margin. Present a range instead of mixing them into one false-precision number.

The final successful run is only one line. Add scaling pilots, data ablations, tokenizer experiments, failed runs, checkpoint evaluation, post-training, and reruns caused by faults. The BLOOM planning paper published in 2022 explicitly framed model choice around a one-million-A100-GPU-hour allocation, illustrating why the experiment portfolio must be planned alongside the target run. Use that figure only as evidence about that project and date, not a current price.

Non-compute categories include data acquisition and governance, deduplication and filtering, storage, inter-region transfer, checkpoint replicas, evaluation infrastructure, observability, security, and engineering time. Opportunity cost matters when scarce cluster capacity delays other work. A contingency line should reflect evidence from pilot failure and preemption rates.

Energy is device power integrated over time plus other equipment. Facility power usage effectiveness accounts for cooling and overhead. Carbon depends on where and when electricity is generated; multiplying by a generic global factor hides location. The 2022 BLOOM carbon study separates dynamic training consumption from broader life-cycle boundaries, a useful reminder that carbon totals depend on scope.

A calculator should accept variables rather than hardcode a current cloud price: parameters, tokens, architecture factor, peak throughput, utilization range, device count, hourly price dated by the user, retry fraction, storage, staff, and contingency. Output device-hours, elapsed time at the chosen device count, direct compute cost, program range, and sensitivity. Changing device count ideally changes elapsed time, not total ideal device-hours, until scaling efficiency changes.

Reconcile estimates after the run. Compare planned versus actual tokens, utilization, device-hours, failures, and spending. Explain variance and feed it into the next budget. A cost model that is never reconciled is marketing arithmetic.

The durable lesson is accounting discipline. FLOPs estimate physical work; utilization maps work to hardware time; price maps time to money; program scope adds everything needed to make the run happen; environmental scope adds energy and emissions. Every headline number is meaningful only with its date, assumptions, and boundary attached.