What a real pretraining run costs

01 · Concept

Concept

Someone will ask you what it cost to train Qwen3.8-27B. The honest answer has two parts, and both matter: nobody outside the team knows, and here is how you would estimate it by supplying the missing inputs. This lesson builds that estimate end to end, labelling every assumption as it goes, so that what emerges is a method you can apply rather than a figure you can quote.

For a dense decoder, the common Kaplan first-order shortcut for the model core is

with $N_{core}$ the non-embedding parameters and $D$ training tokens. The factor of six approximates forward and backward matrix work — roughly two operations per parameter per token forward, and twice that again backward. This shortcut estimates the core; it does not say that every operation outside $N_{core}$ is free.

Start with the rough version people often produce. Take the marketed 27 billion parameters, choose 4 trillion tokens as a round illustrative input, and multiply:

The shortcut instead uses the non-embedding count. Lesson 5.3 showed that the 248,320-row input embedding and its untied output head together hold about 2.54 billion parameters, and there is a vision tower besides. Using about 24.5 billion for $N_{core}$ gives

That 5.88e23 figure is the core approximation, not a better total. The untied output head still multiplies every final hidden state by a $248{,}320\times5{,}120$ matrix to produce logits. Chinchilla’s Appendix F counts that final logits projection separately. The head has about 1.27 billion weights, so under the same forward-plus-backward convention its contribution is

Adding the core shortcut and the head gives about $6.19\times10^{23}$ FLOPs before architecture-specific corrections. Treating the input embedding lookup as though it were another dense projection over the full vocabulary would also be wrong; direct operator accounting is what separates these costs.

The largest uncertainty is $D$. There is no published token count for this model — no technical report exists, and the model card does not disclose training tokens, training compute, utilization, hardware, or cost. The 4 trillion above is a round assumption used only to demonstrate the method; it is not a claim about Qwen3.8-27B. Change the assumption to 2 trillion and every number below halves.

A further caveat sits underneath the shortcut. Here, 48 of the 64 layers are Gated DeltaNet rather than full attention, which changes the operation count per token and how cost grows with sequence length. Treat $6.19\times10^{23}$ as a first-order construction, not a measurement; a precise estimate would count the actual operators in every path.

Now convert work into time. Realized device-hours are

where $P_{peak}$ is applicable peak operations per second in the dtype actually used and $u$ is measured utilization. For a worked estimate, assume an order-of-magnitude $10^{15}$ bf16 FLOP/s and 40% model FLOPs utilization. Those are illustrative inputs, not facts about Qwen’s hardware or run. With them:

On 1,024 accelerators that is about 420 hours, or 17.5 days of continuous training. At an illustrative rented rate of USD 2.50 per device-hour — a figure that must carry its date, region, and purchasing model — the final run costs roughly

That is one line of the budget, and it is the line everyone quotes as though it were the total.

Those non-compute lines are substantial: data acquisition and governance, deduplication and filtering at corpus scale, storage and inter-region transfer, checkpoint replicas — and note from lesson 5.7 that a single checkpoint of this model carries about 432 GB of state, so retention policy is a real storage decision — plus evaluation infrastructure, observability, security, and engineering salaries. Opportunity cost matters too when scarce cluster capacity delays other work. The BLOOM planning paper framed model choice explicitly around a one-million-A100-GPU-hour allocation, which is the right way to think about it: the experiment portfolio is planned alongside the target run, not discovered afterwards. Use that figure as evidence about that project at that date, never as a current price.

Energy follows the same discipline. Device power integrated over time gives the base: at an illustrative 700 W per accelerator, 430,000 device-hours are about 301 MWh at the chips. A facility multiplier covering hosts, networking, and cooling pushes delivered energy higher. Carbon then depends on where and when that electricity was generated; multiplying by a generic global intensity factor hides the single largest source of variation. The BLOOM carbon study is useful precisely because it separates dynamic training consumption from broader life-cycle boundaries and says which it is reporting.

Build the estimate as a calculator with exposed variables rather than a hardcoded total: parameters, tokens, architecture factor, peak throughput, a utilization range, device count, a dated hourly price, retry fraction, storage, staff, and contingency. Report device-hours, elapsed time at the chosen device count, direct compute cost, a program range, and a sensitivity analysis showing which input moves the answer most. In our example that input is unambiguously $D$, which is also the one that is unknown.

Then reconcile after the run — planned versus actual tokens, utilization, device-hours, failures, and spend — and feed the variance into the next budget. A cost model that is never reconciled against reality is marketing arithmetic wearing a spreadsheet.

The durable lesson is accounting discipline. FLOPs estimate physical work; utilization maps work to hardware time; price maps time to money; program scope adds everything needed to make the run happen; environmental scope adds energy and emissions. Every headline number is meaningful only with its date, its assumptions, and its boundary attached — and for the model this course follows, the most important thing that chain produces is a clear view of exactly which number nobody has published.