LLM Risk: What the Chat Box Hides
Large language models are powerful, but they are not free, neutral, or harmless by default.
A simple prompt can hide four things: expensive infrastructure, energy and cooling demand,
privacy and security exposure, and the risk of confident wrong answers. This page helps you
see the real process underneath: training builds the model, inference serves every prompt,
and responsible use means checking cost, data, accuracy, and dependence.
Training cost
Building frontier-scale models can require millions of GPU-hours before any user sends a prompt.
Inference cost
Every chat, summary, or code request uses serving hardware, memory, networking, and electricity.
Information risk
Private data, biased outputs, hallucinations, and over-trust can create real harm.
Good practice
Use LLMs where they add value, verify important outputs, and avoid unnecessary repeated calls.
Most people see a clean chat box—not the model replicas, accelerators, memory, networking, grid connections, cooling, backup capacity, safety systems, and engineering needed to produce each answer. The subscription price hides most of the physical system.
OpenAI model tiers and API prices checked against official developer documentation on August 3, 2026. Estimates and derived calculations are labelled.
1. The Physical Cost
LLMs feel like software, but they run on physical infrastructure: GPUs, memory, storage, power, cooling, and networks. These examples show the scale.
Power draw per top-end GPU
Official
700 W
NVIDIA lists the H100 SXM with up to 700W TDP and 80 GB memory. Even one 8-GPU server means 5.6 kW for GPUs alone, before CPUs, networking, storage, and cooling.
Training scale
Model card
30.84M H100 GPU-hours
Meta reports Llama 3.1 405B used 30.84 million H100 GPU-hours for training, with 15T+ pretraining tokens. That is industrial-scale compute, not desktop-scale computing.
Cooling water
Research
700,000 L direct
5.4M L total
The 2023 water-footprint paper estimated GPT-3 training in Microsoft U.S. data centers could directly consume 700,000 liters of freshwater, and about 5.4 million liters total when indirect water is included.
Model storage
Derived
~810 GB weights
A 405B-parameter model stored at FP16 needs about 810 GB just for weights. Replicas, checkpoints, optimizer state, KV cache, and backup copies push real storage and memory requirements much higher.
Infrastructure investment
IEA 2026
$400B+ in 2025
The IEA reports that capital expenditure by five large technology companies exceeded $400 billion in 2025. This is broader than one model’s training bill: it reveals the enormous infrastructure build-out behind AI services.
Inference can dominate
HotCarbon
25x training emissions
Under a high-traffic scenario, Chien et al. estimated that serving a ChatGPT-like system could require about 25 times the GPU-hours of training GPT-3 once. It is a scenario estimate, not a measurement of today’s service.
Strong takeaway: the interface is simple, but the system behind it is large. Cost does not disappear because the prompt box is clean.
2. The Everyday Cost
Training is expensive, but repeated inference is where everyday usage becomes a recurring bill. Change the numbers to see the effect.
Estimated API cost per day
$0
Estimated API cost per month
$0
Estimated API cost per year
$0
What the Three Tiers Mean—and What Remains Hidden
- Luna: OpenAI’s cost-sensitive tier for efficient, high-volume workloads.
- Terra: the middle tier intended to balance intelligence and cost.
- Sol: the flagship tier for complex professional reasoning and coding.
- For the same token volume, Sol’s listed input and output prices are 25× Luna’s. Higher reasoning effort, pro mode, long context, tool calls, images, and repeated agent steps may add more work, latency, and billed tokens.
- Hidden information: OpenAI does not publish parameter counts, accelerator counts, energy per prompt, cooling water per prompt, or the complete infrastructure cost for these tiers. Do not invent those numbers from price alone.
Reading these bars: the bill uses official token prices. The energy and operating-risk bars are illustrative workload-pressure indicators—not measured energy use or scientific risk percentages—because providers do not publish enough per-request infrastructure data to calculate those values.
Visible inference bill
0%
Illustrative infrastructure pressure
0%
Illustrative lock-in/operating pressure
0%
A moderate-volume workload already produces a real recurring bill. The hidden part is that the API bill is only one layer; the underlying power, cooling, storage, and capacity footprint is larger still.
3. Why the Risk Stays Hidden
What most users see
- A chat box and a quick answer.
- A simple subscription or token price.
- No direct view of GPU clusters or cooling systems.
- No obvious sign of storage replication, checkpointing, or traffic spikes.
What sits underneath
- High-end accelerator hardware with large power draw.
- Cooling water or equivalent cooling infrastructure.
- Large model weights, caches, checkpoints, and replicas.
- Recurring inference traffic that can outweigh one-time training impacts over time.
Main teaching point
- Training is expensive, but repeated inference at scale can be even more expensive over time.
- Water and power matter because LLM infrastructure is physical, not magical.
- Model size affects not just quality, but memory, storage, cooling, and cost.
- Best practice: use LLMs where they add high value, then reuse outputs locally when possible.
4. Use LLMs Deliberately
A strong LLM workflow is not "never use AI". It is: use it where it helps, protect sensitive data, verify important claims, and control repeated cost.
Check the data
Do not paste private, confidential, medical, financial, legal, or student-identifiable data unless the system is approved for that use.
Check the answer
LLMs can sound certain when they are wrong. Verify facts, citations, calculations, and code before using the output.
Check the bias
Training data can contain stereotypes or gaps. Review outputs for unfair assumptions, missing perspectives, and cultural context.
Check the cost
Repeated prompts, long context, and long answers multiply token use. Cache, reuse, summarise, and batch where possible.
Check the dependence
If a workflow only works with one provider or one large model, there is lock-in risk. Keep exports, fallbacks, and human knowledge.
Check the value
Use the smallest capable model and the shortest useful prompt. The best prompt is not always the biggest prompt.
Sources and Notes
- NVIDIA H100 specs: up to 700W TDP and 80 GB memory. nvidia.com
- Meta’s official Llama 3.1 model card: the 405B model used 30.84M H100 GPU-hours; the whole model family used 39.3M GPU-hours and had estimated location-based training emissions of 11,390 tonnes CO2e. Meta Llama model card
- IEA, 2026: data-centre electricity use grew 17% in 2025; AI-focused data-centre electricity grew 50%; five large technology companies invested more than $400B in 2025; global data-centre electricity is projected to grow from 485 TWh in 2025 to about 950 TWh in 2030. IEA executive summary
- Li et al., 2023, "Making AI Less 'Thirsty'": GPT-3 training estimated at 700,000 L direct water and ~5.4M L total water footprint; about 500 mL of water for 10-50 prompts depending on where and when inference runs. arxiv.org
- Chien et al., HotCarbon 2023: under a modelled ChatGPT-like high-traffic scenario, annual inference GPU-hours were estimated at about 25× the GPU-hours used to train GPT-3 once. This is scenario analysis, not a direct measurement of a current service. HotCarbon paper
- OpenAI model catalog checked August 3, 2026: GPT-5.6 Sol $5 input/$30 output, Terra $2/$12, and Luna $0.20/$1.20 per million tokens. The catalog describes Sol as frontier, Terra as balanced, and Luna as cost-sensitive/high-volume. OpenAI model catalog
- OpenAI model guidance: pro mode applies more model work, increases latency, and bills the aggregated tokens at the selected model’s standard rates. OpenAI model guidance
- Derived estimates on this page: 21.6 GWh for Llama 3.1 405B training GPU draw is from 30.84M GPU-hours x 0.7 kW; 810 GB model storage is from 405B parameters x 2 bytes/parameter at FP16.
- No parameter-count estimates are assigned to current proprietary OpenAI tiers because those values are not disclosed. API price is not a reliable measurement of model size or energy per request.