Environment

Piotr Mazurek and Felix Gabriel have an amazing post up on LLM Inference Economics from First Principles, which I found on Bluesky. They go into a huge amount of detail about how inference works and how that affects processing speed. But I saw the graph above and thought “we can get energy from that”. And so I asked chatgpt-o3: “Looking at these graphs of throughput at different batch sizes on a 4xH100 80gb cluster, what ranges of power per token do they equate to?” ...

We have some actual numbers on tokens! In their commentary on the latest earnings report, Microsoft report: We processed over 100 trillion tokens this quarter, up 5X year-over-year – including a record 50 trillion tokens last month alone So we’ve got a baseline, and can estimate a growth rate. Assuming exponential growth and a three month token split of 19/31/50 we get 62% month on month growth, or, if it keeps going 100 T → 424 T → 1.8 P → 7.6 P, which is lower than the 5x yoy growth MS quote. ...

As Andy Massick explains well, if you are making a few requests per day to a simple non-reasoning text based AI model the carbon impact is, to use a technical term, bugger all 1. I probably ask ChatGPT and Claude around 8 questions a day on average. Over the course of a year of using them, this uses up the same energy as running a single space heater in my room for 2 hours in total. Not 2 hours per day, just a one-off use of a single space heater for 2 hours […] By being vegan, I have as much climate impact as not prompting ChatGPT 400,000 times each year ...