The phrase "inverse carrying charges" is the whole explanation compressed into two words. Holding a physical commodity costs money, and the CFTC's entry for carrying charges lists the components: insurance, storage and interest on the funds tied up. Those costs push later delivery months above nearer ones, which is the ordinary shape and the subject of the entry on contango. Backwardation is the market paying somebody to give up the goods now rather than charging them to hold the goods later, and it takes a real shortage to do that. Whoever needs the commodity this month is bidding above what the market will pay for delivery in six.
That is why the condition tends not to last. A shortage is a temporary state in most markets: high prices for immediate delivery draw supply out of storage and pull production forward, and once the shortage eases the curve can revert to charging for carry. The practical consequence is that a curve shape is a description of current conditions, not a durable characteristic of a commodity, and a strategy that depends on the curve staying inverted is exposed to the shape changing rather than only to the price moving.
Backwardation reverses the arithmetic of the roll, which is the part that reaches ordinary investors. Almost nobody wants delivery, so anyone holding continuous futures exposure has to sell the contract about to expire and buy a later one. In contango that swap costs money each time. In backwardation the later contract is cheaper than the one being sold, so the same exposure is re-established at a lower price, and if the market price then holds up as the new contract approaches its own delivery month, the difference is a gain. The CFTC's entry for convergence names the tendency that makes this work: "the tendency for prices of physicals and futures to approach one another, usually during the delivery month."
It is not a free lunch, and the reason is in the same sentence. The roll gain only appears if the spot price does not fall while the position is held. A commodity in short supply is often expensive by historical standards, and the same conditions that invert the curve can reverse quickly. A backwardated market rewards the holder for the roll and still exposes them to the entire price move of the underlying, which is much the larger number.
The condition is not confined to physical commodities either. Any market with a term structure of futures prices can display it. What does not transfer is the storage explanation: where there is nothing to store, an inverted curve reflects who wants the exposure and at what price rather than a warehouse running dry.