It is a negative number on the form, and that is the whole idea. Regulation Z has the Deposit line disclosed as a negative figure inside the calculation of estimated cash to close, because the money has already been committed. It is not a fee, it buys no service, and nobody keeps it in the ordinary case. A buyer who budgets the deposit and the down payment as two separate outlays has double-counted, since the deposit is normally applied to the down payment or the closing costs at settlement.
What actually protects it is the contract, not the escrow. The neutral holder is a safeguard against the seller having the money in hand, but it settles nothing about who is entitled to it. That question is answered by the contingencies the buyer negotiated: a financing contingency, an inspection or due-diligence contingency, an appraisal contingency, sometimes a contingency on the sale of the buyer's current home. Each one names a condition, a deadline and a consequence, and a buyer who terminates inside a contingency's deadline for a reason the contingency covers is generally entitled to the deposit back on the contract's terms. A buyer who walks away after the deadlines have passed, or for a reason no contingency covers, generally is not. Waiving contingencies to make an offer more competitive is therefore a decision about the deposit as much as about anything else, and it is the mechanism by which a waived contingency turns a refundable deposit into an exposed one.
Release is a contract question, and it is where disputes actually happen. The holder is neutral, which means it usually cannot simply hand the money to whichever party asks first. Purchase agreements commonly require written instructions signed by both sides, or a defined dispute procedure, before the holder may release anything. The practical consequence is that a contested deposit can sit undisbursed for a long time even where one side's entitlement looks obvious, because the holder's obligation runs to the instructions rather than to the merits.
One tax consequence that catches people out, in the buyer's favor. IRS Publication 936 sets out a nine-part test for deducting mortgage points in the year they are paid, and one part requires that the funds the buyer provided at or before closing, plus any points the seller paid, be at least as much as the points charged. The publication is explicit that those funds "aren't required to have been applied to the points" and that they "can include a down payment, an escrow deposit, earnest money, and other funds you paid at or before closing for any purpose." So a deposit committed at contract signing can be what makes the points deductible in full that year, months later, through a rule the buyer never connected to it.