The structural point comes first, because it decides more outcomes than the law does: the clause bites only a beneficiary. A forfeiture provision takes away what the instrument gave. Someone the instrument gave nothing has nothing to lose by challenging it, so the clause is invisible to exactly the person most likely to be aggrieved. That is why the drafting move that actually uses the clause is to leave the person a bequest large enough that losing it would hurt, and it is why a document that cuts someone out completely and then adds a no-contest clause has done two things that do not work together.
The legal ceiling is probable cause, and the two statutory formulations differ in scope. Minnesota's is the whole uniform sentence: "A provision in a will purporting to penalize an interested person for contesting the will or instituting other proceedings relating to the estate is unenforceable if probable cause exists for instituting proceedings." Note how wide that is. It reaches an interested person rather than only a beneficiary, and it reaches "other proceedings relating to the estate" rather than only a challenge to the will, so the protection covers filings that are not attacks on validity at all.
California builds the same idea into a much more detailed structure. Section 21311(a) provides that a no contest clause "shall only be enforced against" three things: a direct contest brought without probable cause; a pleading challenging a transfer of property on the ground that it was not the transferor's property at the time of the transfer, and then only "if the no contest clause expressly provides for that application"; and the filing or prosecution of a creditor's claim, again only if the clause expressly so provides. Two of the three therefore depend on the drafter having thought of them, which means a standard clause reaches only the first.
"Direct contest" is a defined term, and the definition is the useful list. Section 21310(b) says a direct contest is one alleging the invalidity of a protected instrument or of one or more of its terms on any of six grounds: forgery; lack of due execution; lack of capacity; menace, duress, fraud, or undue influence; revocation, under the sections governing revocation of a will, of a trust, or of another instrument; and disqualification of a beneficiary under three named provisions. Anything outside that list is not a direct contest, so a petition asking the court to interpret an ambiguous provision, or to remove a trustee, is not the thing section 21311(a)(1) permits a clause to punish.
Probable cause has a statutory test rather than a feel to it. Section 21311(b): "probable cause exists if, at the time of filing a contest, the facts known to the contestant would cause a reasonable person to believe that there is a reasonable likelihood that the requested relief will be granted after an opportunity for further investigation or discovery." Two features of that sentence matter. It is measured at filing, on the facts then known, so a contest that turns out to be wrong is not for that reason without probable cause. And it expressly contemplates that the evidence may not be in hand yet, since the belief is about what further investigation or discovery might produce.
Three further sections finish the picture and are easy to miss. Section 21312 directs that in determining the transferor's intent, a no contest clause "shall be strictly construed," which places the interpretive thumb against enforcement. Section 21314 provides that the part "applies notwithstanding a contrary provision in the instrument," so a clause cannot draft its way around the limits. And section 21313 is candid that the part "is not intended as a complete codification of the law governing enforcement of a no contest clause," with the common law governing to the extent the part does not apply, which matters because section 21315 confines the part to instruments that became irrevocable on or after January 1, 2001.
What all of this adds up to in practice is a provision whose value is mostly in the shadow it casts. A beneficiary weighing a challenge is weighing a certain loss against an uncertain gain, and the clause makes the certain loss concrete. Whether it is a good idea in a particular family is not a legal question. It deters the marginal challenge and the frivolous one, and it also deters the meritorious one brought by someone who cannot afford to be wrong, which is a real cost and not a rhetorical one.