Vanity metrics vs. decisions: the filter test
Most dashboards are full of numbers that feel like progress but change nothing you do. The filter test is one question that sorts every metric into "run the business" or "delete from the report."
Fastest path: one prompt, end to end
🤖 AI prompt — paste into ChatGPT / Claude
You are a DTC analytics coach. Use ONLY the metrics I list; invent no numbers.
Metrics I currently track: [paste your list, e.g. followers, impressions, total revenue, email open rate, ...]
My business stage: [pre-launch / under $10k mo / scaling]
For each metric, run the filter test:
1. Ask: "if this number moved 20% up or down, what would I DO differently this week?"
2. If the answer is "nothing", label it VANITY and name the decision metric to watch instead.
3. If it drives a concrete action, label it DECISION and name the action.
4. Group the survivors by funnel stage (acquisition, conversion, retention).
Do not guess my values; if you need a number to judge, ask me for it. If you cannot browse to check a definition, say so and use mine.
Output: a two-column table (VANITY -> replace with X | DECISION -> drives Y), then my 5 keep-metrics.
Or do it in 3 steps
- Apply the filter to each metric. Ask the one question: "if this number changed 20%, what would I do differently?" If the honest answer is "nothing", it's vanity. Followers, impressions, total-revenue-without-margin, page views, and "reach" almost always fail this test.
- Swap each vanity trap for its decision metric. Followers -> email/SMS subscribers you own. Impressions -> click-through and cost per click. Total revenue -> contribution-margin dollars. Open rate -> revenue per email. The decision metric is the one tied to money or a next action.
- Keep one decision metric per stage. Acquisition: CAC and payback. Conversion: add-to-cart and checkout completion rate. Retention: repeat-purchase rate and 90-day revenue per customer. Five is plenty; more is just noise dressed as rigor.
Worked example (labeled): A founder tracks 47k Instagram followers and $80k monthly revenue and feels great. Filter test: followers +20% -> nothing changes (VANITY).
Revenue looks fine, but at 12% contribution margin and a $38 CAC every order loses money, so the DECISION metric (contribution-per-order minus CAC) is negative. The two feel-good numbers hid a bleeding funnel; the two decision numbers exposed it.
Run the filter on every metric before it earns a spot on your dashboard; if it can't change an action, it's decoration. See north-star-metrics for choosing the one metric that sits above all of these.