Signal mining and data snooping

What signal mining is, how testing millions of rules produces convincing but false results, and how to tell a real fundamental signal from data-mining luck.

What it is

Signal mining means testing a large number of rules against historical data and keeping the best. A computer can try millions of combinations, such as "hold when revenue growth is above X and forward P/E is below Y", in minutes.

The problem

Test enough rules and some will look brilliant purely by chance. If you flip 1,000 coins ten times each, about one will land heads all ten times. That coin is not special. A rule chosen as the best of millions is very likely to be one of those lucky coins. This is data snooping.

Signs a result is luck

  • It was the single best of a huge search.
  • A small change to a threshold ruins it.
  • It works in only one period or one market.
  • Nobody can explain why it should work.

Mining responsibly

  1. Start from a reason. Test ideas with an economic story, such as "profitable, growing companies bought at reasonable prices do well".
  2. Hold data back. Search on one period, then check the winners on data never used. See Out-of-sample testing.
  3. Prefer plateaus. A rule that works across a range of thresholds beats a sharp peak.
  4. Count the tries. The more rules you tested, the higher the bar.
  5. Prefer fundamentals. Signals based on business results have a clearer reason to persist than short-term price patterns.

A fundamentals signal miner is on this site's roadmap, built with these checks.

See also

  • Overfitting in investing What overfitting is, why strategies tuned to past data fail in live trading, what a large study of community strategies found, and how to measure it.
  • Out-of-sample testing How out-of-sample and walk-forward testing check a strategy on data it was not built on, how to split your data, and how to read the results honestly.
  • What is backtesting? What backtesting a strategy means, how a backtest is built, the common biases that make backtests look better than reality, and how to use results wisely.
  • If-then signals How if-then signal rules work in rules-based strategies, the common condition types, how rules nest into decision trees, and how to keep them robust.

Last updated September 30, 2026. Education only, not investment advice.