What We Call “Working” Is Not What You Call “Making Money”

Both previous articles ended at zero. Nine chart pattern types: zero cells passed. Forty-nine technical signal types: also zero.

And the reader response has been almost uniformly the same question: then why do people still make money with technical analysis?

It is a good question and it deserves a real answer, not "those people just got lucky." The answer comes in two halves:

• The first half is in the definitions — what we measured is not the same proposition as "making money." That is this article.
• The second half is in other people's data — what published research actually observes about traders who profit over the long run. That is the next one.

This article introduces no new data. It takes the word "working" apart and shows what it actually meant in our tests, how much time it covered, and where it diverges from the problem a real trader faces.

This is a methodology article. It explains the definitions and boundaries of a research programme. It is not a recommendation regarding any security or any method of analysis, and it is not investment advice.

1. We never measured profit

This is the most basic point and the easiest one to skip past.

What the first two articles measured is information content: after this pattern or signal appears, does the distribution of subsequent returns differ detectably from when it does not.

What was never measured includes:

So "no detectable information" and "loses money when traded" are two different propositions, and they can point in opposite directions.

A signal that carries information may be untradable because of costs. A signal with no detectable information may still be a useful trigger inside a process that includes sizing and stops. The first two articles answer only the first question.

2. Defining "working": the benchmark decides the answer

This section is the core of the article, and it can be demonstrated directly from the data.

Take the 49 signal types from the second article. Hold the events, the period, and the parameters constant. Change only the benchmark being compared against:

Same signals. Under the most permissive benchmark, 81% of test cells "win more than half the time." Under the strictest, only 22% do.

(To be precise about the terms: "hit rate" here means the share of events within a test cell whose excess return was positive — not anyone's trading win rate. A "test cell" is one signal type × one timeframe × one holding period × one benchmark.)

Where does the gap come from? Look at the raw return before any benchmark: over this period, buying a randomly chosen constituent and holding 20 trading days returned +1.08% on average. Market adjustment removes the index but not "which stocks this signal tends to fire on, and when." The matched control removes that too — it compares against stocks in the same sector, on the same day, in the same market regime, that did not produce a signal.

What this means for a reader: when anyone says a method works, the first question is not what the win rate is. It is "compared to what?" A win rate with no stated benchmark is not a claim, however good the number looks.

Incidentally, this is why both previous articles report their headline conclusions on the matched control. Not because stricter is automatically safer, but because the other two layers leave known, nameable contamination behind.

3. Our time scale: one trading day to three months

The holding periods tested were 1, 5, 10, 20, and 60 trading days, on daily, weekly, and monthly bars.

Translated into ordinary language:

The top row deserves emphasis: the research contains no intraday data whatsoever.

We use daily OHLCV — open, high, low, close, volume, one row per day. So for anything opened and closed within a single day, we did not test it and find no effect. We never looked. That is not an oversight; it is where the data ends.

The second row carries a caveat too. A one-day hold in our calculation means "today's close to tomorrow's close," and the second article's finding was that the overnight segment accounts for 60.6% of total daily variance, while signal confirmation days are 5.76× more likely than an ordinary day to carry a large gap. So even the one-day result is substantially made of movement that happened while the market was shut and nobody could act.

Sample density varies sharply by timeframe:

And by holding period, effect sizes grow with time while hit rates do not improve:

A larger effect size does not mean a better signal — hold anything longer and returns disperse more. The hit rate stays between 48% and 49% throughout.

4. Does the market regime change anything

"Signals have to be read against the market direction" is one of the most common claims in practice. The second article's multi-timeframe conditioning tested exactly that: the weekly trend was expanded into four states, and every daily signal was tested separately within each.

At a 20-trading-day hold, under two benchmarks side by side:

This table replays Section 2's point.

Under market adjustment, signals appear to work best in uptrends (52.7%) — precisely the kind of number that gets written up as "trend-following works." Switch to the matched control and that advantage disappears: all four states land between 48.7% and 50.3%, a spread of under 1.6 percentage points.

The reason is not mysterious. In an uptrend everything is rising. Market adjustment removes the index but not the shared rise of same-sector peers over the same window; the matched control removes that as well, and what was attributable to the signal itself is gone.

So for these signals, under this definition: conditioning on market regime still leaves zero significant cells. That does not deny that reading market direction has other value — risk management, for one — but that value does not come from the information content of these signals.

5. The asymmetry in sample size — which I think is the real answer

Everything above is still methodology. This section is where I think the genuine reconciliation lies between "zero result" and "people do profit."

Our test asks:

After this signal has occurred 2,661 times (the median event count of a reliable test cell; the largest single cell has 80,681), does anything detectable remain on average, after subtracting three layers of benchmark?

A trader faces:

Did my few dozen trades, added up, make money?

These are not the same proposition mathematically.

A signal with zero average effect can perfectly well be profitable across someone's few dozen uses — because those uses are not a random sample. They were filtered: acted on only under particular conditions, with a stop attached, exited early when wrong, sized differently each time. None of that filtering exists in our test, and it happens to be the part practitioners consider most important.

The converse also holds: a zero average effect means you cannot obtain that effect by following the rule mechanically, many times. That is what the zero result genuinely rules out — the mechanical, repeatable, judgement-free use.

Section 7 of the first article argued that one setup used a hundred times beats a hundred setups used once each. The other face of that argument is here: what a large sample can prove and what a small sample can achieve are different things. We did the former.

6. So what these two articles cannot answer

The boundaries, listed plainly:

Apart from the last row, those four "no"s are the part readers actually want to know. I cannot answer them with my own data — but other people's research can answer some of them.

The next article

The next instalment takes on the second half: what published research actually observes about people who profit from trading over the long run.

It will use public, verifiable empirical studies and documented cases. And the stance needs stating in advance: it will describe what studies observed, and will not assert that any individual "used method X" — unless that person has published their own approach. Reasoning backwards from a result to a method is exactly the kind of inference this whole series argues against.

---

Every figure in this article comes from the result files of the two previous studies. Study window 1 January 2000 to 31 December 2017, universe is point-in-time S&P 500 membership. Figures are recomputed by the analysis pipeline rather than transcribed by hand.