Research methodology

The sequence

Stages run in order. A stage cannot be skipped, and a refused attempt is recorded rather than discarded.

1. Step 1 — observe

   Write down what was noticed, and when.

2. Step 2 — data qa

   Check coverage, gaps, staleness and revisions before any test runs.

3. Step 3 — hypothesis

   State the claim so that it can fail, together with the null it must beat.

4. Step 4 — strategy spec

   Turn the claim into explicit machine-readable rules with bounded parameters.

5. Step 5 — code test

   Implement the rules in deterministic code and test the code itself.

6. Step 6 — backtest

   Measure the rules over history that the rules were allowed to see.

7. Step 7 — walk forward

   Re-test in rolling sequence, fitting only on the past of each fold.

8. Step 8 — holdout

   Test once against the reserved period, which was never used for tuning.

9. Step 9 — robustness

   Perturb parameters, cut the sample, raise costs, and see what survives.

10. Step 10 — red team

    Try to break the conclusion on purpose.

11. Step 11 — risk review

    Review exposure, concentration and worst-case behaviour at portfolio level.

12. Step 12 — research report

    Write the evidence packet: claims tied to sources and to specific test runs.

13. Step 13 — promotion proposal

    Propose it to a person. A human signature is required; nothing promotes itself.

Anti-overfitting controls

• Point-in-time data. A signal formed on a given day may only use data visible on that day, and may only act on the following period.
• Locked holdout. A period of history is reserved and time-stamped before any tuning, and is never reused for tuning afterwards.
• Walk-forward testing. Rolling folds, each fitted only on its own past.
• Costs always on. Fees, spread and slippage are applied in every run, and every conclusion is re-checked with costs raised well above expectation.
• Counted attempts. The number of variants tested is recorded, and headline results are adjusted for the fact that repeated testing inflates them — the Deflated Sharpe and backtest-overfitting-probability diagnostics.
• Baselines. Nothing is judged in isolation. Buy-and-hold is always in the comparison.

What the red team tries to prove

A separate review stage is tasked with breaking the conclusion. Any one of these findings blocks a strategy outright, whatever its headline numbers.

• Look-ahead bias

  Does any feature use information dated at or after the label it predicts?

• Data leakage

  Could the target have influenced the features, directly or through a revision?

• Survivorship bias

  Does the universe include instruments that were removed during the test window?

• Overfitting

  Do the results depend on a narrow parameter neighbourhood?

• Too many trials

  How many configurations were tested before this one was reported?

• Unstable parameters

  Does performance collapse under small parameter perturbation?

• Unrealistic fills

  Are assumed fills achievable given the bar data and spread?

• Liquidity and capacity

  Could the position sizes be traded without moving the price?

• Hidden concentration

  Is the result driven by one instrument, sector or period?

• Regime dependency

  Does it only work in one detected regime?

Honest limitations

• Our current price history is short and drawn from our own logged readings. Short history limits confidence.
• We do not use paid vendor data, option chains or order-book data. Any claim needing them is out of scope.
• Everything runs in simulation. Simulated fills are optimistic compared with real execution at size.
• Surviving these tests makes a claim better evidenced. It does not make it true, and it guarantees nothing.