Your empirical result is showing genuine mis‑calibration and some mis‑ordering, not just a cosmetic “narrow range” issue.

---

1. Is “narrow range” the real cause of non‑monotonic calibration?

No. Narrow range can explain under/over‑confidence, but not your kind of non‑monotonicity.

- A logistic map of any scalar score is strictly monotone.

If the underlying `regime_score` is a good 1‑D proxy for risk, then:

- higher score ⇒ higher true hit probability, and

- with your sample sizes (≈30k / bucket for OLD, ≈10k / bucket for NEW_v2), the bucket hit‑rates should be almost monotone increasing apart from tiny noise.

- Your deviations are large:

- OLD 10‑day: 53.19 → 50.47 → 52.99 → 51.02 → 56.85

Drops of ~2–3 percentage points with ~30k obs per bucket are multiple‑sigma effects, not sampling noise.

- NEW_v2 60‑day: 63.38 → 54.91 → 55.56 → 60.52 → 63.10

Drops of up to ~8.5 points (~10k per bucket) are definitely real.

This means:

- The issue is not that probabilities are compressed to [0.32, 0.69] or [0.57, 0.69].

- The deeper issue is: the 1‑D score you’re feeding into sigmoid does not order cases cleanly by true hit probability.

In other words, the mapping “regime_score → P(hit)” is not truly monotone in the data, so empirical buckets break monotonicity.

Typical established causes of such non‑monotonic calibration:

1. Mis‑specified 1‑D score

Wrong functional form / weights (w1, w2, ε) → mixture of different patterns getting projected onto a single axis.

2. Mixture of regimes with different response curves

E.g. combining different market regimes, sectors, or volatility states where “score = 0.6” can mean different true probabilities.

3. Feature interactions ignored by the scalar regime_score

If the true decision boundary is non‑linear in your inputs and you compress them into a linear regime_score, you can easily get “mid scores” behaving worse than some lower ones.

4. Time‑dependence / regime drift

If the mapping from score to hit‑rate changed across sub‑periods (e.g., pre‑ and post‑some market event) and you pool them, you can see non‑monotonic bucket behaviour.

So: “narrow range” is at most a secondary symptom (weak separation, under‑confidence), not the primary cause of the non‑monotonic buckets.

---

2. What to adjust: w1/w2/epsilon vs scale (temperature)?

You have two conceptually different levers:

1. Scale (temperature) inside the sigmoid:

`p = 1 / (1 + exp( -scale * regime_score ))`, with scale currently = 1.0.

- Changing scale (or adding a bias term) is a pure calibration transform:

- It does not change ranking of cases.

- It only stretches/compresses probabilities (e.g. going from [0.32, 0.69] to something closer to [0.1, 0.9]).

- This is appropriate when the reliability curve is smooth and monotone but systematically under‑ or over‑confident.

- In your case, the curve is non‑monotone, so:

- A global scale change cannot fix “Bucket‑1 has higher hit‑rate than Bucket‑2”.

- It will just move all points up/down and change their slopes.

Conclusion: scale alone is not the right primary fix for your NEW_v2 60‑day pattern.

2. w1 / w2 / epsilon (i.e., the regime_score itself):

- Changing these does change ranking; you are effectively redefining the 1‑D projection of your underlying signals.

- In established practice, if you want a probabilistic score from a linear combination, you usually:

- Treat the components as features and

- Fit a logistic model:

`logit(P(hit)) = β0 + β1 feature1 + β2 feature2 + ...`

where your `regime_score` is a special case with hand‑picked βs.

- Because your bucket behaviour shows real mis‑ordering, you should treat w1, w2, ε as parameters to be estimated, not fixed heuristics.

Recommended hierarchy:

- Step A – Fix the score first (w1, w2, ε or more flexible mapping).

- Fit or at least tune these weights using a proper objective (log‑loss, Brier score, or even Sharpe/IC if that’s more aligned with your trading target).

- Check that hit‑rate vs raw regime_score is roughly monotone before sigmoid/bucketization.

- Step B – Then apply a monotone calibrator on top:

- Simple: temperature + bias (Platt scaling) estimated on a held‑out set.

- More flexible: isotonic regression mapping score → probability, which enforces monotonicity non‑parametrically.

Temperature scaling is for “already monotone but mis‑scaled”; you currently have “not cleanly monotone”, so the main work is on the underlying score, not just the scale.

---

3. Why is NEW_v2’s range even narrower (0.57–0.69), and is that expected?

Your own guess is essentially right and does have an established interpretation.

- If Section‑3C “regime hard‑wire” means the system only fires in strongly bullish regimes, then:

- The conditional base‑rate of hits in this filtered universe is already high (e.g. 55–65%).

- The model no longer sees the truly bad regimes (e.g. 30–40% hit‑rate); they are excluded upstream.

- As a result, the predicted probabilities naturally cluster in a narrow band above 0.5, even if the ranking is good.

This is expected behaviour when:

- The score is applied after a strong filter (only high‑prior cases survive).

- Or your inputs themselves have low variance in this filtered subset.

So:

- Narrower range in NEW_v2 is not automatically a separate red flag.

It is consistent with: “we only act in already‑favourable regimes, and we just fine‑tune within that subset.”

- The real red flag is specifically the shape of the reliability curve:

- Bucket‑1 (lowest predicted) having 63.38% vs Bucket‑2 at 54.91% is structurally wrong.

- That says: within that filtered bullish subset, your 1‑D regime_score is not aligned with actual odds.

In other words: “narrow-ness” for NEW_v2 is explainable; non‑monotonicity is not.

---

4. What should you do now in the workflow: go to Ablation (Step‑3) or refine the score first?

Given what you’ve observed, this is exactly the right moment for “later‑refine”, not for jumping ahead.

Ablation analysis is most meaningful when:

- Your base score is at least a reasonable ranker (higher score ≈ higher hit‑rate), even if it’s poorly calibrated in absolute terms.

Your current evidence:

- Significant, consistent rank inversions across buckets, especially in NEW_v2 60‑day.

- That suggests that the current `regime_score` is not yet a clean ordering statistic.

Under that condition:

- An ablation study on this regime_score will mix up:

- Genuine signal,

- Mis‑weighted components,

- And artefacts from regime mixing / mis‑specification.

- You will get conclusions like “feature X improves performance” which may be contaminated by the current bad projection rather than the feature’s true value.

Pragmatic established sequence:

1. Refine the core score first. For example:

- Break out the components that went into regime_score (those behind w1, w2, ε).

- Fit a proper logistic (or similar) model per horizon (10/20/60‑day) on a chronological train / validation split.

- Ensure hit‑rate vs score is monotone in each horizon on a hold‑out set.

2. Calibrate probabilities.

- Start with simple scale + bias (Platt) on hold‑out.

- If needed, move to isotonic regression for each horizon.

3. Re‑check bucket calibration and monotonicity.

- Specifically re‑check the worrying NEW_v2 60‑day case.

4. Then run ablation.

- Now, when you remove/alter components, you’re ablation‑testing against a reasonable baseline score, so the conclusions are much more trustworthy.

So in terms of your own earlier terminology: yes, this is the “later‑refine” moment; calibration + score refinement should come before your serious Step‑3 ablation.

---

If you have any further queries, please connect with us on 022-6290-10141 (Timings : 09.00 AM to 05.00 PM) or you can email us on info@cniinfoxchange.com