fix(autonomy): disable evidence staleness rollback by default - #315
Merged
Conversation
A promoted direct-API provider previously lost its "ready" status once its recorded evidence aged past max_history_age_days (7) — even with a perfect pass history — forcing a re-probe to stay promoted. Operators want a proven provider to STAY ready until evidence actually regresses, not to silently roll back for going quiet. Make the freshness window opt-in: - DEFAULT_MAX_HISTORY_AGE_DAYS = 0, where 0 (or any non-positive value) DISABLES expiry. AutonomyPolicy.max_history_age now returns timedelta | None; None means "freshness is not a promotion requirement". - Both readiness paths (CLI --autonomy-readiness via provider_smoke_commands and runtime_commands) and the runtime promotion gate (direct_api_autonomy._fresh_history_complete) treat None as always-fresh: no provider_history_stale warning, no fresh_provider_history blocker, max_history_age_seconds rendered as null / "disabled (no expiry)". - The freshness check stays fully configurable: the `strict` preset keeps a 3-day window, and AUTO_CODE_AUTONOMY_<PROVIDER>_MAX_HISTORY_AGE_DAYS=<N> re-enables an N-day window per provider. `lax` no longer needs its 30-day seed (it inherits the no-expiry default). Tests: split each stale-history test into a default (no rollback) case and a window-enabled case (env override / monkeypatched module constant) so the freshness mechanism stays covered when explicitly turned on. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Oleg Miagkov <mrobenner@gmail.com>
Contributor
📝 WalkthroughWalkthroughThis PR disables provider autonomy readiness history freshness expiry by default. Evidence no longer expires after 7 days; instead, staleness checks are bypassed entirely unless explicitly configured. All consumers of the autonomy policy (agent runtime gate, provider readiness evaluation, runtime CLI) now handle None max-history-age values, returning success immediately when freshness is disabled. Test expectations updated throughout to reflect the new default. ChangesDisable evidence freshness expiry by default
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
![coderabbitai[bot]](https://cdn.github.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/backend/core/autonomy_policy.py`:
- Around line 84-93: The parser currently clamps negative values for the
max_history_age_days knob which conflicts with max_history_age()'s contract that
non-positive values (<= 0) disable freshness; locate the code that
parses/coerces max_history_age_days (search for the parsing block around the
knob name, e.g., the function or logic that handles "max_history_age_days" in
the config parsing code) and stop dropping or clamping negative integers—accept
integers as provided, only enforce type/sanity (raise on non-numeric types) and
preserve negative values (e.g., -1) so max_history_age() can return None; update
any docstring/comments near the parser to reflect that non-positive values
disable freshness.
In `@tests/test_runtime_modes_command.py`:
- Around line 1205-1243: The test
test_runtime_modes_policy_gate_keeps_ready_when_expiry_disabled depends on
ambient environment variables (e.g.
AUTO_CODE_AUTONOMY_OPENAI_MAX_HISTORY_AGE_DAYS) and must explicitly isolate
them; before calling handle_runtime_modes_command in that test, use the pytest
monkeypatch fixture to remove or unset the relevant env keys
(monkeypatch.delenv("AUTO_CODE_AUTONOMY_OPENAI_MAX_HISTORY_AGE_DAYS",
raising=False) and any other default max-age envs your code reads) so the test
exercises the code path where max_history_age is None and is deterministic
across CI/shell environments.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: dbbdb20b-dd87-492d-a771-c402cff8a27d
📒 Files selected for processing (8)
apps/backend/agents/runtime/direct_api_autonomy.pyapps/backend/cli/provider_smoke_commands.pyapps/backend/cli/runtime_commands.pyapps/backend/core/autonomy_policy.pytests/test_agent_runtime.pytests/test_autonomy_policy.pytests/test_provider_smoke_command.pytests/test_runtime_modes_command.py
Comment on lines
+84
to
93
| def max_history_age(self) -> timedelta | None: | ||
| """Return the freshness window, or ``None`` when it is disabled. | ||
|
|
||
| A non-positive ``max_history_age_days`` means recorded evidence never | ||
| expires; callers treat ``None`` as "freshness is not a promotion | ||
| requirement" and never roll a provider back from ``ready``. | ||
| """ | ||
| if self.max_history_age_days <= 0: | ||
| return None | ||
| return timedelta(days=self.max_history_age_days) |
Contributor
There was a problem hiding this comment.
Honor the “non-positive disables freshness” contract in knob parsing.
max_history_age_days now documents/implements <= 0 as disabled, but parser coercion still drops negative values (Lines 308-319). That makes -1 overrides silently ignored, which can unexpectedly keep stricter preset windows active.
💡 Suggested patch
def _parse_knob(knob: str, value: Any) -> Any:
"""Parse a single knob value according to its declared type."""
parser = _KNOB_PARSERS[knob]
if parser == "_int":
- return _coerce_int(value)
+ if knob == "max_history_age_days":
+ return _coerce_signed_int(value)
+ return _coerce_int(value)
@@
def _coerce_int(value: Any) -> int | None:
@@
return None
+
+
+def _coerce_signed_int(value: Any) -> int | None:
+ if isinstance(value, bool):
+ return None
+ if isinstance(value, int):
+ return value
+ if isinstance(value, str):
+ try:
+ return int(value.strip())
+ except ValueError:
+ return None
+ return None🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/backend/core/autonomy_policy.py` around lines 84 - 93, The parser
currently clamps negative values for the max_history_age_days knob which
conflicts with max_history_age()'s contract that non-positive values (<= 0)
disable freshness; locate the code that parses/coerces max_history_age_days
(search for the parsing block around the knob name, e.g., the function or logic
that handles "max_history_age_days" in the config parsing code) and stop
dropping or clamping negative integers—accept integers as provided, only enforce
type/sanity (raise on non-numeric types) and preserve negative values (e.g., -1)
so max_history_age() can return None; update any docstring/comments near the
parser to reflect that non-positive values disable freshness.
Comment on lines
+1205
to
+1243
| def test_runtime_modes_policy_gate_keeps_ready_when_expiry_disabled( | ||
| tmp_path: Path, | ||
| monkeypatch, | ||
| capsys, | ||
| ): | ||
| """Default freshness expiry disabled → old evidence never rolls back.""" | ||
| from cli.runtime_commands import handle_runtime_modes_command | ||
|
|
||
| ancient_run_at = _provider_smoke_run_at(days_ago=365) | ||
| _write_provider_history(tmp_path, _stable_provider_history(ancient_run_at)) | ||
| monkeypatch.chdir(tmp_path) | ||
|
|
||
| handle_runtime_modes_command(output_json=True) | ||
| payload = json.loads(capsys.readouterr().out) | ||
| capability_rows = { | ||
| row["provider"]: row for row in payload["runtime_capability_matrix"] | ||
| } | ||
| policy_rows = { | ||
| (row["phase"], row["provider"]): row for row in payload["runtime_policy_matrix"] | ||
| } | ||
|
|
||
| openai_requirements = capability_rows["openai"][ | ||
| "autonomous_readiness_requirements" | ||
| ] | ||
| assert capability_rows["openai"]["readiness"] == "full_autonomous_candidate" | ||
| assert capability_rows["openai"]["autonomous_policy_gate"] == "passed" | ||
| assert ( | ||
| "provider_history_stale" | ||
| not in capability_rows["openai"]["autonomous_readiness_warnings"] | ||
| ) | ||
| assert ( | ||
| "fresh_provider_history" | ||
| not in capability_rows["openai"]["autonomous_readiness_missing_requirements"] | ||
| ) | ||
| assert openai_requirements["last_run_at"] == ancient_run_at | ||
| assert openai_requirements["max_history_age_seconds"] is None | ||
| assert openai_requirements["history_freshness_complete"] is True | ||
| assert policy_rows[("coder", "openai")]["autonomous_promotion_ready"] is True | ||
|
|
Contributor
There was a problem hiding this comment.
Isolate default-freshness test from ambient env overrides.
Line 1205’s default-behavior test currently relies on external env state. If AUTO_CODE_AUTONOMY_OPENAI_MAX_HISTORY_AGE_DAYS (or default max-age env) is set in CI/shell, this test can fail nondeterministically.
Suggested fix
def test_runtime_modes_policy_gate_keeps_ready_when_expiry_disabled(
tmp_path: Path,
monkeypatch,
capsys,
):
"""Default freshness expiry disabled → old evidence never rolls back."""
from cli.runtime_commands import handle_runtime_modes_command
+ monkeypatch.delenv("AUTO_CODE_AUTONOMY_OPENAI_MAX_HISTORY_AGE_DAYS", raising=False)
+ monkeypatch.delenv("AUTO_CODE_AUTONOMY_DEFAULT_MAX_HISTORY_AGE_DAYS", raising=False)
+
ancient_run_at = _provider_smoke_run_at(days_ago=365)
_write_provider_history(tmp_path, _stable_provider_history(ancient_run_at))
monkeypatch.chdir(tmp_path)As per coding guidelines, "Ensure tests are comprehensive and follow pytest conventions. Check for proper mocking and test isolation."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_runtime_modes_command.py` around lines 1205 - 1243, The test
test_runtime_modes_policy_gate_keeps_ready_when_expiry_disabled depends on
ambient environment variables (e.g.
AUTO_CODE_AUTONOMY_OPENAI_MAX_HISTORY_AGE_DAYS) and must explicitly isolate
them; before calling handle_runtime_modes_command in that test, use the pytest
monkeypatch fixture to remove or unset the relevant env keys
(monkeypatch.delenv("AUTO_CODE_AUTONOMY_OPENAI_MAX_HISTORY_AGE_DAYS",
raising=False) and any other default max-age envs your code reads) so the test
exercises the code path where max_history_age is None and is deterministic
across CI/shell environments.
Source: Coding guidelines
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A promoted direct-API provider lost its ready status once its recorded evidence aged past
max_history_age_days(7) — even with a perfect, stable pass history — forcing a re-probe just to stay promoted. The green readiness badges we just lit would revert after a week of quiet. The ask: a proven provider should stay ready until evidence actually regresses, not roll back for going idle.Fix
Make the freshness window opt-in (disabled by default):
DEFAULT_MAX_HISTORY_AGE_DAYS = 0—0(or any non-positive value) disables expiry.AutonomyPolicy.max_history_agenow returnstimedelta | None;Nonemeans "freshness is not a promotion requirement."Noneas always-fresh — noprovider_history_stalewarning, nofresh_provider_historyblocker,max_history_age_secondsrendered asnull/max_history_age=disabled (no expiry):provider_smoke_commands._provider_readiness_history_freshness_complete)runtime_commands._runtime_provider_readiness_history_freshness_complete)direct_api_autonomy._fresh_history_complete) — this one would otherwiseTypeErroronNone, so it's the most important.strictpreset keeps a 3-day window;AUTO_CODE_AUTONOMY_<PROVIDER>_MAX_HISTORY_AGE_DAYS=<N>re-enables an N-day window per provider.laxdrops its 30-day seed (inherits the no-expiry default).Tests
Each stale-history test is split into:
tests/test_autonomy_policy.py,tests/test_provider_smoke_command.py,tests/test_runtime_modes_command.py,tests/test_agent_runtime.py— 356 passed locally.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Changes