Skip to content

Run edge-case analysis & property testing (P-38) on the highest-risk core areas #1771

Description

@vybe

Summary

Apply the new edge-case analysis & property-testing process (core-team /edge-cases skill, catalog candidate P-38) to the highest-risk areas of the codebase: enumerate edge cases per target (boundary-value analysis + the incident-traced bug-class checklist), generate parametrized pytest + Hypothesis property tests for the uncovered ones, and verify honestly — branch coverage everywhere, a scoped mutation-testing gate (--mutate) on the destructive guards.

Context

Target selection is risk-based (churn × complexity ∩ git fix-density ∩ property-testing suitability ∩ blast radius), and the shortlist mirrors real incident history: destructive retention prunes (#1638, #1644), naive/aware + ISO-format timestamp bugs (#1474, #476), NaN cost-accumulator poisoning (#1155), CAS status races (#1082), replay side-effects (#1083). Method lineage: Anthropic's agentic property-based testing workflow (understand → propose properties → generate → reflect → report).

Targets (ranked)

  1. src/backend/services/cleanup_service.py prune predicates + src/backend/services/retention_guard.pyrun with --mutate; a surviving mutant here is a data-loss bug (bug: retention floor (#1065) silently deletes pre-existing execution history on upgrade — currently dev-only, ships to everyone at next release #1638/bug: retention prunes have no blast-radius guard — a mistyped window deletes most of a table within 5 minutes #1644 class)
  2. Timestamp helpers: src/backend/utils/helpers.py (iso_cutoff, parse_iso_timestamp) + src/scheduler/utils.py — round-trip + format-mixing properties (bug: schedule-triggered executions store naive started_at (no timezone suffix) — UI shows wrong relative time #1474, bug(SUB-003): rate-limit events never age out due to SQLite string-compare bug; retries amplify outages #476, architecture Invariant #16)
  3. src/backend/db/schedules/ — CAS status writers (precondition-guarded terminal transitions) + analytics aggregation (NULL-skip averages, UTC bucketing)
  4. src/backend/services/loop_service.py — budget/deadline/no-progress arithmetic with hostile-float strategies (feat(loops): per-loop cost budget enforced as a hard stop (max_cost_usd) #1155feat(loops): no-progress / doom-loop detection — stop on repeated identical responses #1157)
  5. src/backend/services/credential_paths.py — allow/deny policy; bonus property: byte-identical parity with the vendored agent-server copy on every input
  6. src/backend/services/skill_packaging.py — frontmatter parse + archive member vetting (agent-authored input trust boundary)
  7. src/backend/services/task_execution_service.pyapply_result envelope normalization + status mapping (pure parts only; replay/CAS semantics)

Acceptance Criteria

  • Edge-case matrix produced per target (enumerated / already covered / newly covered / unspecified)
  • New tests land in tests/unit/ as parametrized pytest + Hypothesis properties, CI-bounded (max_examples capped, deadline=None), no Docker/network in unit tests
  • --mutate gate run on target 1; every surviving mutant either killed by a new test or documented as an accepted gap with a reason
  • Any real bug found ships as a strict=True xfail test + its own issue via the normal flow — no silent product-code fixes inside this task
  • hypothesis added to the dev/test requirements if property tests are kept
  • Full local unit run green (dev's required checks don't execute unit tests — verify locally before merge)

Technical Notes

  • One PR per target (or small clusters) rather than one mega-PR, so review stays tractable and a bug finding doesn't block the rest.
  • Property tests should pin each subsumed matrix row with @example(...) so deterministic cases survive even if the random search misses them.
  • Mutation runs must stay scoped to the target file(s) — never repo-wide.

🤖 Generated with Claude Code

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions