distill: best practices from 2026-04-19 cross-project run
Adds 3 new topic files (ai-parallel-agents, api-integration, python-patterns) and extends 21 existing topic files with new gotchas and patterns surfaced from memory across tracked projects. Index updated accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -371,6 +371,16 @@ When an implementation agent is gated by tests, **place tests in a read-only ref
|
||||
|
||||
Use filesystem enforcement, not prompt instructions. In a 3-way model comparison: Sonnet respected "DO NOT MODIFY" instructions; MiniMax edited tests 7 times; Haiku rewrote the entire test file. The filesystem makes modification impossible regardless of model.
|
||||
|
||||
### Test Infrastructure Files Must Be Protected from Agent Modification
|
||||
|
||||
Read-only protection must extend beyond test files to include **test infrastructure**: `conftest.py`, `pyproject.toml`, `pytest.ini`, `setup.cfg`, `tox.ini`. An agent can satisfy tests by adding pytest hooks in a writable `conftest.py` — for example, a `pytest_collection_modifyitems` hook that skips failures, or an autouse fixture that monkeypatches the system under test. Security reviews of agent gate implementations have repeatedly found `conftest.py` as a CRITICAL bypass vector.
|
||||
|
||||
**Rule:** the read-only reference directory must contain all test-discovery and test-configuration files, not just `test_*.py`. At gate-enforcement time, run pytest with `--rootdir` / `--confcutdir` pointed at the read-only tree so writable copies of these files in the working directory cannot override the authoritative configuration.
|
||||
|
||||
### Stand Up Real Test Infrastructure Early
|
||||
|
||||
Deferring a real test database/service (Docker Compose, testcontainers, ephemeral Postgres, etc.) pushes integration tests into a "deselected" bucket that nobody runs. Set up the test backing service in the first phase that touches it, so integration tests execute from day one instead of accumulating as tech debt. The cost of standing up ephemeral infrastructure is almost always lower than the cost of letting integration coverage rot.
|
||||
|
||||
### Model Selection for Implementation Agents
|
||||
|
||||
**Sonnet is the minimum viable model for constrained implementation tasks** (spec + test gate). Smaller and cheaper models modify test files or ignore constraints:
|
||||
@@ -482,6 +492,12 @@ When test files import optional packages (e.g., `sqlalchemy`, `psycopg`) at modu
|
||||
|
||||
Patching an entire module (e.g., `patch("mod.kubernetes.config")`) replaces exception classes with MagicMock objects. `except SomeException` then catches `MagicMock` instead of the real exception, causing tests to pass the wrong code path. Patch individual functions (`load_incluster_config`, `load_kube_config`) and leave exception classes intact so `except` clauses work correctly.
|
||||
|
||||
### MagicMock Returns Truthy in Controller Loops — Always Set Boolean Defaults
|
||||
|
||||
`MagicMock()` return values are truthy by default. If a controller loop calls `mock.should_stop()` or `mock.reconcile_triggered()` and the mock has no explicit `return_value`, the loop never exits — or never sleeps — because every call returns a truthy MagicMock. In one real incident this pattern consumed 40GB RAM before OOM.
|
||||
|
||||
**Always set `mock.method.return_value = False`** for boolean-returning methods used in loop predicates. Combine with `pytest-timeout` (e.g., `timeout = 10` in `pyproject.toml`) on any project with async loops, signal handlers, or sleep patterns so runaway tests die fast instead of starving the test host.
|
||||
|
||||
### Async Migration Requires Full Test Conversion
|
||||
|
||||
When migrating a codebase from sync to async, helper functions get converted but test functions are often left as sync `def`. Every test that calls an async function needs `async def` + `@pytest.mark.asyncio` + `await`. After any async migration, run tests and grep for `RuntimeWarning: coroutine '...' was never awaited` to find remaining sync-to-async gaps.
|
||||
|
||||
Reference in New Issue
Block a user