best-practices/api-integration.md

API Integration

Client-side patterns for integrating with third-party REST/HTTP APIs. Focused on the realities of consuming APIs you do not control — capability gaps, missing filters, absent webhooks, inconsistent REST semantics, and the bidirectional sync problems that arise when two external systems must stay aligned.

Cross-references: API Design covers server-side API design (the APIs you build). Secrets Management covers credential storage for API tokens. Validation & Deployment covers full-chain integration testing.

---

1. Verify API Capabilities Before Locking Architecture

Principle: Probe critical assumptions against live docs or test API calls before committing to an architecture. Assume standard REST conventions do NOT hold on any API you have not personally verified.

Why it matters: Mid-planning discovery of a missing capability can force a full redesign. Real examples: an API that has no modified_at filter (so incremental sync is impossible), no issue-linking primitive (so cross-system references cannot be stored natively), no webhooks (so push-based sync is off the table), or PATCH semantics that replace rather than merge nested objects. Each of these changes the architecture in a way that is expensive to discover after code is written.

How to implement:

• Before sketching an integration, enumerate the capabilities you are relying on: filter/modification support, PATCH vs PUT semantics, webhook availability, custom field types, linking/relationship primitives, pagination cursor stability, rate limits.
• For each capability, find a live docs reference or run a test call against a sandbox. Do not infer from "it's REST, so it should have X."
• Document the actual capability matrix in the plan (or spec) — explicitly list what the API does NOT support, not just what it does.
• When an integration targets multiple APIs, run this probe per API before deciding on a shared abstraction.

Anti-patterns:

• Assuming every REST API supports an updated_since or modified_at filter.
• Assuming PATCH merges; assuming PUT is idempotent on nested structures.
• Assuming webhooks exist because "it's a modern SaaS product."
• Assuming custom fields support reference/relationship types when the docs only mention strings and numbers.
• Deferring capability verification to "we'll find out when we build it."

---

2. Compensate in the Application Layer for Missing API Features

Principle: When an external API lacks a feature you need, design the application layer to compensate rather than forcing the API to behave like you want.

Why it matters: You cannot change a third-party API. The cost of hoping for a capability is architecture that only works in a future that may never come. Compensation patterns are well-understood and usually add acceptable overhead — but only if you plan for them up front.

How to implement:

• No modified_at filter? Hash-compare every record against a stored hash to detect changes. On each run, fetch all records, compute a content hash, diff against the previous hashes, apply changes for records whose hash changed.
• No reference/relationship type for custom fields? Store foreign-key IDs in text fields with a documented format (e.g., ticket:12345). Parse on read.
• No webhooks? Poll on a cursor. Persist the cursor in state so each run resumes from the last processed position.
• No persistent state store in the execution environment? Use a git-backed flat-file store (see section 3).
• No idempotency key support on writes? Implement dedupe at the application layer using a (external_id, operation, hash) tuple stored in your own state.
• No batch endpoints? Implement client-side batching with concurrency caps and retry/backoff.

Anti-patterns:

• "Let's ask the vendor to add X" as the primary plan — maybe they will, maybe they won't, probably not on your timeline.
• Polling every record on every run with no hash/cursor — works at 10 records, implodes at 10,000.
• Storing relationship data in a second external system with no back-reference — creates orphaned state.
• Hand-rolling a webhook receiver when the source doesn't push — build a poller with a cursor instead.

---

3. Polling-Based Sync With Git + SOPS as State Store

Principle: For scheduled or runbook-style syncs where no webhooks are available and no persistent compute exists, use a dedicated git repo with SOPS-encrypted JSON flat files as the state store.

Why it matters: Ephemeral execution contexts (Octopus runbooks, cron jobs, scheduled GitHub Actions, short-lived containers) have no local disk, no database, and no shared cache. Spinning up a database for a low-throughput sync is overkill; using an HTTP KV service adds another dependency with its own credentials and failure mode. A git repo is free infrastructure that every execution context already knows how to use.

How to implement:

• Create a dedicated repo (e.g., <project>-sync-state) separate from the application code.
• Store state as JSON files, one per logical entity (e.g., records.json, cursor.json, hashes.json). Keep files human-readable.
• Encrypt with SOPS + age so secrets in state (external IDs, customer references, email addresses) are never exposed in plaintext.
• Each run: git clone --depth 1 → SOPS decrypt → read/write → SOPS encrypt → git add && git commit && git push.
• Make operations idempotent so a retry after a push conflict is safe. On conflict: re-pull, re-apply, re-push. A second writer racing for the same state is rare for scheduled runbooks, and idempotency handles it without coordination.
• Write a meaningful commit message per run (sync 2026-04-19T08:00Z: 3 created, 1 updated, 0 deleted) — this IS your audit trail.

Benefits:

• No storage infrastructure to provision or secure.
• Free audit trail via git log — every state change is attributable to a run.
• Human-readable state files for debugging.
• Encrypted at rest via SOPS; decrypted only in the execution context.
• Works in any ephemeral runner with git + age installed.
• Rollback is git revert.

Anti-patterns:

• Storing secrets in state files without SOPS ("it's a private repo").
• Non-idempotent operations against the external API — turns push conflicts into data loss.
• Treating state files as a database with complex queries; if you need queries, use a real database.
• Shared state repo across unrelated integrations — blast radius and credential coupling get worse over time.
• No commit message content — you lose the audit trail benefit.

---

4. Bidirectional Sync: Declare a System of Record

Principle: Before implementing any bidirectional sync, pick ONE system as the system of record (SoR) and declare conflicts resolve in its favour. Document the direction explicitly.

Why it matters: Without a canonical side, conflict resolution becomes ad hoc, races between writers produce unpredictable state, and users lose trust in both systems (because they cannot predict which change will win). "Last write wins" is not a conflict resolution strategy — it is a coin flip that sometimes destroys the wrong side's work.

How to implement:

• Pick the SoR based on where humans actually work — the system with the richer UI, the auditable trail, or the regulatory obligation usually wins.
• Document the direction in the integration spec: "SoR is Zendesk. PlanHat fields X, Y, Z are mirrored from Zendesk on every sync. PlanHat-side edits to X, Y, Z are overwritten."
• Implement the sync loop as one-way reads from the SoR and one-way writes to the other system(s). Any "reverse" flow is a separate, explicit sync with its own direction and conflict rule.
• Surface the direction in the user-facing UI when possible: show "read-only, synced from Zendesk" on mirrored fields so users don't waste effort editing them.

Corollaries:

• Only auto-create low-stakes artifacts. Creating a tracking stub in a downstream system is fine. Creating a ticket or case that obligates a human response is not — leave high-stakes creation to humans in the SoR.
• Never mirror fields that could leak cross-tenant or cross-context data. Internal notes, private comments, and any field that assumes a specific audience must not be synced to a system with different access controls. Review every field for leakage before adding it to the sync set.
• Separate read-fields and write-fields in config. The set of fields the sync writes to system B should be a strict subset of the fields it reads from system A, and that mapping should be explicit code/config, not an implicit "copy everything."

Anti-patterns:

• Bidirectional sync with no declared SoR — "both systems edit freely, we'll resolve conflicts later."
• Last-write-wins across two independent writers — destroys work non-deterministically.
• Auto-creating tickets, cases, or escalations in a downstream system without a human in the loop.
• Mirroring entire records rather than an explicit allowlist of fields.
• Documenting the SoR in a diagram only, without enforcing it in code.

---

5. Poll Cadence, Cursor Design, and Idempotency

Principle: Polling-based syncs need a deliberate cadence, a persistent cursor, and idempotent writes so missed runs and retries do not corrupt state.

Why it matters: Ephemeral runners fail. Network requests time out. Push conflicts happen when two runs overlap. The only way these are survivable is if every operation is safe to repeat.

How to implement:

• Cadence: match the business tolerance for staleness, not the API's rate limit ceiling. A 15-minute poll is plenty for most customer-data syncs; sub-minute polls rarely justify their cost. Consider the rate limit as a constraint, not a target.
• Cursor persistence: store the cursor in the same state store as the rest of the sync (section 3). On run start, read the cursor; on run end, advance it only if all writes succeeded. A half-failed run must NOT advance the cursor, or records will be silently skipped.
• Cursor type: prefer a high-water-mark timestamp or an opaque server-provided cursor over page offsets. Offsets break under concurrent writes on the source.
• Idempotent writes: every write to the downstream system must be safe to repeat. Use external IDs, idempotency keys, or upsert semantics. A retry after a transient failure must produce the same end state as a single successful write.
• Rate limit handling: honour Retry-After headers. Implement exponential backoff with jitter. Do not retry in a tight loop — this turns a soft rate limit into a hard ban.
• Observability: log per-run counts (read / created / updated / deleted / skipped / errored). Alert on error counts above a threshold or on consecutive empty runs (which may indicate the cursor has wedged).

Anti-patterns:

• Advancing the cursor before writes complete — skips records on partial failure.
• Polling faster than the business needs — wastes rate limit budget for no benefit.
• Non-idempotent writes paired with at-least-once delivery — creates duplicates on retry.
• Ignoring Retry-After — gets you rate-banned.
• No per-run observability — silent failures accumulate until someone notices data is missing.

---

Summary

Integrating with a third-party API is an exercise in compensating for its limitations. Verify capabilities before designing, build application-layer compensation for missing features, use git + SOPS as a state store when you have no persistent compute, and always declare a system of record for bidirectional syncs. Idempotency and observability are non-negotiable.