Pull requests

Every change lands on two branches: develop (what deploys to the dev environment) and main (what production tracks). The sequence below keeps those branches in sync. Branch from main, open the production PR first, then cherry-pick the same commits onto develop.

The two-PR path looks like this:

flowchart LR
  mainBranch["main"] --> feature["feature branch"]
  feature --> prMain["PR to main"]
  prMain --> cherry["cherry-pick"]
  cherry --> prDev["PR to develop"]
  prDev --> mergeDev["merge develop"]
  mergeDev --> testDev["test on dev"]
  testDev --> mergeMain["merge main"]

git fetch origin
git checkout main
git pull
git checkout -b your-feature-branch

git push -u origin your-feature-branch

./tools/pr-flow          # confirm each step
./tools/pr-flow --yes    # approve all steps (-y also works)

brew install fzf azure-cli
az extension add --name azure-devops
az login

Go validation

.azure/go-checks.yml↗ starts with a Detect Go-relevant changes job on the Microsoft-hosted ubuntu-latest pool. It compares the PR with its target branch and checks whether any file changed under services/, mcp/, cmd/, or common/, or whether go.mod or go.sum changed. If none did, the other three jobs are skipped and the build validation still passes. Manual runs always continue to the other three jobs.

The other three jobs are Go checks and generated clients, Go tests, and Go lint check. They run on go-build-pool. When they run, all three must pass for the required build validation to succeed. They start at the same time on separate agents after the first job finishes. If Azure has only one available agent or parallel-job slot, a job queues until capacity becomes available.

The checks job builds the services in the diff and verifies generated output. See HTTP client generation and validation for the generation checks. go vet is not a separate step; golangci-lint runs it. A diff with no Go packages and no generator input skips the rest of this job after selection, including Go installation and cache restoration. Frontend application source, docs/, and bruno/ are not generator input. A service or mcp pipeline file or k8s values file checks the devops catalog only. A frontend pipeline file does not pass the first job, so it runs no Go jobs.

The lint job runs golangci-lint on the same service selection and reports issues introduced since the merge base with the target branch. The binary is the published release archive for the version pinned in the pipeline, checked against the release checksums. Its analysis cache is restored with the Go build cache.

The test job compares the PR with its target branch and runs make test-ci with the race detector. Each selected service is a separate go test process, and several of those processes run at once. Up to twice the agent CPU count of packages run at once, because most test time is spent waiting on Postgres. Each service gets an equal share of that budget, but never more than the CPU count. When more services are selected than there are CPUs, one process runs per CPU. On an 8-core agent, one or two services each test 8 packages at once, three services each test 5, and eight or more services each test 2. Before tests start, the run prints every selected service and the number of top-level Test and Fuzz functions in their test files. Each finished test is printed with how many top-level tests have finished out of that number, the percentage complete, how many services are done, and which services are still running. Subtests are not counted, because they are created while the tests run. When every service has finished, the log prints one collapsed group per service, in selection order, containing that service's full test output. It also includes services that import changed non-internal Go packages, such as generated clients and event payloads, following Go imports transitively through non-internal packages. Changing a service's internal implementation or tests selects that service without selecting its HTTP callers. This selection scans exact import-path strings conservatively, so a comment containing an import can select extra tests.

A change under common/, cmd/, or go.mod tests every service, plus the packages outside services/ (reported as shared). Those are the only paths that do. Everything else outside a service skips tests. A Go package consumer outside services/ does not widen the run. The log lists every path that forced the full run. Changes limited to docs/, frontend/, bruno/, the root openapi.yaml, or a markdown file in the repository root skip tests, cache restoration, and database setup. Files within a service still select that service, including prompts and test fixtures. Manual and scheduled pipeline runs always run the full suite; this pipeline does not configure a schedule.

The job publishes one JUnit file per service to Azure's Tests tab, including results from failed runs. Skipped tests do not publish a report. It uses committed specs and clients, so it does not repeat generation. It starts a fresh Postgres 16 container with pgvector, matching the major version deployed in dev and production. The data directory is a memory filesystem with fsync off, because every test creates a database, and the container is discarded afterward. max_connections is 800 so the parallel packages can share that server.

Each database test gets its own database, copied from a template database that has all of the service's migrations applied. The first test binary for a service builds the template, and the others wait for it, so migrations run once per service rather than once per test. The template's name includes a hash of the migration files, so changing a migration builds a new template. A test that stops at an earlier migration version runs migrations on an empty database instead. CI=true makes an unavailable database fail the tests instead of skipping them.

Both the checks and test jobs, and the lint job, use Azure's built-in pipeline cache for downloaded Go modules and for compiled build output. Caches are stored outside the checkout. Jobs use separate caches, keyed by operating system, architecture, Go version, job name, and go.sum. The key does not include the commit, so a cache is saved once per dependency set and later runs restore it instead of uploading a new archive every time. If the workspace has less than 8 GB free, the build cache is not restored. If a restored build cache grows past 3 GB, it is cleared before it can be saved. The lint job caches its analysis the same way, and caches the linter binary by version.

Before restoring modules, each job clears its own local module-cache directory with go clean -modcache, which handles Go's read-only module files. This prevents cache extraction from colliding with files retained on reused agents; the remote cache is kept. Each job also removes its local build-cache directory before a restore, for the same reason.

Module and build caches are saved after successful jobs when the cache key is new. PR runs can reuse caches from their own earlier runs or their target branch; caches created by one PR are not shared with other PRs. A successful manual pipeline run on main or develop can populate caches for PRs targeting that branch. Caching reduces downloads and compilation; database setup and test execution still run, and -count=1 prevents Go from reusing test results.

Disk diagnostics report free space and cache sizes at startup, after module restoration, and after the job's work, including failures.

The test job installs Docker, the C compiler, and make if missing, and starts Docker if needed. The Linux agent must support Docker and allow passwordless sudo for package installation, service startup, and Docker commands. Installation is skipped when Docker is already in the agent image. The job starts Postgres on a dynamically assigned loopback port, waits for it to be healthy, and removes its container and volume afterward, including when tests fail. Tests that explicitly require external services or provider credentials remain opt-in. This job does not start the full local stack or run frontend tests.

To run the suite locally, start the local Postgres service, then run make test. Use make test-ci to produce test-results.xml for the full suite. To match the pipeline's per-service run, use make test-ci TEST_SERVICES="billing leases". That writes test-results/<service>.xml. TEST_PACKAGES still selects packages in one process when TEST_SERVICES is unset. Missing local Postgres causes database tests to skip; set CI=true when you want that to fail locally as it does in the pipeline.

An existing required Go checks build-validation policy covers all four jobs. No separate test pipeline or branch policy is needed.

Develop-first validation

.azure/develop-first.yml↗ checks that every commit introduced by a PR to main has already reached develop, either as the same commit or as an equivalent patch. A commit is the same commit when it is an ancestor of develop, for example when the develop PR was merged from the same branch. Cherry-picking changes commit IDs, so otherwise the check compares patch IDs from git patch-id. The patches are taken with no context lines, so a cherry-pick still matches when the lines around a change differ between the branches. For example, an ADR row appended to docs/decisions/README.md matches even when develop already lists ADRs that main does not. It checks the PR's reviewed source commit rather than Azure's temporary merge commit. Commits already on main are excluded, so historical differences between the two branches do not block new work. The logic is in tools/ci/develop-first.sh↗. Run tools/ci/develop-first_test.sh to test it against disposable repositories. Run it manually when changing the script; it is not wired into CI.

The main PR is expected to fail this check until its develop PR merges. If you add more commits to the main PR, cherry-pick those to develop too, merge them, and requeue validation.

This check establishes historical patch presence, not successful deployment or testing. A patch later reverted on develop still exists in its history and can pass. Continue to test on dev before completing the main PR.

Enable the Azure policy

Adding the YAML file does not enable enforcement. An administrator must configure Azure DevOps:

1. Land the pipeline file through the two-PR workflow above: merge into develop, then main.
2. Create an Azure Pipeline using the existing .azure/develop-first.yml file on main.
3. Under Repos → Branches → main → Branch policies → Build validation, add that pipeline. Set the trigger to Automatic, the policy requirement to Required, leave the path filter blank, and set build expiration to Immediately when main is updated.
4. Ensure ordinary contributors cannot Bypass policies when completing pull requests or Bypass policies when pushing. Protect changes to the gate YAML with required reviewers.
5. Verify on a test PR that validation fails before the develop PR merges and passes after it merges and validation is requeued. The test must preserve the individual commits on develop.

During rollout, the policy can be configured as optional for this test, then made required at the enforcement date. Once required, it also applies to existing open PRs. A manual pipeline run outside a PR fails intentionally because it has no reviewed source commit to validate.

See Azure Repos build validation for policy settings and git patch-id for patch comparison.