Credential Inventory

This page records where a credential lives and what it is for. It never contains a credential value, and no value may be added to it. The documentation site is published; treat every page here as public.

1. Why this page exists

Credentials for this estate accumulate in whichever place was convenient at the time — a shell history, a chat transcript, an agent’s memory file, a teammate’s note. That spreads the value across places nobody audits, and it leaves nobody able to answer the two questions that matter in an incident: what is this key for, and where is the authoritative copy.

This page answers both. It is the index; the values live on disk in one place, described below.

2. The storage rule

Credential values live on disk under ~/dev/ems/.local/, which is gitignored, at mode 600 in directories at mode 700. They are split across two locations by blast radius, and the split is enforced by tooling rather than convention.

Tier Location Reachable by an agent

Low blast radius — development and stage

.local/credentials/<name>.txt

Yes, directly. These are read constantly, are cheap to rotate, and compromise costs a throwaway environment.

High blast radius — production, and any identity-provider secret

.local/secrets/<name>.key

No. Usable but not readable — see [Using a protected credential].

The second tier is backed by deny rules in the agent harness configuration, which refuse reads of ./secrets/ and .//.key. That is why the protected files carry a .key extension inside a directory called secrets — *both halves of the name are load-bearing, and renaming either one removes what protection there is.

Know what this guard is and is not. The deny rule is evaluated against the shape of the command, so it reliably stops a direct read — but ordinary shell composition gets past it. A subshell and a pipe are enough. Treat it as defence in depth against accidental or naive disclosure, not as a sandbox boundary.

What it genuinely buys: a session cannot casually cat a production key while doing something else, a prompt-injected instruction phrased as a plain read fails, and the failure is loud enough that a person notices the attempt.

What it does not buy: any protection against a determined caller. An agent with shell access can read anything the operator can read. If a credential must be unreachable by an agent, it cannot live on the operator’s disk at all — it belongs in a secret manager the agent has no path to. See design-journal/2026-05/secrets-management-architecture.adoc for that longer-term direction.

If a tool reports that it cannot read a credential under .local/secrets/, that is the guard working as intended. Do not resolve it by renaming the file, moving it out of that directory, by relaxing the deny rules, or by re-phrasing the command until it slips through. Extend the wrapper described below instead. The point is that a production credential is used rather than read, and every legitimate use has a wrapper.
Concern Rule

Credential inventory

This page. Purpose, consumer, storage path, rotation posture. Never a value.

Non-secret identifiers

This page or Environments. Tenant ids, client ids, issuer URLs and hostnames are configuration, not secrets, and are more useful written down than hidden.

Deployed credentials

Kubernetes Secret objects, applied from the local copy. The cluster is the runtime source of truth; the local copy is for reproducing and rotating it.

Three things must never hold a credential value: a Git repository, a documentation page, and an AI agent’s persistent memory. A memory file or a design document may name the credential and point at its path, which is what makes this inventory the thing worth pointing at.

[[Using a protected credential]] == Using a protected credential

.local/bin/ems-api makes an authenticated admin-service call in any environment. It opens the key file inside the process, where the deny rule does not reach, passes it to curl through a header file rather than argv so it cannot leak through ps, and shreds that file on exit. The value is never printed.

.local/bin/ems-api prod-v2 /management/info
.local/bin/ems-api stage   '/api/events?page=0&size=5'
.local/bin/ems-api prod    /api/events -- -X POST -d @body.json -H 'Content-Type: application/json'

Anything after a bare -- is passed to curl verbatim. prod is the legacy hostname and prod-v2 the current line; see Environments.

.local/bin/woo-api does the same for a customer site’s WooCommerce REST API. It sends the storefront key as HTTP Basic credentials through a curl config file, never on the command line and never in the query string (which would write it into the host’s access logs). It is read-only by default: any request that changes data, or carries a body, is refused unless the command includes --write, so every write is visible in the command that runs it.

.local/bin/woo-api wcsc '/orders/12345?_fields=id,status,total'
.local/bin/woo-api wcsc --write /orders/12345 -- -X DELETE

Paths are relative to the site’s /wp-json/wc/v3 root. A DELETE without force=true moves an order to trash, which can be undone from the WooCommerce admin until the trash is emptied.

The same pattern already exists for the federation integration key, in .local/set-csa-key.sh, which reads the key, patches a cluster secret, verifies by fingerprint comparison and restarts the deployment — all without disclosing the value. When a new access pattern needs a protected credential, add a subcommand or a sibling script. That is the intended way to extend this.

3. Inventory

The Storage column gives the path under ~/dev/ems/.local/. A path under credentials/ is directly readable; a path under secrets/ is protected and must be used through the wrapper.

3.1. Service API keys

admin-service authenticates machine callers with an API key passed as the X-API-KEY header. A key is bound to one organisation and carries a set of permission roles.

Credential Purpose Storage Notes

admin-service production ADMIN key

Management endpoints and administrative API calls against production. Carries ADMIN.

secrets/admin-service-prod-admin.key — protected

Works against both production hostnames. Rejected by stage. Use ems-api prod or ems-api prod-v2.

admin-service stage ADMIN key

Administrative API calls and management endpoints against stage.

credentials/admin-service-stage-admin.txt (older copy also at .local/stage-api-key)

Bound to one organisation but effective across organisations for imports and management. Reset by the nightly production clone — see [Stage keys do not survive the nightly clone].

admin-service dev key, single organisation

The admin-portal development gateway’s outbound key on its proxy route.

credentials/admin-service-dev-org4.txt

Deliberately single-organisation: a key bound to two organisations makes the organisation lookup ambiguous and participant import fails.

CSA integration key

Outbound calls to the external cycling federation membership and licensing API.

.local/csa-api.key — protected by extension

Not directly readable. Applied to a cluster secret by .local/set-csa-key.sh, which patches, verifies by fingerprint and restarts without ever printing the value.

3.2. Customer storefronts

A customer’s WordPress site runs WooCommerce, which issues REST API keys as a consumer key and consumer secret pair, each bound to a WordPress user and to a permission level of Read, Write or Read/Write. These keys belong to the customer’s site, not to admin-service.

Credential Purpose Storage Notes

WooCommerce REST key, Western Cape Schools Cycling production

Reading and maintaining storefront orders on the customer’s live shop — reconciliation against EMS, and removing duplicate orders.

secrets/woocommerce-wcsc-prod.key — protected

One line, consumer_key:consumer_secret. Read/Write, issued under the WordPress user christhonie. Use woo-api wcsc. Reduce it to Read, or revoke it, once the maintenance it was issued for is finished. This is not the event payment plugin’s key: the plugin authenticates to admin-service with a separate credential of its own.

3.3. Identity provider

Credential Purpose Storage Notes

Entra client secret, admin-portal development

The gateway’s OIDC client credential for the development app registration.

secrets/entra-admin-portal-dev-client-secret.txt — protected by directory

Loaded at runtime from an environment variable, never inlined into a configuration file. An identity-provider secret is treated as high blast radius regardless of environment. Flagged for rotation — see Rotation.

The matching non-secret identifiers — tenant id, client id, issuer and discovery URLs, registered redirect URIs, requested scopes — are configuration rather than credentials. They live in the service’s own configuration and in Environments.

3.4. Databases

Credential Purpose Storage Notes

MySQL read user, production cluster

Read-only SQL against the production cluster through a bastion tunnel.

~/dev/ems/claude.pwd

Gitignored explicitly by name, and readable — no deny pattern matches it. If a tunnel is up on the expected port but MySQL returns an access-denied error for the local shell account, the tunnel is fine and only the credentials are missing. Connection procedure is in the mysql-idealogic-prod skill, not here.

MySQL, legacy server

SQL against the legacy database on the external host.

See the mysql-legacy-wpca skill

A separate estate with its own access path.

3.5. Observability

Credential Purpose Storage Notes

Grafana viewer service-account token

Reading logs, metrics and dashboards through the Grafana API and the MCP server.

.local/grafana-claude-christhonie.txt

Per-user token, Viewer role. .local/grafana-claude-admin-apikey.txt in the same directory is dead and returns unauthorised on every endpoint; do not reach for it.

Metrics remote-write tokens

Pushing metrics from outside the cluster into the monitoring stack.

.local/grafana-metrics-write.txt, .local/aaks-rancher-idealogic-metric-ingress-write.txt

Write-scoped. Not needed for reading telemetry.

4. Adding a new credential

  1. Decide the tier. Production, or any identity-provider secret, goes to .local/secrets/<name>.key and is reached through the wrapper. Development and stage go to .local/credentials/<name>.txt and are read directly. Write it with umask 077, using a name that says what it opens rather than what it is called upstream.

  2. Add a row to the table above in the right section: purpose, storage filename, and any constraint a future reader would otherwise have to rediscover.

  3. If the credential is consumed by a workload, apply it to the cluster as a Secret and record which deployment reads it.

  4. Never paste the value into a commit message, a work item, a documentation page, or an agent memory file.

The naming convention matters more than it looks. A file called key.txt is unusable a month later; admin-service-stage-admin.key tells the next reader the service, the environment and the permission level without opening it.

5. Verifying a value without disclosing it

Compare fingerprints rather than printing secrets. This is the check to run when a call is failing and you need to know whether the deployed value matches the local one:

sha256sum <(tr -d '[:space:]' < ~/dev/ems/.local/secrets/<file>) | cut -c1-8

Take the same eight characters from the decoded cluster secret and compare. A mismatch localises the fault immediately; a match rules the credential out and sends you to look at permissions instead.

A trailing newline is the classic false alarm here. It produces exactly the same authentication failure as a wrong key, so strip whitespace on both sides before comparing.

6. Rotation

There is no automated rotation. Rotate on these triggers:

  • Disclosure. A value that has appeared in a chat transcript, a shared terminal recording, or any system outside the operator’s own machine is disclosed and must be rotated. The Entra development client secret listed above is in this state and is flagged for rotation once the integration it supports is verified end to end.

  • Expiry. The Entra client secret carries a provider-set expiry. Confirm the date in the portal rather than assuming it.

  • Departure. Any person-scoped token when that person’s access should end.

When a credential is rotated, update the file under .local/secrets/, apply it to every consumer, and correct this page if the storage path or purpose changed. The value’s location is the only thing recorded here, so a rotation that keeps the same path needs no edit — which is the intended property.

[[Stage keys do not survive the nightly clone]] Stage keys do not survive the nightly clone. The stage database is refreshed from production overnight, which truncates the API key tables and re-inserts them from side tables held only in the stage schema. A permission written to the live table lasts until the next refresh. To make it permanent, write the same row to the corresponding seed table as well. Those seed tables are maintained by hand and are deliberately not tracked in a repository, so nothing in Git tells you what stage’s keys are — read the tables.

7. Related