Almeria Industries

Open-Source, Drop-In Replacement for MinIO KES — Backed by HashiCorp Vault

Server Information

Server Identity

Property	Value
Service Name	vkms-kes
API Compatibility	MinIO KES v1 HTTP API
Default Port	7373
Protocol	HTTP/HTTPS (mTLS supported)
Backend	HashiCorp Vault (Transit + KV v2 engines)
Language / Runtime	Go 1.25+

Architecture

┌──────────────────────────────────────────────────────────┐
│  MinIO S3                                                 │
│  (sends HTTPS requests using client cert or API key)      │
└───────────────────────────┬──────────────────────────────┘
                             │  HTTPS / mTLS  (port 7373)
┌───────────────────────────▼──────────────────────────────┐
│  VKMS-KES (this service)                                  │
│                                                             │
│  ┌─────────────┐  ┌──────────────┐  ┌────────────────┐    │
│  │  Auth Layer │  │  Key Service │  │ Policy Engine  │    │
│  │  (5 modes)  │  │  (Transit)   │  │ (glob rules)   │    │
│  └──────┬──────┘  └──────┬───────┘  └────────────────┘    │
│         │                │                                 │
│  ┌──────▼────────────────▼──────────────────────────┐     │
│  │           Vault Backend Driver                    │     │
│  │   Transit engine → crypto operations               │     │
│  │   KV engine      → key/policy/identity metadata    │     │
│  └──────────────────────────────────────────────────┘     │
└───────────────────────────┬──────────────────────────────┘
                             │  HTTP (Vault API)
┌───────────────────────────▼──────────────────────────────┐
│  HashiCorp Vault                                           │
│  Transit engine: master keys + encrypt/decrypt/wrap        │
│  KV v2 engine:   key metadata, policies, identities         │
└──────────────────────────────────────────────────────────┘

Capabilities

• MinIO KES v1 API compatible — drop-in replacement, no MinIO-side changes beyond the endpoint URL
• Envelope encryption via Vault Transit — generate, encrypt, decrypt, import, and delete master keys without ever persisting plaintext key material
• Five authentication modes — none, mtls, tls, api-key, token (JWT)
• Glob-based policy engine — allow/deny rules per identity, evaluated deny-first then allow
• Zero-bootstrap admin API key — auto-generated on first start and persisted to the data directory; no manual key-generation step required
• SIGHUP hot-reload — reload admin identity, auth mode, logging, and policies without downtime
• Server-Sent Events audit/error streams — real-time structured JSON logs for every request
• In-memory DEK cache — configurable TTL to cut Vault round-trips on repeated decrypt calls
• Prometheus metrics + OpenTelemetry/Jaeger tracing — full observability stack
• Web UI — dashboard, key/policy/identity management, user accounts
• vkms-cli command-line client — manage keys, policies, identities, API keys, and stream logs from the terminal
• Swagger / OpenAPI docs — interactive API reference in development mode

Key Flows

Generate DEK (primary MinIO flow)

MinIO                    VKMS-KES                  Vault Transit
  │                         │                           │
  │─── POST /v1/key/generate/minio-key ──────────────►  │
  │                         │                           │
  │                         │── 1. crypto/rand 32 bytes │
  │                         │                           │
  │                         │── POST /transit/encrypt/minio-key ──►
  │                         │        {"plaintext": base64(32 bytes)}
  │                         │◄── {"ciphertext": "vault:v1:..."}
  │                         │                           │
  │◄── {"plaintext":"<b64>","ciphertext":"vault:v1:..."} │
  │                         │                           │
  │  MinIO encrypts object with plaintext DEK,          │
  │  stores ciphertext alongside object.                │
  │  Plaintext DEK is discarded.                        │

Decrypt DEK (MinIO reading an object)

MinIO                    VKMS-KES                  Vault Transit
  │                         │                           │
  │─── POST /v1/key/decrypt/minio-key ───────────────►  │
  │    {"ciphertext":"vault:v1:..."}                    │
  │                         │                           │
  │                         │  (cache hit? return early)│
  │                         │── POST /transit/decrypt/minio-key ──►
  │                         │        {"ciphertext":"vault:v1:..."}
  │                         │◄── {"plaintext": base64(32 bytes)}
  │                         │                           │
  │◄── {"plaintext":"<b64>"} │                           │
  │                         │                           │
  │  MinIO decrypts object with recovered plaintext DEK.│

Policy Enforcement Flow

Request arrives ──► KESAuth middleware
                        │
                        ▼
               Extract identity hash
               (from cert / API key / JWT)
                        │
                        ▼
               identity == admin? ──YES──► allow
                        │ NO
                        ▼
               GetIdentityData(identity)
               → policy name
                        │
                        ▼
               GetPolicyDocument(policy)
               → allow/deny glob rules
                        │
                        ▼
               Check deny rules (glob match)
               → match? deny (403)
                        │ no match
                        ▼
               Check allow rules (glob match)
               → match? allow
               → no match? deny (403)

Vault Integration

VKMS-KES uses two Vault engines.

1. Transit Engine (cryptographic operations)

The Transit engine performs all key wrapping/unwrapping. VKMS-KES never stores plaintext key material anywhere.

Operation	Vault API Call
Create key	POST /transit/keys/<name> — creates aes256-gcm96 key
Delete key	POST /transit/keys/<name>/config (set deletion_allowed=true), then DELETE /transit/keys/<name>
List keys	LIST /transit/keys
Encrypt DEK	POST /transit/encrypt/<name> with {"plaintext": base64}
Decrypt DEK	POST /transit/decrypt/<name> with {"ciphertext": "vault:v1:..."}

2. KV v2 Engine (metadata storage)

The KV engine stores creation timestamps, policies, and identity assignments. No key material is stored here.

Data	KV Path
Key metadata	<engine>/data/<prefix>/keys/<name> → {created_at, created_by}
Policy	<engine>/data/<prefix>/policies/<name> → {allow, deny, created_at}
Identity	<engine>/data/<prefix>/identities/<identity_hash> → {policy, created_at}

Vault Setup

# Enable Transit engine
vault secrets enable transit

# Enable KV v2 engine
vault secrets enable -version=2 kv

# Create a policy for VKMS-KES
vault policy write vkms-kes - <<EOF
# Transit — manage keys and perform crypto
path "transit/keys/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "transit/encrypt/*" {
  capabilities = ["update"]
}
path "transit/decrypt/*" {
  capabilities = ["update"]
}
path "transit/wrapping_key" {
  capabilities = ["read"]
}

# KV — store metadata
path "kv/data/vkms/*" {
  capabilities = ["create", "read", "update", "delete"]
}
path "kv/metadata/vkms/*" {
  capabilities = ["list", "delete"]
}
EOF

# Create a token (development)
vault token create -policy=vkms-kes -ttl=8760h

API Reference

All endpoints return JSON. Error responses always use {"message": "..."}.

System

Method	Path	Auth Required	Description
GET	/version	No	Server version
GET	/v1/ready	No	Readiness probe — returns 200
GET	/v1/status	Yes	Server uptime, arch, backend state
GET	/v1/api	Yes	List of all supported endpoints
GET	/v1/metrics	No	Prometheus metrics
GET	/v1/log/audit	Yes	SSE stream of audit events
GET	/v1/log/error	Yes	SSE stream of error events (4xx/5xx only)

Key Management

Method	Path	Description
POST	/v1/key/create/{name}	Create a new master key
POST	/v1/key/import/{name}	Import raw key material (base64-encoded 256-bit)
DELETE	/v1/key/delete/{name}	Permanently delete a master key
GET	/v1/key/describe/{name}	Key metadata (name, created_at, created_by)
GET	/v1/key/list/{prefix}	List key names (optional prefix filter)
POST	/v1/key/generate/{name}	Generate DEK → {"plaintext":"<b64>","ciphertext":"vault:v1:..."}
POST	/v1/key/encrypt/{name}	Encrypt arbitrary data → {"ciphertext":"vault:v1:..."}
POST	/v1/key/decrypt/{name}	Decrypt ciphertext → {"plaintext":"<b64>"}

Policy Management

Method	Path	Description
POST	/v1/policy/set/{name}	Create or replace a policy
GET	/v1/policy/describe/{name}	Policy metadata
GET	/v1/policy/read/{name}	Full allow/deny rule list
DELETE	/v1/policy/delete/{name}	Delete a policy
GET	/v1/policy/list/{prefix}	List policies

Identity Management

Method	Path	Description
POST	/v1/identity/assign/{policy}/{identity}	Assign identity hash to a policy
GET	/v1/identity/describe/{identity}	Identity's policy and metadata
DELETE	/v1/identity/delete/{identity}	Remove identity assignment
GET	/v1/identity/list/{prefix}	List identity hashes
GET	/v1/identity/self/describe	Caller's own identity and policy

Authentication Modes

Set via auth.mode in vkms.yaml or the VKMS_AUTH_MODE environment variable.

none — Development Only

No authentication. The server accepts connections only from loopback addresses (127.0.0.1, ::1).

auth:
  mode: "none"

curl http://localhost:7373/v1/key/create/my-key

mtls — Mutual TLS (Production)

Both the server and client present TLS certificates. The client's identity is the SHA-256 hex digest of the DER-encoded SubjectPublicKeyInfo from the client certificate.

auth:
  mode: "mtls"
tls:
  key:  "/etc/vkms/server.key"
  cert: "/etc/vkms/server.crt"
  ca:   "/etc/vkms/ca.crt"     # CA that signed client certificates

curl --cert client.crt --key client.key --cacert ca.crt \
  https://vkms-kes:7373/v1/key/generate/minio-key

api-key — API Key (TLS terminated upstream)

Clients authenticate with an API key in the Authorization header. The key encodes an Ed25519 private seed (32 bytes, base64-encoded). The identity is SHA-256(Ed25519 public key).

Format: Authorization: ApiKey kes:v1:<base64-encoded-seed>

The service listens on plain HTTP — TLS must be terminated by an upstream proxy (e.g. Traefik). No tls.key or tls.cert is required.

auth:
  mode: "api-key"

# tls block not required — TLS is terminated upstream.
tls:
  key:  ""
  cert: ""

curl -H "Authorization: ApiKey kes:v1:<base64-seed>" \
  https://vkms-kes.example.com/v1/key/generate/minio-key

The vkms-cli api-key new command generates non-admin client keys (e.g. for MinIO or other service accounts), which are then assigned to policies via vkms-cli identity assign.

tls — Server TLS + API Key

Identical to api-key mode for identity extraction. Server TLS is required; clients are not required to present a certificate, but they must supply an ApiKey header.

token — JWT Bearer Token

Clients authenticate with a JWT in the Authorization: Bearer <token> header. The identity is SHA-256(value of the configured claim).

auth:
  mode: "token"
  identity_claim: "sub"   # which JWT claim to use (default: "sub")
  issuer: "https://accounts.example.com"
  audience: "vkms"
tls:
  key:  "/etc/vkms/server.key"
  cert: "/etc/vkms/server.crt"

Admin Identity

One identity can be designated as admin in vkms.yaml. The admin bypasses all policy checks and can perform any operation including policy and identity management. The admin identity should be held by operators only, never by MinIO.

admin:
  identity: "3ecfaaa5f68f5d4b3213b9bc4c9a4f7e2a8d0123..."  # hex SHA-256

Override with: VKMS_ADMIN_IDENTITY=<hex-sha256>

TLS and Certificates

Mode	Server TLS	Client Certificate	Identity Source
none	No	No	None (loopback only)
mtls	Yes (service handles it)	Yes (required + verified)	SHA-256(cert.SubjectPublicKeyInfo)
tls	Yes (service handles it)	No	API key header
api-key	No (terminated upstream)	No	API key header
token	No (terminated upstream)	No	JWT claim

In mtls mode, the server loads the CA certificate (tls.ca) and sets ClientAuth: RequireAndVerifyClientCert. The identity hash is computed from the client's public key, not certificate metadata — it remains stable across certificate rotation as long as the key pair is reused.

Generate self-signed certificates for local mTLS testing:

# 1. Create a CA
openssl genrsa -out ca.key 4096
openssl req -new -x509 -days 3650 -key ca.key -out ca.crt \
  -subj "/CN=VKMS-CA"

# 2. Create the server key and certificate
openssl genrsa -out server.key 4096
openssl req -new -key server.key -out server.csr \
  -subj "/CN=localhost"
openssl x509 -req -days 365 -in server.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out server.crt \
  -extfile <(echo "subjectAltName=IP:127.0.0.1,DNS:localhost")

# 3. Create a client key and certificate (for mTLS mode)
openssl genrsa -out client.key 4096
openssl req -new -key client.key -out client.csr \
  -subj "/CN=minio-client"
openssl x509 -req -days 365 -in client.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client.crt

# 4. Compute the client identity hash (needed for admin or policy assignment)
openssl x509 -in client.crt -pubkey -noout \
  | openssl pkey -pubin -outform DER \
  | openssl dgst -sha256 -hex \
  | awk '{print $2}'

Installation

Prerequisites

• Go 1.25+
• task (optional, for Taskfile.yaml tasks)
• HashiCorp Vault (local dev: vault server -dev)

Building from Source

cd src

# Generate Swagger docs, vet, build
task build

# Or directly
go build -o vkms-kes main.go

Run (Development, No Auth)

cd src

export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=root           # dev Vault root token
export VAULT_ENGINE_NAME=kv
export VAULT_TRANSIT_ENGINE=transit
export VAULT_PREFIX=vkms

./vkms-kes -d -p 7373

Start a local Vault dev server and enable required engines, then exercise the key lifecycle:

vault server -dev -dev-root-token-id=root

export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=root
vault secrets enable transit
vault secrets enable -version=2 kv

cd src
VAULT_ADDR=http://127.0.0.1:8200 VAULT_TOKEN=root ./vkms-kes -d -p 7373

# Create a master key
curl -s -X POST http://localhost:7373/v1/key/create/minio-key

# Generate a DEK (the core MinIO operation)
curl -s -X POST http://localhost:7373/v1/key/generate/minio-key | jq .
# → {"plaintext":"<base64>","ciphertext":"vault:v1:<...>"}

# Server status
curl -s http://localhost:7373/v1/status | jq .
# → {"version":"v0.21.4","os":"darwin","arch":"amd64","uptime":12,"keystore":{"state":"Online","type":"vault"}}

Run (mTLS, Self-Signed Certs)

# Compute the client certificate identity hash
CLIENT_IDENTITY=$(openssl x509 -in client.crt -pubkey -noout \
  | openssl pkey -pubin -outform DER \
  | openssl dgst -sha256 -hex \
  | awk '{print $2}')

Set up vkms.yaml:

address: "0.0.0.0:7373"

admin:
  identity: ""   # set to $CLIENT_IDENTITY to give this cert admin access

auth:
  mode: "mtls"

tls:
  key:  "./server.key"
  cert: "./server.crt"
  ca:   "./ca.crt"

VAULT_ADDR=http://127.0.0.1:8200 VAULT_TOKEN=root \
VKMS_ADMIN_IDENTITY=$CLIENT_IDENTITY \
./vkms-kes -p 7373

# Create a key (using mTLS client cert)
curl -s --cert client.crt --key client.key --cacert ca.crt \
  -X POST https://localhost:7373/v1/key/create/minio-key

# Generate a DEK
curl -s --cert client.crt --key client.key --cacert ca.crt \
  -X POST https://localhost:7373/v1/key/generate/minio-key | jq .

Configuration

vkms.yaml

address: "0.0.0.0:7373"

admin:
  identity: ""   # hex SHA-256; empty = no admin bypass

auth:
  mode: "none"              # none | mtls | tls | api-key | token
  issuer: ""                # token mode: OIDC issuer URL
  audience: ""              # token mode: expected audience claim
  identity_claim: "sub"     # token mode: JWT claim used as identity

tls:
  key:  ""    # PEM server private key (required for all modes except none)
  cert: ""    # PEM server certificate
  ca:   ""    # PEM client CA (required for mtls only)

log:
  audit: true   # enable /v1/log/audit SSE stream
  error: true   # enable /v1/log/error SSE stream

cache:
  keys:
    ttl: "0"    # DEK cache TTL; "0" = disabled; e.g. "5m"

backend:
  type: vault
  vault:
    endpoint: "http://127.0.0.1:8200"
    engine:    "transit"   # Transit engine name
    kv_engine: "kv"        # KV v2 engine name
    prefix:    "vkms"      # KV path prefix
    auth:
      method: "token"      # token | approle | kubernetes
      token:  ""           # prefer VAULT_TOKEN env var

policies:
  minio-policy:
    allow:
      - /v1/key/create/*
      - /v1/key/generate/*
      - /v1/key/decrypt/*
      - /v1/key/describe/*
      - /v1/key/list/*
    deny: []

Environment Variables

All VKMS_* variables override the corresponding vkms.yaml field.

Variable	Overrides	Example / Default
VKMS_CONFIG	Config file path	/etc/vkms/vkms.yaml
VKMS_ADDRESS	address	0.0.0.0:7373
VKMS_ADMIN_IDENTITY	admin.identity	3ecfaaa5...
VKMS_AUTH_MODE	auth.mode	mtls (default none)
VKMS_TLS_KEY	tls.key	/etc/vkms/server.key
VKMS_TLS_CERT	tls.cert	/etc/vkms/server.crt
VKMS_TLS_CA	tls.ca	/etc/vkms/ca.crt
VKMS_TOKEN_ISSUER	auth.issuer	https://accounts.example.com
VKMS_TOKEN_AUDIENCE	auth.audience	vkms
VKMS_TOKEN_IDENTITY_CLAIM	auth.identity_claim	sub (default)
VKMS_BACKEND_TYPE	backend.type	vault (default)
VKMS_LOG_AUDIT	log.audit	true (default)
VKMS_LOG_ERROR	log.error	true (default)
STORAGE_DIRECTORY	Persistent data directory	/data (default)
VAULT_ADDR	Vault endpoint	http://vault:8200
VAULT_TOKEN	Vault token	s.xxxxx
VAULT_ENGINE_NAME	KV engine name	kv (default)
VAULT_TRANSIT_ENGINE	Transit engine name	transit (default)
VAULT_PREFIX	KV path prefix	vkms (default)
WEB_UI_ENABLED	Enable the Web UI	false (default)
LOCAL_AUTH_ENABLED	Enable local Web UI authentication	true (default)
LOCAL_AUTH_USERNAME	Bootstrap Web UI admin username	admin (default)
LOCAL_AUTH_PASSWORD	Bootstrap Web UI admin password	changeme (default)
REQUEST_BASE_URL	Base URL used by the Web UI (e.g. OAuth redirect)	https://vkms-kes.example.com
JAEGER_ENDPOINT	Jaeger telemetry endpoint	(empty)

Server CLI Flags

Flag	Short	Default	Description
--port	-p	7373	HTTP port
--devel	-d	(disabled)	Development mode (enables Swagger, relaxes some checks)
--swagger	-s	(disabled)	Enable Swagger UI at /swagger/index.html
--telemetry	-r	(empty)	Telemetry mode: local (stdout) or remote (Jaeger)
--gin-logger	-g	(disabled)	Enable Gin's request logger (unstructured, dev only)
--vault-logging	-v	(disabled)	Enable Vault client debug logging (dev only)
--pprof	-o	(disabled)	Enable pprof profiling endpoints
--timeout	-t	60	Graceful shutdown timeout (seconds)

SIGHUP Hot-Reload

Send SIGHUP to reload vkms.yaml without restarting the server:

kill -HUP $(pgrep vkms-kes)

This reloads:

• admin.identity
• auth.mode and token settings
• log.audit and log.error
• policies (synced into Vault KV)

In a Docker deployment: docker kill --signal=HUP vkms-kes

Web UI

Enable with WEB_UI_ENABLED=true. The Web UI provides:

• Status / Dashboard — server uptime, Vault backend health, keystore state, and the admin API key (masked reveal + copy-to-clipboard) when running in api-key mode
• Keys — create, delete, list, describe master keys, and generate DEKs interactively
• Policies — create/replace, read, describe, and delete allow/deny policies
• Identities — assign identities to policies, describe or derive an identity hash, delete assignments
• Users — local Web UI accounts: create, update, delete, change/reset password
• Login — local username/password authentication (LOCAL_AUTH_*) via cookie session

Navigate to http://localhost:7373/ui (or your configured base URL) to log in.

Command-Line Client (vkms-cli)

vkms-cli is the companion command-line client for operating a running VKMS-KES server — managing keys, policies, identities, API keys, and streaming logs, all against any of the five auth modes.

vkms-cli is the command-line client for the VKMS-KES key management server.

It supports all five authentication modes: none, api-key, tls, mtls, and token.
Credentials and the server endpoint can be supplied via flags or environment variables.

Global Flags

Flag	Env Var	Default	Description
--endpoint	VKMS_ENDPOINT	http://localhost:7373	VKMS-KES server endpoint
--api-key	VKMS_API_KEY	(empty)	API key for api-key/tls auth
--token	VKMS_TOKEN	(empty)	Bearer JWT token for token auth
--tls-cert	VKMS_TLS_CERT	(empty)	Client TLS certificate for mTLS auth
--tls-key	VKMS_TLS_KEY	(empty)	Client TLS private key for mTLS auth
--tls-ca	VKMS_CA	(empty)	CA certificate bundle
--insecure	—	false	Skip TLS certificate verification (development only)

Command Groups

Command	Subcommands	Description
vkms-cli key	create, import, delete, list, generate, encrypt, decrypt, describe	Manage cryptographic master keys
vkms-cli policy	set, describe, read, delete, list	Manage access control policies
vkms-cli identity	assign, describe, delete, list, self	Manage identities and policy assignments
vkms-cli api-key	new, identity	Generate client API keys / derive identity hash from a key
vkms-cli log	audit, error	Stream audit or error logs (SSE)
vkms-cli status	—	Show server status
vkms-cli version	—	Show server version
vkms-cli api	—	List available API endpoints
vkms-cli ready	—	Check server readiness (exit 0 if ready, 1 if not)

Example Usage

export VKMS_ENDPOINT=https://vkms.example.com
export VKMS_API_KEY=kes:v1:...

./vkms-cli status
./vkms-cli key list
./vkms-cli key create minio-backend-default-key

# Generate a dedicated client key (not the admin key)
./vkms-cli api-key new
# api-key     : kes:v1:MINIO...
# identity    : a3f2...

./vkms-cli policy set minio \
  --allow "/v1/key/create/*" \
  --allow "/v1/key/generate/*" \
  --allow "/v1/key/decrypt/*"

./vkms-cli identity assign minio a3f2...

If the server is behind a privately-signed CA (e.g. internal PKI):

export VKMS_CA=/path/to/internal-ca.crt
./vkms-cli key list

MinIO Integration

Configure MinIO to use VKMS-KES with environment variables:

# The KES server endpoint
MINIO_KMS_KES_ENDPOINT=https://vkms-kes:7373

# Server CA certificate (to verify the VKMS-KES server cert)
MINIO_KMS_KES_CAPATH=/etc/minio/certs/vkms-ca.crt

# Client certificate (MinIO's identity — add this to VKMS-KES as an identity)
MINIO_KMS_KES_CERT_FILE=/etc/minio/certs/minio-kes.crt
MINIO_KMS_KES_KEY_FILE=/etc/minio/certs/minio-kes.key

# Default master key name
MINIO_KMS_KES_KEY_NAME=minio-backend-default-key

On VKMS-KES, create the MinIO master key and assign the MinIO identity:

# Compute MinIO's identity from its certificate
MINIO_IDENTITY=$(openssl x509 -in minio-kes.crt -pubkey -noout \
  | openssl pkey -pubin -outform DER \
  | openssl dgst -sha256 -hex | awk '{print $2}')

# Create the master key (using admin cert)
curl --cert admin.crt --key admin.key --cacert ca.crt \
  -X POST https://vkms-kes:7373/v1/key/create/minio-backend-default-key

# Create a MinIO policy
curl --cert admin.crt --key admin.key --cacert ca.crt \
  -X POST https://vkms-kes:7373/v1/policy/set/minio \
  -H "Content-Type: application/json" \
  -d '{
    "allow": [
      "/v1/key/create/*",
      "/v1/key/generate/*",
      "/v1/key/decrypt/*",
      "/v1/key/describe/*",
      "/v1/key/list/*"
    ],
    "deny": []
  }'

# Assign MinIO's identity to the policy
curl --cert admin.crt --key admin.key --cacert ca.crt \
  -X POST "https://vkms-kes:7373/v1/identity/assign/minio/$MINIO_IDENTITY"

DEK Cache

To reduce latency on repeated decrypt calls (common when MinIO reads many objects in parallel), enable the in-memory DEK cache:

cache:
  keys:
    ttl: "5m"   # cache decrypted DEKs for 5 minutes

The cache maps ciphertext → plaintext in memory. A cache hit skips the Vault round-trip entirely. The cache is intentionally not persisted — it is cleared on restart.

Monitoring & Observability

Audit and Error Log Streams

VKMS-KES emits structured JSON audit events for every request via Server-Sent Events.

# Stream all audit events
curl -N --cert client.crt --key client.key --cacert ca.crt \
  https://localhost:7373/v1/log/audit

# Stream error events only (HTTP 4xx / 5xx)
curl -N --cert client.crt --key client.key --cacert ca.crt \
  https://localhost:7373/v1/log/error

Example event:

data: {
  "time": "2025-06-18T10:23:45.123456789Z",
  "request": {
    "path": "/v1/key/generate/minio-key",
    "identity": "3ecfaaa5f68f5d4b..."
  },
  "response": {
    "code": 200,
    "time": "1.24ms"
  }
}

Enable/disable streams via vkms.yaml (log.audit, log.error) or reload with SIGHUP after editing the config file.

Metrics and Tracing

Endpoint	Description
GET /v1/metrics	Prometheus metrics (request counts, latencies)
GET /v1/log/audit	Real-time audit log via SSE
GET /v1/log/error	Real-time error log via SSE
GET /v1/status	Backend health check

Distributed tracing is available via OpenTelemetry. Enable with --telemetry remote and set JAEGER_ENDPOINT.

Deployment

Docker

Build:

cd $GOPATH/src
docker build -t vkms-kes -f go-vkms-kes/Dockerfile .

Data directory — the container exposes a single persistent directory at /data. Mount a named volume or host path here to persist state across restarts:

File	Created by	Purpose
/data/admin.apikey	Service (auto)	Admin API key — generated on first start, reused on every subsequent start
/data/vkms.yaml	Operator (optional)	Custom configuration — if absent the service starts from built-in defaults + env vars

Run with defaults (no config file) — the service starts with auth.mode: api-key, auto-generates the admin API key, and stores it in the named volume:

docker run -d \
  --name vkms-kes \
  -p 7373:7373 \
  -v vkms-data:/data \
  -e VAULT_ADDR=http://vault:8200 \
  -e VAULT_TOKEN=s.xxxxx \
  -e VAULT_ENGINE_NAME=kv \
  -e VAULT_TRANSIT_ENGINE=transit \
  -e VAULT_PREFIX=vkms \
  vkms-kes

Retrieve the generated admin API key:

docker exec vkms-kes cat /data/admin.apikey
# kes:v1:47DEQpj8HBSa...

Run with a custom config file:

# Use the repo's vkms.yaml as a starting point
cp vkms.yaml my-vkms.yaml

# Edit to taste (cache TTL, policies, log settings, auth mode, etc.)
$EDITOR my-vkms.yaml

docker run -d \
  --name vkms-kes \
  -p 7373:7373 \
  -v vkms-data:/data \
  -v $(pwd)/my-vkms.yaml:/data/vkms.yaml:ro \
  -e VAULT_ADDR=http://vault:8200 \
  -e VAULT_TOKEN=s.xxxxx \
  -e VAULT_ENGINE_NAME=kv \
  -e VAULT_TRANSIT_ENGINE=transit \
  -e VAULT_PREFIX=vkms \
  vkms-kes

Hot-reload config without restarting:

docker kill --signal=HUP vkms-kes

Enable the Web UI (admin API key also visible on the dashboard):

docker run -d \
  --name vkms-kes \
  -p 7373:7373 \
  -v vkms-data:/data \
  -e VAULT_ADDR=http://vault:8200 \
  -e VAULT_TOKEN=s.xxxxx \
  -e VAULT_ENGINE_NAME=kv \
  -e VAULT_TRANSIT_ENGINE=transit \
  -e VAULT_PREFIX=vkms \
  -e WEB_UI_ENABLED=true \
  -e LOCAL_AUTH_USERNAME=admin \
  -e LOCAL_AUTH_PASSWORD=changeme \
  vkms-kes

Navigate to http://localhost:7373/ui to log in.

HashiCorp Nomad

TLS is terminated by Traefik (configured via Consul service tags with certresolver). The service receives plain HTTP inside the container, so no TLS cert/key paths are needed on the service itself. Use api-key auth mode — clients authenticate via the Authorization: ApiKey kes:v1:<base64> header over the Traefik-terminated HTTPS connection.

group "vkms" {
  volume "vkms-data" {
    type      = "host"
    source    = "vkms-data"   # defined in the Nomad client config
    read_only = false
  }

  task "vkms-kes" {
    volume_mount {
      volume      = "vkms-data"
      destination = "/data"
    }

    env {
      VAULT_ADDR         = "http://vault.service.consul:8200"
      VAULT_TOKEN        = "<token>"
      VKMS_AUTH_MODE     = "api-key"
      STORAGE_DIRECTORY  = "/data"
      # VKMS_ADMIN_IDENTITY is intentionally omitted — derived from /data/admin.apikey
      # VKMS_TLS_CERT and VKMS_TLS_KEY are intentionally omitted —
      # Traefik handles TLS; the service listens on plain HTTP internally.
    }
  }
}

Traefik service registration (Consul service tags):

service {
  name = "vkms-kes"
  port = "http"

  tags = [
    "traefik.enable=true",
    "traefik.http.routers.vkms-kes.rule=Host(`vkms.example.com`)",
    "traefik.http.routers.vkms-kes.entrypoints=https",
    "traefik.http.routers.vkms-kes.tls=true",
    "traefik.http.routers.vkms-kes-redirect.rule=Host(`vkms.example.com`)",
    "traefik.http.routers.vkms-kes-redirect.entrypoints=http",
    "traefik.http.routers.vkms-kes-redirect.middlewares=redirect-to-https",
  ]
}

After the first start, retrieve the admin API key from the Web UI dashboard, or read it directly from the host:

cat /path/to/nomad/data/vkms-data/admin.apikey
# kes:v1:47DEQpj8HBSa...

CLI usage against the Traefik HTTPS endpoint:

export VKMS_ENDPOINT=https://vkms.example.com
export VKMS_API_KEY=kes:v1:...   # from /data/admin.apikey or the Web UI

./vkms-cli status
./vkms-cli key list

For environments where the service must handle TLS directly (no reverse proxy), use mtls instead:

env {
  VAULT_ADDR          = "http://vault.service.consul:8200"
  VAULT_TOKEN         = "<token>"
  VKMS_AUTH_MODE      = "mtls"
  VKMS_TLS_CERT       = "/secrets/server.crt"
  VKMS_TLS_KEY        = "/secrets/server.key"
  VKMS_TLS_CA         = "/secrets/ca.crt"
  VKMS_ADMIN_IDENTITY = "<hex-sha256>"
}

Security Considerations

• Plaintext key material never persisted — all master keys live in Vault's Transit engine; VKMS-KES only holds plaintext DEKs transiently in memory during a request (and optionally in the DEK cache, see above)
• Deny-first policy evaluation — deny rules are checked before allow rules, so an explicit deny always wins
• Admin identity isolation — reserve the admin identity for operators; service accounts (MinIO, S3-Storage, etc.) should always use scoped, policy-bound identities
• Stable identity on cert rotation — mTLS identity is derived from the public key, not certificate metadata, so rotating a certificate for the same key pair does not require re-assigning policies
• JWT signature not verified internally — token mode must sit behind a validating proxy in production
• File permissions — admin.apikey is written with mode 0600; ensure the mounted volume preserves this
• TLS termination boundary — in api-key/token modes the service trusts that the upstream proxy has already terminated TLS; never expose the plain-HTTP listener directly to untrusted networks

Troubleshooting

Issue	Cause	Solution
403 Forbidden on every call	Identity not assigned to a policy, or policy lacks the required allow rule	Check vkms-cli identity describe <identity> and vkms-cli policy read <policy>
New admin key generated on every restart	$STORAGE_DIRECTORY is not a persistent volume	Mount a named volume / host path at /data (or your configured STORAGE_DIRECTORY)
Connection refused on none mode	Request not originating from loopback	none mode only accepts 127.0.0.1/::1 — use another auth mode for remote clients
mTLS handshake failure	Client cert not signed by the configured CA, or CA path wrong	Verify tls.ca points to the CA that signed the client certificate
Vault errors on key operations	Transit/KV engines not enabled, or token lacks required capabilities	Re-check the Vault Setup policy and enabled secret engines
SIGHUP has no effect on TLS changes	TLS certs/keys are not hot-reloadable by design	Perform a full restart after rotating TLS material

API Documentation

Interactive Swagger/OpenAPI documentation is available in development mode:

./vkms-kes -d -s
# → Swagger UI at http://localhost:7373/swagger/index.html

• Requires: start server with -s or -d flag
• Format: OpenAPI 3.0 specification

System Requirements

Resource	Minimum	Recommended
CPU	1 core	2+ cores
RAM	256 MB	1 GB+
Disk	50 MB (app) + admin.apikey/config	Depends on log retention / cache usage

Runtime dependency: a reachable HashiCorp Vault instance with the Transit and KV v2 secret engines enabled.