# Amaquet Documentation — Full Text > Complete, build-generated Markdown export of the official Amaquet documentation for language models, retrieval systems, and offline reference. Each document includes its canonical site path and an implementation-aligned summary. Source order follows the public documentation route structure. --- ## Amaquet documentation Source: [canonical documentation page](/) Summary: Official Amaquet documentation for installing, configuring, operating, and integrating with the standalone typed in-memory database and its binary protocol. [![CI](https://github.com/newfoundcodes/amaquet/actions/workflows/ci.yml/badge.svg)](https://github.com/newfoundcodes/amaquet/actions/workflows/ci.yml) [![Soak](https://github.com/newfoundcodes/amaquet/actions/workflows/soak.yml/badge.svg)](https://github.com/newfoundcodes/amaquet/actions/workflows/soak.yml) Amaquet is a standalone, Redis-like in-memory database written in Go. It does **not** run on Redis and it does not use Redis as a storage backend. The core database stores Amaquet values directly in process memory and exposes them through the Amaquet binary protocol. The repository includes the database server, a Go client, `amaquet-cli`, an HTTP administration API, optional append-only persistence, API-key authentication, RBAC, Ed25519 protocol identity, Docker deployment assets, deterministic tests, and fuzzing tools. ### Main capabilities Amaquet combines a typed in-memory data plane with a separate HTTP control plane. The following list summarizes the implemented protocol, storage, persistence, administration, and test surfaces. - Custom `amaquet://` protocol on TCP and `amaquets://` on TLS. - 91 stable wire-level data types. - Strongly typed nested values for collections and messaging structures. - 256-shard concurrent keyspace with TTL and key versions. - Optional CRC-protected append-only operation journal. - Organization, member, API-key, RBAC, system, data-access, and audit administration APIs. - HTTP administration API for control-plane operations and metrics. - Per-type Bash test and fuzz entrypoints plus raw-frame fuzzing. ### Where to start Read these pages in order for a first installation; operators and client authors can then branch into the architecture, persistence, and protocol references that match their work. 1. [Install Amaquet](/getting-started/installation/). 2. Follow the [quick start](/getting-started/quickstart/). 3. Read the [configuration reference](/getting-started/configuration/). 4. Review the [data-type catalog](/data-types/). 5. Read the [Amaquet protocol](/protocol/overview/) if you are building a client. 6. Review [architecture](/concepts/architecture/) and [persistence](/concepts/persistence/) before production deployment. ### Process model Amaquet exposes a TCP/TLS data-plane listener and a separate HTTP control-plane listener: ```mermaid flowchart TB Client["Native client"] -->|Amaquet protocol on :13378| Protocol["TCP/TLS protocol server"] Protocol --> Authorization["Dispatcher and RBAC"] Authorization --> Engine["In-memory core engine"] Automation["Automation or curl"] -->|HTTP on :13379| Admin["Admin HTTP API"] Admin -. control-plane operations .-> Authorization Admin -. shared in-memory state .-> Engine ``` The administration API is a control-plane surface. It is not used as a backing data store. --- ## Amaquet architecture Source: [canonical documentation page](/architecture/) Summary: Amaquet architecture reference covering the sharded storage engine, typed values, protocol server, Go client, HTTP control plane, and append-only persistence. This page describes the server's storage engine, protocol boundary, persistence, and administration architecture. ### Storage engine The keyspace is divided across 256 FNV-1a-selected shards. Each shard owns a map and an RW mutex. A stored entry contains: - strongly typed `core.Value`; - creation/update timestamps; - optional expiry timestamp; - monotonic per-key version; - atomic access counter and last-access timestamp; - approximate engine memory size used for memory governance. The engine supports `SET` with NX/XX behavior, `GET`, deletion, TTL, persist, compare-and-set, in-place atomic updates, cursor scans, and an internal deterministic multi-key transaction primitive. Expiration is scheduled through a versioned min-heap and also enforced lazily on access. Engine key count and compression telemetry are maintained incrementally. Complex values have their own fine-grained mutexes. Blocking queues, barriers, and semaphores do not hold a keyspace shard lock while waiting. ### Type and encoding separation `core.DataType` is a stable public identifier. The concrete Go representation can evolve without changing the wire type name. This gives the project room to add packed/small-object encodings later while preserving protocol compatibility. ### Protocol server The TCP server handles one frame-reader loop per connection and can execute multiple request IDs concurrently. Response writes are synchronized, and `CANCEL` can cancel an in-flight request context. It supports: - optional TLS; - API-key authentication; - RBAC permission checks; - command dispatch; - Pub/Sub event frames; - optional Ed25519 identity response in HELLO; - a connection semaphore for the configured connection limit. ### Go client `pkg/amaquet` has a background frame reader and a request-ID routing table. It supports concurrent requests on one connection and routes asynchronous subscription events to dedicated channels. ### Control plane The HTTP control plane is separate from the Amaquet data protocol. It stores only management metadata in `data_dir/admin.json`: - organization; - team members; - roles/RBAC; - hashed API keys; - audit events. It never stores application/database keys in another database. ### Optional AOF The current AOF is a transactional write-ahead journal (`AMQTAOF2`). A mutation is prepared before memory changes and is then committed or aborted. Recovery replays committed prepares only. Time-dependent requests are resolved to stable IDs/timestamps/absolute expirations before journaling, so restart time cannot change historical semantics. AOF records have CRC protection. Background fsync errors become persistence-health failures. Legacy `AMQTAOF1` files are migrated when opened. Verified checkpoints and online compaction are available through the administration API. --- ## Architecture Source: [canonical documentation page](/concepts/architecture/) Summary: Amaquet separates protocol, authorization, dispatch, storage, value implementations, persistence, and administration. Amaquet separates protocol, authorization, dispatch, storage, value implementations, persistence, and administration. ### Components Requests cross a small set of explicit package boundaries. Native clients use the framed protocol server, while HTTP administration requests reach the same dispatcher and engine through the control-plane package. ```mermaid flowchart TB Client["pkg/amaquet client"] -->|Amaquet frames| Protocol["internal/protocol"] Protocol --> TCP["internal/server/tcp"] TCP --> Auth["Authentication and RBAC"] Auth --> Dispatcher["server.Dispatcher"] Dispatcher --> Engine["core.Engine"] Dispatcher -. optional append-only file .-> AOF["internal/persistence"] Engine --> Types["internal/types/*"] AdminHTTP["Admin HTTP"] --> Admin["internal/admin"] Admin --> Dispatcher Admin --> Engine ``` ### Core engine The keyspace is partitioned into 256 shards. FNV-1a hashes a key to a shard. Each shard contains a Go map and an `RWMutex`. Read paths take a shard read lock. Set/update/delete/TTL paths take the shard write lock. Type implementations that can block or maintain their own synchronized state use internal locks and then call `BumpVersion` after mutation. This prevents a blocking queue or semaphore wait from pinning a whole keyspace shard. ### Type identity versus encoding `core.DataType` is the stable type identity. The concrete Go object is the current internal encoding. This is important because an implementation can later replace an internal representation without changing the protocol-visible type name. ### Dispatcher The dispatcher implements common commands (`SET`, `CREATE`, `GET`, `TYPE`, TTL operations, and so on) and a generic `OP` command for type-specific behavior. `CREATE` invokes the type factory; `SET` invokes typed decoding. ### Administration The admin API does not have a separate database. Organization/member/API-key/RBAC/audit state is persisted to `data_dir/admin.json`; data commands run against the same in-memory `core.Engine` used by Amaquet clients. --- ## Adaptive in-memory compression Source: [canonical documentation page](/concepts/compression/) Summary: Amaquet can store selected large values in compressed form while preserving their normal Amaquet data type. Amaquet can store selected large values in compressed form while preserving their normal Amaquet data type. Compression is part of the Amaquet storage engine. It does not use Redis, an external cache, or a separate database. ### Goals The compression layer is designed to increase the amount of logical data that can fit in RAM without forcing every operation through a slow, high-ratio codec. It follows four rules: 1. Small values remain uncompressed. 2. Compression is kept only when it clears the configured minimum saving. 3. Decompression is transparent to `GET` and type operations. 4. The original Amaquet type ID remains unchanged. A compressed key is therefore still reported as `utf8_string`, `json`, `dense_vector`, and so on. Compression is a physical encoding, not a public data type. ### Algorithms Amaquet chooses or applies one of the following codecs to eligible canonical payloads. Their trade-off is compression ratio versus CPU latency. #### `auto` Recommended default. Amaquet evaluates its two cheapest codecs: - **LZ4 block** for general text, JSON, binary payloads, and repeated byte patterns. - **RLE** for highly repetitive buffers such as zero-filled data. The smaller candidate is retained only when it meets `min_savings_percent`. #### `lz4` Uses the built-in LZ4 block codec. It is optimized for low compression and decompression latency. Amaquet stores only the LZ4 block payload because the storage metadata already contains the codec and original byte length. #### `rle` Run-length encoding. It is useful for long repeated byte runs. It has very low CPU overhead but should not be selected for general natural-language or random data. #### `deflate-fast` Uses Go's fast DEFLATE mode. It can provide a better ratio for some content, but it costs more CPU than LZ4 or RLE. Select it when RAM pressure matters more than the lowest possible latency. #### `none` Disables compression while leaving the compression subsystem configured. ### Default configuration The following JSON enables adaptive compression with the repository's default thresholds. ```json { "compression": { "enabled": true, "algorithm": "auto", "min_bytes": 1024, "min_savings_percent": 8 } } ``` `min_bytes` is checked before a codec runs. `min_savings_percent` is checked after compression. A 4 KiB value that compresses by only 2% remains raw when the threshold is 8%. ### Environment variables These variables override the corresponding compression settings after the JSON configuration is loaded. They expose enablement, codec selection, and the minimum payload size; the minimum-savings percentage remains JSON-only. | Variable | Purpose | | ------------------------------- | ---------------------------------------------------------- | | `AMAQUET_COMPRESSION` | Enable or disable compression. `0` and `false` disable it. | | `AMAQUET_COMPRESSION_ALGORITHM` | `auto`, `lz4`, `rle`, `deflate-fast`, or `none`. | | `AMAQUET_COMPRESSION_MIN_BYTES` | Minimum payload size considered for compression. | ### Eligible values Compression is currently applied to payload-oriented values where materialization can be lossless and deterministic: - contiguous binary strings; chunked blob values remain in their block representation - UTF-8 strings - decimals and big decimals when large enough - big integers when large enough - JSON documents - MessagePack - CBOR - dense vectors - sparse vectors - quantized vectors - matrix and tensor payloads Small scalar values, synchronization primitives, active indexes, streams, queues, and other latency-sensitive mutable structures remain in their native representations. This avoids repeatedly rebuilding complex indexes just to save memory. Collection elements continue to use their native `core.Value` representation. Large top-level payloads receive the largest benefit from the current implementation. ### Write path For an eligible value, Amaquet performs these steps: ```mermaid flowchart TB Request["Amaquet request"] --> Decode["Validate and decode logical type"] Decode --> Canonical["Create canonical byte representation"] Canonical --> Size{"Meets size threshold?"} Size -->|No| Native["Store native value"] Size -->|Yes| Codec["Run fast codec"] Codec --> Savings{"Meets minimum saving?"} Savings -->|No| Native Savings -->|Yes| Compressed["Store compressed payload and original type ID"] ``` The compressed payload stores the codec name, original byte size, and compressed bytes. TTL, version, timestamps, and key metadata remain outside the compressed payload. ### Read path `GET` materializes the logical value before it is converted to the Amaquet response. Clients do not need compression support. ```mermaid flowchart LR Entry["Compressed entry"] --> Decompress["Decompress"] Decompress --> Value["Reconstruct typed value"] Value --> Response["Normal Amaquet response"] ``` Read-only type operations use the same materialization step. When a compressed mutable payload is changed, Amaquet recompresses the resulting value before replacing the stored entry. If the new value no longer meets the saving threshold, it is stored uncompressed. ### Inspecting compression Use `MEMORY` for one key: ```bash amaquet-cli -uri amaquet://127.0.0.1:13378 MEMORY '{"key":"large-json"}' ``` A compressed key reports fields similar to: ```json { "key": "large-json", "type": "json", "compressed": true, "algorithm": "lz4", "original_bytes": 1048576, "stored_bytes": 93421, "savings_bytes": 955155, "savings_percent": 91.09, "version": 1, "access_count": 4 } ``` `INFO` includes aggregate compression statistics: ```json { "compression": { "compressed_keys": 120, "original_bytes": 73400320, "stored_bytes": 11848913, "savings_bytes": 61551407, "savings_percent": 83.86, "by_algorithm": { "lz4": 117, "rle": 3 } } } ``` These byte counters cover compressed value payloads. They do not claim to measure Go allocator overhead, map buckets, key strings, index structures, or process RSS. ### Choosing settings For a general-purpose installation, keep `auto`, `min_bytes: 1024`, and `min_savings_percent: 8`. For very latency-sensitive workloads, increase `min_bytes` to 4 KiB or 16 KiB so only large values are compressed. For memory-constrained workloads with repetitive documents, reduce `min_savings_percent` carefully. For archival-like large blobs where CPU is less important, benchmark `deflate-fast`. Do not assume compression increases capacity by a fixed factor. Encrypted data, already-compressed images, video, random bytes, and compressed archives often provide little or no saving and will normally remain raw because of the saving threshold. --- ## Engine internals Source: [canonical documentation page](/concepts/engine-internals/) Summary: This page describes the behavior implemented by internal/core, including constraints that are not visible in the short command reference. This page describes the behavior implemented by `internal/core`, including constraints that are not visible in the short command reference. ### Sharding and locking The engine creates 256 shards. FNV-1a over the complete key selects a shard, and every shard owns a Go map protected by an `RWMutex`. Independent shards can proceed concurrently. Key writes acquire one shard's write lock; reads acquire its read lock and atomically update access telemetry. Composite values have their own locks. The dispatcher obtains the value, performs its operation through that object, and then updates the engine's version and size metadata. Blocking queues, barriers, and semaphores therefore wait on a request context without pinning the keyspace shard lock. ### Entry metadata and versions Each entry records: - the stable `DataType` plus its physical Go value; - creation, update, and optional expiration timestamps; - a monotonically increasing per-key version; - atomic access count and last-access time; - approximate stored bytes used by memory governance. Replacing an existing key preserves `CreatedAt` and increments its version. A deleted then recreated key starts again at version 1. Successful `EXPIRE` and `PERSIST` also increment the version. `CAS` requires a positive expected version and atomically replaces only that version. ### Expiration algorithm TTL is enforced in two ways: 1. Reads and writes lazily remove an expired entry encountered on its shard. 2. One background goroutine maintains an indexed min-heap ordered by expiration time. There is at most one heap node per currently expiring key. Updating TTL fixes that node in place; `PERSIST` removes it. The heap item includes the entry version, so the expirer cannot delete a newer replacement based on an obsolete deadline. `TTL` returns `-1` for an existing persistent key and `NOT_FOUND` for a missing/expired key. A non-positive relative expiration is rejected. During deterministic replay, an already-passed absolute expiration leaves the key absent. ### Scans `SCAN` encodes the current shard number and last returned key into an opaque base64url cursor. For each visited shard it copies the live matching keys under a read lock, sorts that shard-local page, and continues lexicographically after the cursor key. It does not create one globally sorted snapshot. Consequences of a live, non-snapshot scan: - concurrent insertions/deletions can change later pages; - ordering is deterministic within the observed shard pages, not a global lexical ordering across all shards; - callers must treat the cursor as opaque and stop when it becomes empty; - `count` is clamped to 10,000 and defaults to 100. `KEYS` repeatedly consumes `SCAN`, defaults to 1,000, and is also capped at 10,000. ### Approximate memory accounting The engine estimates each entry from the key, a fixed metadata allowance, and either an implementation-provided approximate size, a snapshot JSON size, or a conservative fallback. This counter is designed for admission and eviction decisions; it is not allocator accounting or process RSS. Writers reserve estimated positive growth before making a mutation visible. Reservations participate in concurrent capacity checks. The exact estimated delta is applied when metadata is committed, and the reservation is then released. Memory-reducing operations require no reservation. When `max_memory_bytes` is non-zero, the Go runtime soft memory limit is set to 125% of that value. This is headroom, not a hard operating-system limit. ### Eviction Supported policies are: | Policy | Candidate comparison | | -------------- | ---------------------------------------------------------- | | `noeviction` | Reject growth with `MEMORY_LIMIT` | | `allkeys-lru` | Oldest sampled last-access time | | `allkeys-lfu` | Lowest sampled access count | | `volatile-ttl` | Nearest sampled expiration; persistent keys are ineligible | Victim selection is approximate. One eviction attempt visits up to all 256 shards but samples at most two eligible map entries per shard. Go map iteration provides the spread; it is not a strict global LRU/LFU ordering. The key currently being mutated is excluded from victim selection. ### Engine transactions The internal `Engine.Transaction` API sorts and locks all involved shard IDs to prevent lock-order deadlocks. It provides staged copies of entry metadata and permits only non-pointer/non-map/non-slice composite payloads, because mutating a shared object would defeat rollback. An error from the callback commits nothing. The public `BATCH` command is different: it reduces round trips but executes requests sequentially and does not roll back earlier successes. ### Compression boundary Compression changes only physical `Value.Data`; the stable type ID stays unchanged. Eligible values are converted to canonical bytes before codec selection and reconstructed before normal reads/operations. Compression statistics are updated incrementally as entries are installed, replaced, or removed. See [Adaptive compression](/concepts/compression/). ### Shutdown Closing the engine stops and joins the expiration goroutine. Process shutdown separately stops the admin server, closes protocol connections, aborts incomplete upload state, flushes the AOF, and closes control-plane state. --- ## Keyspace and lifecycle Source: [canonical documentation page](/concepts/keyspace/) Summary: This page explains how Amaquet stores keys, applies expiration, and performs conditional lifecycle operations. This page explains how Amaquet stores keys, applies expiration, and performs conditional lifecycle operations. ### Keys Keys are non-empty strings. There is one global in-process keyspace. The URI parser rejects non-empty path/database components because logical databases are not implemented. ### Common commands Every currently registered logical type participates in the common lifecycle commands: `TYPE`, `EXISTS`, `TTL`, `EXPIRE`, `PERSIST`, `DEL`, top-level `GET`, and cursor `SCAN`. `GET` returns either the direct value or a synchronized public snapshot, while `CAS` performs an optimistic versioned replacement. ### TTL behavior `EXPIRE` accepts positive milliseconds. Reads enforce lazy expiry. The background expiration worker uses an indexed min-heap ordered by absolute expiry; it does not scan every key once per second. Each expiring key owns at most one scheduled heap node. Replacing the key, changing its TTL, or calling `PERSIST` updates or removes that node instead of accumulating stale expiry records. `EXPIRE` and `PERSIST` first reject/purge a logically expired key, so an expired value cannot be revived by removing or changing its TTL. `TTL` returns milliseconds remaining. `-1` means a live key has no expiry. ### `NX`, `XX`, and `CAS` `SET` and `CREATE` accept `nx` and `xx`. `CAS` compares the current per-key version with `expected_version` and applies the new typed value only when they match. ### Enumeration `KEYS` remains useful for bounded administration but collects its result by consuming shard-local sorted scan pages. The combined result is not globally lexical. For application iteration, use `SCAN`. Its opaque cursor encodes progress through shard-local sorted views, avoiding a complete keyspace materialization on every page. --- ## Memory and concurrency model Source: [canonical documentation page](/concepts/memory-model/) Summary: This page explains entry metadata, shard-level concurrency, object-level synchronization, and transaction locking. This page explains entry metadata, shard-level concurrency, object-level synchronization, and transaction locking. ### Key entries Each key maps to an `Entry` containing: - strongly typed `Value`; - creation and update timestamps; - optional expiry timestamp; - monotonically increasing version; - atomic access counter. ### Sharding Amaquet currently creates 256 shards. The shard is selected with FNV-1a over the complete key. Independent shards can be accessed concurrently. ### Versions A new key starts at version 1. Replacing a key, changing TTL, removing TTL, scalar mutation, or successful in-place type mutation advances its version. `CompareAndSet` can replace a value only if the observed version still matches. ### Atomic multi-key transactions `Engine.Transaction` computes the distinct shards for all involved keys, sorts shard indexes, locks them in stable order, builds a view of non-expired entries, and runs the callback while those shard locks remain held. Stable lock ordering prevents lock-order inversion between transactions. ### Composite object locking Many data types have their own `sync.RWMutex` or synchronization primitive. Their operation executes without a shard lock. After success, the engine takes the short shard lock needed to update the entry version and update timestamp. ### Blocking types Blocking queues, barriers, and semaphores must not hold keyspace shard locks while waiting. The dispatcher retrieves the object, performs the wait on the object itself, then updates key metadata after the operation completes. ### Compressed physical encodings A key's logical `DataType` and its physical storage encoding are separate. Large eligible values can be represented by a compressed payload while the entry retains the same public type ID. Reads materialize the native value only for the operation. See [Adaptive in-memory compression](/concepts/compression/). --- ## Persistence and recovery Source: [canonical documentation page](/concepts/persistence/) Summary: Amaquet keeps the active data set in its own in-memory engine. Optional persistence uses the Amaquet append-only write-ahead journal (AOF); it does not depend… Amaquet keeps the active data set in its own in-memory engine. Optional persistence uses the Amaquet append-only write-ahead journal (AOF); it does not depend on Redis or another database. ### Transactional AOF v2 Current journals begin with: ```text AMQTAOF2\n ``` After the header, each physical record is a 4-byte big-endian payload length, a 4-byte big-endian IEEE CRC-32 of the payload, and the JSON payload itself. A record payload is limited to 64 MiB and contains the transaction kind, transaction ID, resolved request when applicable, and record timestamp. A logical mutation uses these phases: 1. `prepare`: persist the fully resolved mutation before changing memory. 2. Apply the mutation to the in-memory engine. 3. `commit`: mark the prepare as committed. 4. `abort`: record that a prepared mutation failed before commit. Recovery replays only prepares that have a matching commit. An uncommitted prepare is ignored. If the journal commit fails after memory has changed, Amaquet enters a persistence-degraded fail-stop state and rejects further durable mutations until the persistence problem is resolved or the process is restarted from valid durable state. Legacy `AMQTAOF1` files are detected and migrated to the v2 journal format when they are opened. Journals created before the Amaquet rename are also recognized by their historical v1/v2 signatures and rewritten with the current `AMQTAOF2` header before use. ### Deterministic replay Journal records resolve time-dependent values before the mutation is written. Examples include: - absolute key expiration timestamps; - `CAS` expiration timestamps; - automatically generated stream IDs; - time-series sample timestamps; - persistent-topic and event-log timestamps; - reliable-queue enqueue and claim times; - delayed-queue readiness checks. Replay reconstructs the historical mutation instead of generating a new value from the restart clock. If an absolute expiration has already passed during recovery, the key remains absent. ### Integrity and failure state Every record is protected by CRC-32. A partial final record can be ignored as an interrupted tail write. A checksum failure in a complete record is corruption and stops normal replay. For `fsync: everysec`, background flush errors are retained as persistence-health failures. Readiness reports the degraded state, and new durable mutations are rejected instead of silently running without the requested durability. ### Durability modes `persistence.fsync` controls when buffered journal phases are forced toward durable storage. The modes trade write latency for the amount of recently acknowledged data that can be lost after an operating-system or machine failure. - `always`: flush and fsync every journal phase. Highest durability and highest write latency. - `everysec`: flush approximately once per second. This is the normal balance for many deployments. - `no`: use buffered and operating-system writeback. Lowest durability. ### Compact verified checkpoints `POST /api/persistence/checkpoint` creates a compact recovery image rather than copying the live journal byte-for-byte. Amaquet: 1. serializes with AOF writers and flushes the active journal to capture a committed prefix; 2. reads that stable prefix while later appends wait; 3. resolves committed requests; 4. removes aborted and uncommitted records; 5. squashes obsolete histories where command semantics permit it and retains only the latest completed blob-upload sequence for each blob key; 6. writes a fresh `AMQTAOF2` image to a temporary file; 7. fsyncs and replays the temporary image for verification; 8. atomically publishes the checkpoint and fsyncs the destination directory. Incomplete chunked uploads are not restored as live uploads. ### Online compaction Online AOF compaction uses the same verified rewrite principles. The replacement uses a temporary file, verification, backup/rollback protection, atomic rename, and directory fsync. It removes aborted/uncommitted transactions and collapses supported superseded histories while preserving observable committed state. Compaction is not a substitute for backup. Keep independent checkpoint copies on separate storage. ### Offline restore Use `amaquet-restore` while the target Amaquet process is stopped: ```bash amaquet-restore \ -source /backups/amaquet-checkpoint.aof \ -target /var/lib/amaquet/appendonly.aof ``` The restore command validates the source, copies it to a temporary target, fsyncs it, verifies it again, then atomically installs it with rollback protection. ### Runtime state that is not recovered Ephemeral process coordination is intentionally not restored when doing so could recreate stale ownership. Examples include live Pub/Sub subscriptions, HTTP cookie sessions, active protocol connections, and process-local synchronization ownership. See [Backup and recovery](/operations/backups/) for operational procedures. --- ## Security model Source: [canonical documentation page](/concepts/security/) Summary: Amaquet separates transport security, machine credentials, human member identities, HTTP cookie sessions, RBAC, and optional server identity. Amaquet separates transport security, machine credentials, human member identities, HTTP cookie sessions, RBAC, and optional server identity. ### API keys API keys are machine credentials. A key contains a random secret and a stable key ID. Amaquet stores the SHA-256 digest, role, expiration, last-use time, and revocation state. Authentication hashes the presented token with SHA-256 and uses the digest as an index into API-key and member-credential maps, avoiding a scan of all credentials. Bootstrap-token digest comparison uses `subtle.ConstantTimeCompare`. The stored hashes are credential verifiers rather than encryption; the generated high-entropy tokens remain the security boundary. Revoking a key prevents new authentication and invalidates the actor on existing sessions/connections. The server also closes matching active Amaquet connections through the revocation hook. For remote protocol access, use `amaquets://`. Authenticated plaintext on a non-loopback listener is rejected unless `allow_insecure_auth` is explicitly enabled. ### Human member identities A member record has a role and status. Administrators can create a one-time invite with `POST /api/members/{id}/invite`. The member sends the invite to `POST /api/identity/accept-invite`. A successful acceptance rotates any previous member access credential and returns the new access token once. Member access tokens create server-side sessions and always use the member's current RBAC role. Disabled or deleted members are immediately invalid. #### TOTP MFA An authenticated member can start MFA setup with `POST /api/identity/mfa`. Amaquet returns a base32 TOTP secret and an `otpauth://` URI. Confirm setup with `PUT /api/identity/mfa` and a current six-digit code. After confirmation, new member sessions require both the access token and `mfa_code`. Codes use HMAC-SHA-1, a 30-second step, six decimal digits, and a ±1-step verification window. `DELETE /api/identity/mfa` disables TOTP for the current member. TOTP secrets are sensitive and are stored in the crash-safe administration state file. Protect its backups. ### One-time bootstrap The bootstrap credential only initializes a new control plane. If startup needs to generate it, Amaquet writes the secret to `data_dir/bootstrap-token` with file mode `0600` and logs the path, not the secret. When an existing administration state still has the untouched organization default from before the Amaquet rename, the schema migration updates that default to `Amaquet` / `amaquet`. User-customized organization names and all existing credential hashes remain unchanged. After bootstrap creates the first `admin` API key, bootstrap authentication is disabled in persisted administration state. ### HTTP cookie sessions `POST /api/session` exchanges a login credential for a 12-hour server-side session. The client receives an `HttpOnly`, `SameSite=Strict` cookie. HTTPS also sets the `Secure` flag. Sessions are bounded, expired sessions are pruned, and actor validity is checked on use. API-key revocation or member revocation removes matching sessions through the revocation hook. ### RBAC Built-in roles are `admin`, `auditor`, and `developer`. Their initial permissions are: | Role | Default permissions | | ----------- | ---------------------------------------------------------------------------------------------- | | `admin` | `*` | | `auditor` | `data.read`, `org.read`, `members.read`, `rbac.read`, `keys.read`, `system.read`, `audit.read` | | `developer` | `data.read`, `data.write`, `org.read`, `members.read`, `system.read` | Protocol commands map to `data.read` or `data.write`. Admin routes use permissions for organization, members, RBAC, keys, system configuration, persistence, audit, and data access. The RBAC map is editable, so these are defaults rather than permanent role definitions. Role changes take effect on the next actor validation. Existing sessions do not retain an old role snapshot indefinitely. ### TLS `amaquets://` uses the Amaquet TLS listener. A non-loopback authenticated protocol listener requires TLS unless the explicit insecure override is enabled. The admin listener has independent HTTPS settings. Non-loopback plain HTTP is rejected by default. ### Ed25519 server identity The protocol can load a separate Ed25519 identity. `HELLO` can sign a client nonce and return the public key and fingerprint. This proves possession of the configured identity key. It does not replace TLS certificate validation. ### Crash-safe control-plane state Configuration and administration state use a temporary write, file fsync, atomic rename, and directory fsync. Administration state currently uses schema version 3 and has explicit migrations from earlier schemas; a state file created by a newer unsupported schema is rejected instead of being partially interpreted. The in-file audit history retains the newest 5,000 events. API-key last-use timestamps are buffered and flushed to disk on a five-second loop (and at orderly state shutdown) so authentication does not force an fsync for every request. --- ## Why Amaquet? Source: [canonical documentation page](/concepts/why-amaquet/) Summary: Compare Amaquet and Redis across protocol compatibility, typed data models, concurrency, persistence, security, operations, and deployment fit. Redis is a mature data-structure server with a large client and operations ecosystem. Amaquet is a different product: a standalone, typed in-memory database written in Go with its own binary protocol, HTTP control plane, and optional append-only persistence. Amaquet is not a Redis-compatible server and should not be selected when Redis compatibility is the requirement. The useful question is therefore not which product is universally better. It is which data model, protocol, topology, security boundary, and operational contract matches the application you are building. ### The short answer Amaquet is a strong candidate when a new service needs an explicit typed data contract, a single-node in-memory data plane, and a control plane that is part of the same distribution. - Choose Amaquet when the application can use the [Amaquet binary protocol](/protocol/overview/), especially from Go, and should not inherit Redis command or wire compatibility as an architectural constraint. - Choose Amaquet when values such as queues, streams, time series, vectors, indexes, locks, leases, semaphores, and other structures should have stable type identities and documented operations rather than being assembled from generic strings or hashes. - Choose Amaquet when one process and one logical keyspace are an acceptable topology, with internal 256-shard concurrency, TTLs, versions, CAS, memory limits, eviction policy, and optional AOF persistence. - Choose Amaquet when API keys, human members, roles, TOTP MFA, sessions, audit history, and the HTTP administration API should be delivered as one product boundary. - Choose Redis when existing Redis clients, RESP compatibility, Redis commands, Redis modules, managed Redis offerings, replication, or Redis Cluster are central requirements. ### Comparison at a glance The following comparison describes the current Amaquet implementation alongside the capabilities documented by Redis. It intentionally avoids throughput, latency, memory-efficiency, or reliability claims that would require a controlled benchmark or a deployment-specific evaluation. | Aspect | Redis | Amaquet | Best fit | | --- | --- | --- | --- | | **Client and wire compatibility** | Clients communicate with Redis through [RESP](https://redis.io/docs/latest/develop/reference/protocol-spec/), with a broad ecosystem of libraries and tools. | Clients use Amaquet’s binary-framed protocol, JSON payloads, `HELLO` negotiation, request IDs, and the `amaquet://` or `amaquets://` URI schemes. | Redis for an existing Redis integration; Amaquet for a new, controlled client contract. | | **Data model** | Redis provides native structures such as strings, hashes, lists, sets, sorted sets, streams, JSON, geospatial, probabilistic, time-series, and vector data. | Amaquet currently exposes 91 stable wire-level types with explicit type identity, typed nested values, `CREATE`, and type-specific `OP` operations. | Redis for Redis-native conventions; Amaquet when application values and operations should be explicit protocol types. | | **Concurrency** | Redis provides command-level atomicity and optimistic locking through mechanisms such as `WATCH`, `MULTI`, and `EXEC`. | Amaquet tracks key versions, supports CAS, gives commands request IDs, supports cancellation, and documents `BATCH` as a round-trip optimization rather than a rollback transaction. | Redis when existing Redis transaction semantics are required; Amaquet when versioned typed mutations and cancellable requests fit the design. | | **Memory governance** | Redis supports `maxmemory` and configurable eviction policies such as `noeviction`, LRU, LFU, random, and TTL-based policies. | Amaquet exposes approximate stored-data accounting, an optional memory ceiling, `noeviction`, LRU, LFU, and TTL-based eviction, plus adaptive compression for eligible values. | Either can fit a bounded cache; compare the exact accounting and eviction semantics with the workload. | | **Persistence** | Redis supports RDB snapshots, AOF, no persistence, or combinations of persistence modes. | Amaquet keeps the active dataset in its own memory engine and optionally writes a CRC-protected `AMQTAOF2` journal with prepare/commit recovery, deterministic replay, checkpoints, compaction, and offline restore. | Redis for an established Redis persistence and recovery workflow; Amaquet for the repository-owned AOF and checkpoint contract. | | **Topology** | Redis can run as a standalone instance and can scale out through [Redis Cluster](https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/), replication, and related deployment components. | Amaquet is single-node with one logical keyspace. Its 256 shards are internal concurrency partitions, not a cluster or replication protocol. | Redis for multi-node availability, failover, or horizontal scale; Amaquet when a single-node topology is intentional. | | **Security and administration** | Redis provides ACLs and optional TLS; deployments commonly combine these with the surrounding Redis or platform operations model. | Amaquet has a separate HTTP administration listener, API keys, members, roles, editable RBAC, TOTP MFA, sessions, audit history, TLS enforcement, and optional Ed25519 protocol identity. | Redis for an existing Redis security stack; Amaquet when the data plane and control plane should ship together. | The product boundaries matter more than the number of rows in this table. Redis and Amaquet both provide in-memory data structures, expiration, persistence choices, authentication, TLS, and memory policies, but the interfaces and operational semantics are not interchangeable. ### When Amaquet is the better fit Amaquet is designed for applications that value an explicit server-side data contract and a focused deployment topology. #### Typed application state Amaquet makes the public type identity part of the protocol and storage contract. Scalar values, collections, queues, messaging objects, time series, geospatial values, vectors, indexes, and synchronization structures are registered types rather than undocumented application conventions. A nested value also carries its own `type` and `value` envelope. This is useful when several services or tools must agree on whether a value is an integer, decimal, queue entry, vector, timestamp, or structured object. The [data-type catalog](/data-types/) and [operation reference](/data-types/operations/) document the accepted arguments, results, defaults, persistence behavior, and failure behavior for each type. Redis is often the better choice when the application already has a Redis data model built around RESP commands, strings, hashes, lists, streams, or Redis modules. Amaquet’s typed model is an advantage only when adopting a new protocol and type contract is acceptable. #### Go services with an owned protocol Amaquet ships a supported Go client that negotiates protocol version 1, multiplexes requests by request ID, routes asynchronous event frames, supports TLS, and exposes typed helpers alongside a general `Command` method. The server and the client are maintained in the same repository, so a Go service can use the documented Amaquet contract without introducing Redis command compatibility into its own API boundary. That does not mean every language can use Amaquet equally easily. A non-Go client must implement the Amaquet framing, negotiation, authentication, request, response, and event contracts. Redis is the simpler choice when the target language, framework, or platform already has a mature Redis client. #### Queues, messaging, and coordination Amaquet includes first-class types for FIFO, blocking, delayed, reliable, priority, and ring-buffer queues, as well as streams, consumer groups, Pub/Sub, persistent topics, and event logs. It also includes synchronization and admission structures such as locks, leases, barriers, semaphores, and limit buckets. The benefit is a documented object-level contract for the behavior the application needs. For example, the reliable-queue documentation covers claiming, acknowledgement, expiry, retry, and dead-letter behavior, while the synchronization types document their own state and operations. Redis can be the better fit when workers already depend on Redis Lists, Streams, Pub/Sub, or a Redis-based coordination library. Amaquet’s structures are server-side Amaquet types; they are not Redis command aliases, and Amaquet’s single-node boundary should not be mistaken for a distributed lock or coordination service. #### A single-node data plane with a separate control plane Amaquet deliberately separates the native TCP/TLS data-plane listener from the HTTP administration listener. Data commands use the Amaquet protocol; administration covers organization state, members, API keys, RBAC, persistence operations, runtime inspection, metrics, and audit history. This fits a service that wants one deployable server with an explicit operational API instead of assembling the data server, identity model, administration endpoints, and audit storage from separate components. The administration state is stored in `data_dir/admin.json`; it is not mixed into the user keyspace or the data AOF. #### Controlled memory and persistence behavior Amaquet’s engine supports TTLs, key versions, CAS, configurable memory admission, eviction policies, and transparent adaptive compression for eligible values. Optional AOF persistence uses resolved mutation data so time-dependent behavior such as expirations, generated stream IDs, and timestamps can be replayed deterministically. Verified checkpoints and `amaquet-restore` provide an explicit recovery path. This is a good fit when the application wants to choose between cache-like operation and a journaled in-memory service while keeping the persistence format and recovery tooling within the Amaquet distribution. It is not a reason to treat Amaquet’s in-memory dataset as a replacement for independent backups or durable storage outside the service. ### When Redis is the better fit Redis remains the more appropriate choice when compatibility, ecosystem, or distributed deployment requirements dominate. #### Existing Redis clients and commands If an application already speaks RESP, uses Redis command names, depends on Redis modules, or relies on Redis-aware frameworks, switching to Amaquet is not a transparent replacement. Amaquet explicitly does not implement Redis RESP compatibility. Migration would require a client and data-model change, not only a new server address. #### Replication, failover, and cluster operation Amaquet currently has no clustering, replication, automatic failover, or multi-node keyspace. Redis documents standalone operation alongside replication and Redis Cluster, which automatically distributes keys across nodes and uses replicas for failover scenarios. Choose Redis when the service must continue through node failure using a Redis replication or Cluster topology, or when the dataset must scale across multiple Redis nodes. Choose Amaquet only when the single-node boundary is acceptable and is offset by the simpler topology or its typed/control-plane features. #### Redis ecosystem and managed operations Redis has an established ecosystem of clients, command references, operational guides, monitoring integrations, modules, and managed offerings. Those integrations can be more valuable than Amaquet’s product-specific features when a team already operates Redis or needs a vendor-supported Redis deployment model. Amaquet is a better fit when the team prefers a smaller, source-available Go service whose server, client, data types, protocol, tests, persistence, and documentation are in one repository. That is an ownership and integration choice, not a claim that Amaquet has the breadth or maturity of the Redis ecosystem. ### Decision guide by use case Use the following as a starting decision, then validate the exact operations, recovery objectives, and deployment constraints for the application. | Use case | Prefer Amaquet when… | Prefer Redis when… | | --- | --- | --- | | New Go service state | The service wants typed values, an owned Go client, versioned mutations, and a single-node server. | The service should use existing Redis libraries, RESP, or Redis commands. | | Cache with TTL and bounded memory | Amaquet’s explicit memory limits, eviction choices, and optional compression match the service’s operational model. | The team already standardizes on Redis eviction, monitoring, or managed cache services. | | Work queues | The application wants Amaquet queue types with documented retry, lease, blocking, delayed, or priority behavior. | Existing workers already use Redis Lists or Streams and their ecosystem is the main constraint. | | Pub/Sub and event delivery | A service wants Amaquet event frames, typed payloads, persistent topics, or event logs within one server. | Existing applications and tooling already use Redis Pub/Sub or Streams. | | Coordination | A single Amaquet node is the authority for locks, leases, barriers, semaphores, or admission controls. | Coordination must use an existing Redis deployment or a distributed Redis topology. | | Durable in-memory service | Amaquet’s optional transactional AOF, deterministic replay, verified checkpoints, and offline restore match the recovery design. | Redis RDB/AOF workflows, replication, or managed durability are already established. | | High availability and scale-out | The service does not require replication, cluster routing, or automatic failover. | Redis replication, Redis Cluster, or a managed Redis topology is required. | | Existing production Redis | A migration is acceptable and the application benefits from Amaquet’s typed/control-plane model. | Compatibility and migration avoidance are more valuable than changing the server contract. | ### Important boundaries Amaquet’s advantages are meaningful only when its boundaries are understood before deployment. - Amaquet is not a drop-in Redis replacement. It does not implement RESP, Redis commands, Redis logical databases, Redis clustering, or replication. - The 256 internal shards improve concurrent access within one process; they do not provide horizontal scaling or failover. - The active dataset remains in memory. AOF is optional, and its `always`, `everysec`, and `no` fsync modes make different durability tradeoffs. - `BATCH` reduces round trips but is not a rollback transaction. Do not infer Redis `MULTI`/`EXEC` semantics from it. - Amaquet’s memory accounting is approximate stored-data accounting, not a hard RSS limit for the complete process. - Explicit Amaquet index values are user-managed. Writing a document does not automatically update a separate index. ### Learn more Read the [architecture](/concepts/architecture/), [memory and concurrency model](/concepts/memory-model/), [persistence and recovery](/concepts/persistence/), [security model](/concepts/security/), [protocol overview](/protocol/overview/), and [implementation coverage](/reference/implementation-coverage/) pages for Amaquet’s current behavior. For the Redis side of the comparison, consult the official documentation for [data types](https://redis.io/docs/latest/develop/data-types/), [RESP](https://redis.io/docs/latest/develop/reference/protocol-spec/), [persistence](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/), [ACLs](https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/), [TLS](https://redis.io/docs/latest/operate/oss_and_stack/management/security/encryption/), [eviction](https://redis.io/docs/latest/develop/reference/eviction/), and [Redis Cluster](https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/). --- ## Data types Source: [canonical documentation page](/data-types/) Summary: Browse Amaquet's 91 stable wire-level data types, their categories, construction commands, supported operations, and links to detailed references. Amaquet exposes **91 stable wire-level types**. Type IDs are stable protocol/storage identities and are separate from internal encodings. Use the [construction and operation reference](/data-types/operations/) for the complete argument, default, result, persistence, and implementation contract behind every operation listed here. Direct JSON encodings are defined in [Wire values and snapshots](/protocol/wire-values/). | Type | Category | Construction | Operations | | ------------------------------------------------------------------- | ----------------------------------- | ------------ | ---------: | | [`binary_string`](/data-types/types/binary_string/) | Scalar | SET | 1 | | [`utf8_string`](/data-types/types/utf8_string/) | Scalar | SET | 1 | | [`integer`](/data-types/types/integer/) | Scalar | SET | 1 | | [`unsigned_integer`](/data-types/types/unsigned_integer/) | Scalar | SET | 1 | | [`float`](/data-types/types/float/) | Scalar | SET | 1 | | [`decimal`](/data-types/types/decimal/) | Scalar | SET | 3 | | [`boolean`](/data-types/types/boolean/) | Scalar | SET | 1 | | [`null`](/data-types/types/null/) | Scalar | SET | 1 | | [`timestamp`](/data-types/types/timestamp/) | Scalar | SET | 1 | | [`duration`](/data-types/types/duration/) | Scalar | SET | 1 | | [`uuid`](/data-types/types/uuid/) | Scalar | SET | 1 | | [`big_integer`](/data-types/types/big_integer/) | Scalar | SET | 1 | | [`big_decimal`](/data-types/types/big_decimal/) | Scalar | SET | 3 | | [`symbol`](/data-types/types/symbol/) | Scalar | SET | 1 | | [`list`](/data-types/types/list/) | Collections | CREATE | 6 | | [`array`](/data-types/types/array/) | Collections | CREATE | 5 | | [`deque`](/data-types/types/deque/) | Collections | CREATE | 6 | | [`ring_buffer`](/data-types/types/ring_buffer/) | Collections | CREATE | 3 | | [`tuple`](/data-types/types/tuple/) | Collections | CREATE | 3 | | [`set`](/data-types/types/set/) | Collections | CREATE | 5 | | [`sorted_set`](/data-types/types/sorted_set/) | Collections | CREATE | 4 | | [`ordered_set`](/data-types/types/ordered_set/) | Collections | CREATE | 3 | | [`multiset`](/data-types/types/multiset/) | Collections | CREATE | 3 | | [`hashmap`](/data-types/types/hashmap/) | Collections | CREATE | 4 | | [`ordered_map`](/data-types/types/ordered_map/) | Collections | CREATE | 4 | | [`multimap`](/data-types/types/multimap/) | Collections | CREATE | 3 | | [`json`](/data-types/types/json/) | Structured | CREATE / SET | 2 | | [`messagepack`](/data-types/types/messagepack/) | Structured | CREATE / SET | 1 | | [`cbor`](/data-types/types/cbor/) | Structured | CREATE / SET | 1 | | [`record`](/data-types/types/record/) | Structured | CREATE | 2 | | [`matrix`](/data-types/types/matrix/) | Structured | CREATE | 3 | | [`tensor`](/data-types/types/tensor/) | Structured | CREATE | 3 | | [`fifo_queue`](/data-types/types/fifo_queue/) | Queues and messaging | CREATE | 4 | | [`lifo_stack`](/data-types/types/lifo_stack/) | Queues and messaging | CREATE | 4 | | [`priority_queue`](/data-types/types/priority_queue/) | Queues and messaging | CREATE | 3 | | [`blocking_queue`](/data-types/types/blocking_queue/) | Queues and messaging | CREATE | 4 | | [`delayed_queue`](/data-types/types/delayed_queue/) | Queues and messaging | CREATE | 3 | | [`reliable_queue`](/data-types/types/reliable_queue/) | Queues and messaging | CREATE | 5 | | [`stream`](/data-types/types/stream/) | Queues and messaging | CREATE | 7 | | [`consumer_group`](/data-types/types/consumer_group/) | Queues and messaging | CREATE | 1 | | [`pubsub`](/data-types/types/pubsub/) | Queues and messaging | CREATE | 2 | | [`persistent_topic`](/data-types/types/persistent_topic/) | Queues and messaging | CREATE | 2 | | [`event_log`](/data-types/types/event_log/) | Queues and messaging | CREATE | 3 | | [`bitmap`](/data-types/types/bitmap/) | Bit structures | CREATE | 3 | | [`bit_field`](/data-types/types/bit_field/) | Bit structures | CREATE | 2 | | [`bit_set`](/data-types/types/bit_set/) | Bit structures | CREATE | 3 | | [`roaring_bitmap`](/data-types/types/roaring_bitmap/) | Bit structures | CREATE | 5 | | [`hyperloglog`](/data-types/types/hyperloglog/) | Probabilistic | CREATE | 2 | | [`bloom_filter`](/data-types/types/bloom_filter/) | Probabilistic | CREATE | 2 | | [`counting_bloom_filter`](/data-types/types/counting_bloom_filter/) | Probabilistic | CREATE | 3 | | [`cuckoo_filter`](/data-types/types/cuckoo_filter/) | Probabilistic | CREATE | 3 | | [`count_min_sketch`](/data-types/types/count_min_sketch/) | Probabilistic | CREATE | 2 | | [`top_k`](/data-types/types/top_k/) | Probabilistic | CREATE | 2 | | [`t_digest`](/data-types/types/t_digest/) | Probabilistic | CREATE | 2 | | [`timeseries`](/data-types/types/timeseries/) | Time series | CREATE | 5 | | [`counter_series`](/data-types/types/counter_series/) | Time series | CREATE | 5 | | [`gauge_series`](/data-types/types/gauge_series/) | Time series | CREATE | 5 | | [`histogram`](/data-types/types/histogram/) | Time series | CREATE | 2 | | [`timeseries_labels`](/data-types/types/timeseries_labels/) | Time series | CREATE | 2 | | [`aggregated_series`](/data-types/types/aggregated_series/) | Time series | CREATE | 2 | | [`geo_point`](/data-types/types/geo_point/) | Geospatial | SET | 1 | | [`geo_spatial_index`](/data-types/types/geo_spatial_index/) | Geospatial | CREATE | 5 | | [`bounding_box`](/data-types/types/bounding_box/) | Geospatial | SET | 1 | | [`polygon`](/data-types/types/polygon/) | Geospatial | SET | 1 | | [`dense_vector`](/data-types/types/dense_vector/) | Vectors | SET | 1 | | [`sparse_vector`](/data-types/types/sparse_vector/) | Vectors | SET | 1 | | [`vector_set`](/data-types/types/vector_set/) | Vectors | CREATE | 4 | | [`vector_metadata`](/data-types/types/vector_metadata/) | Vectors | CREATE | 3 | | [`quantized_vector`](/data-types/types/quantized_vector/) | Vectors | SET | 1 | | [`btree_index`](/data-types/types/btree_index/) | Indexes and search | CREATE | 4 | | [`hash_index`](/data-types/types/hash_index/) | Indexes and search | CREATE | 3 | | [`radix_tree`](/data-types/types/radix_tree/) | Indexes and search | CREATE | 3 | | [`trie`](/data-types/types/trie/) | Indexes and search | CREATE | 3 | | [`inverted_index`](/data-types/types/inverted_index/) | Indexes and search | CREATE | 3 | | [`hnsw_index`](/data-types/types/hnsw_index/) | Indexes and search | CREATE | 3 | | [`flat_vector_index`](/data-types/types/flat_vector_index/) | Indexes and search | CREATE | 4 | | [`geospatial_secondary_index`](/data-types/types/geospatial_secondary_index/) | Indexes and search | CREATE | 5 | | [`secondary_index`](/data-types/types/secondary_index/) | Indexes and search | CREATE | 2 | | [`full_text_document`](/data-types/types/full_text_document/) | Indexes and search | CREATE | 5 | | [`tag_field`](/data-types/types/tag_field/) | Indexes and search | CREATE | 2 | | [`numeric_field`](/data-types/types/numeric_field/) | Indexes and search | CREATE | 2 | | [`text_field`](/data-types/types/text_field/) | Indexes and search | CREATE | 2 | | [`adjacency_set`](/data-types/types/adjacency_set/) | Graph, synchronization, and network | CREATE | 3 | | [`atomic_counter`](/data-types/types/atomic_counter/) | Graph, synchronization, and network | CREATE | 3 | | [`lease`](/data-types/types/lease/) | Graph, synchronization, and network | CREATE | 4 | | [`lock`](/data-types/types/lock/) | Graph, synchronization, and network | CREATE | 3 | | [`barrier`](/data-types/types/barrier/) | Graph, synchronization, and network | CREATE | 1 | | [`semaphore`](/data-types/types/semaphore/) | Graph, synchronization, and network | CREATE | 4 | | [`rate_limit_bucket`](/data-types/types/rate_limit_bucket/) | Graph, synchronization, and network | CREATE | 2 | | [`cidr_network`](/data-types/types/cidr_network/) | Graph, synchronization, and network | SET | 1 | | [`url`](/data-types/types/url/) | Graph, synchronization, and network | SET | 2 | ### Nested typed values Composite structures carry nested values as `{"type":"integer","value":42}`, so nested values keep their type. ### Common lifecycle All keys support existence checks, type inspection, TTL, expiry, persistent TTL removal, deletion, and version tracking. --- ## Bit structures Source: [canonical documentation page](/data-types/bit-structures/) Summary: Bit structures provide compact bit-level and integer-set storage. Bit structures provide compact bit-level and integer-set storage. See [Type construction and operations](/data-types/operations/#bit-structure-operations) for addressing, width/endian rules, return values, and the adaptive array/bitmap/run containers used by `roaring_bitmap`. | Type | Construction | Principal operations | | ------------------------------------------- | ------------ | ------------------------------------------ | | [`bitmap`](/data-types/types/bitmap/) | CREATE | SET, GETBIT, COUNT | | [`bit_field`](/data-types/types/bit_field/) | CREATE | SET, GET | | [`bit_set`](/data-types/types/bit_set/) | CREATE | SET, GETBIT, COUNT | | [`roaring_bitmap`](/data-types/types/roaring_bitmap/) | CREATE | ADD, REMOVE, CONTAINS, VALUES, CARDINALITY | --- ## Collections Source: [canonical documentation page](/data-types/collections/) Summary: Collections are synchronized mutable structures manipulated with OP. Collections are synchronized mutable structures manipulated with `OP`. See [Type construction and operations](/data-types/operations/#collection-operations) for exact arguments, results, edge cases, and implementation costs. In particular, list front operations use slices, deques use a circular buffer, ring buffers overwrite the oldest entry at capacity, and sorted-set range reads sort the current entries by score. | Type | Construction | Principal operations | | ------------------------------------- | ------------ | ------------------------------------------------------- | | [`list`](/data-types/types/list/) | CREATE | PUSH_FRONT, PUSH_BACK, POP_FRONT, POP_BACK, RANGE, LEN | | [`array`](/data-types/types/array/) | CREATE | APPEND, SET, GET, VALUES, LEN | | [`deque`](/data-types/types/deque/) | CREATE | PUSH_FRONT, PUSH_BACK, POP_FRONT, POP_BACK, VALUES, LEN | | [`ring_buffer`](/data-types/types/ring_buffer/) | CREATE | PUSH, VALUES, LEN | | [`tuple`](/data-types/types/tuple/) | CREATE | GET, VALUES, LEN | | [`set`](/data-types/types/set/) | CREATE | ADD, REMOVE, CONTAINS, MEMBERS, LEN | | [`sorted_set`](/data-types/types/sorted_set/) | CREATE | ADD, REMOVE, SCORE, RANGE | | [`ordered_set`](/data-types/types/ordered_set/) | CREATE | ADD, REMOVE, VALUES | | [`multiset`](/data-types/types/multiset/) | CREATE | ADD, REMOVE, COUNT | | [`hashmap`](/data-types/types/hashmap/) | CREATE | SET, GET, DELETE, KEYS | | [`ordered_map`](/data-types/types/ordered_map/) | CREATE | SET, GET, DELETE, ENTRIES | | [`multimap`](/data-types/types/multimap/) | CREATE | ADD, GET, REMOVE | --- ## Geospatial Source: [canonical documentation page](/data-types/geospatial/) Summary: Geospatial types use latitude/longitude coordinates and spatial predicates. Geospatial types use latitude/longitude coordinates and spatial predicates. See [Type construction and operations](/data-types/operations/#geospatial-operations) for coordinate validation, Haversine distance, 0.25-degree grid candidate selection, exact post-filtering, result ordering, and polygon/antimeridian limitations. | Type | Construction | Principal operations | | ------------------------------------------------- | ------------ | --------------------------------- | | [`geo_point`](/data-types/types/geo_point/) | SET | DISTANCE | | [`geo_spatial_index`](/data-types/types/geo_spatial_index/) | CREATE | ADD, REMOVE, RADIUS, BOX, POLYGON | | [`bounding_box`](/data-types/types/bounding_box/) | SET | CONTAINS | | [`polygon`](/data-types/types/polygon/) | SET | CONTAINS | --- ## Graph, synchronization, and network Source: [canonical documentation page](/data-types/graph-synchronization-and-network/) Summary: These types cover graph adjacency, atomic coordination, synchronization, rate limiting, and network values. These types cover graph adjacency, atomic coordination, synchronization, rate limiting, and network values. See [Type construction and operations](/data-types/operations/#graph-synchronization-and-network-operations) for token/TTL rules, cancellation behavior, permit accounting, token-bucket refill, persistence exclusions, CIDR parsing, and URL accessors. | Type | Construction | Principal operations | | ------------------------------------------------- | ------------ | ------------------------------------ | | [`adjacency_set`](/data-types/types/adjacency_set/) | CREATE | ADD, REMOVE, NEIGHBORS | | [`atomic_counter`](/data-types/types/atomic_counter/) | CREATE | ADD, LOAD, CAS | | [`lease`](/data-types/types/lease/) | CREATE | ACQUIRE, RENEW, RELEASE, HELD | | [`lock`](/data-types/types/lock/) | CREATE | TRY_LOCK, RENEW, UNLOCK | | [`barrier`](/data-types/types/barrier/) | CREATE | WAIT | | [`semaphore`](/data-types/types/semaphore/) | CREATE | ACQUIRE, TRY_ACQUIRE, RELEASE, STATS | | [`rate_limit_bucket`](/data-types/types/rate_limit_bucket/) | CREATE | ALLOW, REMAINING | | [`cidr_network`](/data-types/types/cidr_network/) | SET | CONTAINS | | [`url`](/data-types/types/url/) | SET | HOST, SCHEME | --- ## Indexes and search Source: [canonical documentation page](/data-types/indexes-and-search/) Summary: Index/search types provide exact, range, prefix, full-text, vector, and secondary lookup paths. Index/search types provide exact, range, prefix, full-text, vector, and secondary lookup paths. See [Type construction and operations](/data-types/operations/#index-and-search-operations) for request contracts and implementation notes covering B-tree splits/rebuilds, byte-wise radix traversal, tokenization/ranking, and explicit index maintenance. Index objects are not updated automatically when another key changes. | Type | Construction | Principal operations | | ------------------------------------------------------------------- | ------------ | --------------------------------------------- | | [`btree_index`](/data-types/types/btree_index/) | CREATE | INSERT, DELETE, SEARCH, RANGE | | [`hash_index`](/data-types/types/hash_index/) | CREATE | ADD, REMOVE, LOOKUP | | [`radix_tree`](/data-types/types/radix_tree/) | CREATE | INSERT, EXACT, PREFIX | | [`trie`](/data-types/types/trie/) | CREATE | INSERT, EXACT, PREFIX | | [`inverted_index`](/data-types/types/inverted_index/) | CREATE | INDEX, REMOVE, SEARCH | | [`hnsw_index`](/data-types/types/hnsw_index/) | CREATE | ADD, REMOVE, SEARCH | | [`flat_vector_index`](/data-types/types/flat_vector_index/) | CREATE | ADD, REMOVE, SEARCH, LEN | | [`geospatial_secondary_index`](/data-types/types/geospatial_secondary_index/) | CREATE | ADD, REMOVE, RADIUS, BOX, POLYGON | | [`secondary_index`](/data-types/types/secondary_index/) | CREATE | ADD, LOOKUP | | [`full_text_document`](/data-types/types/full_text_document/) | CREATE | ADD_TEXT, ADD_TAG, ADD_NUMERIC, GET, SNAPSHOT | | [`tag_field`](/data-types/types/tag_field/) | CREATE | SET, GET | | [`numeric_field`](/data-types/types/numeric_field/) | CREATE | SET, GET | | [`text_field`](/data-types/types/text_field/) | CREATE | SET, GET | --- ## Type construction and operation reference Source: [canonical documentation page](/data-types/operations/) Summary: Reference the Amaquet protocol contract for constructing and operating on typed values, including arguments, defaults, results, persistence, and failures. This is the complete protocol contract for constructing and operating on Amaquet's 91 registered types. The individual type pages provide copyable creation examples; this page defines arguments, defaults, results, and the implementation behavior behind them. ### `OP` envelope Use this command envelope to invoke a type-specific operation on an existing key. ```json { "command": "OP", "args": { "key": "jobs", "operation": "ENQUEUE", "args": { "value": { "type": "json", "value": { "id": "job-1" } } } } } ``` Operation names are case-insensitive at dispatch. Argument names are case-sensitive JSON object keys. A nested typed value always uses `{"type":"…","value":…}`; see [Wire values](/protocol/wire-values/). Read operations do not increment the key version. A successful mutation reserves approximate memory first and increments the version after the type object changes. A failed validation or operation does not increment it. Empty reads generally return `null`, an empty collection, or `false` as documented below. Read-classified paths can still perform internal lazy maintenance. In particular, reliable-queue `STATS` and snapshots requeue or dead-letter expired leases without a version increment or a separate AOF mutation record. The word `GET` in a directly encoded type's individual page means the top-level key command. It is not automatically an `OP GET`. Only operations listed in the tables below are accepted by `OP`. ### Construction Direct values use `SET`; their JSON representations are defined in [Wire values](/protocol/wire-values/). `json`, `messagepack`, and `cbor` support both direct `SET` and an empty `CREATE` initializer. Other composite/specialized values use: ```json { "command": "CREATE", "args": { "key": "example", "type": "list", "options": {}, "ttl_ms": 60000, "nx": true } } ``` `CREATE` and `SET` both accept optional `ttl_ms`, `nx`, and `xx`. `nx` fails if a live key exists; `xx` fails if no live key exists. Supplying both is legal to decode but can never produce a useful conditional write. `expires_at_unix_nano` is the absolute-time form used by deterministic replay and advanced clients. #### Collection and structured options These creation options define the initial shape and constraints of collection and structured values. | Type | Options and defaults | Constraints | | -------------------------------------------------------------------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------- | | `list`, `set`, `sorted_set`, `ordered_set`, `multiset`, `hashmap`, `ordered_map`, `multimap` | none | Start empty | | `array` | `typed:false`, `element_type` omitted | A typed array requires each nested value to match `element_type` | | `deque` | `capacity:16` | Values below 8 become 8; grows by doubling | | `ring_buffer` | `capacity:1024` | Minimum 1; overwrites the oldest value when full | | `tuple` | `values:[]` | Array of nested typed values; fixed after creation | | `json` | none | Starts as `{}` | | `messagepack`, `cbor` | none | Start as empty opaque bytes | | `record` | `schema:[]` | Each field has `name`, registered `type`, and `nullable` | | `matrix` | `rows:1`, `cols:1`, `dtype:"float64"`, zero-filled `data` | `data` length must equal rows × columns | | `tensor` | `shape:[1]`, `dtype:"float64"`, zero-filled `data` | Every dimension positive; product must equal data length | The type package defines dtype constants `float64`, `float32`, `int64`, and `int32`. The constructor currently stores any supplied dtype string without validating it, and values are represented internally as `float64`; use the defined constants for portable data because dtype is descriptive metadata rather than an enforced conversion on every assignment. #### Queue, stream, and messaging options Messaging structures begin empty unless their row identifies retention, delivery-attempt, or group-name state. These values are creation-time options; runtime behavior is controlled through the operations documented later on this page. | Type | Options and defaults | Constraints | | ---------------------------------------------------------------------------------------------------------------- | -------------------- | ---------------------------------------------------------------------------- | | `fifo_queue`, `lifo_stack`, `priority_queue`, `blocking_queue`, `delayed_queue`, `stream`, `pubsub`, `event_log` | none | Start empty | | `reliable_queue` | `max_attempts:5` | Values below 1 become 5 | | `consumer_group` | `name:"default"` | Standalone snapshot object; stream groups are normally created on a `stream` | | `persistent_topic` | `max_messages:0` | `0` means no retention-count cap | #### Bit and probabilistic options Bit and probabilistic structures allocate or grow their internal state from these sizing parameters. Defaults favor general-purpose use, while constructor validation or fallback behavior is listed explicitly in the constraints column. | Type | Options and defaults | Constraints | | --------------------------------------- | --------------------------------------------- | ------------------------------------------------------------ | | `bitmap`, `bit_set`, `roaring_bitmap` | none | Grow as values/bits are added | | `bit_field` | `size_bytes:0` | Negative values become 0; later writes grow it | | `hyperloglog` | `precision:14` | Precision 4…18 | | `bloom_filter`, `counting_bloom_filter` | `expected:100000`, `false_positive_rate:0.01` | Expected > 0; rate strictly between 0 and 1 | | `cuckoo_filter` | `capacity:100000` | Allocates power-of-two four-slot buckets sized from capacity | | `count_min_sketch` | `width:2048`, `depth:5` | Both positive | | `top_k` | `k:10` | Values below 1 become 1 | | `t_digest` | `compression:100` | Values below 20 become 100 | #### Time-series, geo, vector, and index options These constructors define retention, aggregation, dimensions, distance metrics, and index tuning. Some string options are stored at creation and validated only when an operation needs them, as noted after the table. | Type | Options and defaults | Constraints | | ----------------------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------- | | `timeseries`, `counter_series`, `gauge_series` | `labels:{}`, `retention_ms:0` | `0` means no retention trimming | | `histogram` | `bounds:[]` | Bounds are sorted; one overflow bucket is added | | `timeseries_labels` | `labels:{}` | Mutable string map | | `aggregated_series` | `bucket_ms:60000`, `aggregation:"avg"` | Source key is supplied at read time | | `geo_spatial_index`, `geospatial_secondary_index` | none | Start empty | | `vector_set`, `flat_vector_index` | required positive `dim`, `metric:"cosine"` | Metrics: `cosine`, `dot`, `euclidean`, `manhattan` | | `vector_metadata` | none | Starts as an empty JSON-like map | | `hnsw_index` | required positive `dim`, `metric:"cosine"`, `m:16`, `ef_search:64` | `m<2` becomes 16; `ef_search Cmd["cmd/"] Cmd --> Server["amaquet/"] Cmd --> CLI["amaquet-cli/"] Cmd --> Keygen["amaquet-keygen/"] Cmd --> Restore["amaquet-restore/"] Repository --> Internal["internal/"] Internal --> Admin["admin/"] Internal --> Compression["compression/"] Internal --> Config["config/"] Internal --> Core["core/"] Internal --> Persistence["persistence/"] Internal --> Protocol["protocol/"] Internal --> Security["security/"] Internal --> TCPServer["server/"] Internal --> Types["types/"] Repository --> Package["pkg/amaquet/"] Repository --> Scripts["scripts/"] Repository --> Tests["tests/"] Repository --> Docs["docs/"] Docs --> AstroConfig["astro.config.mjs"] ``` The `cmd` directory contains the server, CLI, identity generator, and offline restore executables. `internal` contains the HTTP administration API and state, compression, configuration, sharded core engine, append-only persistence, protocol framing, identity, TCP dispatch, and concrete data structures. `pkg/amaquet` is the public Go client; `scripts`, `tests`, and `docs` contain project automation, integration coverage, and the Astro/Starlight documentation site respectively. The `internal` boundary prevents external Go packages from depending directly on implementation details. Public applications should use `pkg/amaquet` or implement the Amaquet protocol. --- ## Bash testing and fuzzing Source: [canonical documentation page](/development/testing-fuzzing/) Summary: Amaquet includes a black-box Bash contract and a mutation-fuzz entrypoint for every stable wire data type. Amaquet includes a black-box Bash contract and a mutation-fuzz entrypoint for **every stable wire data type**. The harness talks to the server through `amaquet-cli`; it does not call Go type methods directly. This tests the same protocol, dispatcher, engine, serialization, TTL, and authorization boundaries used by normal clients. ### Layout The Bash harness is organized into shared helpers, per-type contracts and fuzzers, protocol checks, and suite runners. ```mermaid flowchart TB Tests["tests/"] --> Bash["bash/"] Bash --> Common["lib/common.sh"] Bash --> Manifest["type-manifest.json"] Bash --> Types["types/"] Types --> Contracts["test_type.sh: 91 deterministic contracts"] Bash --> Fuzz["fuzz/"] Fuzz --> TypeFuzzers["fuzz_type.sh: 91 type fuzzers"] Bash --> Protocol["protocol/"] Protocol --> ProtocolContract["test_uri_and_commands.sh"] Protocol --> Frames["fuzz_frames.sh"] Bash --> RunAll["run_all.sh"] Bash --> RunFuzz["run_fuzz.sh"] Bash --> RunProtocolFuzz["run_protocol_fuzz.sh"] ``` `type-manifest.json` is the machine-readable coverage catalog. The documentation checker requires one deterministic script, one fuzzer, and one Astro reference page for each of its 91 entries. ### Requirements The harness requires Bash, the Go toolchain when binaries are not already built, and Python 3 for portable helpers. If `jq` is available, scalar JSON extraction uses it because it is much faster. Python remains the fallback, so `jq` is optional. ### Deterministic type suite Run this suite to exercise the public protocol contract for every stable wire type. ```bash ./tests/bash/run_all.sh ``` The runner starts one isolated Amaquet process on free loopback ports and passes its `AMAQUET_URI` to each type script. Scripts use unique keys and can run concurrently. The default is four workers: ```bash AMAQUET_TEST_JOBS=8 ./tests/bash/run_all.sh ``` Each contract verifies, as applicable: 1. the documented `SET` or `CREATE` construction path; 2. the exact `TYPE` wire identifier; 3. `EXISTS`; 4. `EXPIRE` and a positive `TTL`; 5. `PERSIST` and the `-1` no-expiry result; 6. one or more meaningful type-specific operations and their results; and 7. `DEL` cleanup. The tests use current timestamps for retention-sensitive time-series contracts so old synthetic values are not pruned by design. ### Run one type Invoke an individual contract directly when developing or diagnosing a specific type. ```bash ./tests/bash/types/test_hnsw_index.sh ./tests/bash/types/test_reliable_queue.sh ./tests/bash/types/test_timeseries.sh ``` When `AMAQUET_URI` is not set, an individual script builds missing binaries, starts a disposable local Amaquet process, runs the contract, and removes its temporary state. To inspect a failed local run: ```bash AMAQUET_KEEP_TMP=1 ./tests/bash/types/test_vector_set.sh ``` The harness prints the temporary directory path through its normal diagnostics. The directory contains the generated configuration, data directory, and `server.log`. To test an existing server instead: ```bash AMAQUET_URI=amaquet://127.0.0.1:13378 \ ./tests/bash/types/test_vector_set.sh ``` ### Per-type fuzzing Every stable data type has a dedicated script: ```bash AMAQUET_FUZZ_ITERATIONS=100 ./tests/bash/run_fuzz.sh ``` For each iteration, the fuzzer sends a randomized unsupported operation and requires Amaquet to reject it without damaging the key. It then checks that the key remains addressable. Every tenth iteration it resets the key, reconstructs the type, and executes the valid semantic contract. Resetting prevents stateful types such as queues, consumer groups, locks, and streams from producing false failures only because a previous valid iteration changed state. Run a single type fuzzer: ```bash AMAQUET_FUZZ_ITERATIONS=1000 ./tests/bash/fuzz/fuzz_vector_set.sh ``` ### Raw Amaquet frame fuzzing The protocol fuzzer works below `amaquet-cli` and opens TCP connections directly: ```bash AMAQUET_FRAME_FUZZ_ITERATIONS=1000 ./tests/bash/protocol/fuzz_frames.sh ``` It exercises malformed or adversarial frames, including: - invalid `AMQT` magic; - unsupported protocol versions; - invalid frame kinds and opcodes; - truncated headers and payloads; - inconsistent declared lengths; - malformed JSON request payloads; - random byte sequences; - randomized request IDs; - single-byte frame mutation; and - reconnection after malformed input. The fuzzer ends with a valid `PING`. A run is not considered successful if the server stops accepting valid protocol traffic. The combined protocol runner also tests URI and command behavior: ```bash AMAQUET_FRAME_FUZZ_ITERATIONS=250 ./tests/bash/run_protocol_fuzz.sh ``` ### Environment variables The Bash runners use these variables to select an existing server, tune concurrency, and bound randomized work. Unless a row says otherwise, unset variables use the runner defaults shown below. | Variable | Purpose | Default | | ------------------------------- | --------------------------------------------------------- | ----------------- | | `AMAQUET_URI` | Target an existing server instead of creating a local one | unset | | `AMAQUET_BIN` | Override the `amaquet` server binary | `bin/amaquet` | | `AMAQUET_CLI` | Override the CLI binary | `bin/amaquet-cli` | | `AMAQUET_TIMEOUT` | Per-CLI request timeout | `5s` | | `AMAQUET_KEEP_TMP` | Keep disposable test state when set to `1` | `0` | | `AMAQUET_TEST_JOBS` | Concurrent type scripts in suite runners | `4` | | `AMAQUET_FUZZ_ITERATIONS` | Mutations per type fuzzer | `25` in runner | | `AMAQUET_FRAME_FUZZ_ITERATIONS` | Random raw-frame cases | `250` | ### CI coverage The main GitHub Actions workflow separates ordinary Go and documentation checks from bounded developer-fuzz and chaos jobs. Across those jobs it runs: ```text go test ./... go test -race ./... go vet ./... go build ./cmd/amaquet ./cmd/amaquet-cli ./cmd/amaquet-keygen ./cmd/amaquet-restore ./tests/bash/run_all.sh AMAQUET_FUZZ_ITERATIONS=1 ./tests/bash/run_fuzz.sh AMAQUET_FRAME_FUZZ_ITERATIONS=50 ./tests/bash/run_protocol_fuzz.sh AMAQUET_SCENARIO_ITERATIONS=1 ... ./tests/bash/run_scenarios.sh AMAQUET_SCENARIO_ITERATIONS=1 ... ./tests/bash/run_isolated_fuzz.sh AMAQUET_GO_FUZZ_TIME=1s ./scripts/run_go_fuzz.sh ... ./tests/bash/run_chaos.sh AMAQUET_BENCH_TIME=100ms ./scripts/run_benchmarks.sh python3 docs/scripts/check_docs.py make docs-build ``` The ellipses stand for the additional bounded environment settings recorded in `.github/workflows/ci.yml`; they are not literal shell arguments. The one-iteration fuzz settings are a smoke layer. The scheduled workflow runs longer chaos, native fuzz, Bash fuzz, and benchmark jobs. ### Scope The Bash suite is black-box integration coverage. It complements, but does not replace, Go unit tests, the race detector, static analysis, load testing, or Go's native coverage/fuzz tooling. ### Cross-feature developer scenarios The per-type fuzzers are complemented by 28 cross-feature developer workflow scripts plus isolated security/persistence tests. These cover concurrent clients, malformed arguments, large values, compression transitions, key expiry races, blocking queues, barriers, reliable queue dead-letter behavior, mixed workloads, AOF restart/recovery, RBAC, API-key lifecycle, protocol multiplexing, partial TCP writes, and actual Pub/Sub event delivery. Run them with: ```bash AMAQUET_SCENARIO_ITERATIONS=10 ./tests/bash/run_scenarios.sh ./tests/bash/run_isolated_fuzz.sh ``` For the full combined black-box suite: ```bash ./tests/bash/run_developer_fuzz.sh ``` See [Developer workflow fuzzing](/development/developer-fuzzing/) for the complete matrix, reproduction variables, and release-test recommendations. ### Native Go coverage-guided fuzzing Seed corpora for eight native Go fuzz targets run under `go test ./...`. To run the coverage-guided engine: ```bash AMAQUET_GO_FUZZ_TIME=10s ./scripts/run_go_fuzz.sh ``` These targets fuzz protocol frames and URIs, compression/decompression, typed wire decoding, and the core engine lifecycle. Input sizes are bounded inside each fuzz target. --- ## Configuration Source: [canonical documentation page](/getting-started/configuration/) Summary: Configure Amaquet listeners, authentication, TLS, persistence, memory limits, request admission, and compression for local or production deployments. Amaquet configuration controls two listeners, authentication, optional persistence, memory governance, request admission, upload handling, and adaptive in-memory compression. The server reads JSON, applies supported environment overrides, validates the merged result, creates required runtime directories, and only then starts serving traffic. The [configuration reference](/reference/configuration/) is the exhaustive field-by-field reference. This page explains how to choose values for common deployments and what each group means operationally. ### Configuration lifecycle Amaquet applies configuration in this order: 1. Start with compiled defaults. 2. Read the JSON file named by `-config`, defaulting to `./amaquet.json`. A missing file is allowed and leaves the defaults in place. 3. Apply supported `AMAQUET_*` environment variables. An environment value wins over the JSON value for the field it controls. 4. Validate ports, limits, enum values, TLS material, and listener security rules. 5. Create `data_dir`, load administration state, initialize persistence and compression, and start both listeners. Invalid JSON or an invalid merged value stops startup. Invalid numeric environment strings are ignored by the loader and leave the earlier JSON/default value in place; the resulting configuration is still validated. Relative paths are resolved from the server process's current working directory, so service managers should set a fixed working directory or use absolute paths. Generate a complete starting file rather than hand-writing every default: ```bash ./bin/amaquet --init-config --config ./amaquet.json ``` This writes a random `security.bootstrap_token`, prints it once, and exits. It does not apply environment overrides. Keep the file private. During normal startup, an empty bootstrap value causes Amaquet to reuse or create `data_dir/bootstrap-token` with mode `0600` and log the protected path rather than the secret. ### Local development profile The generated configuration is a good local starting point because both listeners bind to loopback and native authentication is enabled. A minimal hand-written equivalent is: ```json { "protocol": { "host": "127.0.0.1", "port": 13378, "require_auth": true }, "admin": { "host": "127.0.0.1", "port": 13379 }, "data_dir": "./data" } ``` Omitted fields receive the defaults listed below. Use the token generated by `--init-config`, or the protected token file created on first start, for the first authenticated request. Do not set `require_auth` to `false` merely to avoid handling credentials unless the process is completely isolated and disposable. ### Networked deployment profile A networked deployment should use separate TLS policies for the data plane and control plane, private bind addresses, explicit persistent paths, and a memory budget below the host or container limit. The following is an illustrative shape; replace every certificate, key, token, and storage path with deployment-specific values: ```json { "protocol": { "host": "10.0.10.15", "port": 13378, "tls": true, "tls_cert_file": "/etc/amaquet/tls/protocol.crt", "tls_key_file": "/etc/amaquet/tls/protocol.key", "require_auth": true, "read_timeout_ms": 120000, "write_timeout_ms": 30000 }, "admin": { "host": "10.0.20.15", "port": 13379, "tls": true, "tls_cert_file": "/etc/amaquet/tls/admin.crt", "tls_key_file": "/etc/amaquet/tls/admin.key", "metrics_require_auth": true }, "security": { "public_key_file": "/etc/amaquet/identity/public.pem", "private_key_file": "/etc/amaquet/identity/private.pem" }, "persistence": { "aof_enabled": true, "aof_path": "/var/lib/amaquet/amaquet.aof", "fsync": "everysec" }, "limits": { "max_connections": 2000, "max_memory_bytes": 6442450944, "eviction_policy": "noeviction", "max_inflight_requests": 4096, "max_inflight_per_connection": 128, "command_timeout_ms": 30000 }, "data_dir": "/var/lib/amaquet" } ``` `protocol.allow_insecure_auth` and `admin.allow_insecure` remain false in this profile. They are validation escape hatches for deliberately isolated plaintext networks, not substitutes for TLS. ### Protocol listener The `protocol` object configures the binary Amaquet data-plane endpoint used by `amaquet-cli` and the Go client. Plaintext clients use `amaquet://`; TLS clients use `amaquets://`. #### `protocol.host` and `protocol.port` `host` is the local bind address and defaults to `127.0.0.1`; `port` defaults to `13378` and must be between `1` and `65535`. Loopback is appropriate for a local process. A wildcard address such as `0.0.0.0` exposes the service on every IPv4 interface and should be avoided unless firewalling and TLS are already designed. #### `protocol.tls`, `protocol.tls_cert_file`, and `protocol.tls_key_file` Set `tls` to `true` to encrypt native protocol connections. Both certificate and private-key paths are required, readable by the service account, and expected to match the hostnames used by clients. These files are for transport TLS; optional Ed25519 identity files are a separate application-level identity mechanism. #### `protocol.require_auth` The default is `true`. New native connections must authenticate with `AUTH` before protected commands are accepted. If false, new connections are anonymous administrative actors, which is suitable only for an isolated local test process. It is not a replacement for authorization in a shared environment. #### `protocol.allow_insecure_auth` This is false by default and does not disable authentication. It permits the specific combination of a non-loopback host, required authentication, and plaintext transport. Enable it only when another trusted network boundary provides the protection; TLS is the recommended solution. #### `protocol.read_timeout_ms` and `protocol.write_timeout_ms` The read timeout defaults to `120000` milliseconds and limits incomplete frames or idle input. The write timeout defaults to `30000` milliseconds and limits response or event writes to slow readers. `0` disables the respective deadline; use that only when connection lifetime is controlled elsewhere. ### Administration listener The `admin` object configures the HTTP control plane. It serves `/api/*` and `/metrics`, not a user interface. Keep it on loopback or a private management network whenever possible. #### `admin.host` and `admin.port` The defaults are `127.0.0.1` and `13379`. The admin port is independent of the native protocol port. A non-loopback admin host must use HTTPS or explicitly set `admin.allow_insecure`. #### `admin.tls`, `admin.tls_cert_file`, and `admin.tls_key_file` Enable `admin.tls` for HTTPS and provide a certificate/key pair. Admin TLS has independent paths from protocol TLS so control-plane certificates can use a separate hostname, trust chain, and rotation schedule. #### `admin.allow_insecure` Allows plaintext HTTP on a non-loopback admin address. It is intended only for an isolated development network or a deployment where an explicitly trusted boundary provides the transport protection. It does not make the admin API unauthenticated. #### `admin.metrics_require_auth` The default is `true`, so `/metrics` requires an authenticated admin request. Set it to false only when the metrics listener is already isolated and exposing runtime information to the monitoring network is intentional. #### `admin.max_sessions` The default is `10000` server-side HTTP sessions. Sessions are independent of native protocol connections and API keys and normally expire after twelve hours. Lower this value on a constrained admin process; raise it only when the number of simultaneous browser/session clients justifies the memory use. #### `admin.auth_failures_per_minute` The default is `20` failed authentication attempts per source address per minute. It must be positive. This throttle helps slow credential guessing but should be combined with TLS, firewall restrictions, strong secrets, and monitoring. ### Security and identity These settings control the first administrative credential and the optional cryptographic identity that clients can verify after connecting. #### `security.bootstrap_token` This is the initial administrative credential. `--init-config` generates one. With an empty value, startup reads or creates `data_dir/bootstrap-token`; an environment value from `AMAQUET_BOOTSTRAP_TOKEN` takes precedence at runtime. Use the bootstrap credential to create a permanent API key, then remove it from routine application configuration. After the first admin API key is created, bootstrap authentication is permanently disabled in persisted admin state. #### `security.public_key_file` and `security.private_key_file` These paths enable the optional Ed25519 protocol identity. Generate matching files with `amaquet-keygen`, protect the private key, and configure both paths. Clients can verify the server identity fingerprint/signature after `HELLO`. This key pair does not replace X.509 certificates or provide encryption. ### Persistence Amaquet is still an in-memory database when AOF is enabled. AOF records committed mutations for restart recovery; it does not turn the engine into a disk-backed database or eliminate the need for backups. #### `persistence.aof_enabled` The default is `false`. With AOF disabled, a restart begins with an empty data keyspace, while control-plane state may remain in `data_dir/admin.json`. With AOF enabled, committed records are replayed before listeners accept traffic. Enable it when restart recovery is required and size storage for journal growth and checkpoints. #### `persistence.aof_path` The default is `./data/amaquet.aof`. The parent directory must be writable and should be on persistent storage. Use an absolute path in a service deployment so changing the process working directory cannot select a different journal. #### `persistence.fsync` `always` provides the strongest acknowledged-write durability with the highest write latency. `everysec` flushes approximately once per second and is the usual balance. `no` relies on operating-system writeback and provides the weakest crash-durability guarantee. All modes retain transactional replay and CRC integrity checks. ### Limits and memory These limits bound connections, frames, memory reservations, concurrency, uploads, and command execution. Set them with the host's file-descriptor, memory, and workload ceilings in mind. #### `limits.max_connections` The default is `10000` native connections. Set it below the file-descriptor and memory capacity of the host, accounting for client pools, TLS sockets, and administration traffic. #### `limits.max_payload_bytes` The default and hard maximum are `67108864` bytes (64 MiB) per protocol frame. Lowering it reduces the largest single decode allocation. Chunked blob operations are the path for logical values that need multiple frames. #### `limits.max_memory_bytes` The default `0` means no configured engine data budget. A positive value is an approximate budget for stored values and reservation decisions, not a process-RSS cap. Leave headroom for Go runtime memory, network buffers, indexes, AOF state, and temporary request data. #### `limits.eviction_policy` `noeviction` rejects an allocating write at the memory limit. `allkeys-lru` and `allkeys-lfu` evict sampled candidates regardless of TTL. `volatile-ttl` considers only expiring keys and prefers the nearest expiry. Eviction is approximate; if no eligible candidate exists, the write is rejected. `DEL` and other memory-reducing operations remain available. #### `limits.max_inflight_requests` and `limits.max_inflight_per_connection` The defaults are `4096` active commands server-wide and `128` active commands per connection. The global cap protects the process overall; the per-connection cap prevents one multiplexed client from consuming all command capacity. #### `limits.max_inflight_payload_bytes` The default is `268435456` bytes (256 MiB) across active inbound frames. It must be at least `max_payload_bytes`. Lower it when many clients can submit large requests concurrently. #### `limits.max_pending_upload_bytes` The default is `1073741824` bytes (1 GiB) reserved by unfinished chunked uploads. It protects upload spool and reservation capacity from clients that declare large uploads and never complete them. #### `limits.upload_ttl_ms` The default is `600000` milliseconds (10 minutes), with a minimum of `1000`. An upload that remains inactive past this lifetime is discarded. Increase it only for slow links and keep the aggregate pending-upload budget bounded. #### `limits.command_timeout_ms` The default is `30000` milliseconds. This server-side deadline covers one command and also caps blocking type operations. It is independent of a client's own dial/request timeout, so clients should choose a compatible value. ### In-memory compression Compression trades CPU time for lower in-memory value size. It is applied only when the configured size and savings thresholds make it worthwhile. #### `compression.enabled` Compression is enabled by default for supported contiguous values. It is adaptive: values below `min_bytes`, incompressible values, and results that do not meet the savings threshold remain uncompressed. Disable it when CPU latency is more important than memory reduction. #### `compression.algorithm` `auto` tries RLE and LZ4 and retains the smaller candidate. `lz4` favors fast block compression, `rle` favors repeated-byte data, `deflate-fast` favors a higher ratio at additional CPU cost, and `none` disables compression. The value is validated at startup. #### `compression.min_bytes` The default is `1024` bytes. Values smaller than this are not considered for compression. The setting must be non-negative and can be raised to avoid spending CPU on small values. #### `compression.min_savings_percent` The default is `8`, meaning a compressed result must save at least eight percent before being retained. It must be at least `0` and below `100`. Higher thresholds avoid marginal compression wins; lower thresholds trade more CPU for memory savings. ### `data_dir` The default `./data` is the runtime root for `admin.json`, generated bootstrap-token material, upload spools, checkpoints/backups, and related security/runtime files. Amaquet creates it with restrictive permissions. Put it on persistent storage when administration state, upload recovery, or backups must survive restarts. Back it up separately from the AOF journal. ### Environment overrides Environment values are applied after JSON and are useful for containers and service managers. Only the variables below are supported: | Variable | Overrides | Notes | | ------------------------------------- | ------------------------------------ | -------------------------------------- | | `AMAQUET_HOST` | `protocol.host` | Text address. | | `AMAQUET_PORT` | `protocol.port` | Invalid integers are ignored. | | `AMAQUET_ADMIN_HOST` | `admin.host` | Text address. | | `AMAQUET_ADMIN_PORT` | `admin.port` | Invalid integers are ignored. | | `AMAQUET_ALLOW_INSECURE_AUTH` | `protocol.allow_insecure_auth` | Exact `1` or `true` enables it. | | `AMAQUET_ADMIN_ALLOW_INSECURE` | `admin.allow_insecure` | Exact `1` or `true` enables it. | | `AMAQUET_BOOTSTRAP_TOKEN` | `security.bootstrap_token` | Runtime-only secret override. | | `AMAQUET_TLS_CERT` | `protocol.tls_cert_file` | Also sets `protocol.tls=true`. | | `AMAQUET_TLS_KEY` | `protocol.tls_key_file` | Also sets `protocol.tls=true`. | | `AMAQUET_COMPRESSION` | `compression.enabled` | Disabled only by exact `0` or `false`. | | `AMAQUET_COMPRESSION_ALGORITHM` | `compression.algorithm` | Must be a supported algorithm. | | `AMAQUET_COMPRESSION_MIN_BYTES` | `compression.min_bytes` | Invalid integers are ignored. | | `AMAQUET_AOF_ENABLED` | `persistence.aof_enabled` | Exact `1` or `true` enables it. | | `AMAQUET_AOF_PATH` | `persistence.aof_path` | Runtime path. | | `AMAQUET_AOF_FSYNC` | `persistence.fsync` | `always`, `everysec`, or `no`. | | `AMAQUET_MAX_MEMORY_BYTES` | `limits.max_memory_bytes` | Parsed as a signed 64-bit integer. | | `AMAQUET_EVICTION_POLICY` | `limits.eviction_policy` | Must be a supported policy. | | `AMAQUET_MAX_CONNECTIONS` | `limits.max_connections` | Must remain positive after merging. | | `AMAQUET_MAX_INFLIGHT_REQUESTS` | `limits.max_inflight_requests` | Global active-command cap. | | `AMAQUET_MAX_INFLIGHT_PER_CONNECTION` | `limits.max_inflight_per_connection` | Per-connection active-command cap. | | `AMAQUET_MAX_INFLIGHT_PAYLOAD_BYTES` | `limits.max_inflight_payload_bytes` | Must be at least `max_payload_bytes`. | | `AMAQUET_MAX_PENDING_UPLOAD_BYTES` | `limits.max_pending_upload_bytes` | Aggregate unfinished-upload budget. | | `AMAQUET_UPLOAD_TTL_MS` | `limits.upload_ttl_ms` | Minimum `1000`. | | `AMAQUET_COMMAND_TIMEOUT_MS` | `limits.command_timeout_ms` | Must be positive. | There are no environment overrides for admin TLS paths, `admin.max_sessions`, `admin.auth_failures_per_minute`, `limits.max_payload_bytes`, or `compression.min_savings_percent`; configure those in JSON. Environment bootstrap secrets are not copied into the JSON file by runtime configuration saves. ### Validation and unsafe combinations Both ports must be valid. Positive resource limits cannot be zero or negative, `max_inflight_payload_bytes` cannot be smaller than one frame, and `upload_ttl_ms` cannot be below one second. Compression and eviction values must match their supported enums. TLS requires both a certificate and a key for the listener being secured. An authenticated non-loopback native listener requires TLS unless `protocol.allow_insecure_auth` is explicitly true. A non-loopback admin listener requires HTTPS unless `admin.allow_insecure` is explicitly true. These checks happen before the sockets are opened, so a configuration error fails closed rather than starting a partially protected service. ### Runtime changes and restart behavior The administration configuration endpoint validates and saves a new public configuration using a temporary file, fsync, and atomic rename. It does not reinitialize listeners, engine memory policy, persistence, or other runtime components in place. When the submitted configuration differs from the running configuration, the response reports `restart_required: true`; restart the process under the approved service supervisor to apply it. The bootstrap token is intentionally removed from the public configuration representation. Treat a configuration response as safe to inspect, but continue to protect the original JSON file, environment, token file, private keys, and AOF paths. ### Related guidance Use [Installation](/getting-started/installation/) for a source build and first authenticated request, [Authentication and identity](/protocol/authentication/) for `HELLO`/`AUTH` and credential behavior, [Persistence and recovery](/concepts/persistence/) for AOF durability, and [Production deployment](/operations/deployment/) for supervision, storage, and network topology. --- ## Installation Source: [canonical documentation page](/getting-started/installation/) Summary: Install Amaquet from source, configure its server and CLI binaries, secure the first startup, and verify an authenticated request. This guide builds Amaquet from the repository, creates a secure local configuration, starts the server, and verifies the first authenticated request. Amaquet is a source-built Go server with a separate command-line client, key generator, and offline restore utility; it does not require Redis or another database. ### What gets installed The normal build produces four executables in `bin/`: | Executable | Purpose | | --------------------- | ---------------------------------------------------------------------------------------- | | `bin/amaquet` | Database server, native protocol listener, and HTTP administration listener. | | `bin/amaquet-cli` | Command-line client for native protocol commands and type operations. | | `bin/amaquet-keygen` | Generates the optional Ed25519 server identity key pair. | | `bin/amaquet-restore` | Validates and atomically installs a verified AOF checkpoint while the server is stopped. | The server keeps active values in memory. AOF persistence and the control-plane files under `data_dir` are optional but should be placed on persistent, protected storage for a durable deployment. ### Prerequisites Source builds require Go 1.23 or newer. Git is needed to clone the repository. Node.js 22.12 or newer and npm are only required when building or serving the Astro documentation site; they are not required to run the Amaquet server or CLI. Check the installed toolchains before starting: ```bash go version git --version node --version # only required for docs npm --version # only required for docs ``` The server needs permission to bind its configured protocol and admin ports, create `data_dir`, and read any configured TLS or identity key files. A production service also needs enough file descriptors for its connection limit and enough memory for the engine plus runtime overhead. ### Obtain the source Clone the repository and work from its root so the default relative paths such as `./amaquet.json` and `./data` resolve predictably: ```bash git clone https://github.com/newfoundcodes/amaquet.git cd amaquet ``` If the source is already checked out, update it and review the release or deployment changes before rebuilding. Keep the existing `data_dir` and AOF path when upgrading an installation that must retain its data. ### Build the binaries The recommended build creates `bin/` and writes all four executables there: ```bash make build ``` The equivalent explicit commands are: ```bash mkdir -p bin go build -trimpath -o bin/amaquet ./cmd/amaquet go build -trimpath -o bin/amaquet-cli ./cmd/amaquet-cli go build -trimpath -o bin/amaquet-keygen ./cmd/amaquet-keygen go build -trimpath -o bin/amaquet-restore ./cmd/amaquet-restore ``` Confirm that the binaries are executable and report the expected server version: ```bash ./bin/amaquet -version ./bin/amaquet-cli -h ``` `make build` does not install files into a system directory. Either invoke the binaries with their `./bin/` paths or add the repository's `bin/` directory to a controlled service or shell `PATH`. ### Create the first configuration Generate a default configuration and a high-entropy bootstrap administrator credential: ```bash ./bin/amaquet --init-config --config ./amaquet.json ``` The command writes the JSON file, prints the bootstrap token once, and exits. Save the token in a password manager or secret store immediately. The generated file contains the token and is written with restrictive permissions; do not commit it or include it in a public support bundle. The generated defaults bind both listeners to loopback, require native protocol authentication, keep AOF disabled, and use `./data` for runtime state. Review the file before starting the server. The complete field-by-field explanation is in [Configuration](/getting-started/configuration/); for a first local installation, the generated defaults are safer than copying a network-exposed example. If `security.bootstrap_token` is empty during normal startup, Amaquet reuses `data_dir/bootstrap-token` or creates it with mode `0600`. Startup logs the protected path rather than printing the new secret. This fallback is useful for unattended first boot, but a managed deployment should inject or provision a deliberate secret and rotate it into a permanent API key. ### Start a local server Start the server in the foreground so startup errors and the listener addresses remain visible: ```bash ./bin/amaquet --config ./amaquet.json ``` With the generated defaults, the endpoints are: ```text Native protocol: amaquet://127.0.0.1:13378 Admin API: http://127.0.0.1:13379 Data directory: ./data ``` The process creates `data_dir`, loads the administration state, optionally replays the AOF, and then starts both listeners. Keep the process running while verifying it. Press `Ctrl-C` for a graceful shutdown; the server closes listeners and flushes the AOF according to its configured durability mode. ### Verify the installation The admin health endpoint is a convenient first check because it confirms that the HTTP listener is responding: ```bash curl --fail --silent --show-error http://127.0.0.1:13379/api/health ``` Use the bootstrap token printed during initialization for the first native protocol request. Store it in a shell variable without placing it directly into a committed script: ```bash export AMAQUET_BOOTSTRAP_TOKEN='paste-the-token-from-initialization' ./bin/amaquet-cli \ -uri amaquet://127.0.0.1:13378 \ -api-key "$AMAQUET_BOOTSTRAP_TOKEN" \ PING '{}' ``` A successful response confirms network reachability, protocol negotiation, and authentication. Use the same credential to inspect the server and type registry: ```bash ./bin/amaquet-cli -uri amaquet://127.0.0.1:13378 -api-key "$AMAQUET_BOOTSTRAP_TOKEN" INFO '{}' ./bin/amaquet-cli -uri amaquet://127.0.0.1:13378 -api-key "$AMAQUET_BOOTSTRAP_TOKEN" TYPES '{}' ``` For a complete data-plane check, store and read one typed value: ```bash ./bin/amaquet-cli -uri amaquet://127.0.0.1:13378 -api-key "$AMAQUET_BOOTSTRAP_TOKEN" \ SET '{"key":"install:check","type":"utf8_string","value":"Amaquet is running"}' ./bin/amaquet-cli -uri amaquet://127.0.0.1:13378 -api-key "$AMAQUET_BOOTSTRAP_TOKEN" \ GET '{"key":"install:check"}' ``` ### Create a permanent API key The bootstrap credential is intended for initial control-plane setup, not for application traffic. Use the authenticated admin API to create a permanent API key with the narrowest role and permissions required by each application. Amaquet stores hashes of generated API-key secrets and shows each raw secret only once. After the first admin API key is created, bootstrap authentication is permanently disabled in the persisted administration state. Follow [Admin API](/reference/admin-api/) for the exact API-key request and role model. Keep the generated secret outside `amaquet.json`, provide it to clients through a secret manager, and use `-api-key` or the Go client's API-key option rather than embedding credentials in a URI. ### Configure network access and TLS The generated configuration is intentionally loopback-only. Before exposing either listener, read [Configuration](/getting-started/configuration/) and [Production deployment](/operations/deployment/): - Use a specific private bind address instead of `0.0.0.0` when the topology allows it. - Enable `protocol.tls` for native traffic and use `amaquets://` client URIs. - Enable `admin.tls` for the administration API and protect its private key. - Keep `protocol.require_auth` and `admin.metrics_require_auth` enabled. - Leave `protocol.allow_insecure_auth` and `admin.allow_insecure` disabled unless a deliberately isolated network requires them. - Restrict firewall access to application and management networks, and monitor `/api/ready` and `/metrics` according to the authentication policy. Generate the optional Ed25519 protocol identity separately from TLS certificates: ```bash mkdir -p keys ./bin/amaquet-keygen \ --public ./keys/amaquet-public.pem \ --private ./keys/amaquet-private.pem ``` Set both generated paths under `security.public_key_file` and `security.private_key_file` when clients need application-level identity verification. The key pair does not replace an X.509 certificate, does not encrypt traffic, and should be readable only by the service account where possible. ### Environment-based deployment JSON is convenient for a checked-in template, while environment overrides are useful for containers and service managers. Supported variables are documented in [Configuration](/getting-started/configuration/). Overrides are applied at startup after JSON is loaded; invalid numeric values are ignored, while the final merged configuration is still validated. For example, this starts with a JSON file but moves the secret and listener choices into the process environment: ```bash export AMAQUET_BOOTSTRAP_TOKEN="$BOOTSTRAP_SECRET" export AMAQUET_HOST=127.0.0.1 export AMAQUET_PORT=13378 export AMAQUET_ADMIN_HOST=127.0.0.1 export AMAQUET_ADMIN_PORT=13379 ./bin/amaquet --config /etc/amaquet/amaquet.json ``` Do not assume that an environment override is persisted. A later restart needs the same environment, a JSON value, or the generated `data_dir/bootstrap-token` fallback. ### Docker installation The repository includes development and production-oriented Compose files. Build and start the loopback-only development deployment with: ```bash docker compose build docker compose up -d docker compose logs -f amaquet ``` The Compose configuration persists `/app/data` in the `amaquet-data` volume. Do not remove that volume if AOF data or administrator state must survive container replacement. The production Compose file is a template: provide the expected TLS and identity files, review host bindings and firewall rules, and inject the bootstrap secret through the deployment platform. See [Docker deployment](/operations/docker/) for the port, volume, user, and TLS details. ### Documentation tools Node.js and npm are needed only for the documentation site. From the repository root: ```bash make docs-install make docs-build make docs-serve ``` `make docs-build` validates the source-derived documentation inventories, checks heading descriptions, builds the static site, and checks generated links. `make docs-serve` starts the local Astro development server for editing the documentation; it does not start Amaquet. ### Upgrades and backups Before replacing a server binary or changing persistence settings, stop Amaquet cleanly and back up both the AOF and `data_dir`, including `admin.json`, bootstrap-token material, upload state, and identity files. Keep the configuration path and AOF path stable unless the move is deliberate. For an AOF checkpoint, use the administration persistence endpoint and verify the resulting file before storing it separately. To install a verified checkpoint, stop the target process and use `amaquet-restore`; never copy over a live AOF while the server is writing it. Read [Persistence and recovery](/concepts/persistence/) and [Backup and recovery](/operations/backups/) before treating AOF as a backup strategy. ### Troubleshooting first startup Use this table to narrow down failures during the first launch. Once the process is running, the server log and the health endpoint usually provide the next useful detail. | Symptom | Likely cause and action | | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Failed to load configuration` | The JSON is malformed or the `-config` path is wrong. Validate the file and use an absolute path while diagnosing. | | `invalid port` | A listener port is outside `1`–`65535` or collides with another process. | | `TLS requires ...` | TLS is enabled without both certificate and key paths, or the process cannot read one of the files. | | `authenticated Amaquet on a non-loopback address requires TLS` | Enable native TLS or deliberately set `protocol.allow_insecure_auth`; TLS is the recommended fix. | | `admin API on a non-loopback address requires TLS` | Enable admin HTTPS or keep the admin host on loopback/private management access. | | CLI connection refused | Confirm the server is running, the URI port matches `protocol.port`, and the listener is bound to an address reachable from the client. | | authentication failure | Use the token printed by `--init-config`, the current `data_dir/bootstrap-token`, or a permanent API key; do not assume a JSON token is still active after bootstrap is disabled. | | AOF replay or permission failure | Stop the process, verify ownership and permissions of the AOF/data directory, preserve the original files, and follow the recovery guide before retrying. | ### Next steps Continue with [Quick start](/getting-started/quickstart/) for typed values and operations, [Configuration](/getting-started/configuration/) for every server setting, [Authentication and identity](/protocol/authentication/) for protocol credentials, and [Production deployment](/operations/deployment/) for supervision and durable service layouts. --- ## Quick start Source: [canonical documentation page](/getting-started/quickstart/) Summary: Run Amaquet locally, create a first key, connect with amaquet-cli, and move from an unauthenticated example to a secure deployment. This example uses authentication disabled only to keep the first local session short. Production deployments should enable authentication and TLS. ### 1. Local configuration Create `amaquet.json`: ```json { "protocol": { "host": "127.0.0.1", "port": 13378, "tls": false, "require_auth": false }, "admin": { "host": "127.0.0.1", "port": 13379 }, "security": { "bootstrap_token": "replace-this" }, "persistence": { "aof_enabled": false, "aof_path": "./data/amaquet.aof", "fsync": "everysec" }, "limits": { "max_connections": 10000, "max_payload_bytes": 67108864 }, "data_dir": "./data" } ``` ### 2. Start Amaquet Start the server with the local configuration created in the previous step. The process remains in the foreground and owns both configured listeners. ```bash ./bin/amaquet -config ./amaquet.json ``` ### 3. Ping From another terminal, use the CLI to verify that the native protocol listener accepts commands. ```bash ./bin/amaquet-cli -uri amaquet://127.0.0.1:13378 PING '{}' ``` ### 4. Store a typed scalar Store a directly encoded value with `SET`, then retrieve it with `GET` to confirm its type and payload. ```bash ./bin/amaquet-cli -uri amaquet://127.0.0.1:13378 SET \ '{"key":"user:1:name","type":"utf8_string","value":"Nathanne"}' ``` Read it: ```bash ./bin/amaquet-cli -uri amaquet://127.0.0.1:13378 GET \ '{"key":"user:1:name"}' ``` ### 5. Create a collection Composite values are initialized with `CREATE`; use `OP` to apply a type-specific mutation afterward. ```bash ./bin/amaquet-cli -uri amaquet://127.0.0.1:13378 CREATE \ '{"key":"queue:jobs","type":"fifo_queue","options":{}}' ``` Add a typed value: ```bash ./bin/amaquet-cli -uri amaquet://127.0.0.1:13378 OP \ '{"key":"queue:jobs","operation":"ENQUEUE","args":{"value":{"type":"utf8_string","value":"build"}}}' ``` ### 6. Add TTL Expiration is expressed in positive milliseconds. This command gives the scalar key a 60-second lifetime without changing its value. ```bash ./bin/amaquet-cli -uri amaquet://127.0.0.1:13378 EXPIRE \ '{"key":"user:1:name","ttl_ms":60000}' ``` ### 7. Inspect server metadata Use these read-only commands to inspect runtime information and the registered type catalog. ```bash ./bin/amaquet-cli -uri amaquet://127.0.0.1:13378 INFO '{}' ./bin/amaquet-cli -uri amaquet://127.0.0.1:13378 TYPES '{}' ``` Next, read [Amaquet commands](/protocol/commands/) and [data types](/data-types/). --- ## Backup and recovery Source: [canonical documentation page](/operations/backups/) Summary: This page explains which Amaquet state must be protected and how to create, validate, and restore recovery artifacts. This page explains which Amaquet state must be protected and how to create, validate, and restore recovery artifacts. ### What to protect Protect these files according to your security and retention policy: - the active AOF; - verified checkpoint files; - `admin.json`; - the JSON configuration; - TLS private keys and certificates; - optional Ed25519 identity keys; - the protected bootstrap-token file while bootstrap is still active. Member access hashes and TOTP secrets are part of `admin.json`. Treat an administration-state backup as sensitive security material. ### Create an online checkpoint Use `POST /api/persistence/checkpoint`. The AOF serializes the checkpoint with journal writers, flushes the file, and captures a committed prefix while later appends wait. Unlike online compaction, the checkpoint callback is not wrapped in the dispatcher's broader engine mutation barrier. The checkpoint process: 1. flushes the active journal; 2. resolves committed AOF transactions; 3. compacts semantics-preserving obsolete history; 4. writes a fresh AMQTAOF2 recovery image; 5. fsyncs the output; 6. replays the temporary result to validate framing, CRCs, transaction structure, and request decoding; 7. atomically publishes the checkpoint. The checkpoint is therefore a compact recovery image, not a blind copy of a file that is changing underneath the backup process. ### Compact the active AOF `POST /api/persistence/compact` rewrites the active journal online. The compactor removes aborted and uncommitted records and collapses overwritten state where this is safe. For chunked blobs, only the latest completed upload sequence for the key is retained. Incomplete upload sequences are not recovery state. ### Restore offline Stop Amaquet before replacing its active AOF. ```bash amaquet-restore \ -source /backups/checkpoint-20261002T120000Z.aof \ -target /var/lib/amaquet/amaquet.aof ``` The restore tool: 1. validates the source by replay parsing; 2. copies it to a temporary target; 3. fsyncs the temporary file; 4. validates the copied result; 5. moves the old target to a temporary rollback file when present; 6. atomically installs the restored AOF; 7. fsyncs the destination directory; 8. removes the rollback file after success. Do not run `amaquet-restore` against an AOF that an active Amaquet process has open. ### Recovery drill A production backup is not complete until it has been restored in a separate environment. Periodically test: - checkpoint creation; - `amaquet-restore` into a clean data directory; - server startup and AOF replay; - key counts and application-level invariants; - administration state and RBAC recovery; - TLS/identity key availability. Preserve a corrupt source file before manual repair. A checksum mismatch in a complete record is treated as corruption and is not silently skipped. --- ## Capacity planning Source: [canonical documentation page](/operations/capacity/) Summary: Amaquet keeps the active database in memory. Plan capacity from the stored data, indexes, Go object overhead, protocol buffers, temporary work space,… Amaquet keeps the active database in memory. Plan capacity from the stored data, indexes, Go object overhead, protocol buffers, temporary work space, persistence buffers, connections, and operating-system headroom. ### Memory budget and process guard Set `limits.max_memory_bytes` for the engine data budget. Choose `noeviction` when losing an arbitrary key is unacceptable, or a supported eviction policy when bounded cache behavior is desired. The engine tracks an approximate stored-data budget. It also exposes Go heap statistics and sets a Go runtime soft memory limit with additional headroom when a Amaquet memory limit is configured. This is not an exact RSS limit. TLS, the runtime, page cache, file mappings, stacks, temporary buffers, and the operating system can make process RSS differ from the configured data budget. Do not set the Amaquet data budget equal to machine RAM. Keep explicit headroom and enforce an operating-system/container memory limit as the final safety boundary. ### Expiration TTL expiration uses lazy expiry plus an indexed min-heap. There is at most one scheduled heap node for the current expiration state of a key, so repeated `EXPIRE`/`PERSIST` cycles do not accumulate stale expiration records indefinitely. ### Eviction LRU/LFU/TTL eviction uses bounded candidate sampling instead of scanning the full keyspace for every victim. Sampling keeps eviction work bounded, but it is approximate by design. ### Key iteration Use cursor `SCAN` for large databases. `KEYS` is convenient for bounded administrative use, but it still collects every matching key. Both commands sort within each shard and append shards in shard-index order; neither promises one global lexical ordering. `SCAN` returns an opaque continuation cursor without materializing the complete database in one result. ### Large binary values A physical Amaquet frame remains limited to 64 MiB. Chunked upload lets a final binary value reach the hard 1 GiB per-object limit while each request remains bounded; `limits.max_pending_upload_bytes` separately controls the aggregate reservation for active uploads. Upload behavior is deliberately different from the final in-memory representation: - incoming chunks are spooled to bounded temporary files instead of preallocating the declared object size in the Go heap; - zero-length chunks are rejected; - overlapping chunks are rejected; - the number of spans/chunks is capped; - `connection_scoped: true` binds an upload to the exact native-protocol connection; the default upload is not connection-scoped; - abandoned uploads expire, and connection-scoped uploads are also removed when their owner connection closes; - pending upload bytes have a global reservation limit; - incomplete uploads found during recovery are discarded; - final large values use bounded `ChunkedBinary` blocks rather than one contiguous object-sized byte slice; - clients read large values with `BLOB_READ` ranges instead of forcing one large response allocation. For sustained large-object workloads, also budget the data directory for upload spool files and persistence checkpoints. ### Network and command concurrency Amaquet has per-connection and global in-flight request limits plus a global inbound-payload budget. These limits protect the process from a client declaring many maximum-size frames at once. Configure them according to available RAM and expected concurrency. `limits.command_timeout_ms` bounds ordinary command execution. Blocking primitives remain bounded by the server command context and their own requested timeout. ### Compression Adaptive compression can reduce stored payload bytes for repetitive text, JSON, zero-filled numeric data, and similar values. Compression does not reduce every structure: active indexes, synchronization structures, graph links, key strings, and runtime metadata can remain native. Measure both logical and stored bytes. Very small or incompressible values normally remain uncompressed. ### Indexes and algorithms Production-oriented implementations include multi-level HNSW with bounded neighbor graphs, grid-backed geospatial candidate lookup, B-tree traversal for ranges, hybrid Roaring containers, bounded-memory Top-K candidates, and one-pass t-digest compression. Benchmark these structures with your real dimensions, cardinalities, and query distributions before setting production limits. --- ## Production deployment Source: [canonical documentation page](/operations/deployment/) Summary: Use this guidance to deploy Amaquet as a secure, monitored single-node stateful service. Use this guidance to deploy Amaquet as a secure, monitored single-node stateful service. ### Recommended topology Run Amaquet as a single-node stateful service and keep both listeners as narrow as the application permits. These controls address the implementation's in-memory data model and its separate native-protocol and administration surfaces. - Bind the Amaquet listener only on interfaces required by applications. - Enable TLS and API-key authentication. - Keep the admin listener private or behind an authenticated reverse proxy. - Put `data_dir`, AOF, identity private key, and TLS private key on protected storage. - Set OS file-descriptor limits above expected connection counts. - Monitor process RSS because primary data resides in memory. ### Restart behavior The Go server handles `SIGINT` and `SIGTERM`, gracefully shuts down the admin HTTP server, closes the Amaquet listener, waits for active connection goroutines, and closes the engine. AOF close flushes buffered records. ### Stateless versus durable mode With AOF disabled, a restart starts with an empty data keyspace while administrative state may remain in `admin.json`. With AOF enabled, replay reconstructs journaled data operations before listeners begin serving. ### Process supervision Use systemd, Docker, Kubernetes, or another supervisor. Configure restart policies carefully: repeated restart loops on AOF corruption should alert an operator instead of masking the underlying file problem. ### Migrating an existing installation to Amaquet Treat the product rename as an application upgrade and back up the administration state and AOF before replacing binaries. 1. Stop the existing server so its journal and administration state are quiescent. 2. Deploy the `amaquet`, `amaquet-cli`, `amaquet-keygen`, and `amaquet-restore` binaries and rename the JSON configuration file to `amaquet.json`. 3. Update configuration paths, service units, scripts, and containers to use the `AMAQUET_*` environment variables, `amaquet://` or `amaquets://` endpoint scheme, and current binary names. 4. Update HTTP automation to use `X-Amaquet-Admin-Token`, monitoring queries to use `amaquet_*` metrics, and Go applications to import `github.com/newfoundcodes/amaquet/pkg/amaquet`. 5. Keep the existing `data_dir` and AOF path pointed at the real persisted files for the first start. Amaquet recognizes pre-rename v1 and v2 journal signatures and atomically rewrites them to `AMQTAOF2`. Administration schema migration changes only the untouched historical default organization name; customized organization data and credential hashes are preserved. 6. Start Amaquet, verify `/api/health`, `/api/ready`, authenticated `/metrics`, and an application read before removing the backup. The protocol magic is now `AMQT`; endpoint schemes and the four-byte wire magic are not negotiated aliases. Clients and servers therefore need to be upgraded together. Existing API-key and member secrets remain valid because authentication is based on their persisted hashes, not their display prefixes. Compose files pin the project name to `amaquet`. If upgrading a Compose deployment, explicitly attach or copy data from the previous named volume into `amaquet-data`; changing the Compose project or volume name does not move data. The container runtime user is also named `amaquet`, so ensure mounted files are readable and writable by its configured UID/GID before startup. --- ## Docker Source: [canonical documentation page](/operations/docker/) Summary: The repository contains Dockerfile, a loopback-only development docker-compose.yml, and a TLS-oriented docker-compose.production.yml. The repository contains `Dockerfile`, a loopback-only development `docker-compose.yml`, and a TLS-oriented `docker-compose.production.yml`. Both Compose files build the same image and persist `/app/data` in the `amaquet-data` volume. Typical workflow: ```bash docker compose build docker compose up -d ``` Persist `data_dir` with a volume if AOF or admin state must survive container replacement. Mount production TLS and identity keys read-only where possible. Do not store production bootstrap tokens directly in a public Compose file. Inject secrets through the deployment platform. After start, check: ```bash ./bin/amaquet-cli -uri amaquet://127.0.0.1:13378 \ -api-key "$AMAQUET_BOOTSTRAP_TOKEN" PING '{}' curl http://127.0.0.1:13379/api/health ``` The Compose configuration requires `AMAQUET_BOOTSTRAP_TOKEN` and enables protocol authentication, so the protocol check supplies that credential explicitly. The image uses a Go build stage to produce four statically linked binaries, and the final Alpine image runs as the unprivileged `amaquet` user. Compose binds host ports only to `127.0.0.1`, persists `/app/data` in `amaquet-data`, enables AOF `everysec`, and uses explicit insecure-listener overrides only because Docker must bind `0.0.0.0` inside the isolated container network. The production Compose file is a template rather than a zero-configuration launch. It publishes both ports on the host's configured interfaces, enables TLS for both listeners, and expects the four certificate/key files under `./certs`. Its configuration also names Ed25519 identity files under `/app/data/keys`, so generate those files in the persistent volume or change the paths before first startup. Review host firewall rules and certificate names before exposing either published port. --- ## Observability Source: [canonical documentation page](/operations/observability/) Summary: This page describes the native and HTTP surfaces that expose Amaquet health, capacity, and runtime metrics. This page describes the native and HTTP surfaces that expose Amaquet health, capacity, and runtime metrics. ### Amaquet `INFO` `INFO` returns server identity, protocol version, live key count, uptime, type count, compression totals, approximate engine memory usage, configured memory limit, eviction policy, and eviction count. `MEMORY {"key":"..."}` reports per-key version/access metadata, approximate bytes, and physical compression information. ### Health endpoints The administration server exposes: - `/api/health` for liveness; - `/api/ready` for readiness plus key/memory statistics and persistence health; - `/metrics` for Prometheus/OpenMetrics-style process and database metrics. Readiness becomes unhealthy when AOF persistence reports a background flush/fsync failure. Memory statistics are informational: crossing the configured engine-memory budget does not by itself make this endpoint return HTTP 503. ### Metrics The exact metrics are: - `amaquet_keys`; - `amaquet_memory_bytes` and `amaquet_memory_limit_bytes`; - `amaquet_evicted_keys_total`; - `amaquet_compressed_keys` and `amaquet_compression_saved_bytes`; - `amaquet_go_heap_bytes` and `amaquet_go_goroutines`; - `amaquet_ready` (`1` or `0`). The endpoint uses Prometheus text format version 0.0.4. It requires `system.read` by default and can be made public with `admin.metrics_require_auth:false`. The engine-memory value is an application estimate. Monitor process RSS/working-set separately because Go runtime pages, temporary buffers, TLS, connections, filesystem buffers, and native/runtime overhead are outside that counter. ### Logs The process uses `github.com/charmbracelet/log` with structured key/value fields for listener startup, persistence replay/open status, identity information, configuration failures, and shutdown events. Production deployments should collect these logs with timestamps and retain persistence or authentication errors as security/availability signals. --- ## Production hardening Source: [canonical documentation page](/operations/production-hardening/) Summary: Amaquet applies bounded-resource, fail-stop, and revocation rules for a single-node production deployment. Amaquet applies bounded-resource, fail-stop, and revocation rules for a single-node production deployment. ### Memory governance `limits.max_memory_bytes` is the storage-engine data budget. New and replacement values use their **net** stored-size change. A same-size replacement does not require a second full value in the engine accounting model. The Go runtime also receives a soft process memory limit with headroom when `max_memory_bytes` is non-zero. `INFO` exposes aggregate engine memory, `MEMORY` exposes approximate per-key storage, and `/metrics` adds Go heap allocation. The process limit is a runtime guard, not a byte-exact RSS guarantee because TLS, kernel socket buffers, mapped libraries, and non-Go allocations are outside the engine counter. Supported eviction policies are `noeviction`, `allkeys-lru`, `allkeys-lfu`, and `volatile-ttl`. Eviction uses bounded sampling. Memory-reducing operations remain available when the database is at its limit. Composite mutations reserve capacity before state becomes visible. The engine checks the exact stored-size delta when the mutation metadata is committed. ### Expiration scheduling Each expiring key has one indexed expiration-heap node. `EXPIRE` updates that node. `PERSIST` removes it. Repeated TTL changes therefore do not create unbounded stale heap entries. ### Network resource limits The protocol server enforces: - maximum connections; - maximum in-flight requests for the server; - maximum in-flight requests for one connection; - maximum frame payload size; - maximum aggregate in-flight payload bytes; - duplicate active request-ID rejection; - configurable command execution timeout; - read and write deadlines; - authentication failure throttling. `limits.command_timeout_ms` is the default server-side command deadline. A cancellation frame can cancel a compatible active request before that deadline. ### Large binary values A single Amaquet frame remains bounded. Large `binary_string` values use the chunked upload commands. The upload path has these safeguards: - upload data is spooled to a private temporary file before commit; - zero-byte chunks are rejected; - overlapping chunks are rejected; - one upload can contain at most 65,536 accepted spans; - span lookup uses ordered binary search instead of sorting the complete span list after every chunk; - total pending upload bytes are bounded; - uploads expire after `limits.upload_ttl_ms` of inactivity; - connection-scoped uploads require an exact owner match; - abandoned connection-scoped uploads are removed when the connection closes; - incomplete replayed uploads are removed before traffic is accepted. At `BLOB_COMMIT`, Amaquet reads the temporary file into bounded 1 MiB in-memory blocks. It does **not** allocate one contiguous buffer equal to the complete object size. `BLOB_READ` returns a bounded range and is the preferred read path for large values. A normal `GET` returns metadata instead of materializing a large chunked value into one protocol response. The current maximum declared blob size is 1 GiB. `BLOB_READ.length` is limited to 32 MiB per request. ### Credential revocation API-key revocation is applied to new and existing access paths. - New authentication with the key fails. - Existing server-side sessions are revalidated against the actor state. - Registered revocation hooks remove matching HTTP cookie sessions. - Matching active Amaquet connections are closed by the server process. - Every protocol command also revalidates its authenticated actor. - Active Pub/Sub delivery revalidates the actor before each event. Member disable/delete, member credential rotation, and MFA activation use the same actor-revocation mechanism. ### Pub/Sub write deadlines Normal responses and asynchronous Pub/Sub event frames refresh the socket write deadline before writing. An idle subscription therefore does not inherit an expired deadline from its original subscribe response. ### Persistence fail-stop behavior AOF v2 uses `prepare`, `commit`, and `abort` records with CRC validation. Mutations are serialized through a mutation barrier when AOF is active. If the journal commit fails after an in-memory mutation has already been applied, persistence health changes to failed. New durable writes are then rejected. `/api/ready` reports the node as not ready. This fail-stop rule prevents the server from continuing to accept writes after known durable-state divergence. ### Compact checkpoints and restore An online checkpoint is not a blind file copy. Amaquet resolves committed transactions, removes obsolete histories where the operation semantics permit it, writes a new AMQTAOF2 recovery image, fsyncs it, and replays it for verification before publishing the destination. Blob compaction retains only the latest committed upload sequence for a key. Incomplete and aborted uploads are omitted. Use `amaquet-restore` while the server is stopped to install a checkpoint. The tool validates the source, writes a temporary destination, fsyncs it, verifies it again, and atomically swaps it into place with rollback protection. ### Control-plane bounds The admin server provides: - bounded HTTP cookie sessions; - expired-session cleanup; - session invalidation on actor revocation; - authentication failure throttling; - bounded failure-tracking maps; - optional authentication on `/metrics`; - crash-safe admin-state writes; - batched persistence of API-key `last_used_at` metadata; - explicit admin-state schema versions and migrations. HTTP sessions use `HttpOnly`, `SameSite=Strict` cookies. The cookie is also `Secure` when HTTPS is active. ### Human member identities Team members are not only metadata. An administrator can create a one-time invitation for a member. The member accepts it once and receives a member access token. Member tokens inherit the member's current RBAC role and active/disabled state. Members can enable TOTP MFA. After MFA is active, a member access token alone cannot create a new interactive session; the six-digit TOTP code is also required. Enabling MFA revokes existing actor sessions/connections so the next login uses the new factor. API keys remain the preferred machine-to-machine credential. ### Bootstrap credential If no bootstrap token is supplied, startup creates one under `data_dir/bootstrap-token` with mode `0600` and logs only the file path. The token value is not written to the application log. The file is reused across restarts until bootstrap is disabled. When bootstrap creates the first administrator API key, bootstrap authentication is permanently disabled in admin state. API configuration writes never copy an environment-provided bootstrap token into the JSON configuration file. ### Docker defaults The supplied development Compose file publishes the Amaquet protocol and administration API ports on `127.0.0.1`. Remote production deployments should use `amaquets://` and admin HTTPS. ### Chaos and soak testing Use: ```bash make chaos-smoke make chaos ``` The suite covers kill/restart recovery, abandoned uploads, memory eviction, connection churn, TTL churn, and mixed sustained workloads. Scheduled CI also runs longer chaos, fuzz, and benchmark jobs. --- ## TLS and protocol identity Source: [canonical documentation page](/operations/tls/) Summary: Configure TLS for transport security and optional Ed25519 keys for stable application-level server identity. Configure TLS for transport security and optional Ed25519 keys for stable application-level server identity. ### Amaquet TLS Configure: ```json { "protocol": { "tls": true, "tls_cert_file": "/run/secrets/amaquet.crt", "tls_key_file": "/run/secrets/amaquet.key" } } ``` Connect with `amaquets://`. The Go client uses normal TLS server-name verification based on the URI host. ### Protocol identity Generate a separate Ed25519 identity with `amaquet-keygen` and configure `security.public_key_file` and `security.private_key_file`. TLS protects the transport and authenticates through certificates. The Amaquet identity signs application nonces and gives clients a stable application-level key fingerprint. Use both when you require both guarantees. --- ## Troubleshooting Source: [canonical documentation page](/operations/troubleshooting/) Summary: Use these symptoms and corrective actions to diagnose common startup, authentication, persistence, TLS, and test failures. Use these symptoms and corrective actions to diagnose common startup, authentication, persistence, TLS, and test failures. ### `authentication required` The server has `protocol.require_auth=true` and the connection did not authenticate. Pass `amaquet-cli -api-key "$AMAQUET_API_KEY"`, use `DialWithOptions` with `DialOptions.APIKey`, or send the `AUTH` opcode. URI credentials are rejected by the bundled clients unless the legacy opt-in is enabled. ### `permission denied` The API key authenticated but its role lacks the command's required permission. Inspect `/api/rbac` with an authorized administrator. ### `wrong value type` The operation targeted a key whose stable type does not support that operation. Run `TYPE` and use the relevant type reference. ### Server does not start with TLS Verify both certificate/key paths and file permissions. TLS requires both configured files. ### AOF replay fails Check the error for invalid header, record size, JSON decode, replay error, or CRC mismatch. Make a copy of the AOF before manual diagnosis. ### Bootstrap-token file is missing `data_dir` is also resolved from the process working directory. A token file exists only when `security.bootstrap_token` is empty at startup; `--init-config` instead embeds a generated token in the JSON file. From `bin/`, a relative `./data/bootstrap-token` is therefore `bin/data/bootstrap-token`. Check the effective configuration and working directory before assuming the token was not generated. ### Bash test fails Run one script directly with `AMAQUET_KEEP_TMP=1` to preserve the generated test directory and `server.log`: ```bash AMAQUET_KEEP_TMP=1 tests/bash/types/test_vector_set.sh ``` --- ## Amaquet protocol specification v1 Source: [canonical documentation page](/protocol/) Summary: Amaquet protocol v1 reference for connection URIs, binary frames, multiplexing, negotiation, authentication, commands, Pub/Sub events, and RBAC. This specification defines Amaquet v1 connection URIs, binary frames, authentication, commands, and event delivery. ### URI schemes Use these URI forms to select plaintext TCP or TLS when connecting a client to the native protocol listener. - `amaquet://host[:port]` — Amaquet over TCP. - `amaquets://host[:port]` — Amaquet over TLS. - Default port: `13378`. - API keys should be passed out of band through `DialOptions.APIKey` or the CLI's `-api-key` flag. URI credentials exist only as explicit opt-in legacy compatibility. Examples: ```text amaquet://127.0.0.1:13378 amaquet://example.com amaquets://db.example.com:13378 ``` ### Frame format Every frame starts with a fixed 24-byte header. Multi-byte integers are big-endian. | Offset | Size | Field | | -----: | ---: | ---------------------- | | 0 | 4 | ASCII magic `AMQT` | | 4 | 1 | protocol version (`1`) | | 5 | 1 | frame kind | | 6 | 2 | flags | | 8 | 2 | opcode | | 10 | 2 | reserved | | 12 | 8 | request ID | | 20 | 4 | payload length | The payload follows immediately. The current hard frame payload limit is 64 MiB. Frame kinds: - `1`: request - `2`: response - `3`: asynchronous event Opcodes: - `1`: command - `2`: hello - `3`: authentication - `4`: subscribe - `5`: unsubscribe - `6`: cancel an active command request The protocol is not RESP and has no Redis dependency. ### Multiplexing The 64-bit request ID allows multiple in-flight requests on one connection. Responses carry the same request ID. Pub/Sub events use the subscription request ID and frame kind `event`. ### HELLO Request payload (the negotiation fields default to protocol 1 when omitted): ```json { "nonce": "client-generated-nonce", "min_protocol": 1, "max_protocol": 1 } ``` Response fields include server/version/protocol/auth requirements. If an Ed25519 identity is configured, it also returns: ```json { "identity_public_key": "base64-raw-public-key", "identity_fingerprint": "ed25519:...", "nonce_signature": "base64-raw-signature" } ``` The Go client exposes `ServerInfo.VerifyNonce`. ### AUTH Authentication request: ```json { "api_key": "amaquet_..." } ``` Response: ```json { "authenticated": true, "role": "developer" } ``` If authentication is required, normal commands and subscriptions are rejected until AUTH succeeds. ### COMMAND Command payload: ```json { "command": "SET", "args": { "key": "answer", "type": "integer", "value": 42 } } ``` Success response envelope: ```json { "ok": true, "result": true } ``` Error response envelope: ```json { "ok": false, "error": { "code": "WRONG_TYPE", "message": "wrong value type" } } ``` Known error classes include `NOT_FOUND`, `WRONG_TYPE`, `EXISTS`, `CONFLICT`, `MEMORY_LIMIT`, `BUSY`, `DUPLICATE_REQUEST`, `FORBIDDEN`, `UNAUTHORIZED`, and `ERROR`. ### Direct values vs composite values Scalar/directly serialized values use `SET`. Examples include strings, integers, decimals, timestamps, UUIDs, JSON, geospatial value objects, standalone vectors, CIDRs, and URLs. Composite data structures use `CREATE` followed by `OP`. ```json { "command": "CREATE", "args": { "key": "users", "type": "set", "options": {} } } ``` Then: ```json { "command": "OP", "args": { "key": "users", "operation": "ADD", "args": { "value": { "type": "utf8_string", "value": "alice" } } } } ``` ### Common keyspace commands `PING`, `SET`, `CAS`, `BATCH`, `CREATE`, `GET`, `DEL`, `EXISTS`, `TYPE`, `TTL`, `EXPIRE`, `PERSIST`, `KEYS`, `SCAN`, `BLOB_BEGIN`, `BLOB_CHUNK`, `BLOB_COMMIT`, `BLOB_READ`, `BLOB_ABORT`, `TYPES`, `INFO`, `MEMORY`, and `OP` are implemented by the v1 dispatcher. `GET` returns a safe snapshot instead of exposing internal mutable pointers. ### SUBSCRIBE / UNSUBSCRIBE Subscription request: ```json { "key": "bus", "channel": "updates", "buffer": 64 } ``` Events are Amaquet event frames: ```json { "channel": "updates", "payload": { "type": "utf8_string", "value": "changed" }, "published_at": "2026-10-01T00:00:00Z" } ``` ### RBAC The protocol server maps keyspace reads to `data.read` and mutations to `data.write`. `OP` operations use their read/mutation classification. The current role permission sets are controlled by the admin state and can be changed through the administration API. --- ## Amaquet authentication and identity Source: [canonical documentation page](/protocol/authentication/) Summary: This page describes protocol negotiation, API-key authentication, permissions, and optional Ed25519 identity verification. This page describes protocol negotiation, API-key authentication, permissions, and optional Ed25519 identity verification. ### `HELLO` `HELLO` does not require authentication. Send: ```json { "min_protocol": 1, "max_protocol": 1, "nonce": "client-generated-random-value" } ``` Omitted protocol bounds default to version 1. The response reports `name`, server `version`, frame `protocol`, `selected_protocol`, capability names, `auth_required`, and optional Ed25519 identity fields. There is currently only protocol version 1; non-overlapping bounds return an error. ### `AUTH` Payload: ```json { "api_key": "amaquet_..." } ``` Despite the field name, the credential may be a bootstrap token, API key, or an active member token that does not have MFA enabled. Member tokens with MFA enabled must use the HTTP session endpoint because native Amaquet `AUTH` has no MFA-code field. On success, the connection stores the current role and actor ID. Later commands revalidate the actor and authorize it against the current RBAC table, so revocation, expiration, disabling, and role changes take effect without reconnecting. When `protocol.require_auth` is false, a new connection is an anonymous `admin` actor and can issue commands without `AUTH`. This mode should be confined to trusted environments. ### Read/write classification `PING`, `GET`, `EXISTS`, `TYPE`, `TTL`, `KEYS`, `SCAN`, `INFO`, `TYPES`, `MEMORY`, `BLOB_READ`, and the fixed read-only `OP` set require `data.read`. All other commands/operations require `data.write`. `SUBSCRIBE` requires `data.read`. `UNSUBSCRIBE` closes an already established local subscription without a separate authorization check. Authentication failures are throttled per remote IP using `admin.auth_failures_per_minute`, which is shared with the administration server setting. ### Client identity verification With identity keys configured, verify the returned nonce signature before trusting the application-level server identity. For network confidentiality and certificate-based server verification, also use TLS through `amaquets://`. --- ## Command reference Source: [canonical documentation page](/protocol/commands/) Summary: These commands operate on the keyspace, server metadata, or execution envelope rather than on one type-specific implementation. Commands use opcode `COMMAND` and a JSON request: ```json { "command": "GET", "args": { "key": "example" } } ``` ### Common commands These commands operate on the keyspace, server metadata, or execution envelope rather than on one type-specific implementation. Command names are case-insensitive; argument property names remain case-sensitive. | Command | Important arguments | Result | | ------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------ | | `PING` | none | `PONG` | | `SET` | `key`, `type`, `value`; optional `ttl_ms`, `nx:false`, `xx:false` | `true` | | `CREATE` | `key`, `type`; optional `options:{}`, `ttl_ms`, `nx:false`, `xx:false` | `true` | | `CAS` | `key`, positive `expected_version`, `type`, `value`; optional `ttl_ms` | `true` or `CONFLICT` | | `BATCH` | `requests`; optional `continue_on_error:false` | ordered `{ok,result}` / `{ok,error}` items | | `GET` | `key` | `{key,type,value,version}` plus `expires_at` when present | | `DEL` | `keys` | deleted count | | `EXISTS` | `key` | boolean | | `TYPE` | `key` | stable wire type | | `TTL` | `key` | remaining whole milliseconds or `-1` for no expiration | | `EXPIRE` | `key`, positive `ttl_ms` | boolean | | `PERSIST` | `key` | boolean | | `KEYS` | optional `prefix:""`, `limit:1000` | bounded list; use `SCAN` for large keyspaces | | `SCAN` | optional opaque `cursor:""`, `prefix:""`, `count:100` | `{keys,cursor}` page | | `BLOB_BEGIN` | `key`, `total_size`; optional `upload_id`, `ttl_ms`, `nx`, `xx`, `connection_scoped` | upload ID, size, upload TTL, scope flag | | `BLOB_CHUNK` | `upload_id`, `offset`, base64 `data` | received bytes | | `BLOB_COMMIT` | `upload_id` | boolean | | `BLOB_ABORT` | `upload_id` | boolean | | `BLOB_READ` | `key`; optional `offset:0`, `length:1048576` | key/version/offset/total size and base64 range | | `TYPES` | none | registered type names | | `INFO` | none | server, compression, and memory information | | `MEMORY` | `key` | key/type/version/access count/approximate bytes and compression metadata | | `OP` | `key`, `operation`, `args` | type-specific result | ### Server command deadline Normal command execution is bounded by `limits.command_timeout_ms`. The default is 30 seconds. A protocol `CANCEL` frame can cancel a compatible in-flight request earlier. Blocking data-type operations therefore cannot hold a server request forever even if the caller requests a longer wait. ### Optimistic `CAS` Read `version` with `GET`, then replace only that version: ```json { "command": "CAS", "args": { "key": "account:1", "expected_version": 4, "type": "json", "value": { "balance": 120 }, "ttl_ms": 60000 } } ``` A stale version returns a conflict. AOF replay stores the resolved absolute expiration, so restart does not extend a CAS TTL. `SET`, `CREATE`, and `CAS` are full replacements. A positive relative TTL is converted to an absolute `expires_at_unix_nano` before journaling. `nx` requires the key to be absent; `xx` requires it to exist. Direct wire types and composite construction options are defined in [Type construction and operations](/data-types/operations/). ### `BATCH` A batch accepts up to 1024 normal Amaquet requests. It reduces round trips. It is **not** a rollback transaction. Earlier successful requests remain applied if a later request fails. With `continue_on_error:false`, the returned array ends at the first error; with `true`, later requests are attempted. Each nested mutation crosses the ordinary AOF transaction boundary independently. ### Cursor `SCAN` `SCAN` uses an opaque cursor. Start with an empty cursor and continue until the returned cursor is empty. Do not parse or construct cursors in application code. The implementation scans shard pages instead of materializing and sorting the complete keyspace for every page. Ordering is lexical within each of 256 shards, not globally lexical and not a point-in-time snapshot. Inserts, deletes, and expirations during iteration can affect later pages. `count` and `KEYS limit` are capped at 10,000; non-positive values select their defaults. ### Large binary values A protocol frame is limited to 64 MiB. Chunked upload permits a `binary_string` value up to 1 GiB while keeping each request bounded. Typical sequence: 1. `BLOB_BEGIN` 2. one or more non-empty, non-overlapping `BLOB_CHUNK` requests 3. `BLOB_COMMIT`, or `BLOB_ABORT` `total_size` can be 0 through 1 GiB. If `upload_id` is omitted, the dispatcher generates a random 128-bit hexadecimal ID before persistence. Chunks are standard base64, must be non-empty, cannot overlap, and cannot extend beyond the declared size. An upload accepts at most 65,536 spans. Pending uploads have a global byte budget and inactivity TTL. A connection-scoped upload can only be modified by its exact owner connection; ordinary uploads are not owner-bound. At commit, the object is read from the spool file into bounded 1 MiB in-memory blocks. It is not copied into one contiguous 1 GiB Go slice. Use `BLOB_READ` for large values: ```json { "command": "BLOB_READ", "args": { "key": "archive", "offset": 0, "length": 1048576 } } ``` `length` is limited to 32 MiB per request and reads at end-of-object may be shorter. A normal `GET` returns large chunked-binary metadata for objects above 32 MiB instead of creating an oversized response. The Go client exposes `UploadBlob` and `BlobRead` helpers. ### `OP` `OP` dispatches operations implemented by the stored logical type. See the [data-type catalog](/data-types/) for type-specific operations. ### Introspection results `TYPES` returns all 91 stable type names. `INFO` returns `name`, `protocol_version`, live `keys`, `uptime_seconds`, registered `types`, aggregate `compression`, and engine `memory` statistics. `MEMORY` returns `compressed:false` for native storage; compressed payloads add codec, original/stored sizes, and saving information. Top-level command names and operation names are normalized to uppercase. JSON argument property names are case-sensitive. Unknown commands, missing/invalid arguments, and unsupported type operations return an error rather than being ignored. --- ## Binary framing Source: [canonical documentation page](/protocol/framing/) Summary: Every Amaquet frame has a fixed 24-byte header followed by a bounded payload. Every Amaquet frame has a fixed 24-byte header followed by a bounded payload. | Offset | Size | Field | Encoding | | -----: | ---: | ---------------- | ------------------------------------ | | 0 | 4 | magic | ASCII `AMQT` | | 4 | 1 | protocol version | currently `1` | | 5 | 1 | kind | request `1`, response `2`, event `3` | | 6 | 2 | flags | unsigned big-endian | | 8 | 2 | opcode | unsigned big-endian | | 10 | 2 | reserved | currently zero | | 12 | 8 | request ID | unsigned big-endian | | 20 | 4 | payload length | unsigned big-endian | ### Opcodes The opcode identifies the operation carried by a frame and determines how the server handles its payload. | Opcode | Name | Purpose | | -----: | ------------- | ----------------------------------------- | | 1 | `COMMAND` | Run an Amaquet command | | 2 | `HELLO` | Server metadata / nonce signature | | 3 | `AUTH` | Authenticate an API key | | 4 | `SUBSCRIBE` | Subscribe to Pub/Sub | | 5 | `UNSUBSCRIBE` | Remove subscription | | 6 | `CANCEL` | Cancel an in-flight command by request ID | Requests carry independent request IDs. The server can execute multiple command requests concurrently on one connection and serialize frames safely on the socket. Responses can therefore complete in a different order from requests; clients must correlate by request ID. Only `COMMAND` requests execute concurrently. Other opcodes are handled synchronously in connection read order. Reusing a request ID while its command is still active returns `DUPLICATE_REQUEST`. Per-connection/global active-request semaphores return a framed overload error; if the global declared-payload reservation cannot be acquired, the reader stops and the connection closes before the payload is admitted. A `CANCEL` payload is `{"request_id":42}`; its own frame has a distinct request ID. The result reports the target ID and whether an active request was found. Cancellation is cooperative: the request context is cancelled, so an operation must observe that context to stop early. Invalid magic, unknown versions/opcodes, malformed request frames, truncated input, and payloads above the configured/hard limit are rejected. Read deadlines prevent a peer from holding a connection indefinitely with a partial frame. ### Response payload Command responses use a common success-or-error JSON envelope inside the binary frame payload. ```json { "ok": true, "result": {} } ``` Errors use: ```json { "ok": false, "error": { "code": "NOT_FOUND", "message": "key not found" } } ``` ### Pub/Sub frames `SUBSCRIBE` sends `{"key":"bus","channel":"updates","buffer":16}`. The response confirms the key/channel, then event frames use kind `3`, opcode `SUBSCRIBE`, and the original subscription request ID. An event payload contains `channel`, typed `payload`, and `published_at`. Re-subscribing the same connection to the same key/channel replaces its old server subscription. `UNSUBSCRIBE` sends `{"key":"bus","channel":"updates"}` and returns `{"unsubscribed":true|false}`. Connection close removes every subscription. Server and Go-client channels are bounded and non-blocking, so slow consumers can lose events; `pubsub` is deliberately best effort. --- ## Amaquet protocol overview Source: [canonical documentation page](/protocol/overview/) Summary: Amaquet is Amaquet's binary framed protocol. It is not RESP and it does not proxy Redis. Amaquet is Amaquet's binary framed protocol. It is not RESP and it does not proxy Redis. Two URI schemes are recognized: - `amaquet://` for plain TCP. - `amaquets://` for TLS. The default port is `13378`. ### Request flow Connections negotiate the single supported protocol version before normal command traffic. Authentication follows negotiation when enabled, after which request IDs let command responses complete independently. 1. Open TCP/TLS connection. 2. Send `HELLO` to negotiate protocol version and inspect server metadata/identity (the Go client does this automatically). 3. If authentication is required, send `AUTH`. 4. Send command frames. Request IDs allow responses to be matched to requests. 5. For Pub/Sub, `SUBSCRIBE` creates an event stream whose event frames reuse the subscription request ID. ### Payload encoding Frame payloads are JSON. Framing is binary; command values are typed JSON structures. Protocol version 1 capabilities advertised by `HELLO` are `cancel`, `batch`, `cas`, `scan`, `chunked_blob`, and `pubsub`. The protocol supports multiplexed command responses but makes no cross-command transaction guarantee. ### Hard limits The protocol implementation has a 64 MiB hard payload ceiling. The server can configure a smaller per-connection maximum with `limits.max_payload_bytes`. The configured read timeout applies while waiting for the next frame/header/payload; successful reads refresh it. The write timeout applies to responses and Pub/Sub event frames. `limits.command_timeout_ms` creates a context deadline for each command and synchronous non-command opcode handler. --- ## Connection URIs Source: [canonical documentation page](/protocol/uri/) Summary: Use Amaquet connection URIs to select a protocol transport and identify the target server endpoint. Use Amaquet connection URIs to select a protocol transport and identify the target server endpoint. ### Syntax An Amaquet connection URI specifies the transport scheme, host, and optional port for one server keyspace. ```text amaquet://host[:port] amaquets://host[:port] ``` The default port is `13378`. Examples: ```text amaquet://127.0.0.1:13378 amaquet://example.com amaquets://db.example.com:13378 amaquet://user:secret@127.0.0.1:13378 amaquet://127.0.0.1:13378?api_key=amaquet_... ``` The parser recognizes credentials in URI user-info and the `api_key` query parameter for legacy compatibility. If user-info contains a password, the password becomes the API key; with a username only, the username does. `api_key` overrides user-info. The public Go client and CLI reject URI-embedded credentials by default. Pass `DialOptions.APIKey` or `amaquet-cli -api-key` instead. Enabling `AllowURISecrets`/`-allow-uri-secrets` is explicit opt-in because URLs can leak through logs, shell history, process listings, and telemetry. An empty path or a bare trailing `/` is accepted. Any path component after that slash is rejected because Amaquet has one keyspace and does not implement logical database names. --- ## Wire values and snapshots Source: [canonical documentation page](/protocol/wire-values/) Summary: The type string must be one of the names returned by TYPES. Unknown names and malformed representations are rejected before mutation. Amaquet frames carry JSON payloads. A top-level `SET` identifies its value with sibling `type` and `value` arguments, while values nested inside collections use an explicit envelope: ```json { "type": "utf8_string", "value": "hello" } ``` The `type` string must be one of the names returned by `TYPES`. Unknown names and malformed representations are rejected before mutation. ### Direct `SET` representations This table defines the JSON representation and validation rules for values stored directly with `SET`. | Type | JSON representation | Validation/notes | | ------------------------ | ----------------------------------------- | ------------------------------------------------------------------------- | | `binary_string` | base64 string | Standard padded base64 | | `utf8_string` | string | Must contain valid UTF-8 | | `integer` | JSON integer | Signed 64-bit | | `unsigned_integer` | JSON integer | Unsigned 64-bit | | `float` | JSON number | Go `float64` | | `decimal`, `big_decimal` | decimal string | Exact coefficient and scale; no exponent notation | | `boolean` | boolean | `true` or `false` | | `null` | `null` | Explicit null value | | `timestamp` | string | RFC 3339 with optional fractional nanoseconds | | `duration` | string | Go duration syntax such as `250ms` or `2h45m` | | `uuid` | string | Canonical/hyphenated or 32 hexadecimal digits | | `big_integer` | decimal string | Arbitrary-precision base-10 integer | | `symbol` | string | Non-empty and contains no whitespace | | `json` | any JSON value | Numbers are decoded with their textual precision preserved | | `messagepack`, `cbor` | base64 string | Stored as opaque bytes; Amaquet does not validate the external encoding | | `geo_point` | `{"lat":n,"lon":n}` | Latitude −90…90, longitude −180…180 | | `bounding_box` | `{minLat,minLon,maxLat,maxLon}` | Input matching is case-insensitive; bounds must be ordered and geographic | | `polygon` | array of points | At least three points | | `dense_vector` | number array | Float32 elements | | `sparse_vector` | `{"dim":n,"values":{"0":v}}` | Positive dimension; indexes in range; zero entries omitted | | `quantized_vector` | `{"scale":n,"zero_point":n,"data":[...]}` | Signed 8-bit data elements | | `cidr_network` | string | Parsed and canonicalized by Go's CIDR parser | | `url` | string | Request URI with a required scheme | `json`, `messagepack`, and `cbor` also support `CREATE`, which initializes `{}` or empty opaque bytes before later type-specific operations. Other registered types are created with `CREATE` because they need internal structure or construction options. ### Command request and response The following examples show the JSON command envelope, successful result, and snapshot returned by the protocol. ```json { "command": "SET", "args": { "key": "answer", "type": "integer", "value": 42 } } ``` The protocol wraps the command result: ```json { "ok": true, "result": true } ``` `GET` returns: ```json { "key": "answer", "type": "integer", "value": 42, "version": 1, "expires_at": "2026-10-04T12:00:00Z" } ``` `expires_at` is omitted for persistent keys. The value version begins at 1 and increases for successful replacement, TTL changes, persistence changes, and successful mutating `OP` calls. Pure reads do not increase it. Because `BoundingBox` currently has no explicit JSON field tags, its `GET` value uses Go's exported field names: `MinLat`, `MinLon`, `MaxLat`, and `MaxLon`. The lower-camel input form in the table is accepted case-insensitively. ### Snapshot behavior `GET` never exposes internal Go pointers. Mutable values return a synchronized snapshot. Important bounded snapshots include: - streams and event logs return at most their first 1,000 entries through ordinary `GET`; - large chunked binary strings above 32 MiB return `{chunked,size,read_command}` metadata and must be read with `BLOB_READ`; - probabilistic structures return configuration/summary data rather than their complete backing arrays; - active Pub/Sub values return channel-to-subscriber counts, not buffered messages; - HNSW, B-tree, hash, and inverted indexes return summary metadata; radix trees cap their snapshot at 1,000 prefix results. Geo indexes, exact/flat vector sets, secondary indexes, and adjacency sets currently return complete logical snapshots, so ordinary `GET` can be large for those types. Use the type-specific read operation when it provides more appropriate pagination/range semantics. ### Nested identity and equality Collections store a complete `core.Value`, not an untyped JSON scalar. Sets and map-like structures derive internal equality keys from the stable type ID plus a deterministic representation of the value. Values with different Amaquet types are distinct even if their JSON rendering looks similar; for example, `integer:1` and `unsigned_integer:1` are different members. ### JSON number preservation JSON documents use `json.Decoder.UseNumber` when decoded and copied. This prevents an integer-looking JSON token from being eagerly converted to a binary float merely by document storage. JSON remains a document value; it is not recursively converted into Amaquet typed values. --- ## Administration API Source: [canonical documentation page](/reference/admin-api/) Summary: Operate Amaquet through its HTTP administration API for authentication, organizations, members, API keys, RBAC, data access, metrics, and audit events. HTTP clients can authenticate each request with `Authorization: Bearer ` or `X-Amaquet-Admin-Token`. A client that needs MFA or cookie-based authentication can exchange its credential through the server-side session endpoint. Most structured request bodies use the shared decoder, which caps input at 2 MiB and rejects unknown fields. The data-command proxy instead accepts up to 8 MiB with ordinary JSON decoding, and the member-invitation endpoint uses an uncapped ordinary decoder for its optional `ttl_hours` object. JSON-producing routes set `Content-Type: application/json`; errors normally have the common shape: ```json { "ok": false, "error": { "code": "BAD_REQUEST", "message": "diagnostic text" } } ``` The administration listener serves its embedded OpenAPI 3.1 document at `GET /openapi.json`; `HEAD` returns the same headers with no body. This discovery endpoint is public and does not require authentication. Bearer authentication accepts bootstrap, API-key, and member credentials. A member with MFA enabled cannot use its token as a direct bearer credential; it must create an HTTP cookie session with the token plus a TOTP code. Failed authentication is limited per `RemoteAddr` IP by `admin.auth_failures_per_minute`. The methods listed below are the supported API contract and the methods published in OpenAPI. Several read-only handlers currently dispatch by path without an explicit method check, so an alternate verb can reach the same handler; clients must not depend on that implementation detail. ### Authentication and identity These endpoints establish, inspect, and manage administrator and member authentication state. | Method | Path | Purpose | | ------ | ----------------------------- | --------------------------------------------------------------------- | | POST | `/api/session` | exchange bootstrap/API/member credential for an HTTP cookie session | | DELETE | `/api/session` | destroy the current HTTP cookie session | | GET | `/api/auth/check` | verify the current session/token | | POST | `/api/identity/accept-invite` | accept a one-time member invitation and receive a member access token | | POST | `/api/identity/mfa` | start TOTP MFA setup for the current member | | PUT | `/api/identity/mfa` | confirm TOTP MFA with a six-digit code | | DELETE | `/api/identity/mfa` | disable TOTP MFA for the current member | `POST /api/session` accepts: ```json { "token": "amaquet_...", "mfa_code": "123456" } ``` `mfa_code` is required only for member identities with TOTP enabled. The response includes `ok`, `role`, and `expires_at` and sets `amaquet_session`. Sessions last 12 hours, are server-side, and are limited by `admin.max_sessions`. The cookie is `HttpOnly`, `SameSite=Strict`, scoped to `/`, and `Secure` on HTTPS. Revoked API-key actors, disabled/deleted members, rotated member credentials, and MFA changes invalidate matching sessions through the revocation path. Invite acceptance accepts `{"invite":"amaquet_invite_…"}` and returns the member plus a one-time `access_token`. Starting MFA returns a base32 `secret` and an `otpauth` URI; confirmation accepts `{"code":"123456"}`. The MFA route requires `org.read` at the wrapper and then rejects any actor that is not a member. ### Organization and members These endpoints manage the organization record, members, invitations, and role-permission assignments. | Method | Path | Permission | | ------ | -------------------------- | --------------- | | GET | `/api/organization` | `org.read` | | PUT | `/api/organization` | `org.write` | | GET | `/api/members` | `members.read` | | POST | `/api/members` | `members.write` | | PATCH | `/api/members/{id}` | `members.write` | | DELETE | `/api/members/{id}` | `members.write` | | POST | `/api/members/{id}/invite` | `members.write` | | GET | `/api/rbac` | `rbac.read` | | PUT | `/api/rbac` | `rbac.write` | An invitation response contains the invitation token only once. `ttl_hours` defaults to 24. Organization updates accept `{"name":"Acme","slug":"acme"}`. Member creation accepts `name`, `email`, and `role`. A member patch must include a valid `role` on every request and may include `status`; roles are `admin`, `auditor`, or `developer`. The server stores a non-empty status as supplied, but only the exact status `active` can authenticate; API clients should use `active` and `disabled`. The RBAC update body is the complete role-to-permissions map, for example `{"developer":["data.read","data.write"]}`, and all three role keys are required. ### API keys These endpoints create, list, and revoke API-key credentials for protocol and administration access. | Method | Path | Permission | | ------ | -------------------- | ------------ | | GET | `/api/api-keys` | `keys.read` | | POST | `/api/api-keys` | `keys.write` | | DELETE | `/api/api-keys/{id}` | `keys.write` | A created API-key secret is returned once. Revocation invalidates new authentication, existing HTTP cookie sessions, and matching active Amaquet connections. Creation accepts: ```json { "name": "CI developer", "role": "developer", "expires_at": "2027-01-01T00:00:00Z" } ``` `expires_at` may be omitted or `null`. The response is `{"api_key":{…},"secret":"amaquet_…","warning":"…"}`. List results omit stored credential hashes. ### System and persistence These endpoints expose the running configuration and start server-side persistence maintenance actions. | Method | Path | Permission | | ------ | ----------------------------- | -------------- | | GET | `/api/system` | `system.read` | | PUT | `/api/system` | `system.write` | | POST | `/api/persistence/checkpoint` | `system.write` | | POST | `/api/persistence/compact` | `system.write` | `GET /api/system` returns `{"config":{…},"restart_required":false}`. `PUT` accepts the complete configuration object, validates and crash-safely saves it, clears `security.bootstrap_token`, and returns the public config with `restart_required`. The flag is true whenever the public submitted config differs from the current config; the endpoint persists changes but does not rebind listeners or rebuild engine components in place. Checkpoint creation produces a compact, verified AMQTAOF2 recovery image. Offline restore uses `amaquet-restore`; live in-place restore is intentionally not offered through HTTP. The checkpoint request optionally accepts `{"name":"nightly.aof"}`. Only the base filename is used; an omitted name becomes a UTC timestamped filename under `data_dir/backups`. The response contains its server-local `path`. Compaction has no request body and returns `{"ok":true}`. Both return HTTP 409 with `DISABLED` when AOF is off. ### Data operations These endpoints provide an HTTP control-plane proxy for key inspection, deletion, and native command execution. | Method | Path | Permission | | ------ | ------------------- | -------------------------------- | | GET | `/api/data/keys` | `data.read` | | GET | `/api/data/key` | `data.read` | | DELETE | `/api/data/key` | `data.write` | | POST | `/api/data/command` | dynamic `data.read`/`data.write` | `GET /api/data/keys` accepts `cursor`, `prefix`, and `count`; an omitted or unparsable value stays at the handler default of 250, a non-positive value selects the engine default of 100, and values above 10,000 are capped at 10,000. It returns `{"keys":[…],"cursor":"…"}`. `GET`/`DELETE /api/data/key` require a URL-encoded `key` query parameter. The command body is the native `{"command":"GET","args":{…}}` request and is capped at 8 MiB before ordinary JSON decoding. Top-level read commands and a fixed list of read-only `OP` names use `data.read`; everything else uses `data.write`. The current HTTP classifier does not list `BLOB_READ`, so that command requires `data.write` through this route even though native Amaquet authorization treats it as `data.read`. Success wraps the native result as `{"ok":true,"result":…}`. ### Health and metrics These read-only endpoints expose liveness, readiness, operational summaries, and Prometheus metrics. - `GET /api/health` returns process liveness, service name, and UTC time without authentication. - `GET /api/ready` returns key and memory statistics; it returns HTTP 503 when AOF persistence health is degraded. - `GET /api/overview` requires `data.read` and returns counts plus organization, memory, and compression summaries. - `GET /metrics` emits Prometheus 0.0.4 text. When `admin.metrics_require_auth` is true, the caller needs `system.read`. Metrics are `amaquet_keys`, `amaquet_memory_bytes`, `amaquet_memory_limit_bytes`, `amaquet_evicted_keys_total`, `amaquet_compressed_keys`, `amaquet_compression_saved_bytes`, `amaquet_go_heap_bytes`, `amaquet_go_goroutines`, and `amaquet_ready`. ### Audit `GET /api/audit?limit=200` requires `audit.read`. A positive limit returns at most that many newest-first retained events; a non-positive limit or a value above the retained count returns all retained events. Organization, member, invitation, MFA, RBAC, and API-key mutations create audit entries. Configuration updates and persistence operations currently do not append an audit event. ### Security headers The administration listener serves API and metrics routes only. Unregistered paths, including `/`, return 404. All responses receive `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: no-referrer`, and a same-origin Content Security Policy. Cookie-authenticated requests rely on `SameSite=Strict` as their built-in cross-site-request mitigation; the server does not validate `Origin`/`Referer` and does not issue a separate CSRF token. Bearer credentials are not sent automatically by a browser, but clients are still responsible for keeping them out of untrusted content. --- ## CLI reference Source: [canonical documentation page](/reference/cli/) Summary: Use the Amaquet command-line tools for native protocol commands, API-key authentication, server identity generation, and verified checkpoint restoration. This reference describes the Amaquet client, server, identity-generation, and checkpoint-restore executables. ### `amaquet-cli` `amaquet-cli` sends Amaquet commands. ```bash amaquet-cli \ -uri amaquets://db.example.com:13378 \ -api-key "$AMAQUET_API_KEY" \ GET '{"key":"example"}' ``` For large JSON arguments, use a file or stdin instead of a shell argument: ```bash amaquet-cli SET @request.json cat request.json | amaquet-cli SET - ``` URI-embedded secrets are disabled by default. Pass the secret with `-api-key`. | Flag | Default | Meaning | | -------------------- | --------------------------- | ------------------------------------------------------------ | | `-uri` | `amaquet://127.0.0.1:13378` | `amaquet://` or `amaquets://` endpoint with no database path | | `-api-key` | empty | API key sent with the protocol `AUTH` opcode | | `-allow-uri-secrets` | `false` | explicitly accept legacy URI user-info/query credentials | | `-timeout` | `30s` | dial and request context deadline | The first positional argument is the command name. The optional second argument is a JSON object, `@path` to read that object from a file, or `-` to read it from standard input. JSON numbers are decoded with `UseNumber`, avoiding an automatic conversion to `float64` before transmission. Output is pretty-printed when it is valid JSON. Usage/input errors exit with status 2; dial/command errors exit with status 1. ### `amaquet` The server executable accepts the following startup, initialization, and version-reporting flags. | Flag | Default | Meaning | | -------------- | ---------------- | ------------------------------------------------------------------------------ | | `-config` | `./amaquet.json` | JSON configuration path | | `-init-config` | `false` | save defaults with a generated bootstrap token, print the token once, and exit | | `-version` | `false` | print the server version and exit | Normal startup loads JSON if it exists, overlays supported environment variables, validates the result, creates `data_dir`, replays the AOF when enabled, then starts the admin and Amaquet listeners. If neither JSON nor the environment supplies a bootstrap token, startup reuses or creates `data_dir/bootstrap-token` with mode `0600`. ### `amaquet-keygen` Generate the optional Ed25519 server identity: ```bash amaquet-keygen \ --public ./keys/amaquet-public.pem \ --private ./keys/amaquet-private.pem ``` `-public` defaults to `amaquet-public.pem`; `-private` defaults to `amaquet-private.pem`. Existing targets are replaced. The command prints the public-key fingerprint after saving the PEM files. ### `amaquet-restore` Restore a verified AOF checkpoint while Amaquet is stopped: ```bash amaquet-restore \ -source /backup/checkpoint.aof \ -target /var/lib/amaquet/amaquet.aof ``` The tool validates the source and copied destination before the atomic swap. It maintains a temporary rollback file during replacement. Both `-source` and `-target` are required. Run the restore command only while the target Amaquet process is stopped. --- ## Configuration reference Source: [canonical documentation page](/reference/configuration/) Summary: Reference every Amaquet JSON configuration field, including defaults, accepted values, validation rules, security implications, and operational effects. This table documents every supported JSON configuration field, its accepted type, default value, and operational effect. | Path | Type | Default | Notes | | ------------------------------------ | ------- | -------------------- | ----------------------------------------------------------------------------------- | | `protocol.host` | string | `127.0.0.1` | Amaquet bind address | | `protocol.port` | integer | `13378` | 1–65535 | | `protocol.tls` | boolean | `false` | enables Amaquet TLS | | `protocol.tls_cert_file` | string | empty | required with Amaquet TLS | | `protocol.tls_key_file` | string | empty | required with Amaquet TLS | | `protocol.require_auth` | boolean | `true` | require Amaquet authentication | | `protocol.allow_insecure_auth` | boolean | `false` | permit authenticated plaintext on non-loopback only when explicitly set | | `protocol.read_timeout_ms` | integer | `120000` | frame/idle read deadline; `0` disables | | `protocol.write_timeout_ms` | integer | `30000` | response/event write deadline; `0` disables | | `admin.host` | string | `127.0.0.1` | admin bind address | | `admin.port` | integer | `13379` | 1–65535 | | `admin.tls` | boolean | `false` | enable HTTPS | | `admin.tls_cert_file` | string | empty | required with admin TLS | | `admin.tls_key_file` | string | empty | required with admin TLS | | `admin.allow_insecure` | boolean | `false` | permit non-loopback HTTP only when explicitly set | | `admin.metrics_require_auth` | boolean | `true` | require RBAC authentication for `/metrics` | | `admin.max_sessions` | integer | `10000` | maximum stored HTTP cookie sessions | | `admin.auth_failures_per_minute` | integer | `20` | failed-login throttle per source address | | `security.bootstrap_token` | string | empty | supplied bootstrap secret; if empty, startup uses a protected file under `data_dir` | | `security.public_key_file` | string | empty | Ed25519 public identity key | | `security.private_key_file` | string | empty | Ed25519 private identity key | | `persistence.aof_enabled` | boolean | `false` | transactional AOF journal | | `persistence.aof_path` | string | `./data/amaquet.aof` | AOF path | | `persistence.fsync` | enum | `everysec` | `always`, `everysec`, or `no` | | `limits.max_connections` | integer | `10000` | accepted Amaquet connections | | `limits.max_payload_bytes` | integer | 64 MiB | per-frame payload; maximum 64 MiB | | `limits.max_memory_bytes` | integer | `0` | engine data budget; `0` means unlimited | | `limits.eviction_policy` | enum | `noeviction` | `noeviction`, `allkeys-lru`, `allkeys-lfu`, `volatile-ttl` | | `limits.max_inflight_requests` | integer | `4096` | server-wide active command cap | | `limits.max_inflight_per_connection` | integer | `128` | active command cap for one connection | | `limits.max_inflight_payload_bytes` | integer | 256 MiB | aggregate inbound frame budget | | `limits.max_pending_upload_bytes` | integer | 1 GiB | total declared bytes for active blob uploads | | `limits.upload_ttl_ms` | integer | `600000` | upload inactivity TTL; minimum 1000 ms | | `limits.command_timeout_ms` | integer | `30000` | server-side command execution deadline | | `compression.enabled` | boolean | `true` | adaptive compression for supported contiguous value types | | `compression.algorithm` | enum | `auto` | `auto`, `lz4`, `rle`, `deflate-fast`, `none` | | `compression.min_bytes` | integer | `1024` | smaller values remain native | | `compression.min_savings_percent` | number | `8` | required saving before a compressed encoding is retained | | `data_dir` | string | `./data` | admin state, upload spools, backups, and runtime security files | The table is a compact inventory. The sections below explain what each key changes, when the setting is read, and what to consider before changing it. ### How configuration is loaded Amaquet starts from built-in defaults, reads the JSON file named by `-config`, overlays the supported `AMAQUET_*` environment variables, validates the resulting configuration, and only then starts either listener. The default path is `./amaquet.json`; a missing file is allowed and therefore produces a defaults-plus-environment configuration. A malformed file, invalid value, missing TLS material, or unsafe non-loopback listener stops startup. JSON is the durable configuration source. Environment variables are process-start overrides and take precedence over values read from JSON for the fields they support. They are not written back by `amaquet --init-config` or by the administration configuration endpoint. Relative paths are interpreted by the server process from its current working directory, so service units should set an explicit `WorkingDirectory` or use absolute paths. `amaquet --init-config` writes the default structure and a generated bootstrap token, then exits. It does not start the server. Normal startup may create `data_dir` and, when no bootstrap token is configured, create `data_dir/bootstrap-token` with mode `0600`. ### Protocol listener The `protocol` object controls the native Amaquet binary protocol. Applications connect to this listener with `amaquet://` for plaintext or `amaquets://` for TLS. It is the data-plane endpoint used by `amaquet-cli` and the Go client; it is separate from the HTTP administration listener. #### `protocol.host` The local address on which the native protocol socket listens. `127.0.0.1` keeps the listener local to the machine and is the safest development default. Use a specific private interface in a multi-interface deployment when possible. Binding `0.0.0.0` exposes the listener on every IPv4 interface and should be paired with TLS, authentication, firewall rules, and a connection limit. #### `protocol.port` The TCP port for the native protocol. It must be between `1` and `65535`; the default is `13378`. Changing this value also changes the URI passed to clients. The port must not collide with the admin listener or another process. #### `protocol.tls` Enables TLS for the native protocol. When enabled, clients must use an `amaquets://` URI and validate the configured certificate in the normal way. TLS protects credentials and data in transit; it does not replace protocol authentication. TLS startup fails unless both certificate and key paths are supplied. #### `protocol.tls_cert_file` Filesystem path to the certificate presented by the native protocol listener. The file must be readable by the Amaquet process and contain a certificate whose names match the hostnames clients use. The setting is only used when `protocol.tls` is enabled, although the environment variable `AMAQUET_TLS_CERT` also enables TLS automatically. #### `protocol.tls_key_file` Filesystem path to the private key corresponding to `protocol.tls_cert_file`. Protect this file with filesystem permissions and do not place it in a public repository or image layer. Amaquet refuses to start TLS without both certificate and key files. #### `protocol.require_auth` Controls whether new native-protocol connections must authenticate before issuing protected commands. The default is `true`. Authentication accepts a bootstrap credential, API key, or eligible member token through the protocol `AUTH` flow. Setting this to `false` makes new connections anonymous administrative actors, so use it only on an isolated trusted loopback or test deployment. #### `protocol.allow_insecure_auth` Explicitly permits authenticated plaintext protocol traffic when the listener is not loopback. It does not disable authentication and does not enable TLS. This is a narrowly scoped development escape hatch for a protected network; production deployments should leave it `false` and use TLS instead. Validation rejects a non-loopback authenticated plaintext listener unless this key is enabled. #### `protocol.read_timeout_ms` Sets the read deadline for an incomplete frame or an idle connection, in milliseconds. The default is `120000` (two minutes). A value of `0` disables this deadline. Increase it only for clients that legitimately stream or assemble requests slowly; disabling it allows abandoned connections to consume resources indefinitely. #### `protocol.write_timeout_ms` Sets the response and event write deadline, in milliseconds. The default is `30000` (30 seconds); `0` disables the deadline. This protects the server from clients that stop reading responses, including event or Pub/Sub delivery. A value that is too small can disconnect slow but healthy clients. ### Administration listener The `admin` object controls the HTTP control-plane listener. It serves `/api/*` and `/metrics`; it does not serve the documentation site or a browser UI. Keep it on loopback or a private management network, and expose it remotely only with TLS or an explicitly controlled reverse proxy. #### `admin.host` The local address for the HTTP administration server. The default `127.0.0.1` prevents remote access. A non-loopback address requires `admin.tls` or the explicit `admin.allow_insecure` override during validation. #### `admin.port` The HTTP administration port. It must be between `1` and `65535`; the default is `13379`. Health checks, metrics scrapers, and administration clients must use this port, not the native protocol port. #### `admin.tls` Enables HTTPS for the administration server. When enabled, the admin listener requires its own certificate and key settings. Keep the admin certificate's names aligned with the management hostname. HTTPS protects bootstrap credentials, API keys, sessions, and configuration operations in transit. #### `admin.tls_cert_file` Path to the certificate used by the HTTPS administration listener. It is independent of `protocol.tls_cert_file`, allowing different trust and hostname policies for data-plane and control-plane traffic. #### `admin.tls_key_file` Path to the private key paired with `admin.tls_cert_file`. The Amaquet process must be able to read it at startup; protect it as a secret. `admin.tls` fails validation unless both admin TLS paths are present. #### `admin.allow_insecure` Allows plaintext admin HTTP on a non-loopback address. This setting is intended for a deliberately isolated development network or a reverse-proxy arrangement where the proxy provides the required protection. It does not authenticate requests by itself. Leave it `false` for production and use HTTPS. #### `admin.metrics_require_auth` Controls whether `/metrics` requires administration authentication. The default is `true`, keeping operational data and labels behind the same access boundary as other admin endpoints. Set it to `false` only when the listener is already isolated and the metrics exposure is intentionally public to the monitoring network. #### `admin.max_sessions` Caps the number of server-side HTTP cookie sessions kept in memory. The default is `10000`. Sessions are created by the admin login/session flow and are distinct from native protocol connections and API keys. Lower this value on a constrained control-plane deployment; increasing it consumes more memory and does not increase the number of API keys. #### `admin.auth_failures_per_minute` Limits failed authentication attempts per source address per minute for the admin and native authentication hardening paths. The default is `20`; it must be positive. This is a throttle, not a replacement for TLS, network access control, or strong credentials. ### Security and identity The `security` object contains bootstrap and optional protocol-identity material. Bootstrap credentials authorize the first administration actions; Ed25519 identity keys let clients verify that they reached the expected server identity after protocol negotiation. Identity keys are not TLS certificates and do not encrypt traffic. #### `security.bootstrap_token` The initial administrative credential. `amaquet --init-config` generates one and writes it into the JSON file. If it is empty during normal startup, Amaquet reads or creates `data_dir/bootstrap-token` with restrictive permissions and logs the protected file path rather than the secret itself. Use the bootstrap credential to create a permanent API key, then stop using it; persisted administration state permanently disables bootstrap authentication after the first admin API key is created. Do not commit this value, pass it in shell history, or put it in a public Compose file. `AMAQUET_BOOTSTRAP_TOKEN` overrides it at runtime without copying the secret into the JSON configuration. #### `security.public_key_file` Path to the Ed25519 public identity key generated by `amaquet-keygen`. It is loaded together with `security.private_key_file` and its fingerprint is advertised during protocol negotiation. Supplying only one identity path is not enough to enable identity loading. #### `security.private_key_file` Path to the matching Ed25519 private identity key. Restrict access to the Amaquet service account. A client should verify the advertised fingerprint/signature when it needs application-level server identity; use TLS as well when confidentiality and certificate validation are required. ### Persistence Persistence is optional. The active keyspace remains in memory in both modes; AOF provides recovery after restart, not a disk-backed replacement for the engine. Keep the journal and administration state under protected storage and back them up independently. #### `persistence.aof_enabled` Enables transactional append-only journaling. On startup Amaquet replays committed mutations before opening listeners. When `false`, the data keyspace starts empty after a restart, although administrative state can still remain in `data_dir/admin.json`. Enabling AOF adds write and storage overhead, so select the durability mode with the workload's recovery objective in mind. #### `persistence.aof_path` Path to the AOF journal. The parent directory must be writable by the service account. Use an explicit path on persistent storage in production rather than relying on `./data` relative to an unpredictable working directory. Checkpoint and restore workflows should use separate backup copies rather than replacing the live journal by hand. #### `persistence.fsync` Selects how aggressively journal data is flushed to durable storage: - `always` flushes and fsyncs each journal phase, maximizing durability at the cost of write latency. - `everysec` flushes approximately once per second and is the normal latency/durability balance. - `no` relies on buffered and operating-system writeback and provides the weakest crash-durability guarantee. All modes retain transactional recovery semantics. A complete record with a bad CRC is treated as corruption; an incomplete final tail can be discarded as an interrupted write. ### Resource limits The `limits` object protects the process from connection floods, oversized frames, concurrent work, blob-upload reservations, and unbounded data growth. These limits are independent: raising one does not automatically raise the others. #### `limits.max_connections` Maximum number of accepted native protocol connections. The default is `10000` and the value must be positive. Size it with the operating system's file-descriptor limit, expected client pool count, TLS overhead, and admin connections in mind. #### `limits.max_payload_bytes` Maximum payload in one protocol frame. The default is `67108864` bytes (64 MiB), which is also the hard upper bound. Lower it to reduce per-request memory pressure. Values larger than one frame should use the chunked blob commands where supported; this limit does not define the maximum logical value size for every data type. #### `limits.max_memory_bytes` Approximate memory budget for data held by the Amaquet engine. `0` disables the configured engine budget. It is not a process-RSS cap: Go runtime metadata, network buffers, indexes, temporary decoding buffers, AOF state, and the admin server consume additional memory. Leave headroom between this budget and the container or host memory limit. #### `limits.eviction_policy` Selects what happens when a mutating operation would exceed `max_memory_bytes`: - `noeviction` rejects the allocating write. - `allkeys-lru` removes the least-recently-used sampled candidate, whether or not it has a TTL. - `allkeys-lfu` removes a low-frequency sampled candidate. - `volatile-ttl` considers only keys with expiration and prefers the key nearest expiry; if no eligible key exists, the write is rejected. Eviction is approximate and sampled rather than a perfect global ordering. Destructive or memory-reducing commands such as `DEL` remain available when the engine is over budget. #### `limits.max_inflight_requests` Server-wide cap on active command executions. The default is `4096`. It limits total concurrent work across all protocol connections and helps prevent a small number of clients from exhausting worker capacity. #### `limits.max_inflight_per_connection` Per-connection cap on active commands. The default is `128`. This works with the global cap to prevent one multiplexed client from monopolizing the server while allowing other connections to make progress. #### `limits.max_inflight_payload_bytes` Aggregate budget for inbound payload bytes associated with active requests. The default is `268435456` bytes (256 MiB). It must be at least `max_payload_bytes`; otherwise a single legal frame could never be admitted. Lower it when many concurrent clients can submit large requests. #### `limits.max_pending_upload_bytes` Maximum total declared size reserved by active chunked blob uploads. The default is `1073741824` bytes (1 GiB). This protects disk spool and memory-accounting paths from many unfinished uploads. It is separate from the maximum size of one frame and the AOF path. #### `limits.upload_ttl_ms` Inactivity lifetime for an unfinished blob upload. The default is `600000` milliseconds (10 minutes), and the minimum is `1000` milliseconds. Expired uploads are discarded; lower the value when clients are unreliable, or raise it for genuinely slow upload links while keeping the aggregate pending-upload budget bounded. #### `limits.command_timeout_ms` Server-side deadline for one command. The default is `30000` milliseconds and the value must be positive. It also bounds blocking data-type operations at the protocol request level. This is separate from the client's own dial/request timeout; both deadlines can terminate a request. ### In-memory compression Compression is applied to supported contiguous values after they cross the configured size threshold. The compressed representation is retained only when it is smaller enough to satisfy the savings threshold; enabling compression therefore does not force every value into a compressed form. #### `compression.enabled` Master switch for adaptive in-memory compression. The default is `true`. Disable it when CPU latency matters more than memory, or when the workload consists mostly of values that do not compress well. #### `compression.algorithm` Controls the compressor selection. `auto` tries the low-latency RLE and LZ4 candidates and keeps the smaller result; `lz4` favors fast block compression; `rle` favors repeated-byte data; `deflate-fast` uses a fast DEFLATE setting; `none` disables compression regardless of the enabled flag. Values are validated at startup. #### `compression.min_bytes` Minimum original value size eligible for compression. The default is `1024` bytes. Smaller values remain in their native representation to avoid metadata and CPU overhead. The value must be non-negative. #### `compression.min_savings_percent` Minimum percentage reduction required before a compressed representation is kept. The default is `8`; valid values are at least `0` and below `100`. A high threshold reduces CPU and representation churn for marginal wins; a low threshold saves more memory at the cost of compressing data that barely shrinks. ### Filesystem root This directory contains runtime state that is separate from the primary JSON configuration and the AOF journal. #### `data_dir` Root directory for runtime files that are not the primary JSON configuration: generated bootstrap-token material, `admin.json` control-plane state, upload spools, checkpoint/backups, and other runtime data. Amaquet creates it with restrictive directory permissions during startup. Put it on persistent storage when administration state or upload recovery must survive process replacement, and back it up separately from the AOF journal. ### Choosing a safe starting profile For a single-process local development server, keep both hosts on `127.0.0.1`, use the default authentication and TLS settings, and use `amaquet --init-config` so the bootstrap secret is generated rather than invented. For a networked deployment, use private listener addresses, TLS on both surfaces, strong API keys, an explicit persistent `data_dir`, AOF settings appropriate to the recovery objective, and memory/connection limits sized below the host or container ceiling. ### Environment overrides The complete environment-override surface is below. Numeric values that cannot be parsed are ignored. For normal boolean overrides, only the exact strings `1` and `true` enable the option. Compression is the inverse form: it is enabled unless the value is exactly `0` or `false`. | Variable | Configuration path / behavior | | ------------------------------------- | --------------------------------------------------- | | `AMAQUET_HOST` | `protocol.host` | | `AMAQUET_PORT` | `protocol.port` | | `AMAQUET_ADMIN_HOST` | `admin.host` | | `AMAQUET_ADMIN_PORT` | `admin.port` | | `AMAQUET_ALLOW_INSECURE_AUTH` | `protocol.allow_insecure_auth` | | `AMAQUET_ADMIN_ALLOW_INSECURE` | `admin.allow_insecure` | | `AMAQUET_BOOTSTRAP_TOKEN` | `security.bootstrap_token` at runtime | | `AMAQUET_TLS_CERT` | `protocol.tls_cert_file`; also enables protocol TLS | | `AMAQUET_TLS_KEY` | `protocol.tls_key_file`; also enables protocol TLS | | `AMAQUET_COMPRESSION` | `compression.enabled` | | `AMAQUET_COMPRESSION_ALGORITHM` | `compression.algorithm` | | `AMAQUET_COMPRESSION_MIN_BYTES` | `compression.min_bytes` | | `AMAQUET_AOF_ENABLED` | `persistence.aof_enabled` | | `AMAQUET_AOF_PATH` | `persistence.aof_path` | | `AMAQUET_AOF_FSYNC` | `persistence.fsync` | | `AMAQUET_MAX_MEMORY_BYTES` | `limits.max_memory_bytes` | | `AMAQUET_EVICTION_POLICY` | `limits.eviction_policy` | | `AMAQUET_MAX_CONNECTIONS` | `limits.max_connections` | | `AMAQUET_MAX_INFLIGHT_REQUESTS` | `limits.max_inflight_requests` | | `AMAQUET_MAX_INFLIGHT_PER_CONNECTION` | `limits.max_inflight_per_connection` | | `AMAQUET_MAX_INFLIGHT_PAYLOAD_BYTES` | `limits.max_inflight_payload_bytes` | | `AMAQUET_MAX_PENDING_UPLOAD_BYTES` | `limits.max_pending_upload_bytes` | | `AMAQUET_UPLOAD_TTL_MS` | `limits.upload_ttl_ms` | | `AMAQUET_COMMAND_TIMEOUT_MS` | `limits.command_timeout_ms` | There are no environment overrides for admin TLS files, `max_payload_bytes`, `max_sessions`, `auth_failures_per_minute`, or `compression.min_savings_percent`; configure those in JSON. An environment-provided bootstrap secret is runtime-only. An API configuration save does not copy that secret into the JSON configuration file. ### Restart behavior The control-plane system endpoint compares the full public configuration. It persists rather than hot-applies changes, so any submitted public configuration that differs from the running configuration is reported with `restart_required: true`. Validation rejects insecure non-loopback authenticated protocol and admin listeners unless the matching explicit insecure override is enabled. ### Validation constraints Both listener ports must be between 1 and 65535. Connection, in-flight request, pending-upload, command-timeout, HTTP-session, and authentication-rate limits must be positive. `max_payload_bytes` is limited to 1 byte through 64 MiB; the aggregate in-flight payload budget must be at least that large. `max_memory_bytes` and protocol timeouts may be zero but not negative, and upload TTL must be at least 1000 ms. TLS requires both certificate and key paths. Compression minimum bytes cannot be negative, and its required saving must be at least zero and below 100 percent. Enumerated persistence, compression, and eviction values must match the options in the table above. --- ## Error model Source: [canonical documentation page](/reference/errors/) Summary: Amaquet response errors include a stable code and human-readable message. Amaquet response errors include a stable code and human-readable message. Common codes: | Code | Meaning | | ------------------- | ---------------------------------------------------------------------------------------- | | `NOT_FOUND` | key/resource was not found | | `WRONG_TYPE` | operation was incompatible with stored type | | `EXISTS` | conditional create/write found an existing key | | `CONFLICT` | version conflict | | `MEMORY_LIMIT` | a write would exceed `max_memory_bytes` and eviction could not make room | | `BUSY` | an in-flight request, authentication-rate, or other framed server-busy limit was reached | | `DUPLICATE_REQUEST` | a multiplexed connection reused an active request ID | | `FORBIDDEN` | authenticated role lacks permission | | `UNAUTHORIZED` | authentication is required/invalid | | `ERROR` | validation, decoding, operation, or other server error | These codes are produced by native Amaquet response frames. Unknown commands, malformed arguments, timeouts/cancellation, and most data-type validation failures currently use the general `ERROR` code, so clients may need to surface the diagnostic message for those failures. The global inbound-payload budget is enforced while reading frames. If that reservation cannot be acquired, the server closes the affected connection instead of decoding the payload and returning a `BUSY` frame. The administration HTTP API uses the same `{ok:false,error:{code,message}}` shape but has a route-specific vocabulary such as `BAD_REQUEST`, `INVALID`, `METHOD_NOT_ALLOWED`, `RATE_LIMITED`, `SESSION_LIMIT`, `INVALID_INVITE`, `MFA_ERROR`, `SAVE_FAILED`, `DISABLED`, `CHECKPOINT_FAILED`, `COMPACT_FAILED`, and `COMMAND_ERROR`. Use the HTTP status first, then the structured code; do not parse the message. --- ## Glossary Source: [canonical documentation page](/reference/glossary/) Summary: AOF — Append-only file containing journaled Amaquet mutation requests. **AOF** — Append-only file containing journaled Amaquet mutation requests. **DataType** — Stable numeric/wire identity for a Amaquet value. **Encoding** — Internal representation of a value; conceptually separate from DataType. **Amaquet** — Amaquet's framed network protocol. **`amaquet://`** — Plain-TCP Amaquet endpoint URI. **`amaquets://`** — TLS Amaquet endpoint URI. **Typed wire value** — JSON object with `type` and `value` used when nesting values inside composite structures. **Shard** — One of 256 independently locked key maps in the current engine. **Version** — Monotonic per-key revision counter. **Bootstrap token** — Administrative token configured or generated at startup. --- ## Go client library Source: [canonical documentation page](/reference/go-client/) Summary: Integrate Amaquet from Go with the concurrent-safe client, TCP or TLS dialing, request multiplexing, authentication, subscriptions, and typed commands. The public package `github.com/newfoundcodes/amaquet/pkg/amaquet` is the supported Go client. It owns a single TCP/TLS connection, negotiates protocol v1 with `HELLO`, multiplexes requests by 64-bit request ID, routes asynchronous event frames, and is safe for concurrent command calls. `Close` is idempotent. ### Connect Create a client with a bounded context, an explicit API key, and a TLS configuration when using `amaquets://`. ```go import ( "context" "crypto/tls" "time" "github.com/newfoundcodes/amaquet/pkg/amaquet" ) ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() client, err := amaquet.DialWithOptions(ctx, "amaquets://db.example.com:13378", amaquet.DialOptions{ APIKey: "amaquet_...", Timeout: 30 * time.Second, TLSConfig: &tls.Config{MinVersion: tls.VersionTLS12}, }) if err != nil { // handle connection, TLS, HELLO, or AUTH failure } defer client.Close() ``` `Dial` is shorthand for `DialWithOptions` with defaults. URI-embedded credentials are rejected by default because URLs commonly leak through logs, shell history, and telemetry. Prefer `DialOptions.APIKey`. Set `AllowURISecrets` only for explicit legacy compatibility. For `amaquets://`, the client derives `ServerName` from the URI host when it is absent and enforces TLS 1.2 or newer when no higher minimum is supplied. Normal certificate validation remains enabled unless the caller deliberately changes `TLSConfig`. ### Dial options Configure authentication, TLS, timeouts, and the legacy URI-secret opt-in through `DialOptions`. | Field | Meaning | | ----------------- | -------------------------------------------------------------------------- | | `APIKey` | Credential sent by `AUTH` after `HELLO` | | `TLSConfig` | Optional cloned TLS configuration for `amaquets://` | | `Timeout` | Client-side response deadline; defaults to 30 seconds | | `AllowURISecrets` | Permit credentials parsed from URI user-info or `api_key`; default `false` | The dial context bounds TCP connection, TLS handshake, and initial negotiation. Each later call also accepts its own context. On context cancellation or client timeout, ordinary request calls send a best-effort Amaquet `CANCEL` frame for that request ID. `Subscribe` waits for its synchronous acknowledgement using both the supplied context and `Client.Timeout`; callers should still give that context a deadline. ### Public API The convenience methods below cover connection setup, common commands, Pub/Sub, and chunked binary transfer. `Command` remains the escape hatch for every protocol command that does not have a dedicated helper. | Method | Behavior | | ------------------------- | -------------------------------------------------------------------------- | | `Dial`, `DialWithOptions` | Connect, negotiate v1, optionally authenticate, and start the frame reader | | `Close` | Close the connection and release pending requests/subscriptions | | `Command` | Send any `{command,args}` request and decode its result | | `Hello` | Request server metadata and an optional nonce signature | | `ServerInfo.VerifyNonce` | Verify the Ed25519 signature returned for the supplied nonce | | `Ping` | Execute `PING` | | `Set` | Store a directly encoded type with an optional TTL | | `Get` | Return key/type/value/version metadata as `map[string]any` | | `Delete` | Delete one or more keys | | `Create` | Construct a composite/specialized type | | `Op` | Execute a type-specific operation | | `Subscribe` | Open a Pub/Sub event stream backed by a buffered Go channel | | `Subscription.Close` | Send `UNSUBSCRIBE` and close local event delivery | | `BlobRead` | Read at most one bounded binary range | | `UploadBlob` | Stream a reader through begin/chunk/commit with abort-on-failure | ### Typed commands Convenience methods cover common operations; `Command` exposes the complete protocol: ```go if err := client.Set(ctx, "counter", "integer", int64(41), 0); err != nil { // handle error } var value int64 err = client.Op(ctx, "counter", "ADD", map[string]any{"delta": 1}, &value) ``` Nested values use the [wire-value envelope](/protocol/wire-values/): ```go err = client.Create(ctx, "jobs", "fifo_queue", nil) err = client.Op(ctx, "jobs", "ENQUEUE", map[string]any{ "value": map[string]any{ "type": "json", "value": map[string]any{"id": "job-1"}, }, }, nil) ``` `Command` returns protocol failures as Go errors formatted as `CODE: message`. The current client does not expose a separately typed error-code struct, so code that must branch on codes should wrap this boundary or use the lower-level protocol contract deliberately. ### Server identity Verify the optional Ed25519 server identity by supplying an unpredictable application-generated nonce to `HELLO`. ```go nonce := "application-generated-unpredictable-nonce" info, err := client.Hello(ctx, nonce) if err != nil || !info.VerifyNonce(nonce) { // reject an untrusted identity } ``` Nonce verification proves possession of the configured Ed25519 identity key. It does not encrypt traffic or replace TLS hostname/certificate validation. ### Pub/Sub Subscribe to a key and channel to receive asynchronous events through a bounded Go channel. ```go sub, err := client.Subscribe(ctx, "bus", "updates", 64) if err != nil { // handle error } defer sub.Close() for message := range sub.C { // message.Channel, message.Payload, message.PublishedAt } ``` The client uses non-blocking delivery to the configured local channel buffer. If the application does not consume quickly enough, client-side event delivery can be dropped. The server-side Pub/Sub object also uses bounded subscriber channels and reports its own `published`, `delivered`, and `dropped` counters. ### Large binary values Use the blob helpers to stream large binary values without putting the full payload in a single command frame. ```go err = client.UploadBlob(ctx, "archive", "", source, size, 1<<20) chunk, total, err := client.BlobRead(ctx, "archive", 0, 1<<20) ``` `UploadBlob` chooses a `client-` upload ID when empty, defaults chunks to 1 MiB, caps chunks at 32 MiB, reads exactly the declared size, and attempts `BLOB_ABORT` if any stage fails. Its helper does not expose `nx`, `xx`, TTL, or `connection_scoped`; use `Command` directly when those `BLOB_BEGIN` options are required. ### Concurrency and shutdown A background reader is the sole frame reader. A mutex serializes writes, while request IDs allow many callers to await independent responses. Responses may complete out of order. Closing the client closes all pending request and subscription channels; callers receive `net.ErrClosed` where applicable. Avoid mutating a shared result object from multiple calls, and always bound long-running operations with a context even though the client and server both have default timeouts. --- ## Implementation coverage map Source: [canonical documentation page](/reference/implementation-coverage/) Summary: This page maps the production repository to the documentation that describes it. This page maps the production repository to the documentation that describes it. It is an audit aid: a subsystem is not considered documented merely because its package name appears somewhere. Its public behavior, persistence boundary, security boundary, configuration, and operational constraints must have an owning page. ### Runtime components Each production subsystem has an owning page for its externally observable behavior and operational boundary. Combined rows indicate implementation files that participate in one feature and are documented together. | Source | Responsibility | Documentation | | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd/amaquet` | Process startup, configuration, listeners, AOF replay, bootstrap token, graceful shutdown | [Installation](/getting-started/installation/), [Configuration](/reference/configuration/), [Production deployment](/operations/deployment/) | | `internal/config` | Defaults, JSON loading/saving, environment overrides, validation, runtime config manager | [Configuration](/reference/configuration/), [Getting-started configuration](/getting-started/configuration/) | | `internal/core` | Sharded keyspace, entry metadata, TTL scheduler, versions, memory accounting, eviction, scans, staged transactions | [Engine internals](/concepts/engine-internals/), [Keyspace](/concepts/keyspace/), [Capacity](/operations/capacity/) | | `internal/protocol` | Fixed frame encoding/decoding, payload bounds, requests/responses, endpoint URI parsing | [Framing](/protocol/framing/), [Connection URIs](/protocol/uri/), [Wire values](/protocol/wire-values/) | | `internal/server/tcp.go` | Amaquet connection lifecycle, TLS, request multiplexing, cancellation, authorization, limits, Pub/Sub events | [Protocol overview](/protocol/overview/), [Framing](/protocol/framing/), [Authentication](/protocol/authentication/) | | `internal/server/dispatcher.go` | Commands, type operations, journaling boundary, deterministic replay arguments, chunked blobs | [Commands](/protocol/commands/), [Type operations](/data-types/operations/), [Persistence](/concepts/persistence/) | | `internal/server/factory.go` | Direct wire decoding and `CREATE` factories | [Wire values](/protocol/wire-values/), [Type operations](/data-types/operations/) | | `internal/compression` and `internal/server/value_compression.go` | Adaptive codecs, canonical encodings, transparent materialization | [Adaptive compression](/concepts/compression/) | | `internal/persistence` | AMQTAOF1 migration, AMQTAOF2 transactions, replay, checkpoint, compaction, restore | [Persistence](/concepts/persistence/), [Backups](/operations/backups/) | | `internal/admin/state.go` | Organization, members, credentials, TOTP, RBAC, API keys, audit, crash-safe state | [Security](/concepts/security/), [Admin API](/reference/admin-api/) | | `internal/admin/http.go` | Admin routes, sessions, metrics, and security headers | [Admin API](/reference/admin-api/), [Observability](/operations/observability/) | | `internal/security` | Ed25519 identity generation, loading, signing, verification, fingerprinting | [TLS and identity](/operations/tls/) | | `internal/types` | The 91 registered value implementations and snapshots | [Data types](/data-types/), [Type operations](/data-types/operations/) | ### User-facing programs and libraries The supported entrypoints are the public Go client and the four command-line programs. Container assets package the same server rather than introducing a separate runtime implementation. | Source | User-facing surface | Documentation | | ------------------------------ | ------------------------------------------------------------------------------- | ------------------------------------------------------- | | `pkg/amaquet` | Concurrent Go client, TLS, protocol negotiation, commands, subscriptions, blobs | [Go client](/reference/go-client/) | | `cmd/amaquet-cli` | Generic command-line client and JSON input modes | [CLI](/reference/cli/) | | `cmd/amaquet-keygen` | Ed25519 identity key generation | [CLI](/reference/cli/), [TLS and identity](/operations/tls/) | | `cmd/amaquet-restore` | Verified offline checkpoint installation | [CLI](/reference/cli/), [Backups](/operations/backups/) | | `Dockerfile` and Compose files | Container build, local deployment, TLS production example | [Docker](/operations/docker/) | ### Data-structure implementation files The type catalog is organized by implementation family. Each family page summarizes the structures, while the operation reference and individual type pages define the wire contract. | Source file | Implemented families | Detailed pages | | ------------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------- | | `scalars.go` | strings, integers, exact decimals, UUID, symbols, time values | [Scalars](/data-types/scalar/) | | `collections.go` | lists, arrays, deque/ring, sets, maps, tuples | [Collections](/data-types/collections/) | | `structured.go` | JSON, opaque encodings, records, matrices, tensors | [Structured values](/data-types/structured/) | | `queues.go`, `streams.go` | queues, streams/groups, Pub/Sub, retained topics, event logs | [Queues and messaging](/data-types/queues-and-messaging/) | | `bits.go` | bitmaps, bit fields, adaptive Roaring containers | [Bit structures](/data-types/bit-structures/) | | `probabilistic.go` | HLL, Bloom/Cuckoo filters, CMS, Top-K, t-digest | [Probabilistic structures](/data-types/probabilistic/) | | `timeseries.go` | time series, counter/gauge series, histograms, aggregation | [Time series](/data-types/time-series/) | | `geo.go` | points, boxes, polygons, grid-backed spatial index | [Geospatial](/data-types/geospatial/) | | `vectors.go` | vector metrics, exact search, quantization, HNSW | [Vectors](/data-types/vectors/) | | `indexes.go` | B-tree, hash/radix/trie/inverted/secondary indexes, document fields, adjacency | [Indexes and search](/data-types/indexes-and-search/) | | `sync.go`, `network.go` | counters, leases, locks, barriers, semaphores, token buckets, CIDR, URL | [Graph, synchronization, and network](/data-types/graph-synchronization-and-network/) | ### Deliberate boundaries These exclusions are part of the current product contract. They prevent deployment or client documentation from implying distributed, relational, or Redis-compatible behavior that the implementation does not provide. - Amaquet is single-node and has one logical keyspace. It does not implement clustering, replication, logical databases, SQL, Redis RESP compatibility, or an automatic query planner. - Index values are explicit user-managed objects. Writing a document does not automatically update a separate index. - AOF recovery covers journaled database mutations. HTTP cookie sessions, live connections, Pub/Sub subscribers, and synchronization ownership are process state and are not restored. - `admin.json` is the control-plane store. It is not part of the user keyspace or the AOF. - Engine memory accounting is approximate stored-data accounting, not an RSS hard limit. ### Automated documentation invariants `python3 docs/scripts/check_docs.py` checks the documentation against source-derived inventories. It verifies all registered types, catalog operation counts, test/fuzzer pages, top-level commands, protocol opcodes, configuration fields, environment overrides, admin routes, public Go client methods, sidebar entries, frontmatter, explanatory prose for every section, and local links. The Astro production build additionally checks every generated internal link and asset.