"The time has come," the Walrus said, "To talk of many things: Of shoes -- and ships -- and sealing-wax -- Of cabbages -- and kings -- And why the sea is boiling hot -- And whether pigs have wings."
Decentralized object storage built on Walrus and Sui.
Oyster provides a Web2-friendly HTTP API (buckets, blobs, API keys) backed by Walrus for decentralized blob storage and Sui for on-chain state. A companion service, Pearl, handles deterministic key derivation and transaction signing in isolation.
+---------------------+
| Oyster (HTTP) |
| :3000 (Axum) |
| |
| Buckets / Blobs |
| API Keys / Auth |
| Extension Worker |
+----------+-----------+
|
+---------+-------+--------+-----------+
| | | |
v v v v
+---------+ +--------+ +------------+ +--------+
|Database | | Walrus | | Pearl | | Sui |
|(SQLite | | Nodes | | (gRPC) | | RPC |
| or PG) | | | | :50051 | | |
+---------+ +--------+ +------------+ +--------+
| Crate | Type | Purpose |
|---|---|---|
crates/oyster |
HTTP server (lib + bin) | Object storage API, blob store abstraction, extension worker |
crates/pearl |
gRPC server (lib + bin) | Wallet custody, transaction signing, deterministic key derivation |
crates/oyster-cli |
CLI binary | Command-line client for the Oyster HTTP API |
crates/oyster-e2e-tests |
Test crate | Full-stack E2E tests (Sui + Walrus + Pearl + Oyster) |
- Users interact with Oyster's HTTP API to store and retrieve blobs.
- Oyster manages buckets, blob metadata, API keys, and content-addressed deduplication in its database.
- When configured for Walrus, Oyster encodes blob data with the Walrus SDK, builds Sui
Programmable Transaction Blocks (PTBs) for
reserve_space,register_blob, andcertify_blob, then delegates signing to Pearl. - Pearl derives Ed25519 keys deterministically from a master seed via HKDF-SHA256 and signs transactions on request. It is fully stateless -- no database, no balance tracking.
- A background extension worker (
oysterd extend) monitors blob expiry and extends storage on-chain.
Oyster is an Axum-based HTTP server with OpenAPI documentation served at /api/docs. The server
binary is oysterd, which supports two subcommands: oysterd serve (default) starts the HTTP
API, and oysterd extend runs the blob extension background worker.
- Account -- Top-level identity. Has API keys and a Pearl-derived wallet (keyed by
account ID). Each account has a
max_unencoded_bytescap (default 5 × 10⁹ bytes) that gates uploads against on-chain encoded usage; raise or lower it via the adminPUT /accounts/{account_id}/max-storageroute. - API Key -- Bearer token for authentication. Stored as a Blake2s-256 hash. Multiple keys per account.
- Bucket -- Named container scoped to an account. Bucket names are unique per account.
- Blob -- Content-addressed object stored in a bucket. Identified by
object_id(unique per upload) andblob_id(content hash, shared across duplicate uploads). When stored on Walrus, has apooled_blob_object_idpointing at thePooledBlobon-chain object registered under the account'sStoragePool. - Blob tags -- User-defined key/value metadata attached to a blob. Up to 10 tags per blob,
key ≤128 B, value ≤256 B, total ≤2 KiB; charset
[A-Za-z0-9 +\-=._:/@]. Cascaded on blob delete. Compatible with the S3x-amz-taggingheader and thePutObjectTagging/GetObjectTagging/DeleteObjectTaggingoperations. - Storage pool -- One
StoragePoolSui object per account, created lazily on the first blob write. All of the account'sPooledBlobs reserve capacity from and share the same expiration epoch as this pool.
Admin routes (admin-key authenticated, for managing accounts and credentials):
| Method | Path | Auth | Description |
|---|---|---|---|
POST |
/api/v1/accounts |
Admin Key | Create account |
GET |
/api/v1/accounts |
Admin Key | List accounts with active api-key counts |
PUT |
/api/v1/accounts/{account_id}/max-storage |
Admin Key | Raise/lower per-account max_unencoded_bytes cap (on-chain shrink if needed) |
POST |
/api/v1/accounts/{account_id}/api-keys |
Admin Key | Generate API key for account |
GET |
/api/v1/accounts/{account_id}/api-keys |
Admin Key | List api-key metadata for an account |
DELETE |
/api/v1/accounts/{account_id}/api-keys/{key_id} |
Admin Key | Revoke API key |
POST |
/api/v1/accounts/{account_id}/access-keys |
Admin Key | Create S3 access key |
GET |
/api/v1/accounts/{account_id}/access-keys |
Admin Key | List S3 access keys |
DELETE |
/api/v1/accounts/{account_id}/access-keys/{access_key_id} |
Admin Key | Revoke S3 access key |
Data routes (API key-authenticated):
| Method | Path | Auth | Description |
|---|---|---|---|
GET |
/api/v1/account/wallet |
API Key | Get Pearl wallet address |
POST |
/api/v1/buckets |
API Key | Create bucket |
GET |
/api/v1/buckets |
API Key | List buckets (paginated) |
DELETE |
/api/v1/buckets/{bucket_name} |
API Key | Delete empty bucket (409 if non-empty) |
PUT |
/api/v1/buckets/{bucket_name}/blobs/{key} |
API Key | Upload blob |
GET |
/api/v1/buckets/{bucket_name}/blobs |
API Key | List blobs in bucket (paginated) |
GET |
/api/v1/buckets/{bucket_name}/blobs/{key} |
No | Read blob by bucket and key |
GET |
/api/v1/blobs/by-blob-id/{blob_id} |
No | Read blob by content hash |
PATCH |
/api/v1/buckets/{bucket_name}/blobs/{key}/metadata |
API Key | Update content type |
DELETE |
/api/v1/buckets/{bucket_name}/blobs/{key} |
API Key | Delete blob |
GET |
/api/v1/buckets/{bucket_name}/blobs/{key}/tags |
API Key | List blob tags |
PUT |
/api/v1/buckets/{bucket_name}/blobs/{key}/tags |
API Key | Replace all blob tags |
PATCH |
/api/v1/buckets/{bucket_name}/blobs/{key}/tags |
API Key | Merge blob tags |
DELETE |
/api/v1/buckets/{bucket_name}/blobs/{key}/tags |
API Key | Clear all blob tags |
PUT |
/api/v1/buckets/{bucket_name}/blobs/{key}/tags/{tag_key} |
API Key | Upsert a single tag |
DELETE |
/api/v1/buckets/{bucket_name}/blobs/{key}/tags/{tag_key} |
API Key | Delete a single tag |
GET |
/health |
No | Liveness probe |
GET |
/ready |
No | Readiness probe (checks DB and Pearl connectivity) |
GET |
/metrics |
No | Prometheus metrics |
Admin keys are issued via the oysterd app CLI commands (see below). Pagination is cursor-based
-- pass ?limit=N&cursor=TOKEN to paginate; the response includes next_cursor when more pages
exist. Blob endpoints support If-Match / If-None-Match conditional headers for cache
validation and safe concurrent writes.
Oyster selects a blob store at startup based on environment variables:
| Implementation | When selected | On-chain? | Signing? |
|---|---|---|---|
LocalBlobStore |
No Walrus config | No | No |
DirectWalrusBlobStore |
Pearl + Sui RPC + Walrus system/staking objects | Yes (PTBs) | Via Pearl |
DirectWalrusBlobStore is the production path. It:
- Encodes the blob with the Walrus encoding scheme.
- On the account's first blob write, bundles
create_storage_poolinto the upload PTB and persists the resultingStoragePoolObjectIDon the account row (lazy, first-writer wins). - Builds the upload PTB — optionally prepending
increase_storage_pool_capacity(rounded up to Walrus's 1 MiBBYTES_PER_UNIT_SIZE) when the new blob would exceed the pool's remaining capacity — and callsregister_pooled_blobs. Submits via Pearl. If the register PTB aborts withEInsufficientCapacitybecause of cross-replica drift, Oyster refreshes on-chain pool state, reconciles the DB counters, recomputesgrow_by, and retries the register PTB once. - Uploads slivers to Walrus storage nodes and collects a certificate.
- Builds a
certify_pooled_blobsPTB and submits via Pearl. - Returns the content-addressed
blob_idand thepooled_blob_object_idof the registeredPooledBlobSui object.
Deletes are reference-counted: the on-chain delete_pooled_blob call fires only when the last
reference to a given blob_id is removed from the account.
Run as a separate process with oysterd extend. The worker runs a continuous, idempotent loop:
while there is work to drain it sleeps briefly between cycles; once a cycle finds no candidates
it backs off to a longer idle sleep.
Each cycle:
- Atomically claims a batch of
accountsrows whosepool_end_epoch < current_epoch + POOL_EXTEND_LOOKAHEAD_EPOCHSand whoseextend_attempt_after <= now, stamping each claimed row withextend_attempt_after = now + EXTENSION_CLAIM_COOLDOWN_SECSin the same transaction. The same row cannot be re-claimed (or re-notified) for the cooldown window regardless of the attempt's outcome — a single backoff knob covers both worker-coordination and webhook-spam suppression. - For each claimed pool, builds an
extend_storage_poolPTB (extending byPOOL_EXTEND_EPOCHS), signs via Pearl, and submits to Sui. On success bumpspool_end_epochon the account row. - On insufficient-funds failure, computes the WAL + SUI cost the wallet needs and POSTs an
account.funding_requiredwebhook (see docs/src/guides/webhooks.md). The cooldown TTL acts as the only retry/back-off bookkeeping.
POOL_EXTEND_LOOKAHEAD_EPOCHS and POOL_EXTEND_EPOCHS are raw Walrus epochs — operators pick
values appropriate to the deployed network's epoch duration:
| Network | Epoch length | POOL_EXTEND_LOOKAHEAD_EPOCHS |
POOL_EXTEND_EPOCHS |
|---|---|---|---|
| testnet | ≈ 1 day | 7 | 30 |
| mainnet | ≈ 14 days | 1 | 4 |
When the extension worker hits an insufficient-funds error, Oyster POSTs an
account.funding_required event to the owning app's configured receiver URL. The payload
carries a stable event_id (UUID v4, reused across retries), the Pearl-derived address, and
the WAL/SUI amounts the wallet needs. Delivery uses bounded retries with a circuit breaker.
Each delivery is signed with a per-app Ed25519 keypair; receivers verify with the
X-Oyster-Signature (ed25519=<base64(sig)> over the body) and
X-Oyster-Public-Key-Fingerprint headers. App builders register and rotate the receiver URL
self-service via oyster app webhook set <URL> (or
PUT /api/v1/admin/app/webhook); each call generates a fresh keypair and returns the public
key. See docs/src/guides/webhooks.md for the full payload
schema, retry policy, signature verification, and receiver examples.
Supports SQLite and PostgreSQL via the SQLx Any driver; the backend is determined at runtime by
the DATABASE_URL connection string. Migrations are per backend under crates/oyster/migrations/:
migrations/sqlite/001_initial.sqlmigrations/postgres/001_initial.sql
Tables: accounts, api_keys, s3_access_keys, apps, app_admin_keys, buckets, blobs,
blob_tags, audit_events, dead_letter_orphans.
All configuration is via environment variables. Secrets can alternatively be loaded from files
using CLI flags (useful for Kubernetes secrets, Docker Swarm, or secret managers that mount
files to /run/secrets/).
| Variable | Default | Description |
|---|---|---|
BIND_ADDR |
0.0.0.0:3000 |
HTTP listen address |
DATABASE_URL |
sqlite:oyster.db?mode=rwc |
Database connection string (SQLite or PostgreSQL) |
BLOB_STORE_PATH |
blob_store |
Path for LocalBlobStore |
PEARL_GRPC_URL |
-- | Pearl gRPC address (e.g. http://127.0.0.1:50051) |
PEARL_SERVICE_SECRET |
-- | Shared secret for Pearl auth (required) |
SUI_RPC_URL |
-- | Sui RPC endpoint |
WALRUS_SYSTEM_OBJECT |
-- | Walrus system object ID on Sui |
WALRUS_STAKING_OBJECT |
-- | Walrus staking object ID on Sui |
POOL_INITIAL_EPOCHS_AHEAD |
5 |
Epochs ahead when creating a new StoragePool |
POOL_INITIAL_ENCODED_CAPACITY_BYTES |
1048576 |
Initial reserved capacity for a new pool (1 MiB, Walrus BYTES_PER_UNIT_SIZE) |
POOL_EXTEND_EPOCHS |
5 |
Epochs to extend a StoragePool by |
POOL_EXTEND_LOOKAHEAD_EPOCHS |
7 |
Epochs of runway: claim any pool expiring within current_epoch + this |
EXTENSION_IDLE_SLEEP_SECS |
30 |
Sleep when the extension cycle finds no work |
EXTENSION_BUSY_SLEEP_MS |
250 |
Sleep between extension cycles when there's still work to drain |
EXTENSION_CLAIM_BATCH_SIZE |
100 |
Max pool rows claimed per cycle |
EXTENSION_CLAIM_COOLDOWN_SECS |
60 |
Per-row claim TTL; same knob backs off both re-claim and webhook re-notify |
OYSTER_EXTENSION_METRICS_BIND_ADDR |
0.0.0.0:50053 |
Metrics endpoint for the extension worker |
OYSTER_DEFAULT_AVG_BLOB_SIZE |
10000000 |
Default avg_blob_size (unencoded bytes) for new accounts; inflates the storage-cap ceiling so max_unencoded_bytes is a lower bound for blobs of this size. 0 disables inflation |
| CLI flag | Description |
|---|---|
--pearl-service-secret-file PATH |
Read PEARL_SERVICE_SECRET from a file |
Pearl is a tonic-based gRPC service that derives Ed25519 keypairs deterministically from a master seed and signs Sui transactions. It is fully stateless -- no database, no balance tracking. It is intentionally isolated from business logic; it only knows about key derivation and signing.
Defined in crates/pearl/proto/pearl.proto:
| RPC | Description |
|---|---|
GetAddress |
Derive the Sui address for an account ID |
SignTransaction |
Sign BCS-encoded TransactionData, return signed bytes |
Authentication: all RPCs require a Bearer {secret} in the authorization gRPC metadata
header.
Oyster Pearl Sui RPC
| | |
|-- SignTransaction(tx_data, | |
| account_id) ---------->| |
| |-- derive Ed25519 key |
| | (HKDF-SHA256 from seed) |
| |-- sign tx_data |
|<-- signed_transaction ------| |
| | |
|-- execute_transaction_block -------------------------------->|
|<-- SuiTransactionBlockResponse -------------------------------|
| Variable | Default | Description |
|---|---|---|
PEARL_BIND_ADDR |
0.0.0.0:50051 |
gRPC listen address |
PEARL_SERVICE_SECRET |
-- | Shared secret for service auth (required) |
PEARL_MASTER_SEED |
-- | Hex-encoded master seed for key derivation, version 1 (required, >= 32 bytes) |
PEARL_MASTER_SEED_V<N> |
-- | Additional seed versions for rotation (N >= 2); see Pearl Master-Seed Rotation |
PEARL_ACTIVE_KEY_VERSION |
1 |
Seed version stamped onto newly created accounts |
PEARL_METRICS_BIND_ADDR |
0.0.0.0:50052 |
Prometheus metrics endpoint |
PEARL_TLS_CERT_PATH |
-- | TLS certificate path (optional; must pair with key) |
PEARL_TLS_KEY_PATH |
-- | TLS private key path (optional; must pair with cert) |
| CLI flag | Description |
|---|---|
--pearl-service-secret-file PATH |
Read PEARL_SERVICE_SECRET from a file |
--pearl-master-seed-file PATH |
Read PEARL_MASTER_SEED from a file |
--pearl-master-seed-version-file N:PATH |
Read PEARL_MASTER_SEED_V<N> from a file |
A command-line client for the Oyster HTTP API. Install with cargo install --path crates/oyster-cli.
oyster store photo.jpg --bucket media
oyster read photo.jpg --bucket media -o photo.jpg
oyster list-blobs --bucket media
oyster create-bucket backups
oyster list-buckets
oyster wallet
Configuration is read from ./client.yaml, $XDG_CONFIG_HOME/oyster/client.yaml, or
~/.config/oyster/client.yaml. The file holds a map of named contexts (each
with its own url, api_key, and managed app admin keys); pick one with
--context <name>, OYSTER_CONTEXT, or the file's active_context field. CLI
flags (--url, --api-key) override the selected context. Pass --json for
machine-readable output.
Step-by-step walkthrough for a fresh machine, using the public testnet deployment.
The CLI has no config file by default. Create
~/.config/oyster/client.yaml with just an active context and a URL:
mkdir -p ~/.config/oyster
cat > ~/.config/oyster/client.yaml <<'YAML'
active_context: testnet
contexts:
testnet:
url: https://oyster.testnet.mystenlabs.com/api/v1
YAMLVerify the CLI picks it up:
oyster info
# config: /Users/you/.config/oyster/client.yaml
# url: https://oyster.testnet.mystenlabs.com/api/v1
# key: (not set)Notes on the schema:
urlmust include/api/v1.active_contextselects which context's settings are used. You can override per invocation with--context <name>orOYSTER_CONTEXT=<name>. With a single context in the file, that one is auto-selected even withoutactive_context.- The schema is strict (
deny_unknown_fields) — typos in keys fail to parse.
api_key authenticates the per-account data routes (buckets, blobs,
wallet, …). Skip this step if you only need to manage app admin keys.
Otherwise, have an admin issue a key for your account and add it to the
context:
contexts:
testnet:
url: https://oyster.testnet.mystenlabs.com/api/v1
api_key: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdefThe value is the raw 64-char hex Bearer token returned by
POST /api/v1/accounts/{account_id}/api-keys.
Admin keys are separate from api_key — they authenticate admin
app-management calls (creating accounts, issuing API keys / S3 access keys).
Each admin key is a long-lived 64-char hex Bearer token; rotation is
operator-driven via issue-then-revoke. Ask an Oyster operator with access
to the target deployment to run:
# operator runs this and shares the output with you
oysterd app issue-admin-key <app_id>
# prints: <64-char hex admin key>If your app doesn't exist yet, the operator creates it first — oysterd app new auto-issues a first admin key by default:
oysterd app new --name my-app --contact-email me@example.com
# prints: <app_id>
# prints: <admin_key> <- the first admin key for this appPick a local nickname for the app — it's just a key under apps: in your
config file, so use anything you'll remember (it does not have to match the
server-side app name). Then paste the admin key at the prompt:
oyster app import my-app
# Admin key for my-app: <- prompt; characters are hidden when stdin is a tty
# imported my-app into context testnetThe CLI writes the entry into the active context and replaces the config
file atomically. Your client.yaml now looks like:
active_context: testnet
contexts:
testnet:
url: https://oyster.testnet.mystenlabs.com/api/v1
apps:
my-app:
admin_key: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdefFor non-interactive use (CI, scripts), pipe the key on stdin instead of typing it:
echo "$ADMIN_KEY" | oyster app import my-appAdmin keys are long-lived and do not expire — rotation is voluntary. The recommended pattern is AWS-style two-key overlap: issue a fresh key, swap it in, then revoke the old one once you're sure nothing is still using it.
# operator (issues a new key alongside the old one)
oysterd app issue-admin-key <app_id>
# prints: <new admin_key>
# prints: <new key id> # to stderr — needed later to revoke
# you (replace the local entry with the new key)
oyster app import my-app
# operator (after verifying no in-flight callers depend on the old key)
oysterd app revoke-admin-key <OLD_KEY_ID>oysterd app list-admin-keys <app_id> shows all issued keys (including
revoked ones) so an operator can confirm what is live before revoking.
Add more contexts to the same file and switch between them with --context
or OYSTER_CONTEXT:
active_context: testnet
contexts:
testnet:
url: https://oyster.testnet.mystenlabs.com/api/v1
apps:
my-app:
admin_key: ...
devnet:
url: https://oyster.devnet.mystenlabs.com/api/v1oyster --context devnet info
OYSTER_CONTEXT=devnet oyster infoOnce a context has at least one app with an admin_key, oyster app account
gives you a four-verb workflow over the accounts owned by that app. Pass
--app <name> to pick which app to act through; if the active context has
exactly one app, it's auto-selected.
# list accounts owned by the app (id, name, created_at, active_api_key_count)
oyster app account list
# mint a new account + initial api-key, and immediately set it as the
# context's active api_key (atomic save to client.yaml)
oyster app account create --name alice --activate
# rotate the context's api_key onto a different account in one shot —
# mints a fresh key on the target account, writes it to context.api_key
oyster app account use alice
# list api-key metadata for an account (never returns the bearer secret)
oyster app account keys aliceThere's a server-side cap of 3 active api-keys per account. When use
hits 409 limit, it pivots:
- TTY: prompts inline via
inquireto pick which existing key to revoke, then retries the mint. - Non-TTY (CI, scripts,
--json): you must pre-select the key to revoke with--revoke <key_id>or--revoke-oldest.
oyster app account use alice --revoke-oldest
oyster app account use alice --revoke 0123abcd...For interactive switching, select is a TTY-only picker — it lists the
accounts via inquire and dispatches to use:
oyster app account selectDockerfiles are provided for both services:
docker/Dockerfile.oysterd-- Builds theoysterdbinary. Exposes port 3000 (HTTP API + S3).docker/Dockerfile.pearl-- Builds thepearlbinary. Exposes ports 50051 (gRPC) and 50052 (metrics).
- Rust (edition 2024)
protoc(brew install protobufon macOS)
# Terminal 1: start Pearl
PEARL_MASTER_SEED=<hex-encoded-seed> \
PEARL_SERVICE_SECRET=<shared-secret> \
cargo run -p pearl
# Or load secrets from files (useful for Kubernetes / Docker Swarm):
# cargo run -p pearl -- \
# --pearl-master-seed-file /run/secrets/pearl_master_seed \
# --pearl-service-secret-file /run/secrets/pearl_service_secret
# Terminal 2: start Oyster
PEARL_GRPC_URL=http://127.0.0.1:50051 \
PEARL_SERVICE_SECRET=<shared-secret> \
cargo run -p oyster # runs `oysterd serve` by default
# Terminal 3: create an app (auto-issues a first admin key), then use the admin API
cargo run -p oyster -- app new --name dev --contact-email dev@example.com
# Prints: <app_id>
# Prints: <admin_key> <- first admin key, save it
# Create an account and API key via the admin API
curl -X POST http://localhost:3000/api/v1/accounts \
-H "Authorization: Bearer <admin_key>"
# Returns { "account_id": "...", ... }
curl -X POST http://localhost:3000/api/v1/accounts/<account_id>/api-keys \
-H "Authorization: Bearer <admin_key>"
# Returns { "bearer_token": "...", ... }
# Use the API key for data operations
curl -X POST http://localhost:3000/api/v1/buckets \
-H "Authorization: Bearer <api_key>" \
-H "Content-Type: application/json" \
-d '{"name": "my-bucket"}'
curl -X PUT http://localhost:3000/api/v1/buckets/my-bucket/blobs/hello.txt \
-H "Authorization: Bearer <api_key>" \
-H "Content-Type: text/plain" \
-d 'hello world'
curl http://localhost:3000/api/v1/buckets/my-bucket/blobs/hello.txtTo run the extension worker separately:
cargo run -p oyster -- extend
# or: oysterd extendFor manual full-stack development, use scripts/local-testbed.sh against an already-running
Walrus local testbed. This starts Pearl and Oyster in tmux sessions, creates and funds a test
account, and prints connection details.
# Format + lint (project-specific alias)
chk
# Unit and integration tests
cargo test -p pearl
cargo test -p oyster
# E2E tests (boots Sui + Walrus in-process, ~30s startup). These do not require an external testbed
# as they stand up their own in-process Sui and Walrus clusters.
cargo test -p oyster-e2e-tests
# When changing anything, it's always safe/best-practice to just test everything.
cargo test| Layer | Mechanism | Scope |
|---|---|---|
| Oyster API | Bearer API key (Blake2s-256 hashed) | Per-account |
| Pearl gRPC | Shared service secret | Service-to-service |
| Blob reads | Unauthenticated | Public |
| Private keys | Derived from master seed (HKDF-SHA256), in-memory only | Never stored at rest |
Production hardening:
- Secure
PEARL_MASTER_SEEDvia a secret manager (e.g. AWS Secrets Manager, HashiCorp Vault) - Rehearse seed rotation (
oysterd keys migrate) on testnet so a leaked seed can be retired quickly; see Pearl Master-Seed Rotation - TLS for Pearl gRPC (supported via
PEARL_TLS_CERT_PATH/PEARL_TLS_KEY_PATH) - mTLS or service mesh auth for additional Pearl isolation
- Rate limiting and abuse prevention