(Go) Spelunk

Spelunk is a Golang library for extracting secrets from various sources (Kubernetes, Vault, env vars, files) using a unified URI-based string we are calling Secret Coordinates. Here are some example of coordinates:

# Secret from the namespace `ns`,
# stored inside the secret `my-team-secret`
# at data key `the-key`
k8s://ns/my-team-secret/the-key

# Secret provided in the form
# of base64-encoded string
base64://bXktYmlnLXNlY3JldAo=

# Secret stored in a JSON
# file at a specific field
file://kafka-credentials.json?jp=$.kafka.password

Spelunk simplifies the access to secrets by just providing the coordinates for "digging up" configuration values in cloud-native CLI tools and applications.

Its primary application is command line tools, but... you do you! Users point at a secret from any source, providing the right coordinates: your tool/service/software can use Spelunk to adapt dynamically and fetch the secret.

With a single library, the source of secrets is flexible and adapts to your environment, situation and/or needs.

Spelunk can be configured to support more Sources, and users can apply Modifiers to "prepare" the secret in the exact way they need it.

Multi-Module Architecture (since 2.x)

Starting with version v2.x, Spelunk implements a highly efficient Go Multi-Module Workspace architecture.

Previously, importing Spelunk pulled down every single heavy SDK dependency (including the AWS, GCP, Azure, HashiCorp Vault, Kubernetes, and 1Password SDKs) regardless of whether you used them or not.

From v2.0.0 onwards:

• Ultra-Lean Core: The root core module github.com/detro/spelunk/v2 is completely bare and carries virtually zero production dependencies.
• Pay Only For What You Use: Sibling plugins are completely decoupled into isolated submodules. Heavyweight dependencies are only pulled down by Go if you explicitly choose to import and register their corresponding plugin.

Get started

Add the core library to your project:

# Pull the ultra-lean core library
go get github.com/detro/spelunk/v2

# Pull only the specific plugins you want to use
go get github.com/detro/spelunk/plugin/source/kubernetes/v2

Setup a new Spelunker and start digging up secrets:

package main

import (
	"context"
	"github.com/detro/spelunk/v2"
	"github.com/detro/spelunk/v2/types"
	"github.com/detro/spelunk/plugin/source/kubernetes/v2"
	v1 "k8s.io/client-go/kubernetes/typed/core/v1"
)

// Initialize the Kubernetes client...
k8sClient, err := v1.NewForConfig(restConfig)

// Create a Spelunker
spelunker := spelunk.NewSpelunker(
	kubernetes.WithKubernetes(k8sClient),
)

// Get coordinates to a secret from one of many supported sources:
// from Kubernetes... 
coord, err := types.NewSecretCoord("k8s://my-namespace/my-secret/my-data-key")
// ... or from plain text (please don't!)
coord, err := types.NewSecretCoord("plain://MY_PLAINTEXT_SECRET")
// ... or from a local file
coord, err := types.NewSecretCoord("file://secrets.json?jp=$.kafka.password")
// ... or from environment variable
coord, err := types.NewSecretCoord("env://GITHUB_PRIVATE_TOKEN")

// Dig-up secrets!
secret, _ := spelunker.DigUp(ctx, coord)

Examples

Find some useful /examples directory for how to use spelunk with various libraries for configuration or command line arguments parsing.

built-in vs plug-in

Spelunk comes with a bunch of features: Sources and Modifiers, the role of which is explained below. Some are built-in, and a new spelunk.Spelunker instance comes with those enabled; others are plug-in, and you will have to enable them as SpelunkerOption provided at construction time:

package main

import (
	"github.com/detro/spelunk/v2"
	"github.com/detro/spelunk/plugin/source/kubernetes/v2"
	"github.com/detro/spelunk/plugin/source/vault/v2"
)

_ = spelunk.NewSpelunker(
	kubernetes.WithKubernetes(k8sClient),
	vault.WithVault(vaultClient),
)

Key Types

spelunk.Spelunker is the entry point type, and it does its job using the following types.

Coordinates (SecretCoord)

This is the starting point: take a string containing Secret Coordinates as documented above, and use types.NewSecretCoord to turn it into a SecretCoord.

This is a generic, secret-type-agnostic representation of how to find a secret. And it's all that Spelunker needs to dig-up the secret.

From user input to SecretCoord

SecretCoord implements encoding.TextUnmarshaler, so it can be created through the unmarshalling of command-line user input, through json.Unmarshal and any other type-aware process.

For example, when using the awesome Kong library:

package main

import "github.com/detro/spelunk/v2"

type CLI struct {
	Password spelunk.SecretCoord `name:"password" short:"p" help:"your password"`
	// ...
}

Sources (SecretSource)

Sources are places out of which a secret can be "dug-up". Some are built-in to spelunk.Spelunker, others are plug-in and need to be enabled.

Source (of Secrets)	Type (scheme)	Available as	Status	Doc
Environment Variables	env://	built-in	✅	link
File	file://	built-in	✅	link
Plaintext	plain://	built-in	✅	link
Base64 encoded	base64://	built-in	✅	link
Kubernetes Secrets	k8s://	plug-in	✅	link
Vault	vault://	plug-in	✅	link
AWS Secrets Manager	aws://	plug-in	✅	link
GCP Secrets Manager	gcp://	plug-in	✅	link
Azure Key Vault	az://	plug-in	✅	link
1Password	op://	plug-in	✅	link
Bitwarden	bw://	plug-in	👷1	link
Keeper	kp://	plug-in	👷1	link
LastPass	lp://	plug-in	❌ 2
Dashlane	dl://	plug-in	❌ 2

Modifiers (SecretModifier)

Modifiers are optional behaviour applied to a secret after it has been dug-up by Spelunk. It can be seen as a function in the mathematical sense:

$$ Modifier(SecretVal, Input) = ModifiedSecVal $$

Each modifier is applied in the same order provided in the secret coordinates:

<type>://<location>?mod1=A&mod2=B&mod1=C

will result in this sequence:

• Spelunker digs-up the secret <value> of type <type> from the <location>
• mod1 takes the <value> and applies mod1(<value>, A) = <value_A>
• mod2 takes the <value_A> and applies mod2(<value_A>, B) = <value_A_B>
• mod1 takes the <value_A_B> and applies mod1(<value_A_B>, C) = <value_A_B_C>
• client code is returned the final <value_A_B_C>

Modifier (of Secrets)	Type (query)	Available as	Status	Doc
Base64 encoder	?b64	built-in	✅	link
Base64 encoder (alias for ?b64)	?b64e	built-in	✅	link
Base64 decoder	?b64d	built-in	✅	link
JSONPath extractor	?jp=<JSONPath>	plug-in	✅	link
XPath extractor	?xp=<XPath>	plug-in	✅	link
YAML JSONPath extractor	?yp=<JSONPath>	plug-in	✅	link
TOML JSONPath extractor	?tp=<JSONPath>	plug-in	✅	link
SHA-2/3 / BLAKE-2/3 / ... hasher	TBD	plug-in	⏳

Contributing

If you are interested in contributing (for example, you have a brilliant idea for a plug-in), we have some contribution guidelines.

License

This project is shared under the MIT license.

Links

• Architecture documentation: understand how Spelunk works internally
• Contribution guidelines: setting some ground rules
• Agents documentation: helps LLM-agent augmented developers in their contribution journey
• Changelog: track the evolution of the project

Footnotes

1. Untested: Looking for contributors with access to a test account/vault! ↩ ↩²

2. Not Implemented: Not implemented due to the lack of maintained Go SDK, no suitable REST API, and no local Testcontainers for simulation. ↩ ↩²