GitHub - artob/bitcache: 🦀 Bitcache is a distributed content-addressable storage (CAS) system.
Bitcache is a distributed content-addressable storage (CAS) system.
[
] | [
] | [
] | [
] | [
] | [
]
✨ Features
Available both as the command-line tool
and a polyglot library.
Polyglot software also (soon!) for Dart, Python, Ruby, Rust, and TypeScript.
Cuts red tape: 100% free and unencumbered public domain software.
⬇️ Installation
Installation of the CLI
Installation via
cargo binstall -y bitcache
Installation via
mise use -g github:artob/bitcacheInstallation via
cargo install bitcache --locked --features=cliInstallation of the Library
Installation for Rust from Crates.ioInstallation from
cargo add bitcacheInstallation for JavaScript/TypeScript from NPMInstallation from
npm install bitcache.js bun add bitcache.js pnpm add bitcache.js yarn add bitcache.jsInstallation for Dart from Pub.devInstallation from
dart pub add bitcache flutter pub add bitcacheInstallation for Python from PyPIInstallation from
pip install -U bitcache uv add bitcache poetry add bitcache pdm add bitcacheInstallation for Ruby from RubyGemsInstallation from
gem install bitcache bundle add bitcache👉 Examples
📚 Reference
Command-Line Interface
$ bitcache --helpBitcache is a distributed content-addressable storage (CAS) system.Usage: bitcache [OPTIONS] [COMMAND]General commands: id Compute the BLAKE3 hash of the given file(s), or of stdin help Print this message or the help of the given subcommand(s)Current repository commands (`$BITCACHE_URL`, default `./.bitcache/`): init Initialize a new repository in `./.bitcache/` list List the IDs of the blobs in the repository, in ascending order has Check whether the repository contains blob(s) with the given ID(s) get Fetch blob(s) from the repository, writing their contents to stdout put Store the given file(s) into the repository as blob(s) rm Remove blob(s) with the given ID(s) from the repository clear Remove all blobs from the repository export Export all blobs in the repository into a tarballRemote repository commands: push Copy blobs missing from the given remote repositories to them pull Copy blobs missing from the current repository from the given remotes sync Synchronize with the given remote repositories, in both directionsOther commands: compact Compact the repository's physical storageOptions:: --color <COLOR> Set the color output mode [default: auto] [possible values: auto, always, never] -d, --debug Enable debugging output --license Show license information -v, --verbose... Enable verbose output (may be repeated for more verbosity) -V, --version Print version information -C, --cwd <DIR> Change to this directory before executing the command -h, --help Print help (see a summary with '-h')
- Remove all blobs from the repository
- Compact the repository's physical storage
- Export all blobs in the repository into a tarball
- Fetch blob(s) from the repository, writing their contents to stdout
- Check whether the repository contains blob(s) with the given ID(s)
- Compute the BLAKE3 hash of the given file(s)
- Initialize a new repository in ./.bitcache/
- List the IDs of the blobs in the repository, in ascending order
- Copy blobs missing from the current repository from the given remotes
- Copy blobs missing from the given remote repositories to them
- Store the given file(s) into the repository as blob(s)
- Remove blob(s) with the given ID(s) from the repository
- Synchronize with the given remote repositories, in both directions
bitcache clear
$ bitcache clear --helpRemove all blobs from the repository.As a safety measure, this requires the `--force` flag; without it, nothing is removed and the command exits with a usage error.Usage: bitcache clear [OPTIONS]Options: --color <COLOR> Set the color output mode [default: auto] [possible values: auto, always, never] -f, --force Actually perform the operation; without this, nothing is removed -d, --debug Enable debugging output -v, --verbose... Enable verbose output (may be repeated for more verbosity) -C, --cwd <DIR> Change to this directory before executing the command -h, --help Print help (see a summary with '-h')bitcache compact
$ bitcache compact --helpCompact the repository's physical storage.Filesystem repositories rewrite stored blobs using the requested compression scheme and clean up orphaned temporary artifacts. Other repository backends may perform no maintenance.Usage: bitcache compact [OPTIONS]Options: --color <COLOR> Set the color output mode [default: auto] [possible values: auto, always, never] --compress <SCHEME> The target compression scheme for stored blobs. One of `none`, `xz`, `xz:fast`, or `xz:best` (`xz` is an alias for `xz:fast`). Defaults to the `compress` directive of the `[bitcache.compact]` config section, or else `xz`. -d, --debug Enable debugging output -v, --verbose... Enable verbose output (may be repeated for more verbosity) -C, --cwd <DIR> Change to this directory before executing the command -h, --help Print help (see a summary with '-h')bitcache export
$ bitcache export --helpExport all blobs in the repository into a tarball.Without `--output`, the tar stream is written to stdout, so it can be piped to `xz`, `bzip2`, `gzip`, etc.Usage: bitcache export [OPTIONS]Options: --color <COLOR> Set the color output mode [default: auto] [possible values: auto, always, never] -o, --output <FILE> The path to the tarball file to create (default: stdout) -d, --debug Enable debugging output -v, --verbose... Enable verbose output (may be repeated for more verbosity) -C, --cwd <DIR> Change to this directory before executing the command -h, --help Print help (see a summary with '-h')bitcache get
$ bitcache get --helpFetch blob(s) from the repository, writing their contents to stdout.IDs may be given as unambiguous hexadecimal prefixes: each prefix resolves to the first matching blob ID in the repository.Exits with a nonzero status unless all of the given blobs were found in the repository.Usage: bitcache get [OPTIONS] [IDS]...Arguments: [IDS]... The IDs (or unambiguous ID prefixes) of the blob(s) to fetchOptions: --color <COLOR> Set the color output mode [default: auto] [possible values: auto, always, never] -n, --lines <COUNT> Print only the first COUNT lines of each blob -d, --debug Enable debugging output -f, --format <FORMAT> The output format: `raw` (the default) or `base64` Possible values: - raw: The blob's raw contents - base64: ASCII-armored (Base64-encoded) contents, one line per blob [default: raw] -o, --output <FILE> Write the output to this file instead of stdout. With a single blob, raw output, and no line limit, filesystem repositories reflink uncompressed blobs to the output file on supporting filesystems, avoiding a data copy. -v, --verbose... Enable verbose output (may be repeated for more verbosity) -C, --cwd <DIR> Change to this directory before executing the command -h, --help Print help (see a summary with '-h')bitcache has
$ bitcache has --helpCheck whether the repository contains blob(s) with the given ID(s).With `--verbose`, prints `true` or `false` for each ID.Exits with a nonzero status unless all of the given blobs were found in the repository.Usage: bitcache has [OPTIONS] [IDS]...Arguments: [IDS]... The IDs of the blob(s) to check forOptions: --color <COLOR> Set the color output mode [default: auto] [possible values: auto, always, never] -d, --debug Enable debugging output -v, --verbose... Enable verbose output (may be repeated for more verbosity) -C, --cwd <DIR> Change to this directory before executing the command -h, --help Print help (see a summary with '-h')bitcache id
$ bitcache id --helpCompute the BLAKE3 hash of the given file(s), or of stdin.Prints the ID each file would have as a blob, one per line, without accessing or modifying any repository. With no files (or with `-`), reads from standard input.Usage: bitcache id [OPTIONS] [FILES]...Arguments: [FILES]... The paths to the file(s) to hash (`-` or none for stdin)Options: --color <COLOR> Set the color output mode [default: auto] [possible values: auto, always, never] -f, --format <FORMAT> The format to use for the hash output Possible values: - hex: Hexadecimal (aka Base16) - base58: Base58 -d, --debug Enable debugging output -v, --verbose... Enable verbose output (may be repeated for more verbosity) -C, --cwd <DIR> Change to this directory before executing the command -h, --help Print help (see a summary with '-h')bitcache init
$ bitcache init --helpInitialize a new repository in `./.bitcache/`.Creates an empty repository in the `./.bitcache/` directory of the current working directory; `$BITCACHE_URL` is ignored. The given options are recorded in the created `.bitcache/config.toml`; an existingconfiguration file is never overwritten.Usage: bitcache init [OPTIONS]Options: --color <COLOR> Set the color output mode [default: auto] [possible values: auto, always, never] --hashing <ALGORITHM> The content-hashing algorithm to use (only `blake3`) --capacity <COUNT> A capacity hint for how many blobs will be stored. A count with an optional `K`, `M`, `B`, or `T` suffix (e.g., `100M` for one hundred million). -d, --debug Enable debugging output --encoding <FORMAT> The default encoding for displaying blob IDs Possible values: - hex: Hexadecimal (aka Base16) - base58: Base58 -v, --verbose... Enable verbose output (may be repeated for more verbosity) --without-git Skip creating the `.gitattributes` and `.gitignore` files -C, --cwd <DIR> Change to this directory before executing the command -h, --help Print help (see a summary with '-h')bitcache list
$ bitcache list --helpList the IDs of the blobs in the repository, in ascending order.With `--verbose` (repeatable), appends further tab-separated columns to each line: the blob's byte size, media type, creation timestamp, last-update timestamp, last-access timestamp, and expirationtimestamp.Usage: bitcache list [OPTIONS] [PREFIX]Arguments: [PREFIX] List only IDs whose hexadecimal encoding begins with this prefixOptions: --color <COLOR> Set the color output mode [default: auto] [possible values: auto, always, never] -f, --format <FORMAT> The format to use for the hash output Possible values: - hex: Hexadecimal (aka Base16) - base58: Base58 -a, --after <ID> List only IDs ordered strictly after this one -d, --debug Enable debugging output -n, --limit <COUNT> List at most this many IDs -v, --verbose... Enable verbose output (may be repeated for more verbosity) -C, --cwd <DIR> Change to this directory before executing the command -h, --help Print help (see a summary with '-h')bitcache pull
$ bitcache pull --helpCopy blobs missing from the current repository from the given remotes.Every blob present in a remote repository but absent from the current repository is copied into the current repository.Usage: bitcache pull [OPTIONS] [URLS]...Arguments: [URLS]... The URLs of the remote repositories to pull fromOptions: --color <COLOR> Set the color output mode [default: auto] [possible values: auto, always, never] -d, --debug Enable debugging output -v, --verbose... Enable verbose output (may be repeated for more verbosity) -C, --cwd <DIR> Change to this directory before executing the command -h, --help Print help (see a summary with '-h')bitcache push
$ bitcache push --helpCopy blobs missing from the given remote repositories to them.Every blob present in the current repository but absent from a remote repository is copied to that remote repository.Usage: bitcache push [OPTIONS] [URLS]...Arguments: [URLS]... The URLs of the remote repositories to push toOptions: --color <COLOR> Set the color output mode [default: auto] [possible values: auto, always, never] -d, --debug Enable debugging output -v, --verbose... Enable verbose output (may be repeated for more verbosity) -C, --cwd <DIR> Change to this directory before executing the command -h, --help Print help (see a summary with '-h')bitcache put
$ bitcache put --helpStore the given file(s) into the repository as blob(s).Prints the ID of each stored blob, one per line. Since blobs are content addressed, storing already-present content is harmless: the blob is simply retained with the same ID.Usage: bitcache put [OPTIONS] [FILES]...Arguments: [FILES]... The paths to the file(s) to storeOptions: --color <COLOR> Set the color output mode [default: auto] [possible values: auto, always, never] -f, --format <FORMAT> The format to use for the hash output Possible values: - hex: Hexadecimal (aka Base16) - base58: Base58 --compress <SCHEME> The compression scheme for physically storing the blob(s). One of `none`, `xz`, `xz:fast`, or `xz:best` (`xz` is an alias for `xz:fast`). Defaults to the `compress` directive of the `[bitcache.put]` config section, or else `none`. -d, --debug Enable debugging output --ttl <DURATION> Expire the stored blob(s) after the given duration. Accepts a plain number of seconds (e.g. "90") or a human-friendly duration (e.g. "90s", "2m30s", "1h", "7d"). Requires a repository backend that supports blob expiration (e.g., filesystem, Turso, or Valkey); exits with an error otherwise. --media-type <TYPE> Store an explicit media type (MIME type) for the blob(s) -v, --verbose... Enable verbose output (may be repeated for more verbosity) -C, --cwd <DIR> Change to this directory before executing the command -h, --help Print help (see a summary with '-h')bitcache rm
$ bitcache rm --helpRemove blob(s) with the given ID(s) from the repository.Exits with a nonzero status unless all of the given blobs were found in the repository.Usage: bitcache rm [OPTIONS] [IDS]...Arguments: [IDS]... The IDs of the blob(s) to removeOptions: --color <COLOR> Set the color output mode [default: auto] [possible values: auto, always, never] -d, --debug Enable debugging output -v, --verbose... Enable verbose output (may be repeated for more verbosity) -C, --cwd <DIR> Change to this directory before executing the command -h, --help Print help (see a summary with '-h')bitcache sync
$ bitcache sync --helpSynchronize with the given remote repositories, in both directions.Equivalent to a `pull` followed by a `push` for each given remote repository: afterwards, the current repository and every given remote repository all contain the union of their blobs.Usage: bitcache sync [OPTIONS] [URLS]...Arguments: [URLS]... The URLs of the remote repositories to synchronize withOptions: --color <COLOR> Set the color output mode [default: auto] [possible values: auto, always, never] -d, --debug Enable debugging output -v, --verbose... Enable verbose output (may be repeated for more verbosity) -C, --cwd <DIR> Change to this directory before executing the command -h, --help Print help (see a summary with '-h')Configuration File
Storage Adapters
URL SchemeAdapter Cratelsgetputrmclearfile:bitcache-fs✓✓✓✓✓git:bitcache-git✓✓✓✓✓heap:bitcache-heap✓✓✓✓✓opendal+azblob:bitcache-opendal✓✓✓✓xopendal+fs:bitcache-opendal✓✓✓✓✓opendal+ftp:bitcache-opendal✓✓✓✓✓opendal+memcached:bitcache-opendalx✓✓✓xopendal+memory:bitcache-opendal✓✓✓✓✓opendal+mongodb:bitcache-opendalx✓✓✓xopendal+gcs:bitcache-opendal✓✓✓✓✓opendal+http:bitcache-opendalx✓xxxopendal+redis:bitcache-opendalx✓✓✓xopendal+s3:bitcache-opendal✓✓✓✓✓opendal+sftp:bitcache-opendal✓✓✓✓?opendal+sled:bitcache-opendal✓✓✓✓✓redis:bitcache-valkey✓✓✓✓✓sqlite:bitcache-turso✓✓✓✓✓valkey:bitcache-valkey✓✓✓✓✓File System Adapter
BITCACHE_URL=file:.bitcacheBITCACHE_URL=file:/tmp/bitcacheGit Adapter
BITCACHE_URL=git://github.com/asimov-datasets/gutenberg.org.gitHeap (Memory) Adapter
BITCACHE_URL=heap:OpenDAL Adapter
Azure Blob Storage (
) ServiceBITCACHE_URL=opendal+azblob://my-containerConfiguration for Floci AZConfiguration for
BITCACHE_URL=opendal+azblob://my-container?endpoint=http://localhost:4577/devstoreaccount1&skip_signature=trueFile System Service (
)BITCACHE_URL=opendal+fs:///tmp/bitcacheFTP Service (
)BITCACHE_URL=opendal+ftp://localhostConfiguration for pyftpdlibConfiguration for
BITCACHE_URL=opendal+ftp://127.0.0.1:2121?user=anonymous&[email protected] Cloud Storage Service (
)BITCACHE_URL=opendal+gcs://my-bucket/my-rootConfiguration for Floci GCPConfiguration for
BITCACHE_URL=opendal+gcs://my-bucket/my-root?endpoint=http://localhost:4588&skip_signature=trueHTTP Service (
)BITCACHE_URL=opendal+http://localhost:8000Memcached Service (
)BITCACHE_URL=opendal+memcached://localhost:11211Memory Service (
)BITCACHE_URL=opendal+memory://MongoDB Service (
)BITCACHE_URL=opendal+mongodb://localhost:27017/my-database/my-collectionRedis Service (
)BITCACHE_URL=opendal+redis://localhost:6379S3 Service (
)BITCACHE_URL=opendal+s3://my-bucketConfiguration for Floci AWSConfiguration for
BITCACHE_URL=opendal+s3://my-bucket?region=us-east-1&endpoint=http://localhost:4566&skip_signature=trueSFTP Service (
)BITCACHE_URL=opendal+sftp://my-hostSled Service (
)BITCACHE_URL=opendal+sled:///tmp/bitcacheMiscellaneous ServicesOpenDAL supports dozens more additional
; however, if we haven't validated them yet, we won't have a feature flag for them nor URL scheme support in Bitcache directly. (Submit a pull request to add support for your favorite service!)
Turso (aka SQLite) Adapter
BITCACHE_URL=sqlite:/tmp/bitcache.dbValkey (fka Redis) Adapter
BITCACHE_URL=valkey://localhost:6379👨💻 Development
git clone https://github.com/artob/bitcache.git
———