FlowtandBriefAI · Software · ResearchMeet Flowtand

Hugging Face adds mutable S3-style Storage Buckets to the Hub

Announcement

Hugging Face

Browse Infrastructure topic page →

Hugging Face announced Storage Buckets, a non-versioned, S3-like storage container on the Hub for mutable ML artifacts such as checkpoints, processed shards, and agent traces. The company says Buckets are backed by Xet chunk-based deduplication, can be private or public, are reachable via the hf CLI, Python, JavaScript and fsspec, and support pre-warming data near compute in AWS and GCP regions.

What Hugging Face announced and how it works

Hugging Face published a blog post describing Storage Buckets as a non-versioned, S3-like storage container on the Hub, intended for the mutable side of ML work: checkpoints, optimizer states, processed shards, logs, and agent traces. Buckets live under a user or organization namespace, carry standard Hub permissions, and can be private or public. According to the post, they are reachable through the hf CLI (create, sync, copy, remove, list), the Python client since huggingface_hub v1.5.0, the JavaScript package since v2.10.5, and HfFileSystem, the fsspec-compatible interface. The stated mechanism is Xet, Hugging Face's chunk-based backend, which splits files into chunks and deduplicates overlapping content, so successive checkpoints or near-identical datasets skip bytes already stored. The post also describes pre-warming, which moves hot data near the compute region, starting with AWS and GCP. It says Buckets are included in existing Hub storage plans. These are publisher claims; the excerpt contains no independent benchmarks.

A scenario for an ML platform engineer

A team running distributed training could consider Buckets as a staging area for per-run checkpoints and processed shards, keeping versioned model and dataset repos reserved for deliverables. In that setup, a training script would sync its local checkpoint directory to a Bucket path under the organization namespace, and downstream jobs or notebooks would read selected files back via the fsspec interface rather than a full clone. A platform engineer could try using the dry-run or saved-plan sync options to review what would move before a large transfer. Pre-warming might be worth evaluating for jobs whose compute sits in a different region from the data, since the post frames it as a way to avoid cross-region reads on every job start. This is a plausible experiment, not a documented integration for any particular cluster or pipeline.

How to trial it and measure

A team could start with one non-critical artifact stream, such as weekly checkpoints from a single training job. First record a baseline: transfer time, storage consumed, and where the bytes physically sit for the current approach. Then create a private bucket, sync the directory with --dry-run first to inspect the plan, and repeat the same sync under similar conditions. Compare wall-clock transfer time and stored footprint against the baseline, keeping the same data and network conditions. Failure checks should include verifying that overwritten and deleted files behave as expected, that permissions prevent unintended public exposure, and that reads from the fsspec path return correct contents. Because the excerpt offers no published benchmarks for these workloads, any numbers gathered are specific to the team's own setup. A human should review sync plans before applying large transfers and confirm that no regulated or embargoed data leaves its permitted region.

Trade-offs, prerequisites and when to skip it

Buckets trade version history for mutability, so they are a poor fit for anything that needs audit trails or reproducibility by version; the post itself suggests moving stable deliverables into versioned repos. Prerequisites include the named client versions, Hub authentication, and for pre-warming, compute in supported regions (AWS and GCP at launch, with others described as coming). Whether deduplication actually reduces transfer time or storage for a given artifact family depends on how much content overlaps, which the excerpt asserts but does not quantify. Billing behavior is described only in general terms for Enterprise deduplicated storage, and direct transfers between Buckets and repos are roadmap items, not current features. A team should not adopt Buckets for archival, long-term cold storage, or anything requiring immutable versioning — the post notes a cold-storage tier is discussed but not on the short-term roadmap.

Context and limitations

  • Based on a short source excerpt; the full article was not analyzed.
  • Efficiency and speed claims come from Hugging Face and were not independently verified here.
  • Direct Buckets-to-repo transfers are described as roadmap, not as available today.
  • Based on bounded source text, not independent testing. Practical suggestions are editorial proposals.