Skip to content

Publishing & Registration

publish() — the primary flow

publish() takes a validated local cfdb file, pushes it to your S3 storage, and writes the catalogue entry:

cat = envlib.Catalogue(remotes=[rcg_conn])
cat.publish('era5_temp_v1.cfdb', data_conn, rcg_conn)

Internally, in order: validate → write the derived attributes into the file (envlib_dataset_id, envlib_dataset_version_id, and the auto-populated standard_name) → push the cfdb data → verify the pushed objects → write the catalogue entry → push the catalogue. The data goes up before the entry, so the catalogue never references incomplete data.

  • data_conn is where the dataset lives (an ebooklet.S3Connection with your credentials). Set its db_url to the dataset's public HTTPS location if you host it publicly — that URL rides into the entry as data_url so consumers can open the dataset credential-free. It must be a plain public URL: no user:pass@, no query string (presigned URLs are rejected — their signatures must never land in a catalogue).
  • rcg_conn is the catalogue's own S3 location. A catalogue that doesn't exist yet is created on first publish.
  • group_bytes sets the S3 object layout: chunks are packed, in the order they were written, into group objects of up to that many bytes (default 32 MiB, the setting for most datasets, including ones that grow by appending); group_bytes=None stores one object per chunk (e.g. small frozen datasets). Continuously updated station datasets are grouped too, with small groups (a power of two at most half of one time block across all stations; envlib_ingest_base.tsortho.group_bytes_for computes it) and a file written block by block, so each update re-uploads only the current block's groups. Omitted, an existing remote keeps its layout and the group_bytes it records from its last push (see cfdb's S3 guide). The mode itself (grouped or one object per chunk) is set when a dataset is first published: asking an existing remote for the other mode is ignored with a warning, so changing it takes deregister(..., delete_data=True) and a fresh publish.
  • Datasets published by envlib <= 0.1.7 with num_groups (hash-grouped) cannot be opened by envlib 0.1.8 / ebooklet 0.11, not even to read: they raise UnsupportedFormatError. To move one, open it 'w' with ebooklet < 0.11 and load_items() so the local file holds every value, delete_remote(), then publish() the local file again.

Failure handling & object verification: if publish dies between the data push and the catalogue write, just run it again — the data push is idempotent and the entry write is an upsert. A partial push failure (some objects could not be transferred) raises a RuntimeError naming the failed keys rather than claiming success; the pending changes are retained, so fixing the cause and re-running completes it. Re-run promptly: the groups that did upload are already committed, so until the re-run, readers of the dataset see the new data with gaps (missing labels or values). If the local file changed meanwhile (a validation gate of your own, for example), re-validate first.

By default (verify_objects=True), publish also verifies the pushed remote before writing the catalogue entry — it fscks the member store and raises PublishIntegrityError if the committed index references objects that aren't actually there (a silent over-claim: e.g. a storage layer that reported success for an upload that never durably landed). This keeps a broken dataset from ever being advertised. Unlike a partial-push RuntimeError, this is not healed by a plain re-run (the push changelog is a timestamp diff, empty once the bad generation is committed) — recover by retracting (deregister(..., delete_data=True)) and doing a full republish. Because the check lists the member store, the member credentials must allow LIST (ListBucket on S3, listFiles on B2) alongside PUT/GET/HEAD/DELETE — a write-only key fails loud here. Pass verify_objects=False to skip it. (A reader that opens a still-inconsistent remote gets RemoteIntegrityError; the verify is what stops most of those from being published in the first place.)

Updating a dataset is the same call: append new time steps to your local file, publish() again. The catalogue entry's extents refresh; modified_at bumps only if something actually changed (a byte-identical re-publish is a no-op and leaves modified_at alone, so it stays a meaningful recency signal). created_at is set once, at first registration, forever.

Watching a large push: the upload narrates itself on the ebooklet.push logger — per-group records with cumulative rate and ETA, plus exact upfront totals. Opt in before calling publish():

import logging
logging.basicConfig()
logging.getLogger('ebooklet.push').setLevel(logging.INFO)

register() — data pushed outside envlib

If a pipeline already manages the S3 side (or you're cataloguing data that is already on S3), register() does everything publish does except the data push:

cat.register(data_conn, rcg_conn)

data_conn must carry write credentials: first registration writes the self-identification attributes (and any derived standard_name) into the remote dataset — a small metadata-only push.

Concurrent producers

Multiple producers registering into one shared catalogue is safe: the catalogue holds an exclusive lock from open through push, so a second writer blocks (up to ~5 minutes) and then lands its entry on top of the refreshed index.

Never force the lock on a shared catalogue

force_lock=True bypasses that serialization and will destroy the other writer's entries. Use it only when you are certain the competing lock is dead (a crashed process), never against a live shared catalogue.

Deregistration

Two alternatives: call one or the other, not both (once delisted, the entry is gone, so a retract after it raises ValidationError).

# EITHER delist: remove the catalogue entry, keep the data
cat.deregister(dataset_version_id, rcg_conn)

# OR retract: remove the entry AND delete the dataset's objects
cat.deregister(dataset_version_id, rcg_conn,
               delete_data=True,
               access_key_id='...', access_key='...')

Delist (the default) removes only the catalogue entry — the hosted data stays up for existing consumers; the dataset just stops being discoverable. This is also step one of fixing a typo'd identity field: deregister, fix the file's attributes, re-register (a new id).

Retract (delete_data=True) is for data known to be wrong: it also deletes the hosted cfdb file, using the owner credentials you inject. Before deleting anything, envlib verifies that no other catalogue entry references the same storage target — if one does (say you re-published a corrected version to the same S3 path), the call refuses rather than destroy the other entry's data.

Deleting data after a plain delist

Once an entry is delisted, envlib no longer tracks the dataset — so removing the hosted data later happens one of two ways:

  1. Re-register, then retract (recommended): cat.register(...) the still-live remote back into the catalogue, then deregister(..., delete_data=True). The shared-target safety check stays in force.
  2. Direct storage deletion: open the remote with your own credentials and call ebooklet's delete_remote() (or delete the objects at the bucket level). envlib is not involved and no safety check applies.

There is no tombstone: after retraction the id simply resolves to nothing. If you've published DOIs pointing at a dataset, prefer delisting over retraction.