MODULE REFERENCE

download

anomx.data.download

Durable page journals for resumable, credential-scoped dataset downloads.

This module stores raw JSON responses independently of DataFrame materialization. Each committed page is durable before control returns to the caller, so stopping iteration or restarting Python does not discard already downloaded observations. The caller owns authentication and chooses a directory scoped to its server, credentials, and dataset reference. Requested prefix lengths and batch sizes are deliberately absent from journal identity: a short read can later extend into a complete download.

classView source ↗

DatasetDownload

DatasetDownload(directory: str | Path, *, definition: Mapping[str, Any], max_age: float | None = 0, refresh: bool = False)

Keep a validated, append-only prefix of a remote dataset on disk.

Use this object as a context manager. Its exclusive nonblocking lock lasts until context exit; a second writer gets DownloadConflict rather than silently overwriting pages. Operating-system locks are released when a process exits, including an interrupted interpreter.

Parameters

directory: Dedicated directory already scoped to the API origin, authorization identity, and dataset reference. For example, derive it from the unbounded dataset cache path with cache.path.with_suffix('.parts'). Never include an API token directly in the directory name.
definition: Freshly authorized dataset definition. version, cache_token, kind, time_column, and time_unit bind the journal to its source and interpretation. The authorization token is not stored.
max_age: Explicit stale-data interval in seconds for completed downloads when the host has no content token, measured from the oldest included page. None explicitly permits version-pinned snapshot reuse without an expiry; it does not imply freshness against mutable source records. Without a token, completed journals are otherwise re-read unless the interval permits reuse. Matching partial downloads resume by pinned source version even after the interval expires; hosts without content validators cannot guarantee snapshot consistency for mutable records during such a resume.
refresh: Discard the previous journal under the lock before starting again.

Notes

Every page has its own checksum and atomic file replacement. The manifest contains only constant-size progress metadata, never a growing page list. Recovery rebuilds progress from the contiguous valid page prefix, including a page committed just before a crash prevented the manifest update. A bad page discards only that page and its suffix. No pickle is deserialized. Files are created with mode 0600 and directories with mode 0700 on POSIX.

Signature parameters
Parameter	Type	Default
`directory`	`str \| Path`	`required`
`definition`keyword only	`Mapping[str, Any]`	`required`
`max_age`keyword only	`float \| None`	`0`
`refresh`keyword only	`bool`	`False`

Members

page_countproperty

# DatasetDownload.page_count Source ↗

page_count: int

Number of contiguous, committed pages, including empty pages.

Returns int

row_countproperty

# DatasetDownload.row_count Source ↗

row_count: int

Number of observations durably stored in committed pages.

Returns int

next_cursorproperty

# DatasetDownload.next_cursor Source ↗

next_cursor: str | None

Cursor for the next request; inspect complete to distinguish EOF.

Returns str | None

completeproperty

# DatasetDownload.complete Source ↗

complete: bool

Whether the last committed response explicitly reached the source end.

Returns bool

closemethod

# DatasetDownload.close Source ↗

close() -> None

Release the writer lock; committed data remains available for reuse.

Returns None

clearclassmethod

# DatasetDownload.clear Source ↗

clear(directory: str | Path) -> None

Discard saved observations without resolving or contacting their source.

Local deletion does not require a current source definition or working credentials. The existing writer lock and durable invalidation marker still protect against concurrent downloads and interrupted deletion. Keep the lock file in place: unlinking it could let another process acquire a different lock while an existing waiter retains the old one.

Signature parameters
Parameter	Type	Default
`directory`	`str \| Path`	`required`

Returns None

iter_pagesmethod

# DatasetDownload.iter_pages Source ↗

iter_pages() -> Iterator[dict[str, Any]]

Replay committed raw responses in order while this context is open.

Returns Iterator[dict[str, Any]]

appendmethod

# DatasetDownload.append Source ↗

append(response: Mapping[str, Any], *, request_cursor: str | None) -> None

Durably commit a response before yielding its records to application code.

request_cursor must be the current next_cursor. This protects against accidentally committing a response fetched before another page advanced the journal. Response source versions, columns, and cursor progression are validated before any file is written. An I/O failure propagates; reopening recovers any page that did commit successfully.

Signature parameters
Parameter	Type	Default
`response`	`Mapping[str, Any]`	`required`
`request_cursor`keyword only	`str \| None`	`required`

Returns None

invalidatemethod

# DatasetDownload.invalidate Source ↗

invalidate() -> None

Remove this journal's pages and reset progress while retaining its lock.

Call after a host reports a changed strong content token during a transfer. A new context with the new authorized definition can then download a coherent source selection.

Returns None

classView source ↗

DownloadConflict

DownloadConflict

Bases: RuntimeError

Another writer owns a download, or an append uses an obsolete cursor.