`ondio.backends`¶

ondio.backends ¶

Storage backends: one class per platform, each following StorageBackend.

The SDKs load only when a backend is constructed, so importing this package is light.

AwsBackend ¶

StorageBackend for AWS S3 (s3://bucket/key URIs), via boto3.

See StorageBackend for method semantics. Credentials resolve through boto3's native chain; botocore errors are translated into ondio's exception types: credential errors are AuthError, and connection failures and timeouts are NetworkError.

Parameters:

Name	Type	Description	Default
`endpoint_url`	`str \| None`	The S3 endpoint, for a store that is not AWS (e.g. RustFS). `None` uses `boto3`'s own choice.	`None`
`client`		An S3 client to use as given. No other option can be given with it.	`None`
`anonymous`	`bool`	Send unsigned requests, with no credentials, for a public bucket. Only `endpoint_url` and `region_name` can be given with it. No profile is used, also not one that `AWS_PROFILE` names. A request to a bucket that is not public is then an `AuthError`.	`False`
`**session_kwargs`		Forwarded to `boto3.Session` (e.g. `profile_name`, `region_name`).	`{}`

Raises:

Type	Description
`ValueError`	If `client` is given with another option, or `anonymous` with an option other than `endpoint_url` and `region_name`.

GcsBackend ¶

StorageBackend for Google Cloud Storage (gs://bucket/key URIs).

See StorageBackend for method semantics. Requires the gcs extra. Credentials resolve through the google-cloud-storage client's native chain; API errors are translated into ondio's exception types: credential errors are AuthError, and connection failures and timeouts are NetworkError.

Parameters:

Name	Type	Description	Default
`endpoint_url`	`str \| None`	The GCS API endpoint. The client still uses its normal credentials. `None` uses the client's own choice.	`None`
`client`		A `google.cloud.storage.Client` to use as given. No other option can be given with it.	`None`
`anonymous`	`bool`	Send requests with no credentials and no project, for a public bucket. Only `endpoint_url` can be given with it. A request to a bucket that is not public is then an `AuthError`.	`False`
`**client_kwargs`		Forwarded to `google.cloud.storage.Client` (e.g. `project`).	`{}`

Raises:

Type	Description
`ValueError`	If both `endpoint_url` and `client_options` are given, `client` with another option, or `anonymous` with an option other than `endpoint_url`.

HttpBackend ¶

Read-only StorageBackend for plain http(s):// URLs, via requests.

See StorageBackend for method semantics. Requires the http extra. upload, list_files, object_count, and delete raise UnsupportedOperationError — infeasible over plain HTTP. read_range verifies the server honored the Range header (206) and refuses to pass off a full-body 200 as a range.

requests errors are translated: a bad URL or header is ValueError, a connection failure or timeout is NetworkError, and any other is OndioError.

For a bearer token, give headers={"Authorization": "Bearer <token>"}. A redirect to another host gets no Authorization, no cookies, and none of the headers given here. No error message holds an option's value.

Parameters:

Name	Type	Description	Default
`headers`	`dict[str, str] \| None`	Headers sent with every request. A `Range` header is refused: `read` would send it and get part of the file.	`None`
`auth`		Anything `requests` takes as `auth`, e.g. a `(user, password)` pair.	`None`
`cookies`	`dict[str, str] \| None`	Cookies sent with every request.	`None`
`verify`	`bool \| str`	`True` checks the server's TLS certificate, `False` does not, and a path names the CA bundle to check it with.	`True`
`timeout`	`float \| tuple[float, float] \| None`	Seconds to wait, as `requests` takes it: one number, or a `(connect, read)` pair. The read timeout limits each wait for data, not the whole request, so a server that keeps sending slowly is not stopped by it. `None` waits with no limit.	`(10, 60)`
`allow_redirects`	`bool`	Follow redirects. If `False`, a 3xx answer is an `OndioError`.	`True`

Raises:

Type	Description
`ValueError`	If `headers` has a `Range` key, in any case.

LocalBackend ¶

StorageBackend ¶

Bases: Protocol

Byte-level storage primitives, implemented once per platform.

Implementations translate their platform SDK's errors into ondio's exception hierarchy (ObjectNotFoundError, AuthError, UnsupportedOperationError, OndioError). All URIs are fully qualified and addressable by the backend they are passed to.

read ¶

read(uri: str) -> bytes

Read the full object into memory.

Equivalent to read_range(uri, 0, None).

Parameters:

Name	Type	Description	Default
`uri`	`str`	URI of the object.	required

Returns:

Type	Description
`bytes`	The object's bytes.

Raises:

Type	Description
`ObjectNotFoundError`	If the object does not exist.

read_range ¶

read_range(uri: str, start_byte: int, end_byte: int | None) -> bytes

Read an inclusive byte range.

Parameters:

Name	Type	Description	Default
`uri`	`str`	URI of the object.	required
`start_byte`	`int`	First byte to read (>= 0).	required
`end_byte`	`int \| None`	Last byte to read (inclusive); `None` means "to EOF".	required

Returns:

Type	Description
`bytes`	The requested bytes. Ranges extending past EOF return the
`bytes`	available bytes (possibly empty).

Raises:

Type	Description
`ValueError`	If the range is negative or inverted.
`ObjectNotFoundError`	If the object does not exist.
`UnsupportedOperationError`	If the backend cannot honor a range (e.g. an HTTP server that ignores `Range` and answers 200).

size ¶

size(uri: str) -> int

Object size in bytes, without fetching the object.

Parameters:

Name	Type	Description	Default
`uri`	`str`	URI of the object.	required

Returns:

Type	Description
`int`	The size in bytes (S3 `HeadObject`, GCS blob metadata, HTTP `HEAD`
`int`	`Content-Length`, local stat).

Raises:

Type	Description
`ObjectNotFoundError`	If the object does not exist.

write ¶

write(uri: str, data: bytes) -> None

Write bytes to a URI.

Parameters:

Name	Type	Description	Default
`uri`	`str`	Destination URI.	required
`data`	`bytes`	The bytes to write.	required

create ¶

create(uri: str, data: bytes) -> None

Write bytes to a URI only if no object exists there.

A reader sees either no object or the complete one. Backends that cannot do this raise UnsupportedOperationError.

Parameters:

Name	Type	Description	Default
`uri`	`str`	Destination URI.	required
`data`	`bytes`	The bytes to write.	required

Raises:

Type	Description
`ObjectExistsError`	If an object exists at `uri`. It is not changed.

download ¶

download(uri: str, out_path: str | PathLike[str]) -> None

Stream an object to a local path, creating parent directories.

Parameters:

Name	Type	Description	Default
`uri`	`str`	URI of the object.	required
`out_path`	`str \| PathLike[str]`	Local destination path.	required

Raises:

Type	Description
`ObjectNotFoundError`	If the object does not exist.

upload ¶

upload(uri: str, source_path: str | PathLike[str], *, overwrite: bool = True) -> None

Send a local file to a URI without reading it into memory.

With overwrite=True an existing object at uri is replaced. With overwrite=False the file is written only if no object exists there, as create writes bytes. The dispatcher passes overwrite only when it is False, so a backend that never supports that need not accept it.

Parameters:

Name	Type	Description	Default
`uri`	`str`	Destination URI.	required
`source_path`	`str \| PathLike[str]`	Local file to send.	required
`overwrite`	`bool`	Replace an existing object. `False` refuses to.	`True`

Raises:

Type	Description
`ObjectExistsError`	With `overwrite=False`, if an object exists at `uri`. It is not changed. On S3 and GCS it can be the object that this call wrote, if the client retried after a lost response.
`ObjectNotFoundError`	If the destination's bucket does not exist.
`AuthError`	If access to the destination is denied.
`UnsupportedOperationError`	If the backend cannot write, or cannot write this file only if no object exists.
`OndioError`	For any other storage error, as `write` raises. With `overwrite=False` this does not prove that nothing was written.

list_files ¶

list_files(uri_prefix: str, *, recursive: bool = True, max_items: int | None = None, required_prefix: str | None = None, required_ext: str | None = None) -> list[str]

List full URIs (not bare keys) under a prefix, lexicographically sorted.

The prefix has folder semantics on every backend: it matches the object at that exact path (if one exists) and every object under it as a folder, but never sibling objects that merely share leading characters — results/run-1 matches results/run-1 and results/run-1/x but not results/run-10/x.

Parameters:

Name	Type	Description	Default
`uri_prefix`	`str`	Prefix to list under.	required
`recursive`	`bool`	If `False`, list only the immediate level.	`True`
`max_items`	`int \| None`	Cap on the number of results, applied after the filename filters; `None` means unbounded.	`None`
`required_prefix`	`str \| None`	If given, list only objects whose filename starts with this.	`None`
`required_ext`	`str \| None`	If given, list only objects with this file extension (with or without the leading dot).	`None`

Returns:

Type	Description
`list[str]`	Sorted full URIs; empty if nothing matches.

object_count ¶

object_count(uri_prefix: str, *, required_prefix: str | None = None, required_ext: str | None = None) -> int

Count objects under a prefix (always recursive) without building URI lists.

Same folder semantics as list_files (an exact-path object counts, folder placeholder keys do not), so object_count(p) always equals len(list_files(p)) given the same filters.

Parameters:

Name	Type	Description	Default
`uri_prefix`	`str`	Prefix to count under.	required
`required_prefix`	`str \| None`	If given, count only objects whose filename starts with this.	`None`
`required_ext`	`str \| None`	If given, count only objects with this file extension (with or without the leading dot).	`None`

Returns:

Type	Description
`int`	The number of matching objects.

exists ¶

exists(uri: str) -> bool

Whether the URI addresses an existing object.

Parameters:

Name	Type	Description	Default
`uri`	`str`	URI to check.	required

Returns:

Type	Description
`bool`	`True` if the object exists (not a directory/prefix).

delete ¶

delete(uri: str) -> None

Delete a single object. Idempotent: deleting a missing object is a no-op.

Parameters:

Name	Type	Description	Default
`uri`	`str`	URI of the object.	required

ondio.backends¶

ondio.backends ¶

AwsBackend ¶

GcsBackend ¶

HttpBackend ¶

LocalBackend ¶

StorageBackend ¶

read ¶

read_range ¶

size ¶

write ¶

create ¶

download ¶

upload ¶

list_files ¶

object_count ¶

exists ¶

delete ¶

`ondio.backends`¶