oceanum.storage package#

Submodules#

oceanum.storage.filesystem module#

class oceanum.storage.filesystem.FileSystem(*args, **kwargs)[source]#

Bases: AsyncFileSystem

Datamesh storage filesystem.

This follows the fsspec specification and can be used with dask.

You can use this class directly, for example:

fs = FileSystem(token=”my_datamesh_token”) fs.ls(“/myfolder”)

or use fsspec convenience functions with protocol “oceanum”. For example:

of=fsspec.open(“oceanum://myfolder/myfile.txt”, token=”my_datamesh_token”)

static close_session(loop, session)[source]#
property fsid#

Persistent filesystem id that can be used to compare filesystems across sessions.

ls(path='', detail=True, file_prefix=None, match_glob=None, limit=None)#

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

  • full path to the entry (without protocol)

  • size of the entry, in bytes. If the value cannot be determined, will be None.

  • type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters:

  • path (str)

  • detail (bool) – if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

  • kwargs (may have additional backend-specific options, such as version) – information

Returns:

  • List of strings if detail is False, or list of directory information

  • dicts if detail is True.

async set_session()[source]#
storage_args: tuple[Any, ...]#
storage_options: dict[str, Any]#
ukey(path)[source]#

Unique identifier

oceanum.storage.filesystem.exists(path: str, token: str | None = None, service: str | None = None) → bool[source]#

Check if path exists in oceanum storage.

Parameters:

  • path (str) – Path to check.

  • token (str) – Oceanum datamesh token.

  • service (str) – Oceanum storage service URL.

Returns:

True if path exists, False otherwise.

Return type:

bool

oceanum.storage.filesystem.get(source: str, dest: str, recursive: bool = False, token: str | None = None, service: str | None = None)[source]#

Copy remote source to local dest, or multiple sources to directory.

Parameters:

  • source (str) – Path to get.

  • dest (str) – Destination path.

  • recursive (bool) – Get directories recursively.

  • overwrite (bool) – Overwrite existing destination.

  • token (str) – Oceanum datamesh token.

  • service (str) – Oceanum storage service URL.

Notes

  • Directory dest defined with a trailing slash must exist, consistent with gsutil.

  • Non-existing file dest path is allowed, consistent with gsutil (all required intermediate directories are created).

async oceanum.storage.filesystem.get_client(**kwargs)[source]#
oceanum.storage.filesystem.isdir(path: str, token: str | None = None, service: str | None = None) → bool[source]#

Check if path is a directory in oceanum storage.

Parameters:

  • path (str) – Path to check.

  • token (str) – Oceanum datamesh token.

  • service (str) – Oceanum storage service URL.

Returns:

True if path is a directory, False otherwise.

Return type:

bool

oceanum.storage.filesystem.isfile(path: str, token: str | None = None, service: str | None = None) → bool[source]#

Check if path is a file in oceanum storage.

Parameters:

  • path (str) – Path to check.

  • token (str) – Oceanum datamesh token.

  • service (str) – Oceanum storage service URL.

Returns:

True if path is a file, False otherwise.

Return type:

bool

oceanum.storage.filesystem.ls(path: str, recursive: bool, detail: bool = False, token: str | None = None, service: str | None = None, **kwargs)[source]#

List contents in the oceanum storage (the root directory by default).

Parameters:

  • path (str) – Path to list.

  • recursive (bool) – List subdirectories recursively.

  • detail (bool) – Return detailed information about each content.

  • token (str) – Oceanum datamesh token.

  • service (str) – Oceanum storage service URL.

Returns:

contents – List of contents or dictionary with detailed information about each content.

Return type:

list | dict

oceanum.storage.filesystem.put(source: str, dest: str, recursive: bool = False, token: str | None = None, service: str | None = None)[source]#

Copy local source to remote dest, or multiple sources to directory.

Parameters:

  • source (str) – Path to get.

  • dest (str) – Destination path.

  • recursive (bool) – Get directories recursively.

  • token (str) – Oceanum datamesh token.

  • service (str) – Oceanum storage service URL.

oceanum.storage.filesystem.rm(path: str, recursive: bool = False, token: str | None = None, service: str | None = None)[source]#

Remove path file or directory.

Parameters:

  • path (str) – Path to remove.

  • recursive (bool) – Remove directories recursively.

  • token (str) – Oceanum datamesh token.

  • service (str) – Oceanum storage service URL.