Compare commits
9
Commits
914dd729aa
...
v1.3.21
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6ab34a4ca6
|
||
|
|
96bfceba7a
|
||
|
|
2c0f4a75c5
|
||
|
|
3081d7fbb3
|
||
|
|
a2bc9bb642
|
||
|
|
6612d2f660
|
||
|
|
0423913b3d
|
||
|
|
d60221de21
|
||
|
|
4c9954c1fd
|
+33
-16
@@ -2,24 +2,41 @@
|
||||
|
||||
All notable changes to this project will be documented in this file.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Fixed
|
||||
- Reported a failed upload to the caller. A file the server refused was printed and then forgotten, so a batch that lost files still ended in success and scripts could not tell.
|
||||
- Reported a failed single-file download to the caller, which was counted internally but never raised, so only recursive downloads ever ended in failure.
|
||||
- Raised the error when a downloaded file could not be written locally. A permission error was printed and the path was then listed as if the file had arrived.
|
||||
- Wrote a download to a temporary file beside its destination and moved it into place once the whole body had arrived. A transfer that failed part way used to leave a truncated file under the real name, and a destination that could not be opened was then deleted even though nothing had been written to it. A destination the client may not write is now refused before anything is fetched.
|
||||
- Named the file in the message when an upload failed, and counted a failure the API layer did not raise, such as a file that disappeared between the directory walk and its turn to be sent.
|
||||
- Carried on through the remaining sub-folders when a recursive download lost a file or could not be created locally, instead of abandoning the rest of the tree at the first failure. Every failure is now printed against its own file and reported once at the end.
|
||||
- Said why a download failed, rather than printing the path alone.
|
||||
- Serialised the token refresh across processes. Concurrent `mdrs` invocations shared one refresh token and each sent it, so a provider that rotates refresh tokens accepted the first and refused the rest.
|
||||
- Wrote the login cache through a temporary file so a reader can no longer catch it mid-truncation and discard the session.
|
||||
- Joined the base URL and the API's relative path correctly in `ls --json` output, which produced a doubled separator when the configured URL ended with one.
|
||||
## [1.3.21] - 2026-11-13
|
||||
|
||||
### Changed
|
||||
- Accepted bare hostnames such as `localhost` in `config create`/`config update`, and stored the URL without its trailing slash, matching the Rust client so both can share `config.ini`.
|
||||
- Bounded the token refresh request with its own timeout, so a provider that stops answering cannot hold the cross-process lock indefinitely.
|
||||
- **Breaking for embedders:** `CacheInterface` now requires `lock()` and `reload()`. A cache passed to `MdrsClient.from_remote(..., cache=...)` must provide both; `InMemoryCache` implements them as no-ops.
|
||||
- Built access level labels from their bits instead of a fixed table of combinations, so a level the server adds later is labelled too. A project level (bit 16 and up) is named after the project as the server reports it to its members, e.g. `Brain/MINDS 2.0 Open`, and otherwise `Project<n> Open`. A restrict network other than CBS is `Restrict<n> Open`; `0x0200`, formerly `5Kikan Open`, is now `Restrict2 Open`.
|
||||
- `chacl` takes keys built the same way: `cbs_or_pw_open`, `project0_open` and so on. `5kikan_open` is no longer accepted. Its help lists them, an invalid level lists the projects the user belongs to with their keys, and `public` and `storage`, which the server refuses, are refused before any request.
|
||||
- `ls` asks the server for the user's projects once per run when logged in.
|
||||
- Required Python 3.11 or later, as `pydantic-settings` and `validators` do.
|
||||
- Raised the minimum version of `pydantic` to 2.14.3, and of the development tools `black`, `flake8`, `isort` and `pyright` to their current releases.
|
||||
- Removed `FolderAccessLevel.ACCESS_LEVELS` and `FolderAccessLevelItem`. `FolderSimple.access_level_label()` labels a level with the project names given to it.
|
||||
|
||||
### Fixed
|
||||
- Reported an access level that has no meaning as `Unknown (0x…)` instead of `Storage`. Storage is the absence of every bit, so it matched any level. A level made of known bits is no longer labelled by the first table entry it contained either: `0x10104` was `CBS or PW Open`. This changes the `access_level` value in `ls` output, including JSON.
|
||||
- Reported a locked folder that was fetched without its password as `FolderLockedException`, a subclass of `UnauthorizedException`, instead of a pydantic `ValidationError`. The server answers with a short record that has no sub-folders or path. A password-protected DOI folder fails this way before its password is sent, so it can now be opened again.
|
||||
|
||||
## [1.3.20] - 2026-11-04
|
||||
|
||||
### Changed
|
||||
- Raised the minimum versions of `pydantic`, `pydantic-settings`, `python-dotenv`, `PyJWT` and `validators` to their current releases.
|
||||
|
||||
### Security
|
||||
- Required `PyJWT` 2.15.1 or later; releases before 2.14.0 have a known vulnerability.
|
||||
|
||||
## [1.3.19] - 2026-09-04
|
||||
|
||||
### Fixed
|
||||
- Reported failed transfers instead of ending in success, so a run that lost files exits non-zero.
|
||||
- Left an existing local file intact when a download fails part way through.
|
||||
- Opened password-protected DOI folders, which previously failed outright.
|
||||
- Kept a session working across concurrent runs and long server queues.
|
||||
- Other minor fixes to `ls` output and remote URL handling.
|
||||
|
||||
### Changed
|
||||
- Moved packaging to PEP 621 metadata with flit as the build backend.
|
||||
- Updated dependencies.
|
||||
- **Breaking for embedders:** `CacheInterface` now requires `lock()` and `reload()`.
|
||||
|
||||
## [1.3.18] - 2026-07-02
|
||||
|
||||
|
||||
@@ -40,6 +40,15 @@ pip install -e ".[dev]"
|
||||
uv sync
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
The following environment variables can be set, either in the shell or via a `.env` file placed in the working directory.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `MDRS_CLIENT_CONFIG_DIRNAME` | `~/.mdrs-client` | Directory where config and login cache files are stored |
|
||||
| `MDRS_CLIENT_CONCURRENT` | `10` | Number of concurrent file transfers for upload and download |
|
||||
|
||||
## CLI Usage
|
||||
|
||||
### config create
|
||||
@@ -190,12 +199,14 @@ mdrs rm -r neurodata:/NIU/Repository/TEST/dataset
|
||||
|
||||
### chacl
|
||||
|
||||
Change the folder access level
|
||||
Change the folder access level. The level is `private`, `pw_open`, `cbs_open`, `project<n>_open`, or several open levels joined by `_or_`. `<n>` is the project's bit position; an invalid level lists the projects you belong to with their keys.
|
||||
|
||||
```shell
|
||||
mdrs chacl private neurodata:/NIU/Repository/Private
|
||||
mdrs chacl cbs_open -r neurodata:/NIU/Repository/CBS_Open
|
||||
mdrs chacl pw_open -r -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open
|
||||
mdrs chacl cbs_or_pw_open -p SHARING_PASSWORD neurodata:/NIU/Repository/CBS_or_PW_Open
|
||||
mdrs chacl project0_open neurodata:/NIU/Repository/Project_Open
|
||||
```
|
||||
|
||||
### metadata
|
||||
|
||||
@@ -6,7 +6,7 @@ from pydantic.dataclasses import dataclass
|
||||
|
||||
from mdrsclient.api.base import BaseApi
|
||||
from mdrsclient.api.utils import retry_on_expired_token, token_check
|
||||
from mdrsclient.exceptions import UnauthorizedException
|
||||
from mdrsclient.exceptions import FolderLockedException, UnauthorizedException
|
||||
from mdrsclient.models import Folder, FolderSimple
|
||||
|
||||
|
||||
@@ -38,7 +38,12 @@ class FoldersApi(BaseApi):
|
||||
token_check(self.connection)
|
||||
response = self.connection.get(url)
|
||||
self._raise_response_error(response)
|
||||
ret = TypeAdapter(Folder).validate_python(response.json())
|
||||
data = response.json()
|
||||
# A locked folder not yet opened with its password comes back with 200 but only
|
||||
# its listing fields, without `metadata`, `sub_folders` and `path`.
|
||||
if data.get("lock") and "sub_folders" not in data:
|
||||
raise FolderLockedException(f"Folder `{data.get('name', id)}` is locked.")
|
||||
ret = TypeAdapter(Folder).validate_python(data)
|
||||
return ret
|
||||
|
||||
@retry_on_expired_token
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
from dataclasses import field
|
||||
from typing import Final
|
||||
|
||||
import requests
|
||||
@@ -20,6 +21,13 @@ class UsersCurrentResponseLaboratory:
|
||||
role: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class UsersCurrentResponseProject:
|
||||
id: int
|
||||
name: str | None
|
||||
bit_position: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class UsersApiCurrentResponse:
|
||||
id: int
|
||||
@@ -35,6 +43,7 @@ class UsersApiCurrentResponse:
|
||||
is_reviewer: bool
|
||||
last_login: str # ISO8601
|
||||
date_joined: str # ISO8601
|
||||
projects: list[UsersCurrentResponseProject] = field(default_factory=list)
|
||||
|
||||
|
||||
class UsersApi(BaseApi):
|
||||
@@ -50,6 +59,20 @@ class UsersApi(BaseApi):
|
||||
user = User(id=obj.id, username=obj.username, laboratory_ids=laboratory_ids, is_reviewer=obj.is_reviewer)
|
||||
return user
|
||||
|
||||
def project_names(self) -> dict[int, str]:
|
||||
"""Names of the user's projects by bit_position, the bit their access level sets."""
|
||||
url = self.ENTRYPOINT + "current/"
|
||||
response = self.connection.get(url)
|
||||
self._raise_response_error(response)
|
||||
obj = TypeAdapter(UsersApiCurrentResponse).validate_python(response.json())
|
||||
names: dict[int, list[str]] = {}
|
||||
for project in obj.projects:
|
||||
if project.name:
|
||||
# bit_position is not unique on the server, and a folder at a shared bit
|
||||
# belongs to every project holding it, so all of them are named.
|
||||
names.setdefault(project.bit_position, []).append(project.name)
|
||||
return {bit: ", ".join(sorted(x)) for bit, x in names.items()}
|
||||
|
||||
def token(self, username: str, password: str) -> Token:
|
||||
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
|
||||
url = self.ENTRYPOINT + "token/"
|
||||
|
||||
+94
-80
@@ -1,21 +1,40 @@
|
||||
import os
|
||||
from typing import Any
|
||||
from dataclasses import dataclass
|
||||
from unicodedata import normalize
|
||||
|
||||
from mdrsclient.api import DoiApi, FilesApi, FoldersApi, LaboratoriesApi, UsersApi
|
||||
from mdrsclient.api import FilesApi, FoldersApi
|
||||
from mdrsclient.cache import CacheInterface
|
||||
from mdrsclient.config import ConfigInterface
|
||||
from mdrsclient.connection import MDRSConnection
|
||||
from mdrsclient.exceptions import IllegalArgumentException, MDRSException, UnauthorizedException, UnexpectedException
|
||||
from mdrsclient.models import File, Folder, Laboratory, Token, User
|
||||
from mdrsclient.exceptions import IllegalArgumentException
|
||||
from mdrsclient.models import File, Folder, FolderSimple, Laboratory
|
||||
from mdrsclient.models.file import find_file
|
||||
from mdrsclient.services import MdrsService
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TransferEndpoints:
|
||||
"""Both ends of a `cp` or `mv`, resolved: the parents, their contents, and the names."""
|
||||
|
||||
laboratory: Laboratory
|
||||
s_path: str
|
||||
s_basename: str
|
||||
s_parent_folder: Folder
|
||||
s_parent_files: list[File]
|
||||
d_path: str
|
||||
d_basename: str
|
||||
d_parent_folder: Folder
|
||||
d_parent_files: list[File]
|
||||
|
||||
def is_same_place(self) -> bool:
|
||||
"""Whether the destination names what the source already is, leaving nothing to do."""
|
||||
return self.s_parent_folder.id == self.d_parent_folder.id and self.s_basename == self.d_basename
|
||||
|
||||
|
||||
class MdrsClient(MdrsService):
|
||||
"""Service layer client for MDRS."""
|
||||
|
||||
def __init__(self, connection: MDRSConnection, config_class: type[ConfigInterface] | None = None):
|
||||
def __init__(self, connection: MDRSConnection | None, config_class: type[ConfigInterface] | None = None):
|
||||
super().__init__(connection, config_class)
|
||||
|
||||
@classmethod
|
||||
@@ -63,7 +82,8 @@ class MdrsClient(MdrsService):
|
||||
files = self.find_files(folder.id)
|
||||
return folder, files
|
||||
|
||||
def cp(self, src_path: str, dest_path: str, is_recursive: bool = False) -> None:
|
||||
def __resolve_transfer(self, src_path: str, dest_path: str) -> TransferEndpoints:
|
||||
"""Resolve both ends of a transfer, refusing one that crosses a remote or a laboratory."""
|
||||
s_remote, s_laboratory_name, s_path = self.parse_remote_host_with_path(src_path)
|
||||
d_remote, d_laboratory_name, d_path = self.parse_remote_host_with_path(dest_path)
|
||||
if s_remote != d_remote:
|
||||
@@ -71,8 +91,9 @@ class MdrsClient(MdrsService):
|
||||
if s_laboratory_name != d_laboratory_name:
|
||||
raise IllegalArgumentException("Laboratory mismatched.")
|
||||
s_path = s_path.rstrip("/")
|
||||
s_dirname = os.path.dirname(s_path)
|
||||
s_basename = os.path.basename(s_path)
|
||||
# A destination ending in a separator names a folder to put the source into, and
|
||||
# keeps the source's own name.
|
||||
if d_path.endswith("/"):
|
||||
d_dirname = d_path
|
||||
d_basename = s_basename
|
||||
@@ -80,84 +101,77 @@ class MdrsClient(MdrsService):
|
||||
d_dirname = os.path.dirname(d_path)
|
||||
d_basename = os.path.basename(d_path)
|
||||
laboratory = self.find_laboratory(s_laboratory_name)
|
||||
s_parent_folder = self.find_folder(laboratory, s_dirname)
|
||||
s_parent_files = self.find_files(s_parent_folder.id)
|
||||
s_parent_folder = self.find_folder(laboratory, os.path.dirname(s_path))
|
||||
d_parent_folder = self.find_folder(laboratory, d_dirname)
|
||||
d_parent_files = self.find_files(d_parent_folder.id)
|
||||
s_file = find_file(s_parent_files, s_basename)
|
||||
return TransferEndpoints(
|
||||
laboratory=laboratory,
|
||||
s_path=s_path,
|
||||
s_basename=s_basename,
|
||||
s_parent_folder=s_parent_folder,
|
||||
s_parent_files=self.find_files(s_parent_folder.id),
|
||||
d_path=d_path,
|
||||
d_basename=d_basename,
|
||||
d_parent_folder=d_parent_folder,
|
||||
d_parent_files=self.find_files(d_parent_folder.id),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def __check_file_destination(ends: TransferEndpoints) -> None:
|
||||
"""Refuse a destination that a file cannot take the place of."""
|
||||
if find_file(ends.d_parent_files, ends.d_basename) is not None:
|
||||
raise IllegalArgumentException(f"File `{ends.d_basename}` already exists.")
|
||||
if ends.d_parent_folder.find_sub_folder(ends.d_basename) is not None:
|
||||
raise IllegalArgumentException(
|
||||
f"Cannot overwrite non-folder `{ends.d_basename}` with folder `{ends.d_path}`."
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def __check_folder_destination(ends: TransferEndpoints, s_folder: FolderSimple) -> None:
|
||||
"""Refuse a destination that a folder cannot take the place of."""
|
||||
if find_file(ends.d_parent_files, ends.d_basename) is not None:
|
||||
raise IllegalArgumentException(
|
||||
f"Cannot overwrite non-folder `{ends.d_basename}` with folder `{ends.s_path}`."
|
||||
)
|
||||
d_folder = ends.d_parent_folder.find_sub_folder(ends.d_basename)
|
||||
if d_folder is not None:
|
||||
if d_folder.id == s_folder.id:
|
||||
raise IllegalArgumentException(f"`{ends.s_path}` and `{ends.s_path}` are the same folder.")
|
||||
raise IllegalArgumentException(f"Cannot move `{ends.s_path}` to `{ends.d_path}`: Folder not empty.")
|
||||
|
||||
@staticmethod
|
||||
def __find_source_folder(ends: TransferEndpoints) -> FolderSimple:
|
||||
s_folder = ends.s_parent_folder.find_sub_folder(ends.s_basename)
|
||||
if s_folder is None:
|
||||
raise IllegalArgumentException(f"File or folder `{ends.s_basename}` not found.")
|
||||
return s_folder
|
||||
|
||||
def cp(self, src_path: str, dest_path: str, is_recursive: bool = False) -> None:
|
||||
ends = self.__resolve_transfer(src_path, dest_path)
|
||||
s_file = find_file(ends.s_parent_files, ends.s_basename)
|
||||
if s_file is not None:
|
||||
d_file = find_file(d_parent_files, d_basename)
|
||||
if d_file is not None:
|
||||
raise IllegalArgumentException(f"File `{d_basename}` already exists.")
|
||||
d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
|
||||
if d_sub_folder is not None:
|
||||
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
|
||||
file_api = FilesApi(self.connection)
|
||||
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
|
||||
file_api.copy(s_file, d_parent_folder.id, normalize("NFC", d_basename))
|
||||
else:
|
||||
s_folder = s_parent_folder.find_sub_folder(s_basename)
|
||||
if s_folder is None:
|
||||
raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
|
||||
if not is_recursive:
|
||||
raise IllegalArgumentException(f"Cannot copy `{s_path}`: Is a folder.")
|
||||
if find_file(d_parent_files, d_basename) is not None:
|
||||
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
|
||||
d_folder = d_parent_folder.find_sub_folder(d_basename)
|
||||
if d_folder is not None:
|
||||
if d_folder.id == s_folder.id:
|
||||
raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
|
||||
raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
|
||||
folder_api = FoldersApi(self.connection)
|
||||
if s_parent_folder.id != d_parent_folder.id or s_basename != d_basename:
|
||||
folder_api.copy(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
|
||||
self.__check_file_destination(ends)
|
||||
if not ends.is_same_place():
|
||||
FilesApi(self.connection).copy(s_file, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
|
||||
return
|
||||
s_folder = self.__find_source_folder(ends)
|
||||
if not is_recursive:
|
||||
raise IllegalArgumentException(f"Cannot copy `{ends.s_path}`: Is a folder.")
|
||||
self.__check_folder_destination(ends, s_folder)
|
||||
if not ends.is_same_place():
|
||||
FoldersApi(self.connection).copy(s_folder, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
|
||||
|
||||
def mv(self, src_path: str, dest_path: str) -> None:
|
||||
s_remote, s_laboratory_name, s_path = self.parse_remote_host_with_path(src_path)
|
||||
d_remote, d_laboratory_name, d_path = self.parse_remote_host_with_path(dest_path)
|
||||
if s_remote != d_remote:
|
||||
raise IllegalArgumentException("Remote host mismatched.")
|
||||
if s_laboratory_name != d_laboratory_name:
|
||||
raise IllegalArgumentException("Laboratory mismatched.")
|
||||
s_path = s_path.rstrip("/")
|
||||
s_dirname = os.path.dirname(s_path)
|
||||
s_basename = os.path.basename(s_path)
|
||||
if d_path.endswith("/"):
|
||||
d_dirname = d_path
|
||||
d_basename = s_basename
|
||||
else:
|
||||
d_dirname = os.path.dirname(d_path)
|
||||
d_basename = os.path.basename(d_path)
|
||||
laboratory = self.find_laboratory(s_laboratory_name)
|
||||
s_parent_folder = self.find_folder(laboratory, s_dirname)
|
||||
s_parent_files = self.find_files(s_parent_folder.id)
|
||||
d_parent_folder = self.find_folder(laboratory, d_dirname)
|
||||
d_parent_files = self.find_files(d_parent_folder.id)
|
||||
s_file = find_file(s_parent_files, s_basename)
|
||||
ends = self.__resolve_transfer(src_path, dest_path)
|
||||
s_file = find_file(ends.s_parent_files, ends.s_basename)
|
||||
if s_file is not None:
|
||||
d_file = find_file(d_parent_files, d_basename)
|
||||
if d_file is not None:
|
||||
raise IllegalArgumentException(f"File `{d_basename}` already exists.")
|
||||
d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
|
||||
if d_sub_folder is not None:
|
||||
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
|
||||
file_api = FilesApi(self.connection)
|
||||
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
|
||||
file_api.move(s_file, d_parent_folder.id, normalize("NFC", d_basename))
|
||||
else:
|
||||
s_folder = s_parent_folder.find_sub_folder(s_basename)
|
||||
if s_folder is None:
|
||||
raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
|
||||
if find_file(d_parent_files, d_basename) is not None:
|
||||
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
|
||||
d_folder = d_parent_folder.find_sub_folder(d_basename)
|
||||
if d_folder is not None:
|
||||
if d_folder.id == s_folder.id:
|
||||
raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
|
||||
raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
|
||||
folder_api = FoldersApi(self.connection)
|
||||
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
|
||||
folder_api.move(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
|
||||
self.__check_file_destination(ends)
|
||||
if not ends.is_same_place():
|
||||
FilesApi(self.connection).move(s_file, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
|
||||
return
|
||||
s_folder = self.__find_source_folder(ends)
|
||||
self.__check_folder_destination(ends, s_folder)
|
||||
if not ends.is_same_place():
|
||||
FoldersApi(self.connection).move(s_folder, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
|
||||
|
||||
def chacl(
|
||||
self, remote_path: str, access_level: int, is_recursive: bool = False, password: str | None = None
|
||||
|
||||
@@ -1,19 +1,28 @@
|
||||
from argparse import Namespace
|
||||
from typing import Any
|
||||
from typing import Any, Final
|
||||
|
||||
import requests
|
||||
|
||||
from mdrsclient.api import FoldersApi
|
||||
from mdrsclient.commands.base import BaseCommand
|
||||
from mdrsclient.exceptions import IllegalArgumentException
|
||||
from mdrsclient.exceptions import IllegalArgumentException, MDRSException
|
||||
from mdrsclient.models import FolderAccessLevel
|
||||
|
||||
|
||||
class ChaclCommand(BaseCommand):
|
||||
ACCESS_LEVEL_HELP: Final[str] = (
|
||||
"access level: private, pw_open, cbs_open, project<n>_open, "
|
||||
"or several open levels joined by `_or_` (e.g. cbs_or_pw_open). "
|
||||
"<n> is the project's bit position; an invalid level lists the projects you belong to"
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def register(cls, parsers: Any) -> None:
|
||||
chacl_parser = parsers.add_parser("chacl", help="change the folder access level")
|
||||
chacl_parser.add_argument("access_level", help="access level (private, cbs_open, pw_open)")
|
||||
chacl_parser.add_argument("access_level", help=cls.ACCESS_LEVEL_HELP)
|
||||
chacl_parser.add_argument("-r", "--recursive", help="change access levels recursively", action="store_true")
|
||||
chacl_parser.add_argument("-p", "--password", help="password to set when access level is `pw_open`")
|
||||
chacl_parser.add_argument(
|
||||
"-p", "--password", help="password to set when the access level includes `pw` (e.g. pw_open)"
|
||||
)
|
||||
chacl_parser.add_argument("remote_path", help="remote folder path (remote:/lab/path/)")
|
||||
chacl_parser.set_defaults(func=cls.func)
|
||||
|
||||
@@ -22,9 +31,10 @@ class ChaclCommand(BaseCommand):
|
||||
remote_path = str(args.remote_path)
|
||||
access_level = FolderAccessLevel.key2id(str(args.access_level))
|
||||
if access_level is None:
|
||||
raise IllegalArgumentException(
|
||||
"Invalid `access_level` parameter. must be `private`, `cbs_open` or `pw_open`."
|
||||
)
|
||||
raise IllegalArgumentException(cls._invalid_level_message(remote_path))
|
||||
if access_level in (FolderAccessLevel.STORAGE, FolderAccessLevel.PUBLIC):
|
||||
# The server refuses both; saying so here spares a round trip and a bare 400.
|
||||
raise IllegalArgumentException(f"`{args.access_level}` cannot be set as an access level.")
|
||||
password = str(args.password) if args.password else None
|
||||
is_recursive = bool(args.recursive)
|
||||
cls.chacl(remote_path, access_level, is_recursive, password)
|
||||
@@ -36,3 +46,24 @@ class ChaclCommand(BaseCommand):
|
||||
|
||||
client = MdrsClient.from_remote(remote)
|
||||
client.chacl(remote_path, access_level, is_recursive, password)
|
||||
|
||||
@classmethod
|
||||
def _invalid_level_message(cls, remote_path: str) -> str:
|
||||
message = (
|
||||
"Invalid `access_level` parameter. must be `private`, `pw_open`, `cbs_open`, `project<n>_open` "
|
||||
"or open levels joined by `_or_` (e.g. `cbs_or_pw_open`)."
|
||||
)
|
||||
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
|
||||
from mdrsclient.client import MdrsClient
|
||||
|
||||
# The project numbers are known only to the server, and only for the user's own
|
||||
# projects. Not reaching it leaves the message as it is: the error being reported
|
||||
# is the level, not the connection.
|
||||
try:
|
||||
projects = MdrsClient.from_remote(remote).project_names()
|
||||
except (MDRSException, requests.RequestException):
|
||||
return message
|
||||
if not projects:
|
||||
return message
|
||||
lines = [f" project{bit}_open: {name}" for bit, name in sorted(projects.items())]
|
||||
return "\n".join([message, "Projects you belong to:", *lines])
|
||||
|
||||
@@ -1,12 +1,7 @@
|
||||
import os
|
||||
from argparse import Namespace
|
||||
from typing import Any
|
||||
from unicodedata import normalize
|
||||
|
||||
from mdrsclient.api import FilesApi, FoldersApi
|
||||
from mdrsclient.commands.base import BaseCommand
|
||||
from mdrsclient.exceptions import IllegalArgumentException
|
||||
from mdrsclient.models.file import find_file
|
||||
|
||||
|
||||
class CpCommand(BaseCommand):
|
||||
|
||||
@@ -1,12 +1,8 @@
|
||||
import json
|
||||
import os
|
||||
from argparse import Namespace
|
||||
from typing import Any
|
||||
|
||||
from mdrsclient.api import FilesApi
|
||||
from mdrsclient.commands.base import BaseCommand
|
||||
from mdrsclient.exceptions import IllegalArgumentException
|
||||
from mdrsclient.models.file import find_file
|
||||
|
||||
|
||||
class FileMetadataCommand(BaseCommand):
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
from argparse import Namespace
|
||||
from typing import Any
|
||||
|
||||
from mdrsclient.api import LaboratoriesApi
|
||||
from mdrsclient.commands.base import BaseCommand
|
||||
|
||||
|
||||
|
||||
+47
-33
@@ -1,10 +1,11 @@
|
||||
import json
|
||||
from argparse import Namespace
|
||||
from typing import Any
|
||||
from dataclasses import field
|
||||
from typing import Any, Final
|
||||
|
||||
from pydantic.dataclasses import dataclass
|
||||
|
||||
from mdrsclient.api import FilesApi, FoldersApi
|
||||
from mdrsclient.api import FoldersApi
|
||||
from mdrsclient.client import MdrsClient
|
||||
from mdrsclient.commands.base import BaseCommand
|
||||
from mdrsclient.config import build_download_url
|
||||
@@ -26,6 +27,7 @@ class LsCommandContext:
|
||||
is_json: bool
|
||||
is_quiet: bool
|
||||
is_recursive: bool
|
||||
projects: dict[int, str] = field(default_factory=dict)
|
||||
|
||||
|
||||
class LsCommand(BaseCommand):
|
||||
@@ -85,6 +87,7 @@ class LsCommand(BaseCommand):
|
||||
is_json,
|
||||
is_quiet,
|
||||
is_recursive,
|
||||
client.project_names(),
|
||||
)
|
||||
if context.is_json:
|
||||
cls._ls_json(context, folder, files)
|
||||
@@ -95,33 +98,53 @@ class LsCommand(BaseCommand):
|
||||
def _ls_json(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
|
||||
print(json.dumps(cls._folder2dict(context, folder, files), ensure_ascii=False))
|
||||
|
||||
LABELS: Final[dict[str, str]] = {
|
||||
"type": "Type",
|
||||
"acl": "Access",
|
||||
"laboratory": "Laboratory",
|
||||
"size": "Size",
|
||||
"date": "Date",
|
||||
"name": "Name",
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def _ls_plain(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
|
||||
label = {
|
||||
"type": "Type",
|
||||
"acl": "Access",
|
||||
"laboratory": "Laboratory",
|
||||
"size": "Size",
|
||||
"date": "Date",
|
||||
"name": "Name",
|
||||
}
|
||||
length: dict[str, int] = {}
|
||||
for key in label.keys():
|
||||
length[key] = len(label[key]) if not context.is_quiet else 0
|
||||
def _column_widths(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> dict[str, int]:
|
||||
"""Width of each column: the widest of its heading and everything printed under it."""
|
||||
length = {key: len(label) if not context.is_quiet else 0 for key, label in cls.LABELS.items()}
|
||||
for sub_folder in folder.sub_folders:
|
||||
sub_laboratory = context.client.connection.laboratories.find_by_id(sub_folder.laboratory_id)
|
||||
sub_laboratory_name = sub_laboratory.name if sub_laboratory is not None else "(invalid)"
|
||||
length["acl"] = max(length["acl"], len(sub_folder.access_level_name))
|
||||
sub_laboratory_name = cls._laboratory_name(context, sub_folder.laboratory_id)
|
||||
length["acl"] = max(length["acl"], len(sub_folder.access_level_label(context.projects)))
|
||||
length["laboratory"] = max(length["laboratory"], len(sub_laboratory_name))
|
||||
length["size"] = max(length["size"], len(str(folder.size)))
|
||||
length["size"] = max(length["size"], len(str(sub_folder.size)))
|
||||
length["date"] = max(length["date"], len(sub_folder.updated_at_name))
|
||||
length["name"] = max(length["name"], len(sub_folder.name))
|
||||
for file in files:
|
||||
length["size"] = max(length["size"], len(str(file.size)))
|
||||
length["date"] = max(length["date"], len(file.updated_at_name))
|
||||
length["name"] = max(length["name"], len(file.name))
|
||||
length["acl"] = max(length["acl"], len(folder.access_level_name))
|
||||
length["acl"] = max(length["acl"], len(folder.access_level_label(context.projects)))
|
||||
length["laboratory"] = max(length["laboratory"], len(context.laboratory.name))
|
||||
return length
|
||||
|
||||
@classmethod
|
||||
def _ls_plain_children(cls, context: LsCommandContext, folder: Folder) -> None:
|
||||
"""List each sub-folder in turn, passing over the ones the caller cannot open."""
|
||||
print("")
|
||||
folder_api = FoldersApi(context.client.connection)
|
||||
for sub_folder in sorted(folder.sub_folders, key=lambda x: x.name):
|
||||
try:
|
||||
if sub_folder.lock:
|
||||
folder_api.auth(sub_folder.id, context.password)
|
||||
sub_detail = folder_api.retrieve(sub_folder.id)
|
||||
sub_files = context.client.find_files(sub_folder.id)
|
||||
except UnauthorizedException:
|
||||
continue
|
||||
cls._ls_plain(context, sub_detail, sub_files)
|
||||
|
||||
@classmethod
|
||||
def _ls_plain(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
|
||||
label = cls.LABELS
|
||||
length = cls._column_widths(context, folder, files)
|
||||
header = (
|
||||
f"{label['type']:{length['type']}}\t{label['acl']:{length['acl']}}\t"
|
||||
f"{label['laboratory']:{length['laboratory']}}\t{label['size']:{length['size']}}\t"
|
||||
@@ -139,30 +162,21 @@ class LsCommand(BaseCommand):
|
||||
for sub_folder in sorted(folder.sub_folders, key=lambda x: x.name):
|
||||
sub_laboratory_name = cls._laboratory_name(context, sub_folder.laboratory_id)
|
||||
sub_folder_type = "[d]" if sub_folder.lock is False else "[l]"
|
||||
sub_folder_acl = sub_folder.access_level_label(context.projects)
|
||||
print(
|
||||
f"{sub_folder_type:{length['type']}}\t{sub_folder.access_level_name:{length['acl']}}\t"
|
||||
f"{sub_folder_type:{length['type']}}\t{sub_folder_acl:{length['acl']}}\t"
|
||||
f"{sub_laboratory_name:{length['laboratory']}}\t{sub_folder.size:{length['size']}}\t"
|
||||
f"{sub_folder.updated_at_name:{length['date']}}\t{sub_folder.name:{length['name']}}"
|
||||
)
|
||||
for file in sorted(files, key=lambda x: x.name):
|
||||
print(
|
||||
f"{'[f]':{length['type']}}\t{folder.access_level_name:{length['acl']}}\t"
|
||||
f"{'[f]':{length['type']}}\t{folder.access_level_label(context.projects):{length['acl']}}\t"
|
||||
f"{context.laboratory.name:{length['laboratory']}}\t{file.size:{length['size']}}\t"
|
||||
f"{file.updated_at_name:{length['date']}}\t{file.name:{length['name']}}"
|
||||
)
|
||||
|
||||
if context.is_recursive:
|
||||
print("")
|
||||
for sub_folder in sorted(folder.sub_folders, key=lambda x: x.name):
|
||||
folder_api = FoldersApi(context.client.connection)
|
||||
try:
|
||||
if sub_folder.lock:
|
||||
folder_api.auth(sub_folder.id, context.password)
|
||||
folder = folder_api.retrieve(sub_folder.id)
|
||||
files = context.client.find_files(sub_folder.id)
|
||||
cls._ls_plain(context, folder, files)
|
||||
except UnauthorizedException:
|
||||
pass
|
||||
cls._ls_plain_children(context, folder)
|
||||
|
||||
@classmethod
|
||||
def _folder2dict(
|
||||
@@ -173,7 +187,7 @@ class LsCommand(BaseCommand):
|
||||
"pid": folder.pid,
|
||||
"name": folder.name,
|
||||
"size": folder.size,
|
||||
"access_level": folder.access_level_name,
|
||||
"access_level": folder.access_level_label(context.projects),
|
||||
"lock": folder.lock,
|
||||
"laboratory": cls._laboratory_name(context, folder.laboratory_id),
|
||||
"description": folder.description,
|
||||
|
||||
@@ -2,7 +2,6 @@ import json
|
||||
from argparse import Namespace
|
||||
from typing import Any
|
||||
|
||||
from mdrsclient.api import FoldersApi
|
||||
from mdrsclient.commands.base import BaseCommand
|
||||
|
||||
|
||||
|
||||
@@ -1,12 +1,7 @@
|
||||
import os
|
||||
from argparse import Namespace
|
||||
from typing import Any
|
||||
from unicodedata import normalize
|
||||
|
||||
from mdrsclient.api import FoldersApi
|
||||
from mdrsclient.commands.base import BaseCommand
|
||||
from mdrsclient.exceptions import IllegalArgumentException
|
||||
from mdrsclient.models.file import find_file
|
||||
|
||||
|
||||
class MkdirCommand(BaseCommand):
|
||||
|
||||
@@ -1,12 +1,7 @@
|
||||
import os
|
||||
from argparse import Namespace
|
||||
from typing import Any
|
||||
from unicodedata import normalize
|
||||
|
||||
from mdrsclient.api import FilesApi, FoldersApi
|
||||
from mdrsclient.commands.base import BaseCommand
|
||||
from mdrsclient.exceptions import IllegalArgumentException
|
||||
from mdrsclient.models.file import find_file
|
||||
|
||||
|
||||
class MvCommand(BaseCommand):
|
||||
|
||||
@@ -1,11 +1,7 @@
|
||||
import os
|
||||
from argparse import Namespace
|
||||
from typing import Any
|
||||
|
||||
from mdrsclient.api import FilesApi, FoldersApi
|
||||
from mdrsclient.commands.base import BaseCommand
|
||||
from mdrsclient.exceptions import IllegalArgumentException
|
||||
from mdrsclient.models.file import find_file
|
||||
|
||||
|
||||
class RmCommand(BaseCommand):
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
from argparse import Namespace
|
||||
from typing import Any
|
||||
|
||||
from mdrsclient.__version__ import __version__
|
||||
from mdrsclient.commands.base import BaseCommand
|
||||
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import configparser
|
||||
import os
|
||||
import threading
|
||||
from typing import Final, Protocol, runtime_checkable
|
||||
from typing import TYPE_CHECKING, Final, Protocol, runtime_checkable
|
||||
|
||||
import validators
|
||||
|
||||
@@ -39,6 +39,14 @@ def build_download_url(base_url: str | None, path: str) -> str:
|
||||
class ConfigInterface(Protocol):
|
||||
remote: str
|
||||
|
||||
if TYPE_CHECKING:
|
||||
# Declared for the type checker, because the class itself is passed around and
|
||||
# called: without this it describes something that takes no arguments to build.
|
||||
# Kept out of the running class, because a protocol that defines `__init__` loses
|
||||
# the guard that stops it being instantiated, and hands that empty `__init__` to
|
||||
# any implementation that does not write its own.
|
||||
def __init__(self, remote: str) -> None: ...
|
||||
|
||||
def list(self) -> list[tuple[str, str]]: ...
|
||||
@property
|
||||
def url(self) -> str | None: ...
|
||||
|
||||
@@ -28,6 +28,12 @@ class UnauthorizedException(MDRSException):
|
||||
pass
|
||||
|
||||
|
||||
class FolderLockedException(UnauthorizedException):
|
||||
"""Thrown when a folder cannot be opened without its password"""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class TokenExpiredException(UnauthorizedException):
|
||||
"""Thrown when the access token was still valid when sent but had expired on arrival"""
|
||||
|
||||
|
||||
+90
-25
@@ -1,4 +1,5 @@
|
||||
from typing import Any, Final, NamedTuple
|
||||
import re
|
||||
from typing import Any, Final, Mapping
|
||||
from unicodedata import normalize
|
||||
|
||||
from pydantic.dataclasses import dataclass
|
||||
@@ -6,39 +7,99 @@ from pydantic.dataclasses import dataclass
|
||||
from mdrsclient.models.utils import iso8601_to_user_friendly
|
||||
|
||||
|
||||
class FolderAccessLevelItem(NamedTuple):
|
||||
mask: int
|
||||
key: str
|
||||
label: str
|
||||
|
||||
|
||||
class FolderAccessLevel:
|
||||
# Bit Mask
|
||||
# - bit 0: Is Private
|
||||
# - bit 1: Is Public
|
||||
# - bit 2: With Password
|
||||
# - bit 3-7: (Reserved)
|
||||
# - bit 8-15: Restricted Open
|
||||
ACCESS_LEVELS: Final[list[FolderAccessLevelItem]] = [
|
||||
FolderAccessLevelItem(0x0204, "5kikan_or_pw_open", "5Kikan or PW Open"),
|
||||
FolderAccessLevelItem(0x0104, "cbs_or_pw_open", "CBS or PW Open"),
|
||||
FolderAccessLevelItem(0x0200, "5kikan_open", "5Kikan Open"),
|
||||
FolderAccessLevelItem(0x0100, "cbs_open", "CBS Open"),
|
||||
FolderAccessLevelItem(0x0004, "pw_open", "PW Open"),
|
||||
FolderAccessLevelItem(0x0002, "public", "Public"),
|
||||
FolderAccessLevelItem(0x0001, "private", "Private"),
|
||||
FolderAccessLevelItem(0x0000, "storage", "Storage"),
|
||||
]
|
||||
# - bit 8-15: Restricted Open, one bit per restrict network slot (slot 1 is bit 8)
|
||||
# - bit 16-: Project Open, one bit per project bit_position (bit_position 0 is bit 16)
|
||||
STORAGE: Final[int] = 0x0000
|
||||
PRIVATE: Final[int] = 0x0001
|
||||
PUBLIC: Final[int] = 0x0002
|
||||
PASSWORD: Final[int] = 0x0004
|
||||
RESERVED: Final[int] = 0x00F8
|
||||
RESTRICT_SHIFT: Final[int] = 8
|
||||
RESTRICT_SLOTS: Final[int] = 8
|
||||
PROJECT_SHIFT: Final[int] = 16
|
||||
# The server keeps no names for its restrict networks, so the one in use is named here.
|
||||
# Any other slot is named after its number.
|
||||
RESTRICT_NAMES: Final[dict[int, str]] = {1: "CBS"}
|
||||
LEVEL_NAMES: Final[dict[int, str]] = {STORAGE: "Storage", PRIVATE: "Private", PUBLIC: "Public"}
|
||||
|
||||
@staticmethod
|
||||
def restrict_name(slot: int) -> str:
|
||||
return FolderAccessLevel.RESTRICT_NAMES.get(slot, f"Restrict{slot}")
|
||||
|
||||
@staticmethod
|
||||
def project_name(bit_position: int, projects: Mapping[int, str] | None = None) -> str:
|
||||
name = (projects or {}).get(bit_position)
|
||||
return name if name else f"Project{bit_position}"
|
||||
|
||||
@staticmethod
|
||||
def id2label(id: int, projects: Mapping[int, str] | None = None) -> str | None:
|
||||
"""
|
||||
Label an access level from its bits, e.g. 0x0104 is "CBS or PW Open".
|
||||
|
||||
`projects` names the projects by bit_position, as the server tells their members;
|
||||
any other project gets a name made from its bit_position. None for a level that
|
||||
has no meaning: Private or Public with any other bit, or a reserved bit.
|
||||
"""
|
||||
if id in FolderAccessLevel.LEVEL_NAMES:
|
||||
return FolderAccessLevel.LEVEL_NAMES[id]
|
||||
if id < 0 or id & (FolderAccessLevel.PRIVATE | FolderAccessLevel.PUBLIC | FolderAccessLevel.RESERVED):
|
||||
return None
|
||||
names = [
|
||||
FolderAccessLevel.restrict_name(slot)
|
||||
for slot in range(1, FolderAccessLevel.RESTRICT_SLOTS + 1)
|
||||
if id & (1 << (FolderAccessLevel.RESTRICT_SHIFT + slot - 1))
|
||||
]
|
||||
project_bits = id >> FolderAccessLevel.PROJECT_SHIFT
|
||||
names += [
|
||||
FolderAccessLevel.project_name(bit, projects)
|
||||
for bit in range(project_bits.bit_length())
|
||||
if project_bits & (1 << bit)
|
||||
]
|
||||
if id & FolderAccessLevel.PASSWORD:
|
||||
names.append("PW")
|
||||
return " or ".join(names) + " Open"
|
||||
|
||||
@staticmethod
|
||||
def key2id(key: str) -> int | None:
|
||||
acl = next((x for x in FolderAccessLevel.ACCESS_LEVELS if x.key == key), None)
|
||||
return acl.mask if acl is not None else None
|
||||
"""
|
||||
The access level a key names: `private`, `public`, `storage`, or the parts of an
|
||||
open level joined by `_or_` and ending in `_open`, e.g. `cbs_or_pw_open`. A part
|
||||
is `pw`, `cbs`, `restrict<slot>` or `project<bit_position>`.
|
||||
"""
|
||||
fixed = {label.lower(): id for id, label in FolderAccessLevel.LEVEL_NAMES.items()}
|
||||
if key in fixed:
|
||||
return fixed[key]
|
||||
if not key.endswith("_open"):
|
||||
return None
|
||||
id = 0
|
||||
for part in key.removesuffix("_open").split("_or_"):
|
||||
bit = FolderAccessLevel._part2bit(part)
|
||||
if bit is None:
|
||||
return None
|
||||
id |= bit
|
||||
return id
|
||||
|
||||
@staticmethod
|
||||
def id2label(id: int) -> str | None:
|
||||
acl = next((x for x in FolderAccessLevel.ACCESS_LEVELS if (x.mask & id) == x.mask), None)
|
||||
return acl.label if acl is not None else None
|
||||
def _part2bit(part: str) -> int | None:
|
||||
if part == "pw":
|
||||
return FolderAccessLevel.PASSWORD
|
||||
slots = {name.lower(): slot for slot, name in FolderAccessLevel.RESTRICT_NAMES.items()}
|
||||
match = re.fullmatch(r"restrict([1-9][0-9]*)", part)
|
||||
slot = slots.get(part, int(match.group(1)) if match else None)
|
||||
if slot is not None:
|
||||
if slot > FolderAccessLevel.RESTRICT_SLOTS:
|
||||
return None
|
||||
return 1 << (FolderAccessLevel.RESTRICT_SHIFT + slot - 1)
|
||||
match = re.fullmatch(r"project(0|[1-9][0-9]*)", part)
|
||||
if match:
|
||||
return 1 << (FolderAccessLevel.PROJECT_SHIFT + int(match.group(1)))
|
||||
return None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -55,10 +116,14 @@ class FolderSimple:
|
||||
updated_at: str
|
||||
restrict_opened_at: str | None
|
||||
|
||||
def access_level_label(self, projects: Mapping[int, str] | None = None) -> str:
|
||||
"""The access level as shown to a user, naming the projects given by bit_position."""
|
||||
label = FolderAccessLevel.id2label(self.access_level, projects)
|
||||
return label if label is not None else f"Unknown (0x{self.access_level:x})"
|
||||
|
||||
@property
|
||||
def access_level_name(self) -> str:
|
||||
label = FolderAccessLevel.id2label(self.access_level)
|
||||
return label if label is not None else ""
|
||||
return self.access_level_label()
|
||||
|
||||
@property
|
||||
def lock_name(self) -> str:
|
||||
|
||||
+47
-9
@@ -1,16 +1,16 @@
|
||||
import os
|
||||
import re
|
||||
from typing import Any
|
||||
from unicodedata import normalize
|
||||
|
||||
from mdrsclient.api import DoiApi, FilesApi, FoldersApi, LaboratoriesApi, UsersApi
|
||||
from mdrsclient.api.utils import token_check
|
||||
from mdrsclient.cache import CacheInterface
|
||||
from mdrsclient.config import ConfigFile, ConfigInterface
|
||||
from mdrsclient.connection import MDRSConnection
|
||||
from mdrsclient.exceptions import (
|
||||
FolderLockedException,
|
||||
IllegalArgumentException,
|
||||
MissingConfigurationException,
|
||||
UnauthorizedException,
|
||||
UnexpectedException,
|
||||
)
|
||||
from mdrsclient.models import File, Folder, Laboratory, Token, User
|
||||
@@ -20,11 +20,30 @@ from mdrsclient.utils import page_num_from_url
|
||||
class MdrsService:
|
||||
config_class: type[ConfigInterface] = ConfigFile
|
||||
|
||||
def __init__(self, connection: MDRSConnection, config_class: type[ConfigInterface] | None = None):
|
||||
self.connection = connection
|
||||
def __init__(self, connection: MDRSConnection | None, config_class: type[ConfigInterface] | None = None):
|
||||
self.__connection = connection
|
||||
if config_class is not None:
|
||||
self.config_class = config_class
|
||||
|
||||
@property
|
||||
def connection(self) -> MDRSConnection:
|
||||
"""
|
||||
The connection every remote operation goes through.
|
||||
|
||||
Optional to supply, because `config` and `version` do their work without a remote
|
||||
and are reached through the same client. Asking for it when none was given is a
|
||||
mistake in the caller, and says so rather than failing later on `None`.
|
||||
"""
|
||||
if self.__connection is None:
|
||||
raise MissingConfigurationException("This operation requires a remote host.")
|
||||
return self.__connection
|
||||
|
||||
@connection.setter
|
||||
def connection(self, connection: MDRSConnection | None) -> None:
|
||||
# Assignable as it always was: this is a property to check for absence on read,
|
||||
# not to make the connection fixed once the client is built.
|
||||
self.__connection = connection
|
||||
|
||||
@classmethod
|
||||
def create_connection(
|
||||
cls, remote: str, cache: CacheInterface | None = None, config: ConfigInterface | None = None
|
||||
@@ -48,6 +67,17 @@ class MdrsService:
|
||||
def logout(self) -> None:
|
||||
self.connection.logout()
|
||||
|
||||
def project_names(self) -> dict[int, str]:
|
||||
"""
|
||||
Names of the logged-in user's projects by bit_position, for labelling access levels.
|
||||
|
||||
Empty when nobody is logged in: the server names a project only to its members.
|
||||
"""
|
||||
token_check(self.connection)
|
||||
if self.connection.token is None:
|
||||
return {}
|
||||
return UsersApi(self.connection).project_names()
|
||||
|
||||
def whoami(self) -> User:
|
||||
user_api = UsersApi(self.connection)
|
||||
return user_api.current()
|
||||
@@ -73,7 +103,7 @@ class MdrsService:
|
||||
raise UnexpectedException(f"Folder `{path}` not found.")
|
||||
if folders[0].lock:
|
||||
if password is None:
|
||||
raise UnauthorizedException(f"Folder `{path}` is locked.")
|
||||
raise FolderLockedException(f"Folder `{path}` is locked.")
|
||||
folder_api.auth(folders[0].id, password)
|
||||
return folder_api.retrieve(folders[0].id)
|
||||
|
||||
@@ -177,12 +207,20 @@ class MdrsService:
|
||||
)
|
||||
|
||||
folder_api = FoldersApi(self.connection)
|
||||
folder = folder_api.retrieve(doi_resp.folder_id)
|
||||
# The folder is found by its id, so whether it is locked is known only from what
|
||||
# retrieve returns: a full record, or the short one of a locked folder.
|
||||
folder: Folder | None
|
||||
try:
|
||||
folder = folder_api.retrieve(doi_resp.folder_id)
|
||||
except FolderLockedException:
|
||||
folder = None
|
||||
|
||||
if folder.lock:
|
||||
if folder is None or folder.lock:
|
||||
if password is None:
|
||||
raise UnauthorizedException(f"Folder for DOI `{doi_clean}` is locked.")
|
||||
folder_api.auth(doi_resp.folder.id, password)
|
||||
raise FolderLockedException(f"Folder for DOI `{doi_clean}` is locked.")
|
||||
folder_api.auth(doi_resp.folder_id, password)
|
||||
if folder is None:
|
||||
folder = folder_api.retrieve(doi_resp.folder_id)
|
||||
|
||||
lab_api = LaboratoriesApi(self.connection)
|
||||
labs = lab_api.list()
|
||||
|
||||
+83
-68
@@ -154,86 +154,101 @@ class Downloader:
|
||||
) -> bool:
|
||||
"""Fetch what the remote path names, and report whether every file arrived."""
|
||||
excludes_clean = excludes or []
|
||||
# Detect DOI path: "remote:10.xxxx/prefix.ID[/optional/sub/path]"
|
||||
l_dirname = os.path.realpath(local_path)
|
||||
if not os.path.isdir(l_dirname):
|
||||
raise IllegalArgumentException(f"Local directory `{local_path}` not found.")
|
||||
|
||||
# "remote:10.xxxx/prefix.ID[/optional/sub/path]" names a published dataset rather
|
||||
# than a path within a laboratory, and is resolved through the DOI instead.
|
||||
path_component = remote_path.split(":", 1)[1] if ":" in remote_path else ""
|
||||
if self.client.is_doi(path_component):
|
||||
remote, doi, subpath = self.client.parse_doi_remote_host(remote_path)
|
||||
|
||||
l_dirname = os.path.realpath(local_path)
|
||||
if not os.path.isdir(l_dirname):
|
||||
raise IllegalArgumentException(f"Local directory `{local_path}` not found.")
|
||||
doi_folder, laboratory = self.client.find_folder_by_doi(doi, password)
|
||||
|
||||
subpath_clean = subpath.rstrip("/")
|
||||
if not subpath_clean:
|
||||
folder = doi_folder
|
||||
is_folder = True
|
||||
else:
|
||||
r_dirname = os.path.dirname(subpath_clean)
|
||||
r_basename = os.path.basename(subpath_clean)
|
||||
abs_path = doi_folder.path.rstrip("/") + r_dirname
|
||||
r_parent_folder = self.client.find_folder(laboratory, abs_path, password)
|
||||
r_parent_files = self.client.find_files(r_parent_folder.id)
|
||||
file = find_file(r_parent_files, r_basename)
|
||||
if file is not None:
|
||||
if self.__check_excludes(excludes_clean, laboratory, r_parent_folder, file):
|
||||
return True
|
||||
context = DownloadContext(is_skip_if_exists, [])
|
||||
l_path = os.path.join(l_dirname, r_basename)
|
||||
context.files.append(DownloadFileInfo(file, l_path))
|
||||
return self.__multiple_download(context)
|
||||
else:
|
||||
folder_simple = r_parent_folder.find_sub_folder(r_basename)
|
||||
if folder_simple is None:
|
||||
raise IllegalArgumentException(f"File or folder `{subpath_clean}` not found.")
|
||||
folder = FoldersApi(self.client.connection).retrieve(folder_simple.id)
|
||||
is_folder = True
|
||||
|
||||
# For a DOI target the whole folder is the download target.
|
||||
if not is_recursive:
|
||||
# Non-recursive: download only the files at the top level of the DOI folder.
|
||||
files = self.client.find_files(folder.id)
|
||||
context = DownloadContext(is_skip_if_exists, [])
|
||||
for file in files:
|
||||
if self.__check_excludes(excludes_clean, laboratory, folder, file):
|
||||
continue
|
||||
l_path = os.path.join(l_dirname, file.name)
|
||||
context.files.append(DownloadFileInfo(file, l_path))
|
||||
return self.__multiple_download(context)
|
||||
folder_api = FoldersApi(self.client.connection)
|
||||
return self.__multiple_download_pickup_recursive_files(
|
||||
folder_api, laboratory, folder.id, l_dirname, excludes_clean, is_skip_if_exists
|
||||
return self.__download_doi(
|
||||
remote_path, l_dirname, is_recursive, is_skip_if_exists, password, excludes_clean
|
||||
)
|
||||
|
||||
remote, laboratory_name, r_path = self.client.parse_remote_host_with_path(remote_path)
|
||||
r_path = r_path.rstrip("/")
|
||||
r_dirname = os.path.dirname(r_path)
|
||||
r_basename = os.path.basename(r_path)
|
||||
|
||||
l_dirname = os.path.realpath(local_path)
|
||||
if not os.path.isdir(l_dirname):
|
||||
raise IllegalArgumentException(f"Local directory `{local_path}` not found.")
|
||||
laboratory = self.client.find_laboratory(laboratory_name)
|
||||
r_parent_folder = self.client.find_folder(laboratory, r_dirname, password)
|
||||
r_parent_folder = self.client.find_folder(laboratory, os.path.dirname(r_path), password)
|
||||
r_parent_files = self.client.find_files(r_parent_folder.id)
|
||||
file = find_file(r_parent_files, r_basename)
|
||||
if file is not None:
|
||||
if self.__check_excludes(excludes_clean, laboratory, r_parent_folder, file):
|
||||
return True
|
||||
context = DownloadContext(is_skip_if_exists, [])
|
||||
l_path = os.path.join(l_dirname, r_basename)
|
||||
context.files.append(DownloadFileInfo(file, l_path))
|
||||
return self.__multiple_download(context)
|
||||
else:
|
||||
folder = r_parent_folder.find_sub_folder(r_basename)
|
||||
if folder is None:
|
||||
raise IllegalArgumentException(f"File or folder `{r_path}` not found.")
|
||||
if not is_recursive:
|
||||
raise IllegalArgumentException(f"Cannot download `{r_path}`: Is a folder.")
|
||||
folder_api = FoldersApi(self.client.connection)
|
||||
return self.__multiple_download_pickup_recursive_files(
|
||||
folder_api, laboratory, folder.id, l_dirname, excludes_clean, is_skip_if_exists
|
||||
return self.__download_one(
|
||||
excludes_clean, laboratory, r_parent_folder, file, l_dirname, r_basename, is_skip_if_exists
|
||||
)
|
||||
folder = r_parent_folder.find_sub_folder(r_basename)
|
||||
if folder is None:
|
||||
raise IllegalArgumentException(f"File or folder `{r_path}` not found.")
|
||||
if not is_recursive:
|
||||
raise IllegalArgumentException(f"Cannot download `{r_path}`: Is a folder.")
|
||||
return self.__multiple_download_pickup_recursive_files(
|
||||
FoldersApi(self.client.connection), laboratory, folder.id, l_dirname, excludes_clean, is_skip_if_exists
|
||||
)
|
||||
|
||||
def __download_doi(
|
||||
self,
|
||||
remote_path: str,
|
||||
l_dirname: str,
|
||||
is_recursive: bool,
|
||||
is_skip_if_exists: bool,
|
||||
password: str | None,
|
||||
excludes: list[str],
|
||||
) -> bool:
|
||||
"""Fetch what a DOI names: the dataset's folder, or something inside it."""
|
||||
remote, doi, subpath = self.client.parse_doi_remote_host(remote_path)
|
||||
doi_folder, laboratory = self.client.find_folder_by_doi(doi, password)
|
||||
|
||||
subpath_clean = subpath.rstrip("/")
|
||||
if not subpath_clean:
|
||||
folder = doi_folder
|
||||
else:
|
||||
r_basename = os.path.basename(subpath_clean)
|
||||
abs_path = doi_folder.path.rstrip("/") + os.path.dirname(subpath_clean)
|
||||
r_parent_folder = self.client.find_folder(laboratory, abs_path, password)
|
||||
file = find_file(self.client.find_files(r_parent_folder.id), r_basename)
|
||||
if file is not None:
|
||||
return self.__download_one(
|
||||
excludes, laboratory, r_parent_folder, file, l_dirname, r_basename, is_skip_if_exists
|
||||
)
|
||||
folder_simple = r_parent_folder.find_sub_folder(r_basename)
|
||||
if folder_simple is None:
|
||||
raise IllegalArgumentException(f"File or folder `{subpath_clean}` not found.")
|
||||
folder = FoldersApi(self.client.connection).retrieve(folder_simple.id)
|
||||
|
||||
if is_recursive:
|
||||
return self.__multiple_download_pickup_recursive_files(
|
||||
FoldersApi(self.client.connection), laboratory, folder.id, l_dirname, excludes, is_skip_if_exists
|
||||
)
|
||||
# Without -r the dataset's own files are fetched, and its sub-folders are not.
|
||||
context = DownloadContext(is_skip_if_exists, [])
|
||||
for file in self.client.find_files(folder.id):
|
||||
if self.__check_excludes(excludes, laboratory, folder, file):
|
||||
continue
|
||||
context.files.append(DownloadFileInfo(file, os.path.join(l_dirname, file.name)))
|
||||
return self.__multiple_download(context)
|
||||
|
||||
def __download_one(
|
||||
self,
|
||||
excludes: list[str],
|
||||
laboratory: Laboratory,
|
||||
folder: Folder,
|
||||
file: File,
|
||||
l_dirname: str,
|
||||
local_name: str,
|
||||
is_skip_if_exists: bool,
|
||||
) -> bool:
|
||||
"""
|
||||
Fetch a single named file into the local directory.
|
||||
|
||||
Saved under the name the caller asked for rather than the one the server holds:
|
||||
the two are matched case-insensitively, so they need not be spelled alike.
|
||||
"""
|
||||
if self.__check_excludes(excludes, laboratory, folder, file):
|
||||
return True
|
||||
context = DownloadContext(is_skip_if_exists, [])
|
||||
context.files.append(DownloadFileInfo(file, os.path.join(l_dirname, local_name)))
|
||||
return self.__multiple_download(context)
|
||||
|
||||
def __multiple_download_pickup_recursive_files(
|
||||
self,
|
||||
|
||||
+22
-14
@@ -1,5 +1,5 @@
|
||||
[build-system]
|
||||
requires = ["flit_core>=3.12.0,<4.0.0"]
|
||||
requires = ["flit_core>=4.0.2,<5.0.0"]
|
||||
build-backend = "flit_core.buildapi"
|
||||
|
||||
[tool.flit.module]
|
||||
@@ -7,20 +7,19 @@ name = "mdrsclient"
|
||||
|
||||
[project]
|
||||
name = "mdrs-client-python"
|
||||
version = "1.3.18"
|
||||
version = "1.3.21"
|
||||
description = "The mdrs-client-python is python library and a command-line client for up- and downloading files to and from MDRS based repository."
|
||||
authors = [
|
||||
{ name = "Yoshihiro OKUMURA", email = "yoshihiro.okumura@riken.jp" }
|
||||
]
|
||||
license = { text = "MIT" }
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
requires-python = ">=3.11"
|
||||
classifiers = [
|
||||
"Development Status :: 4 - Beta",
|
||||
"Environment :: Console",
|
||||
"Intended Audience :: Developers",
|
||||
"Intended Audience :: Science/Research",
|
||||
"Programming Language :: Python :: 3.10",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Programming Language :: Python :: 3.13",
|
||||
@@ -32,20 +31,20 @@ classifiers = [
|
||||
dependencies = [
|
||||
"requests>=2.34.2",
|
||||
"requests-toolbelt>=1.0.0",
|
||||
"python-dotenv>=1.2.2",
|
||||
"pydantic>=2.13.4",
|
||||
"pydantic-settings>=2.14.2",
|
||||
"PyJWT>=2.13.0",
|
||||
"validators>=0.35.0",
|
||||
"python-dotenv>=1.2.4",
|
||||
"pydantic>=2.14.3",
|
||||
"pydantic-settings>=2.16.2",
|
||||
"PyJWT>=2.15.1",
|
||||
"validators>=0.36.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"black>=26.5.1",
|
||||
"flake8>=7.3.0",
|
||||
"black>=26.10.1",
|
||||
"flake8>=7.5.0",
|
||||
"Flake8-pyproject>=1.2.4",
|
||||
"isort>=8.0.1",
|
||||
"pyright>=1.1.411",
|
||||
"isort>=9.0.2",
|
||||
"pyright>=1.1.414",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
@@ -53,7 +52,7 @@ mdrs = "mdrsclient.__main__:main"
|
||||
|
||||
[tool.black]
|
||||
line-length = 120
|
||||
target-version = ['py310']
|
||||
target-version = ['py311']
|
||||
include = '\.pyi?$'
|
||||
exclude = '''
|
||||
/(
|
||||
@@ -68,6 +67,9 @@ exclude = '''
|
||||
exclude = ".git, .venv, __pycache__, dist"
|
||||
max-complexity = 10
|
||||
max-line-length = 120
|
||||
# E203 flags the space black puts before a slice colon. Black formats this project, so
|
||||
# its output is the standard and flake8 has to accept it.
|
||||
extend-ignore = ["E203"]
|
||||
|
||||
[tool.isort]
|
||||
profile = "black"
|
||||
@@ -77,6 +79,12 @@ line_length = 120
|
||||
|
||||
[tool.pyright]
|
||||
typeCheckingMode = "basic"
|
||||
# This project keeps its environment in `./.venv`. Named explicitly because without it
|
||||
# pyright falls back to the system interpreter, cannot resolve pydantic, and reports every
|
||||
# model field as an unknown argument. An environment kept elsewhere gets that same fallback
|
||||
# and can override it with a local `pyrightconfig.json` or `pyright --pythonpath`.
|
||||
venvPath = "."
|
||||
venv = ".venv"
|
||||
exclude = ["**/__pycache__", "**/.*", "dist"]
|
||||
#reportUnknownMemberType = "warning"
|
||||
#reportUnknownVariableType = "warning"
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from mdrsclient.client import MdrsClient
|
||||
from mdrsclient.exceptions import IllegalArgumentException
|
||||
from mdrsclient.models import File, Folder, FolderSimple, Laboratories, Laboratory
|
||||
|
||||
TIMESTAMP = "2026-01-01T00:00:00+09:00"
|
||||
LABORATORY = Laboratory(id=1, name="mylab", pi_name="PI", full_name="My Laboratory")
|
||||
|
||||
|
||||
def make_file(id: str, name: str) -> File:
|
||||
return File(
|
||||
id=id,
|
||||
name=name,
|
||||
type="text/plain",
|
||||
size=1,
|
||||
thumbnail=None,
|
||||
description="",
|
||||
metadata={},
|
||||
download_url=f"v3/files/{id}/download/",
|
||||
created_at=TIMESTAMP,
|
||||
updated_at=TIMESTAMP,
|
||||
)
|
||||
|
||||
|
||||
def make_sub_folder(id: str, name: str) -> FolderSimple:
|
||||
return FolderSimple(
|
||||
id=id,
|
||||
pid=None,
|
||||
name=name,
|
||||
access_level=1,
|
||||
lock=False,
|
||||
size=0,
|
||||
laboratory_id=1,
|
||||
description="",
|
||||
created_at=TIMESTAMP,
|
||||
updated_at=TIMESTAMP,
|
||||
restrict_opened_at=None,
|
||||
)
|
||||
|
||||
|
||||
def make_folder(id: str, name: str, path: str, sub_folders: list[FolderSimple] | None = None) -> Folder:
|
||||
return Folder(
|
||||
id=id,
|
||||
pid=None,
|
||||
name=name,
|
||||
access_level=1,
|
||||
lock=False,
|
||||
size=0,
|
||||
laboratory_id=1,
|
||||
description="",
|
||||
created_at=TIMESTAMP,
|
||||
updated_at=TIMESTAMP,
|
||||
restrict_opened_at=None,
|
||||
metadata=[],
|
||||
sub_folders=sub_folders if sub_folders is not None else [],
|
||||
path=path,
|
||||
)
|
||||
|
||||
|
||||
class TestLsColumnWidths(unittest.TestCase):
|
||||
"""Each column is as wide as the widest thing printed under it."""
|
||||
|
||||
def test_the_size_column_fits_a_sub_folder_larger_than_its_parent(self):
|
||||
from unittest.mock import MagicMock as _MagicMock
|
||||
|
||||
from mdrsclient.commands.ls import LsCommand, LsCommandContext
|
||||
|
||||
laboratories = Laboratories()
|
||||
laboratories.append(LABORATORY)
|
||||
connection = _MagicMock()
|
||||
connection.laboratories = laboratories
|
||||
client = MdrsClient(connection)
|
||||
|
||||
big = make_sub_folder("s1", "big")
|
||||
object.__setattr__(big, "size", 123456789)
|
||||
parent = make_folder("f1", "root", "/root/", [big])
|
||||
context = LsCommandContext("remote:/mylab", client, LABORATORY, "", False, False, False)
|
||||
|
||||
widths = LsCommand._column_widths(context, parent, [])
|
||||
|
||||
self.assertEqual(widths["size"], len("123456789"))
|
||||
|
||||
|
||||
class TestCopyAndMove(unittest.TestCase):
|
||||
"""`cp` and `mv` resolve both ends the same way and differ only in what they call."""
|
||||
|
||||
def make_client(self, folders_by_path: dict, files_by_folder: dict) -> MdrsClient:
|
||||
client = MdrsClient(MagicMock())
|
||||
client.find_laboratory = MagicMock(return_value=LABORATORY)
|
||||
client.find_folder = MagicMock(side_effect=lambda lab, path, password=None: folders_by_path[path])
|
||||
client.find_files = MagicMock(side_effect=lambda folder_id: files_by_folder.get(folder_id, []))
|
||||
return client
|
||||
|
||||
def make_tree(self, source_has: str) -> tuple[MdrsClient, dict, dict]:
|
||||
sub = make_sub_folder("fsub", "sub")
|
||||
folder_a = make_folder("fa", "a", "/a/", [sub] if source_has == "folder" else [])
|
||||
folder_b = make_folder("fb", "b", "/b/")
|
||||
folders = {"/a": folder_a, "/b": folder_b, "/b/": folder_b}
|
||||
files = {"fa": [make_file("x1", "data.txt")] if source_has == "file" else [], "fb": []}
|
||||
return self.make_client(folders, files), folders, files
|
||||
|
||||
def test_a_file_is_copied_into_the_destination_folder(self):
|
||||
client, _, files = self.make_tree("file")
|
||||
with patch("mdrsclient.client.FilesApi") as files_api:
|
||||
client.cp("myremote:/mylab/a/data.txt", "myremote:/mylab/b/data.txt")
|
||||
files_api.return_value.copy.assert_called_once_with(files["fa"][0], "fb", "data.txt")
|
||||
|
||||
def test_a_file_is_moved_into_the_destination_folder(self):
|
||||
client, _, files = self.make_tree("file")
|
||||
with patch("mdrsclient.client.FilesApi") as files_api:
|
||||
client.mv("myremote:/mylab/a/data.txt", "myremote:/mylab/b/data.txt")
|
||||
files_api.return_value.move.assert_called_once_with(files["fa"][0], "fb", "data.txt")
|
||||
|
||||
def test_a_trailing_separator_keeps_the_source_name(self):
|
||||
client, _, files = self.make_tree("file")
|
||||
with patch("mdrsclient.client.FilesApi") as files_api:
|
||||
client.mv("myremote:/mylab/a/data.txt", "myremote:/mylab/b/")
|
||||
files_api.return_value.move.assert_called_once_with(files["fa"][0], "fb", "data.txt")
|
||||
|
||||
def test_copying_a_folder_needs_the_recursive_flag(self):
|
||||
client, _, _ = self.make_tree("folder")
|
||||
with patch("mdrsclient.client.FoldersApi") as folders_api:
|
||||
with self.assertRaises(IllegalArgumentException) as caught:
|
||||
client.cp("myremote:/mylab/a/sub", "myremote:/mylab/b/sub")
|
||||
self.assertIn("Is a folder", str(caught.exception))
|
||||
folders_api.return_value.copy.assert_not_called()
|
||||
|
||||
def test_a_folder_is_copied_when_recursive(self):
|
||||
client, folders, _ = self.make_tree("folder")
|
||||
with patch("mdrsclient.client.FoldersApi") as folders_api:
|
||||
client.cp("myremote:/mylab/a/sub", "myremote:/mylab/b/sub", is_recursive=True)
|
||||
folders_api.return_value.copy.assert_called_once_with(folders["/a"].sub_folders[0], "fb", "sub")
|
||||
|
||||
def test_a_folder_is_moved_without_the_recursive_flag(self):
|
||||
client, folders, _ = self.make_tree("folder")
|
||||
with patch("mdrsclient.client.FoldersApi") as folders_api:
|
||||
client.mv("myremote:/mylab/a/sub", "myremote:/mylab/b/sub")
|
||||
folders_api.return_value.move.assert_called_once_with(folders["/a"].sub_folders[0], "fb", "sub")
|
||||
|
||||
def test_a_destination_that_already_holds_the_name_is_refused(self):
|
||||
sub = make_sub_folder("fsub", "sub")
|
||||
folders = {"/a": make_folder("fa", "a", "/a/", [sub]), "/b": make_folder("fb", "b", "/b/")}
|
||||
files = {"fa": [], "fb": [make_file("x2", "sub")]}
|
||||
client = self.make_client(folders, files)
|
||||
with self.assertRaises(IllegalArgumentException) as caught:
|
||||
client.mv("myremote:/mylab/a/sub", "myremote:/mylab/b/sub")
|
||||
self.assertIn("Cannot overwrite non-folder", str(caught.exception))
|
||||
|
||||
def test_a_transfer_across_laboratories_is_refused(self):
|
||||
client, _, _ = self.make_tree("file")
|
||||
with self.assertRaises(IllegalArgumentException) as caught:
|
||||
client.cp("myremote:/mylab/a/data.txt", "myremote:/otherlab/b/data.txt")
|
||||
self.assertIn("Laboratory mismatched", str(caught.exception))
|
||||
|
||||
def test_a_source_that_does_not_exist_is_refused(self):
|
||||
client, _, _ = self.make_tree("file")
|
||||
with self.assertRaises(IllegalArgumentException) as caught:
|
||||
client.mv("myremote:/mylab/a/missing.txt", "myremote:/mylab/b/missing.txt")
|
||||
self.assertIn("not found", str(caught.exception))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -23,6 +23,7 @@ from mdrsclient.commands import (
|
||||
VersionCommand,
|
||||
WhoamiCommand,
|
||||
)
|
||||
from mdrsclient.exceptions import IllegalArgumentException, MissingConfigurationException
|
||||
from mdrsclient.models import Folder, Laboratory
|
||||
|
||||
|
||||
@@ -177,6 +178,7 @@ class TestCommands(unittest.TestCase):
|
||||
mock_lab = Laboratory(id=1, name="mylab", pi_name="pi_name", full_name="full_name")
|
||||
|
||||
mock_client.resolve_folder.return_value = (mock_folder, mock_lab)
|
||||
mock_client.project_names.return_value = {}
|
||||
|
||||
mock_file = MagicMock()
|
||||
mock_file.name = "file.txt"
|
||||
@@ -228,6 +230,7 @@ class TestCommands(unittest.TestCase):
|
||||
mock_lab = Laboratory(id=1, name="mylab", pi_name="pi_name", full_name="full_name")
|
||||
|
||||
mock_client.resolve_folder.return_value = (mock_folder, mock_lab)
|
||||
mock_client.project_names.return_value = {}
|
||||
mock_client.connection.laboratories.find_by_id.return_value = mock_lab
|
||||
|
||||
mock_file = MagicMock()
|
||||
@@ -316,6 +319,48 @@ class TestCommands(unittest.TestCase):
|
||||
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||
mock_client.chacl.assert_called_once_with("myremote:/mylab/", 1, True, "secret")
|
||||
|
||||
def test_chacl_command_project_level(self):
|
||||
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||
mock_client = MagicMock()
|
||||
mock_client_class.from_remote.return_value = mock_client
|
||||
|
||||
args = self.parse_args(ChaclCommand, ["chacl", "cbs_or_project0_open", "myremote:/mylab/"])
|
||||
args.func(args)
|
||||
|
||||
mock_client.chacl.assert_called_once_with("myremote:/mylab/", 0x10100, False, None)
|
||||
|
||||
def test_chacl_command_invalid_level_lists_projects(self):
|
||||
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||
mock_client = MagicMock()
|
||||
mock_client_class.from_remote.return_value = mock_client
|
||||
mock_client.project_names.return_value = {0: "Brain/MINDS 2.0"}
|
||||
|
||||
args = self.parse_args(ChaclCommand, ["chacl", "5kikan_open", "myremote:/mylab/"])
|
||||
with self.assertRaises(IllegalArgumentException) as cm:
|
||||
args.func(args)
|
||||
|
||||
self.assertIn("project<n>_open", str(cm.exception))
|
||||
self.assertIn("project0_open: Brain/MINDS 2.0", str(cm.exception))
|
||||
mock_client.chacl.assert_not_called()
|
||||
|
||||
def test_chacl_command_invalid_level_without_server(self):
|
||||
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||
mock_client_class.from_remote.side_effect = MissingConfigurationException("no remote")
|
||||
|
||||
args = self.parse_args(ChaclCommand, ["chacl", "5kikan_open", "myremote:/mylab/"])
|
||||
with self.assertRaises(IllegalArgumentException) as cm:
|
||||
args.func(args)
|
||||
|
||||
self.assertNotIn("Projects you belong to", str(cm.exception))
|
||||
|
||||
def test_chacl_command_refuses_public_and_storage(self):
|
||||
for level in ("public", "storage"):
|
||||
with self.subTest(level=level), patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||
args = self.parse_args(ChaclCommand, ["chacl", level, "myremote:/mylab/"])
|
||||
with self.assertRaises(IllegalArgumentException):
|
||||
args.func(args)
|
||||
mock_client_class.from_remote.assert_not_called()
|
||||
|
||||
def test_metadata_command(self):
|
||||
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||
mock_client = MagicMock()
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from mdrsclient.api import UsersApi
|
||||
from mdrsclient.models import FolderSimple
|
||||
from mdrsclient.models.folder import FolderAccessLevel
|
||||
from mdrsclient.services import MdrsService
|
||||
|
||||
TIMESTAMP = "2026-01-01T00:00:00+09:00"
|
||||
BM2 = {0: "Brain/MINDS 2.0"}
|
||||
|
||||
|
||||
def make_folder(access_level: int) -> FolderSimple:
|
||||
return FolderSimple(
|
||||
id="f1",
|
||||
pid=None,
|
||||
name="data",
|
||||
access_level=access_level,
|
||||
lock=False,
|
||||
size=0,
|
||||
laboratory_id=1,
|
||||
description="",
|
||||
created_at=TIMESTAMP,
|
||||
updated_at=TIMESTAMP,
|
||||
restrict_opened_at=None,
|
||||
)
|
||||
|
||||
|
||||
def current_user(projects: list[dict]) -> dict:
|
||||
return {
|
||||
"id": 1,
|
||||
"username": "user",
|
||||
"first_name": "",
|
||||
"last_name": "",
|
||||
"email": "",
|
||||
"orcid_id": "",
|
||||
"laboratories": [],
|
||||
"is_staff": False,
|
||||
"is_active": True,
|
||||
"is_superuser": False,
|
||||
"is_reviewer": False,
|
||||
"last_login": TIMESTAMP,
|
||||
"date_joined": TIMESTAMP,
|
||||
"projects": projects,
|
||||
}
|
||||
|
||||
|
||||
class TestId2Label(unittest.TestCase):
|
||||
def test_single_levels(self) -> None:
|
||||
cases = {
|
||||
0x0000: "Storage",
|
||||
0x0001: "Private",
|
||||
0x0002: "Public",
|
||||
0x0004: "PW Open",
|
||||
0x0100: "CBS Open",
|
||||
0x0104: "CBS or PW Open",
|
||||
}
|
||||
for id, label in cases.items():
|
||||
with self.subTest(id=hex(id)):
|
||||
self.assertEqual(FolderAccessLevel.id2label(id), label)
|
||||
|
||||
def test_restrict_slot_without_a_name_is_numbered(self) -> None:
|
||||
self.assertEqual(FolderAccessLevel.id2label(0x0200), "Restrict2 Open")
|
||||
self.assertEqual(FolderAccessLevel.id2label(0x8000), "Restrict8 Open")
|
||||
|
||||
def test_project_is_named_from_the_server(self) -> None:
|
||||
self.assertEqual(FolderAccessLevel.id2label(0x10000, BM2), "Brain/MINDS 2.0 Open")
|
||||
self.assertEqual(FolderAccessLevel.id2label(0x10004, BM2), "Brain/MINDS 2.0 or PW Open")
|
||||
|
||||
def test_project_not_named_gets_its_bit_position(self) -> None:
|
||||
self.assertEqual(FolderAccessLevel.id2label(0x10000), "Project0 Open")
|
||||
self.assertEqual(FolderAccessLevel.id2label(0x20000, BM2), "Project1 Open")
|
||||
|
||||
def test_parts_are_restrict_then_project_then_password(self) -> None:
|
||||
self.assertEqual(
|
||||
FolderAccessLevel.id2label(0x30304, BM2), "CBS or Restrict2 or Brain/MINDS 2.0 or Project1 or PW Open"
|
||||
)
|
||||
|
||||
def test_meaningless_levels_are_unlabelled(self) -> None:
|
||||
# Private or Public mixed with another bit, a reserved bit, a negative value.
|
||||
for id in (0x0003, 0x0006, 0x0102, 0x10001, 0x0008, 0x0080, -1):
|
||||
with self.subTest(id=hex(id)):
|
||||
self.assertIsNone(FolderAccessLevel.id2label(id))
|
||||
|
||||
|
||||
class TestKey2Id(unittest.TestCase):
|
||||
def test_keys(self) -> None:
|
||||
cases = {
|
||||
"storage": 0x0000,
|
||||
"private": 0x0001,
|
||||
"public": 0x0002,
|
||||
"pw_open": 0x0004,
|
||||
"cbs_open": 0x0100,
|
||||
"cbs_or_pw_open": 0x0104,
|
||||
"restrict1_open": 0x0100,
|
||||
"restrict2_open": 0x0200,
|
||||
"project0_open": 0x10000,
|
||||
"project1_or_pw_open": 0x20004,
|
||||
}
|
||||
for key, id in cases.items():
|
||||
with self.subTest(key=key):
|
||||
self.assertEqual(FolderAccessLevel.key2id(key), id)
|
||||
|
||||
def test_unknown_keys(self) -> None:
|
||||
for key in ("5kikan_open", "restrict9_open", "restrict0_open", "project01_open", "cbs", "_open", "pw_or_open"):
|
||||
with self.subTest(key=key):
|
||||
self.assertIsNone(FolderAccessLevel.key2id(key))
|
||||
|
||||
|
||||
class TestAccessLevelLabel(unittest.TestCase):
|
||||
def test_known_level(self) -> None:
|
||||
self.assertEqual(make_folder(0x0100).access_level_name, "CBS Open")
|
||||
|
||||
def test_project_names_are_used(self) -> None:
|
||||
self.assertEqual(make_folder(0x10000).access_level_label(BM2), "Brain/MINDS 2.0 Open")
|
||||
self.assertEqual(make_folder(0x10000).access_level_name, "Project0 Open")
|
||||
|
||||
def test_meaningless_level_shows_its_value(self) -> None:
|
||||
self.assertEqual(make_folder(0x0008).access_level_name, "Unknown (0x8)")
|
||||
|
||||
|
||||
class TestProjectNames(unittest.TestCase):
|
||||
def test_anonymous_user_asks_nothing(self) -> None:
|
||||
connection = MagicMock()
|
||||
connection.token = None
|
||||
with patch("mdrsclient.services.UsersApi") as users_api:
|
||||
self.assertEqual(MdrsService(connection).project_names(), {})
|
||||
users_api.assert_not_called()
|
||||
|
||||
def test_names_by_bit_position(self) -> None:
|
||||
connection = MagicMock()
|
||||
connection.get.return_value.json.return_value = current_user(
|
||||
[
|
||||
{"id": 1, "name": "Brain/MINDS 2.0", "bit_position": 0},
|
||||
{"id": 2, "name": "B", "bit_position": 3},
|
||||
{"id": 3, "name": "A", "bit_position": 3},
|
||||
{"id": 4, "name": None, "bit_position": 5},
|
||||
]
|
||||
)
|
||||
with patch.object(UsersApi, "_raise_response_error"):
|
||||
names = UsersApi(connection).project_names()
|
||||
self.assertEqual(names, {0: "Brain/MINDS 2.0", 3: "A, B"})
|
||||
|
||||
def test_server_without_projects(self) -> None:
|
||||
connection = MagicMock()
|
||||
user = current_user([])
|
||||
del user["projects"]
|
||||
connection.get.return_value.json.return_value = user
|
||||
with patch.object(UsersApi, "_raise_response_error"):
|
||||
self.assertEqual(UsersApi(connection).project_names(), {})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,104 @@
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from mdrsclient.api import FoldersApi
|
||||
from mdrsclient.exceptions import FolderLockedException
|
||||
from mdrsclient.models import Doi, Folder, Laboratories, Laboratory
|
||||
from mdrsclient.services import MdrsService
|
||||
|
||||
TIMESTAMP = "2026-01-01T00:00:00+09:00"
|
||||
DOI = "10.60178/cbs.20260429-001"
|
||||
LABORATORY = Laboratory(id=1, name="mylab", pi_name="PI", full_name="My Laboratory")
|
||||
|
||||
|
||||
def make_locked_folder() -> Folder:
|
||||
return Folder(
|
||||
id="f1",
|
||||
pid=None,
|
||||
name="root",
|
||||
access_level=4,
|
||||
lock=True,
|
||||
size=0,
|
||||
laboratory_id=1,
|
||||
description="",
|
||||
created_at=TIMESTAMP,
|
||||
updated_at=TIMESTAMP,
|
||||
restrict_opened_at=None,
|
||||
metadata=[],
|
||||
sub_folders=[],
|
||||
path="/root/",
|
||||
)
|
||||
|
||||
|
||||
# What the server returns for a locked folder not yet opened with its password.
|
||||
SHORT_RECORD = {
|
||||
"id": "f1",
|
||||
"pid": None,
|
||||
"name": "root",
|
||||
"access_level": 4,
|
||||
"lock": True,
|
||||
"size": 0,
|
||||
"laboratory_id": 1,
|
||||
"description": "",
|
||||
"created_at": TIMESTAMP,
|
||||
"updated_at": TIMESTAMP,
|
||||
"restrict_opened_at": None,
|
||||
"application": {"id": 0, "status": 0},
|
||||
}
|
||||
|
||||
|
||||
class TestRetrieveLockedFolder(unittest.TestCase):
|
||||
def retrieve(self, data: dict) -> Folder:
|
||||
connection = MagicMock()
|
||||
connection.get.return_value.json.return_value = data
|
||||
with patch.object(FoldersApi, "_raise_response_error"):
|
||||
return FoldersApi(connection).retrieve("f1")
|
||||
|
||||
def test_short_record_means_locked(self):
|
||||
with self.assertRaisesRegex(FolderLockedException, "Folder `root` is locked."):
|
||||
self.retrieve(SHORT_RECORD)
|
||||
|
||||
def test_full_record_of_a_locked_folder_is_read(self):
|
||||
full = dict(SHORT_RECORD, metadata=[], sub_folders=[], path="/root/")
|
||||
self.assertEqual(self.retrieve(full).path, "/root/")
|
||||
|
||||
|
||||
class TestFindFolderByDoi(unittest.TestCase):
|
||||
"""The DOI response carries the folder id as a field, not as a nested object."""
|
||||
|
||||
def find(self, retrieved: list, password: str | None) -> tuple[MagicMock, tuple]:
|
||||
laboratories = Laboratories()
|
||||
laboratories.append(LABORATORY)
|
||||
|
||||
with (
|
||||
patch("mdrsclient.services.DoiApi") as doi_api,
|
||||
patch("mdrsclient.services.FoldersApi") as folders_api,
|
||||
patch("mdrsclient.services.LaboratoriesApi") as laboratories_api,
|
||||
):
|
||||
doi_api.return_value.retrieve.return_value = Doi(id="20260429-001", doi=DOI, folder_id="f1")
|
||||
folders_api.return_value.retrieve.side_effect = retrieved
|
||||
laboratories_api.return_value.list.return_value = laboratories
|
||||
return folders_api.return_value, MdrsService(MagicMock()).find_folder_by_doi(DOI, password)
|
||||
|
||||
def test_a_locked_doi_folder_is_unlocked_with_its_folder_id(self):
|
||||
api, (folder, laboratory) = self.find([FolderLockedException("locked"), make_locked_folder()], "secret")
|
||||
api.auth.assert_called_once_with("f1", "secret")
|
||||
self.assertEqual(api.retrieve.call_count, 2)
|
||||
self.assertEqual(folder.id, "f1")
|
||||
self.assertEqual(laboratory.name, "mylab")
|
||||
|
||||
def test_a_locked_doi_folder_without_a_password(self):
|
||||
with self.assertRaisesRegex(FolderLockedException, f"Folder for DOI `{DOI}` is locked."):
|
||||
self.find([FolderLockedException("locked")], None)
|
||||
|
||||
def test_a_full_record_still_asks_for_the_password(self):
|
||||
# A locked folder whose record is readable anyway, e.g. its owner's.
|
||||
api, (folder, _) = self.find([make_locked_folder()], "secret")
|
||||
api.auth.assert_called_once_with("f1", "secret")
|
||||
self.assertEqual(api.retrieve.call_count, 1)
|
||||
with self.assertRaises(FolderLockedException):
|
||||
self.find([make_locked_folder()], None)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user