build: release 1.3.21

- Bump the version to 1.3.21 and close the Unreleased section of the
  changelog as 1.3.21.

CHANGELOG.md

@@ -2,6 +2,42 @@
 
 All notable changes to this project will be documented in this file.
 
+## [1.3.21] - 2026-11-13
+
+### Changed
+- Built access level labels from their bits instead of a fixed table of combinations, so a level the server adds later is labelled too. A project level (bit 16 and up) is named after the project as the server reports it to its members, e.g. `Brain/MINDS 2.0 Open`, and otherwise `Project<n> Open`. A restrict network other than CBS is `Restrict<n> Open`; `0x0200`, formerly `5Kikan Open`, is now `Restrict2 Open`.
+- `chacl` takes keys built the same way: `cbs_or_pw_open`, `project0_open` and so on. `5kikan_open` is no longer accepted. Its help lists them, an invalid level lists the projects the user belongs to with their keys, and `public` and `storage`, which the server refuses, are refused before any request.
+- `ls` asks the server for the user's projects once per run when logged in.
+- Required Python 3.11 or later, as `pydantic-settings` and `validators` do.
+- Raised the minimum version of `pydantic` to 2.14.3, and of the development tools `black`, `flake8`, `isort` and `pyright` to their current releases.
+- Removed `FolderAccessLevel.ACCESS_LEVELS` and `FolderAccessLevelItem`. `FolderSimple.access_level_label()` labels a level with the project names given to it.
+
+### Fixed
+- Reported an access level that has no meaning as `Unknown (0x…)` instead of `Storage`. Storage is the absence of every bit, so it matched any level. A level made of known bits is no longer labelled by the first table entry it contained either: `0x10104` was `CBS or PW Open`. This changes the `access_level` value in `ls` output, including JSON.
+- Reported a locked folder that was fetched without its password as `FolderLockedException`, a subclass of `UnauthorizedException`, instead of a pydantic `ValidationError`. The server answers with a short record that has no sub-folders or path. A password-protected DOI folder fails this way before its password is sent, so it can now be opened again.
+
+## [1.3.20] - 2026-11-04
+
+### Changed
+- Raised the minimum versions of `pydantic`, `pydantic-settings`, `python-dotenv`, `PyJWT` and `validators` to their current releases.
+
+### Security
+- Required `PyJWT` 2.15.1 or later; releases before 2.14.0 have a known vulnerability.
+
+## [1.3.19] - 2026-09-04
+
+### Fixed
+- Reported failed transfers instead of ending in success, so a run that lost files exits non-zero.
+- Left an existing local file intact when a download fails part way through.
+- Opened password-protected DOI folders, which previously failed outright.
+- Kept a session working across concurrent runs and long server queues.
+- Other minor fixes to `ls` output and remote URL handling.
+
+### Changed
+- Moved packaging to PEP 621 metadata with flit as the build backend.
+- Updated dependencies.
+- **Breaking for embedders:** `CacheInterface` now requires `lock()` and `reload()`.
+
 ## [1.3.18] - 2026-07-02
 
 ### Added
@@ -37,36 +73,41 @@ All notable changes to this project will be documented in this file.
 
 ## [1.3.15] - 2026-05-01
 
-### Changed
+### Fixed
-- Bumped package version to 1.3.15.
+- Apply NFC normalization to filenames and folder names sent to the server.
 
 ## [1.3.14] - 2026-04-17
 
 ### Changed
 - Simplified `config list` command (removed `-l`/`--long` option, always display URL).
+- Renamed `--quick` option to `--quiet` for `ls` subcommand.
 
 ### Added
 - Added subcommand aliases for config commands (e.g. `ls` alias for list, `rm` alias for delete).
+- Added `version` command.
 
 ## [1.3.13] - 2025-07-02
 
-### Changed
+### Fixed
-- Bumped package version to 1.3.13.
+- Fixed pagination logic for the `file.list` API.
 
 ## [1.3.12] - 2025-05-20
 
-### Changed
+### Fixed
-- Bumped package version to 1.3.12.
+- Fixed bug where file downloading was skipped incorrectly when `-s`/`--skip-if-file-exists` option was present.
 
 ## [1.3.11] - 2025-01-21
 
-### Changed
+### Fixed
-- Bumped package version to 1.3.11.
+- Follow-up fixes for User API specification changes.
 
 ## [1.3.10] - 2024-12-23
 
+### Added
+- Delete broken files and show a summary when a file download fails.
+
 ### Changed
-- Bumped package version to 1.3.10.
+- Updated dependency libraries.
 
 ## [1.3.9] - 2024-10-23
 
@@ -75,8 +116,11 @@ All notable changes to this project will be documented in this file.
 
 ## [1.3.8] - 2024-09-18
 
-### Changed
+### Added
-- Bumped package version to 1.3.8.
+- Implemented `-s`/`--skip-if-file-exists` option for `download` command.
+
+### Fixed
+- Added exception handling for unexpected responses from the server.
 
 ## [1.3.7] - 2024-07-22
 
@@ -85,23 +129,29 @@ All notable changes to this project will be documented in this file.
 
 ## [1.3.6] - 2024-07-08
 
-### Changed
+### Added
-- Bumped package version to 1.3.6.
+- Support cancelling recursive downloads if downloading some files fails.
 
 ## [1.3.5] - 2024-07-08
 
-### Changed
+### Added
-- Bumped package version to 1.3.5.
+- Added authorization token validation checks for file download operations.
+
+### Removed
+- Removed unnecessary debug code.
 
 ## [1.3.4] - 2024-07-04
 
 ### Added
 - Added some aliases for config sub command.
 
+### Fixed
+- Fixed bug when uploading large files.
+
 ## [1.3.3] - 2024-02-13
 
-### Changed
+### Added
-- Bumped package version to 1.3.3.
+- Implemented `-s`/`--skip-if-file-exists` option for `upload` command.
 
 ## [1.3.2] - 2024-02-09
 

README.md

@@ -2,12 +2,53 @@
 
 The mdrs-client-python is python library and a command-line client for up- and downloading files to and from MDRS based repository.
 
-## Installing
+## Installation (Production)
+
+It is recommended to use a virtual environment (`venv`) to keep your Python environment isolated.
+
+### Option 1: Using standard `venv` + `pip`
 
 ```shell
-poetry install
+python3 -m venv .venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+pip install .
 ```
 
+### Option 2: Using `uv`
+
+```shell
+uv venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+uv pip install .
+```
+
+## Development Setup
+
+To set up a local development environment with development tools (testing, formatting, linting):
+
+### Option 1: Using standard `venv` + `pip`
+
+```shell
+python3 -m venv .venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+pip install -e ".[dev]"
+```
+
+### Option 2: Using `uv` (Recommended for developers)
+
+```shell
+uv sync
+```
+
+## Configuration
+
+The following environment variables can be set, either in the shell or via a `.env` file placed in the working directory.
+
+| Variable | Default | Description |
+|---|---|---|
+| `MDRS_CLIENT_CONFIG_DIRNAME` | `~/.mdrs-client` | Directory where config and login cache files are stored |
+| `MDRS_CLIENT_CONCURRENT` | `10` | Number of concurrent file transfers for upload and download |
+
 ## CLI Usage
 
 ### config create
@@ -158,12 +199,14 @@ mdrs rm -r neurodata:/NIU/Repository/TEST/dataset
 
 ### chacl
 
-Change the folder access level
+Change the folder access level. The level is `private`, `pw_open`, `cbs_open`, `project<n>_open`, or several open levels joined by `_or_`. `<n>` is the project's bit position; an invalid level lists the projects you belong to with their keys.
 
 ```shell
 mdrs chacl private neurodata:/NIU/Repository/Private
 mdrs chacl cbs_open -r neurodata:/NIU/Repository/CBS_Open
 mdrs chacl pw_open -r -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open
+mdrs chacl cbs_or_pw_open -p SHARING_PASSWORD neurodata:/NIU/Repository/CBS_or_PW_Open
+mdrs chacl project0_open neurodata:/NIU/Repository/Project_Open
 ```
 
 ### metadata

mdrsclient/api/base.py

@@ -5,7 +5,13 @@ from pydantic import TypeAdapter
 from requests import Response
 
 from mdrsclient.connection import MDRSConnection
-from mdrsclient.exceptions import BadRequestException, ForbiddenException, UnauthorizedException, UnexpectedException
+from mdrsclient.exceptions import (
+    BadRequestException,
+    ForbiddenException,
+    TokenExpiredException,
+    UnauthorizedException,
+    UnexpectedException,
+)
 from mdrsclient.models.error import DRFStandardizedErrors
 
 
@@ -23,6 +29,11 @@ class BaseApi(ABC):
             if response.status_code == requests.codes.bad_request:
                 raise BadRequestException(errors.errors[0].detail)
             elif response.status_code == requests.codes.unauthorized:
+                # A request can wait in the server's queue for longer than the access
+                # token it was sent with lives, so an expired token here does not mean
+                # the session is over - it means this one request arrived too late.
+                if any(e.code == "token_not_valid" for e in errors.errors):
+                    raise TokenExpiredException("Access token expired before the request was served.")
                 raise UnauthorizedException("Login required.")
             elif response.status_code == requests.codes.forbidden:
                 raise ForbiddenException("You do not have enough permissions. Access is denied.")

mdrsclient/api/doi.py

@@ -4,7 +4,7 @@ from pydantic import TypeAdapter
 from pydantic.dataclasses import dataclass
 
 from mdrsclient.api.base import BaseApi
-from mdrsclient.api.utils import token_check
+from mdrsclient.api.utils import retry_on_expired_token, token_check
 from mdrsclient.models.doi import Doi
 
 
@@ -23,6 +23,7 @@ class DoiRetrieveResponse:
 class DoiApi(BaseApi):
     ENTRYPOINT: Final[str] = "v3/doi/"
 
+    @retry_on_expired_token
     def retrieve(self, doi_id: str) -> Doi:
         """Retrieve the folder associated with a DOI suffix ID (GET v3/doi/{id}/)."""
         url = self.ENTRYPOINT + doi_id + "/"

mdrsclient/api/files.py

@@ -1,5 +1,6 @@
 import mimetypes
 import os
+import threading
 from typing import Any, Final
 from unicodedata import normalize
 
@@ -8,8 +9,8 @@ from pydantic.dataclasses import dataclass
 from requests_toolbelt.multipart.encoder import MultipartEncoder
 
 from mdrsclient.api.base import BaseApi
-from mdrsclient.api.utils import token_check
+from mdrsclient.api.utils import retry_on_expired_token, token_check
-from mdrsclient.exceptions import UnexpectedException
+from mdrsclient.exceptions import MDRSException, UnexpectedException
 from mdrsclient.models import File
 
 
@@ -30,6 +31,7 @@ class FilesApi(BaseApi):
     ENTRYPOINT: Final[str] = "v3/files/"
     FALLBACK_MIMETYPE: Final[str] = "application/octet-stream"
 
+    @retry_on_expired_token
     def list(self, folder_id: str, page_num: int) -> FilesApiListResponse:
         url = self.ENTRYPOINT
         token_check(self.connection)
@@ -38,6 +40,7 @@ class FilesApi(BaseApi):
         self._raise_response_error(response)
         return TypeAdapter(FilesApiListResponse).validate_python(response.json())
 
+    @retry_on_expired_token
     def retrieve(self, id: str) -> File:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + id + "/"
@@ -46,6 +49,7 @@ class FilesApi(BaseApi):
         self._raise_response_error(response)
         return TypeAdapter(File).validate_python(response.json())
 
+    @retry_on_expired_token
     def create(self, folder_id: str, path: str) -> str:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT
@@ -66,10 +70,14 @@ class FilesApi(BaseApi):
             raise UnexpectedException(f"Could not open `{path}` file.")
         except MemoryError:
             raise UnexpectedException("Out of memory.")
+        except MDRSException:
+            # Already says what went wrong, and the caller may want to act on the kind.
+            raise
         except Exception as e:
             raise UnexpectedException("Unspecified error.") from e
         return ret.id
 
+    @retry_on_expired_token
     def update(self, file: File, path: str | None) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + file.id + "/"
@@ -96,6 +104,7 @@ class FilesApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def destroy(self, file: File) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + file.id + "/"
@@ -104,6 +113,7 @@ class FilesApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def move(self, file: File, folder_id: str, name: str) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + file.id + "/move/"
@@ -113,6 +123,7 @@ class FilesApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def copy(self, file: File, folder_id: str, name: str) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + file.id + "/copy/"
@@ -122,6 +133,7 @@ class FilesApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def metadata(self, file: File) -> dict[str, Any]:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + file.id + "/metadata/"
@@ -130,22 +142,58 @@ class FilesApi(BaseApi):
         self._raise_response_error(response)
         return response.json()
 
+    @retry_on_expired_token
     def download(self, file: File, path: str) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = file.download_url
         token_check(self.connection)
+        # Refused before anything is fetched. The finished file is moved into place, and a
+        # rename would replace a destination whose mode says it is protected.
+        if os.path.exists(path):
+            try:
+                with open(path, "r+b"):
+                    pass
+            except OSError as e:
+                raise UnexpectedException(f"Cannot write `{path}`: {e}")
         response = self.connection.get(url, stream=True)
         self._raise_response_error(response)
+        # Written beside the destination and moved in once the whole body has arrived, so
+        # a transfer that fails part way leaves whatever was already there untouched and
+        # never leaves a truncated file under the real name.
+        fd, tmp_path = self._open_partial(path)
         try:
-            with open(path, "wb") as f:
+            with os.fdopen(fd, "wb") as f:
                 for chunk in response.iter_content(chunk_size=4096):
                     if chunk:
                         f.write(chunk)
-                        f.flush()
+            os.replace(tmp_path, path)
-        except PermissionError:
+        except BaseException:
-            print(f"Cannot create file `{path}`: Permission denied.")
+            # Only the scratch file goes: anything at the destination was not written here.
+            if os.path.exists(tmp_path):
+                os.unlink(tmp_path)
+            raise
         return True
 
+    @staticmethod
+    def _open_partial(path: str) -> tuple[int, str]:
+        """
+        Create a scratch file beside `path` and return it open for writing.
+
+        Beside it, so moving the finished download into place is a rename within one
+        directory. `0o666` rather than a private mode because the umask is what decided
+        the permissions of a downloaded file before, and still should.
+        """
+        base = f"{path}.{os.getpid()}-{threading.get_ident()}"
+        for attempt in range(100):
+            tmp_path = f"{base}-{attempt}.mdrspart"
+            try:
+                return os.open(tmp_path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o666), tmp_path
+            except FileExistsError:
+                continue
+            except OSError as e:
+                raise UnexpectedException(f"Cannot write `{path}`: {e}")
+        raise UnexpectedException(f"Could not create a temporary file beside `{path}`.")
+
     def _get_mime_type(self, path: str) -> str:
         mt = mimetypes.guess_type(path)
         if mt:

mdrsclient/api/folders.py

@@ -5,8 +5,8 @@ from pydantic import TypeAdapter
 from pydantic.dataclasses import dataclass
 
 from mdrsclient.api.base import BaseApi
-from mdrsclient.api.utils import token_check
+from mdrsclient.api.utils import retry_on_expired_token, token_check
-from mdrsclient.exceptions import UnauthorizedException
+from mdrsclient.exceptions import FolderLockedException, UnauthorizedException
 from mdrsclient.models import Folder, FolderSimple
 
 
@@ -18,6 +18,7 @@ class FoldersApiCreateResponse:
 class FoldersApi(BaseApi):
     ENTRYPOINT: Final[str] = "v3/folders/"
 
+    @retry_on_expired_token
     def list(self, laboratory_id: int, path: str) -> list[FolderSimple]:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT
@@ -30,15 +31,22 @@ class FoldersApi(BaseApi):
             ret.append(TypeAdapter(FolderSimple).validate_python(data))
         return ret
 
+    @retry_on_expired_token
     def retrieve(self, id: str) -> Folder:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + id + "/"
         token_check(self.connection)
         response = self.connection.get(url)
         self._raise_response_error(response)
-        ret = TypeAdapter(Folder).validate_python(response.json())
+        data = response.json()
+        # A locked folder not yet opened with its password comes back with 200 but only
+        # its listing fields, without `metadata`, `sub_folders` and `path`.
+        if data.get("lock") and "sub_folders" not in data:
+            raise FolderLockedException(f"Folder `{data.get('name', id)}` is locked.")
+        ret = TypeAdapter(Folder).validate_python(data)
         return ret
 
+    @retry_on_expired_token
     def create(self, name: str, parent_id: str) -> str:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT
@@ -49,6 +57,7 @@ class FoldersApi(BaseApi):
         ret = TypeAdapter(FoldersApiCreateResponse).validate_python(response.json())
         return ret.id
 
+    @retry_on_expired_token
     def update(self, folder: FolderSimple) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + folder.id + "/"
@@ -61,6 +70,7 @@ class FoldersApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def destroy(self, id: str, recursive: bool) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + id + "/"
@@ -70,6 +80,7 @@ class FoldersApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def auth(self, id: str, password: str) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + id + "/auth/"
@@ -81,6 +92,7 @@ class FoldersApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def acl(self, id: str, access_level: int, recursive: bool, password: str | None) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + id + "/acl/"
@@ -94,6 +106,7 @@ class FoldersApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def move(self, folder: FolderSimple, folder_id: str, name: str) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + folder.id + "/move/"
@@ -103,6 +116,7 @@ class FoldersApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def copy(self, folder: FolderSimple, folder_id: str, name: str) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + folder.id + "/copy/"
@@ -112,6 +126,7 @@ class FoldersApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def metadata(self, id: str) -> dict[str, Any]:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + id + "/metadata/"

mdrsclient/api/laboratories.py

@@ -3,13 +3,14 @@ from typing import Final
 from pydantic import TypeAdapter
 
 from mdrsclient.api.base import BaseApi
-from mdrsclient.api.utils import token_check
+from mdrsclient.api.utils import retry_on_expired_token, token_check
 from mdrsclient.models import Laboratories, Laboratory
 
 
 class LaboratoriesApi(BaseApi):
     ENTRYPOINT: Final[str] = "v3/laboratories/"
 
+    @retry_on_expired_token
     def list(self) -> Laboratories:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT

mdrsclient/api/users.py

@@ -1,3 +1,4 @@
+from dataclasses import field
 from typing import Final
 
 import requests
@@ -8,6 +9,10 @@ from mdrsclient.api.base import BaseApi
 from mdrsclient.exceptions import UnauthorizedException
 from mdrsclient.models import Token, User
 
+# (connect, read) seconds for the token refresh. Uploads are served by a separate
+# instance, so this one is not queued behind them and has no reason to be slow.
+TOKEN_REFRESH_TIMEOUT: Final[tuple[float, float]] = (5.0, 30.0)
+
 
 @dataclass(frozen=True)
 class UsersCurrentResponseLaboratory:
@@ -16,6 +21,13 @@ class UsersCurrentResponseLaboratory:
     role: int
 
 
+@dataclass(frozen=True)
+class UsersCurrentResponseProject:
+    id: int
+    name: str | None
+    bit_position: int
+
+
 @dataclass(frozen=True)
 class UsersApiCurrentResponse:
     id: int
@@ -31,6 +43,7 @@ class UsersApiCurrentResponse:
     is_reviewer: bool
     last_login: str  # ISO8601
     date_joined: str  # ISO8601
+    projects: list[UsersCurrentResponseProject] = field(default_factory=list)
 
 
 class UsersApi(BaseApi):
@@ -46,6 +59,20 @@ class UsersApi(BaseApi):
         user = User(id=obj.id, username=obj.username, laboratory_ids=laboratory_ids, is_reviewer=obj.is_reviewer)
         return user
 
+    def project_names(self) -> dict[int, str]:
+        """Names of the user's projects by bit_position, the bit their access level sets."""
+        url = self.ENTRYPOINT + "current/"
+        response = self.connection.get(url)
+        self._raise_response_error(response)
+        obj = TypeAdapter(UsersApiCurrentResponse).validate_python(response.json())
+        names: dict[int, list[str]] = {}
+        for project in obj.projects:
+            if project.name:
+                # bit_position is not unique on the server, and a folder at a shared bit
+                # belongs to every project holding it, so all of them are named.
+                names.setdefault(project.bit_position, []).append(project.name)
+        return {bit: ", ".join(sorted(x)) for bit, x in names.items()}
+
     def token(self, username: str, password: str) -> Token:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + "token/"
@@ -61,7 +88,10 @@ class UsersApi(BaseApi):
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + "token/refresh/"
         data: dict[str, str | int] = {"refresh": token.refresh}
-        response = self.connection.post(url, data=data)
+        # Bounded on its own: the caller holds a lock that spans processes while this runs,
+        # so a provider that accepts the connection and then goes quiet would otherwise
+        # stall every other request on this machine rather than just this one.
+        response = self.connection.post(url, data=data, timeout=TOKEN_REFRESH_TIMEOUT)
         if response.status_code == requests.codes.unauthorized:
             raise UnauthorizedException("Token is invalid or expired.")
         self._raise_response_error(response)

mdrsclient/api/utils.py

@@ -1,19 +1,124 @@
+import functools
+from typing import Any, Callable, TypeVar, cast
+
 from mdrsclient.api.users import UsersApi
 from mdrsclient.connection import MDRSConnection
-from mdrsclient.exceptions import UnauthorizedException
+from mdrsclient.exceptions import (
+    MDRSException,
+    ServerBusyException,
+    TokenExpiredException,
+    UnauthorizedException,
+)
+from mdrsclient.models import Token
+
+F = TypeVar("F", bound=Callable[..., Any])
+
+SERVER_BUSY_MESSAGE = (
+    "The server took too long to start handling the request and may be overloaded. "
+    "Try again, or reduce the number of parallel transfers."
+)
 
 
 def token_check(connection: MDRSConnection) -> None:
-    try:
+    """
-        connection.lock.acquire()
+    Bring the access token up to date before a request goes out.
-        if connection.token is not None:
+
-            if connection.token.is_refresh_required:
+    Refreshing is a read-modify-write over a cache shared with every other client
-                user_api = UsersApi(connection)
+    process on this machine, and a rotating provider stops honouring the refresh token
-                try:
+    it replaces. Two processes reaching this at once would otherwise both send the same
-                    connection.token = user_api.tokenRefresh(connection.token)
+    token, and the loser would be left holding one the server no longer accepts, so the
-                except UnauthorizedException:
+    whole sequence runs under a lock that spans processes and the cache is re-read
-                    connection.logout()
+    inside it.
-            elif connection.token.is_expired:
+    """
+    with connection.lock:
+        token = connection.token
+        if token is None or not (token.is_refresh_required or token.is_expired):
+            # Nothing to do, which is the answer for almost every request. The lock below
+            # reaches across processes and is held for a round trip, so it is worth
+            # knowing that before taking it.
+            return
+        with connection.cache_lock():
+            connection.reload_cache()
+            token = connection.token
+            if token is None:
+                return
+            if token.is_expired:
                 connection.logout()
-    finally:
+                return
-        connection.lock.release()
+            if not token.is_refresh_required:
+                return
+            user_api = UsersApi(connection)
+            try:
+                connection.token = user_api.tokenRefresh(token)
+            except UnauthorizedException:
+                connection.logout()
+
+
+def token_recover(connection: MDRSConnection, used: Token | None) -> bool:
+    """
+    Get a usable access token after one was refused for having expired.
+
+    A request can sit in the server's queue for longer than its access token lives, and
+    while it waited another thread or process may already have refreshed. Prefer what
+    they left behind: refreshing again would spend a round trip, and with a provider
+    that rotates refresh tokens it would retire one that is still in use.
+
+    Returns whether the caller now holds a token worth retrying with.
+    """
+    with connection.lock, connection.cache_lock():
+        connection.reload_cache()
+        token = connection.token
+        if token is None:
+            return False
+        if used is None or token != used:
+            # Somebody else has been here since the request went out. Assign rather than
+            # just read: the setter is what rewrites the session's Authorization header,
+            # so a plain reload would leave the retry carrying the refused token.
+            connection.token = token
+            return True
+        user_api = UsersApi(connection)
+        try:
+            connection.token = user_api.tokenRefresh(token)
+        except UnauthorizedException:
+            connection.logout()
+            return False
+        return True
+
+
+def retry_on_expired_token(func: F) -> F:
+    """
+    Send a request again once when the server found its access token expired.
+
+    The token is checked before every request, so this only happens when the request was
+    held long enough on the way in for a valid token to lapse - a queue behind uploads
+    that take minutes, most often.
+    """
+
+    @functools.wraps(func)
+    def wrapper(self: Any, *args: Any, **kwargs: Any) -> Any:
+        connection: MDRSConnection = self.connection
+        # Settle the token first so `used` is what the request actually carries. The
+        # wrapped method checks it again, which costs nothing once it is already current.
+        token_check(connection)
+        used = connection.token
+        try:
+            return func(self, *args, **kwargs)
+        except TokenExpiredException:
+            try:
+                recovered = token_recover(connection, used)
+            except MDRSException:
+                raise
+            except Exception as e:
+                # A busy server queues the refresh too, and failing here would report an
+                # endpoint the user never asked for rather than the reason.
+                raise ServerBusyException(SERVER_BUSY_MESSAGE) from e
+            if not recovered:
+                raise
+        try:
+            return func(self, *args, **kwargs)
+        except TokenExpiredException as e:
+            # A second refusal, with a token that was current when it left. The session is
+            # fine; the server is not starting requests before their credentials lapse.
+            raise ServerBusyException(SERVER_BUSY_MESSAGE) from e
+
+    return cast(F, wrapper)

mdrsclient/cache.py

@@ -1,8 +1,12 @@
+import contextlib
 import dataclasses
 import hashlib
 import json
 import os
-from typing import Protocol, runtime_checkable
+import tempfile
+import time
+from contextlib import AbstractContextManager
+from typing import Iterator, Protocol, runtime_checkable
 
 from pydantic import TypeAdapter, ValidationError
 from pydantic.dataclasses import dataclass
@@ -46,6 +50,19 @@ class CacheData:
 
 @runtime_checkable
 class CacheInterface(Protocol):
+    def lock(self) -> AbstractContextManager[None]:
+        """
+        Hold exclusive access to the cache for the duration of the block.
+
+        Refreshing a token is a read-modify-write, and the cache is shared with every
+        other client process using the same remote.
+        """
+        ...
+
+    def reload(self) -> None:
+        """Re-read the cache, discarding anything held from an earlier read."""
+        ...
+
     @property
     def token(self) -> Token | None: ...
     @token.setter
@@ -70,6 +87,14 @@ class InMemoryCache(CacheInterface):
     def __init__(self) -> None:
         self.__data = CacheData()
 
+    @contextlib.contextmanager
+    def lock(self) -> Iterator[None]:
+        # Nothing else can reach this cache, so there is nothing to exclude.
+        yield
+
+    def reload(self) -> None:
+        pass
+
     @property
     def token(self) -> Token | None:
         return self.__data.token
@@ -109,14 +134,46 @@ class CacheFile(CacheInterface):
     __serial: int
     __cache_dir: str
     __cache_file: str
+    __lock_file: str
+    __lock_depth: int
     __data: CacheData
 
     def __init__(self, remote: str) -> None:
         self.__serial = -1
         self.__cache_dir = os.path.join(CONFIG_DIRNAME, "cache")
         self.__cache_file = os.path.join(self.__cache_dir, remote + ".json")
+        self.__lock_file = os.path.join(self.__cache_dir, remote + ".lock")
+        self.__lock_depth = 0
         self.__data = CacheData()
 
+    @contextlib.contextmanager
+    def lock(self) -> Iterator[None]:
+        # Re-entrant, because every write takes it and a refresh is a write made while
+        # already holding it. A second flock on the same file from the same process would
+        # wait for a release that cannot come.
+        if self.__lock_depth > 0:
+            self.__lock_depth += 1
+            try:
+                yield
+            finally:
+                self.__lock_depth -= 1
+            return
+        # A separate file, so that replacing the cache cannot disturb the lock holders.
+        self.__ensure_cache_dir()
+        with open(self.__lock_file, "a") as f:
+            FileLock.lock(f)
+            self.__lock_depth = 1
+            try:
+                yield
+            finally:
+                self.__lock_depth = 0
+                FileLock.unlock(f)
+        os.chmod(self.__lock_file, 0o600)
+
+    def reload(self) -> None:
+        self.__serial = -1
+        self.__load()
+
     @property
     def token(self) -> Token | None:
         self.__load()
@@ -124,9 +181,10 @@ class CacheFile(CacheInterface):
 
     @token.setter
     def token(self, token: Token) -> None:
-        self.__load()
+        with self.lock():
-        self.__data.token = token
+            self.reload()
-        self.__save()
+            self.__data.token = token
+            self.__save()
 
     @token.deleter
     def token(self) -> None:
@@ -139,9 +197,10 @@ class CacheFile(CacheInterface):
 
     @user.setter
     def user(self, user: User) -> None:
-        self.__load()
+        with self.lock():
-        self.__data.user = user
+            self.reload()
-        self.__save()
+            self.__data.user = user
+            self.__save()
 
     @user.deleter
     def user(self) -> None:
@@ -154,9 +213,10 @@ class CacheFile(CacheInterface):
 
     @laboratories.setter
     def laboratories(self, laboratories: Laboratories) -> None:
-        self.__load()
+        with self.lock():
-        self.__data.laboratories = laboratories
+            self.reload()
-        self.__save()
+            self.__data.laboratories = laboratories
+            self.__save()
 
     def __clear(self) -> None:
         self.__data.clear()
@@ -164,8 +224,7 @@ class CacheFile(CacheInterface):
 
     def __load(self) -> None:
         if os.path.isfile(self.__cache_file):
-            stat = os.stat(self.__cache_file)
+            serial = self.__stat_serial()
-            serial = hash((stat.st_uid, stat.st_gid, stat.st_mode, stat.st_size, stat.st_mtime))
             if self.__serial != serial:
                 try:
                     with open(self.__cache_file) as f:
@@ -185,15 +244,44 @@ class CacheFile(CacheInterface):
 
     def __save(self) -> None:
         self.__ensure_cache_dir()
-        with open(self.__cache_file, "w") as f:
+        self.__data.update_digest()
-            FileLock.lock(f)
+        payload = json.dumps(dataclasses.asdict(self.__data))
-            self.__data.update_digest()
+        # Written aside and moved into place: opening the cache for writing truncates it
-            f.write(json.dumps(dataclasses.asdict(self.__data)))
+        # first, and a reader landing in that window would find the file empty and treat
-            FileLock.unlock(f)
+        # the session as broken.
+        fd, tmp_file = tempfile.mkstemp(dir=self.__cache_dir, prefix=".tmp-")
+        try:
+            with os.fdopen(fd, "w") as f:
+                f.write(payload)
+            # ensure file is secure.
+            os.chmod(tmp_file, 0o600)
+            self.__replace(tmp_file, self.__cache_file)
+        except BaseException:
+            if os.path.exists(tmp_file):
+                os.unlink(tmp_file)
+            raise
+        self.__serial = self.__stat_serial()
+
+    @staticmethod
+    def __replace(source: str, destination: str) -> None:
+        # Windows refuses the rename while another process still has the destination
+        # open, which a reader briefly does, so give it a moment rather than failing the
+        # save outright. On POSIX the rename always succeeds and the loop ends at once.
+        deadline = time.monotonic() + 5.0
+        while True:
+            try:
+                os.replace(source, destination)
+                return
+            except PermissionError:
+                if time.monotonic() >= deadline:
+                    raise
+                time.sleep(0.05)
+
+    def __stat_serial(self) -> int:
         stat = os.stat(self.__cache_file)
-        self.__serial = hash((stat.st_uid, stat.st_gid, stat.st_mode, stat.st_size, stat.st_mtime))
+        # st_ino and st_mtime_ns both move when the file is replaced, which a refresh
-        # ensure file is secure.
+        # that happens to produce the same number of bytes otherwise would not show.
-        os.chmod(self.__cache_file, 0o600)
+        return hash((stat.st_ino, stat.st_uid, stat.st_gid, stat.st_mode, stat.st_size, stat.st_mtime_ns))
 
     def __ensure_cache_dir(self) -> None:
         if not os.path.exists(self.__cache_dir):

mdrsclient/client.py

@@ -1,21 +1,40 @@
 import os
-from typing import Any
+from dataclasses import dataclass
 from unicodedata import normalize
 
-from mdrsclient.api import DoiApi, FilesApi, FoldersApi, LaboratoriesApi, UsersApi
+from mdrsclient.api import FilesApi, FoldersApi
 from mdrsclient.cache import CacheInterface
 from mdrsclient.config import ConfigInterface
 from mdrsclient.connection import MDRSConnection
-from mdrsclient.exceptions import IllegalArgumentException, MDRSException, UnauthorizedException, UnexpectedException
+from mdrsclient.exceptions import IllegalArgumentException
-from mdrsclient.models import File, Folder, Laboratory, Token, User
+from mdrsclient.models import File, Folder, FolderSimple, Laboratory
 from mdrsclient.models.file import find_file
 from mdrsclient.services import MdrsService
 
 
+@dataclass(frozen=True)
+class TransferEndpoints:
+    """Both ends of a `cp` or `mv`, resolved: the parents, their contents, and the names."""
+
+    laboratory: Laboratory
+    s_path: str
+    s_basename: str
+    s_parent_folder: Folder
+    s_parent_files: list[File]
+    d_path: str
+    d_basename: str
+    d_parent_folder: Folder
+    d_parent_files: list[File]
+
+    def is_same_place(self) -> bool:
+        """Whether the destination names what the source already is, leaving nothing to do."""
+        return self.s_parent_folder.id == self.d_parent_folder.id and self.s_basename == self.d_basename
+
+
 class MdrsClient(MdrsService):
     """Service layer client for MDRS."""
 
-    def __init__(self, connection: MDRSConnection, config_class: type[ConfigInterface] | None = None):
+    def __init__(self, connection: MDRSConnection | None, config_class: type[ConfigInterface] | None = None):
         super().__init__(connection, config_class)
 
     @classmethod
@@ -63,7 +82,8 @@ class MdrsClient(MdrsService):
         files = self.find_files(folder.id)
         return folder, files
 
-    def cp(self, src_path: str, dest_path: str, is_recursive: bool = False) -> None:
+    def __resolve_transfer(self, src_path: str, dest_path: str) -> TransferEndpoints:
+        """Resolve both ends of a transfer, refusing one that crosses a remote or a laboratory."""
         s_remote, s_laboratory_name, s_path = self.parse_remote_host_with_path(src_path)
         d_remote, d_laboratory_name, d_path = self.parse_remote_host_with_path(dest_path)
         if s_remote != d_remote:
@@ -71,8 +91,9 @@ class MdrsClient(MdrsService):
         if s_laboratory_name != d_laboratory_name:
             raise IllegalArgumentException("Laboratory mismatched.")
         s_path = s_path.rstrip("/")
-        s_dirname = os.path.dirname(s_path)
         s_basename = os.path.basename(s_path)
+        # A destination ending in a separator names a folder to put the source into, and
+        # keeps the source's own name.
         if d_path.endswith("/"):
             d_dirname = d_path
             d_basename = s_basename
@@ -80,84 +101,77 @@ class MdrsClient(MdrsService):
             d_dirname = os.path.dirname(d_path)
             d_basename = os.path.basename(d_path)
         laboratory = self.find_laboratory(s_laboratory_name)
-        s_parent_folder = self.find_folder(laboratory, s_dirname)
+        s_parent_folder = self.find_folder(laboratory, os.path.dirname(s_path))
-        s_parent_files = self.find_files(s_parent_folder.id)
         d_parent_folder = self.find_folder(laboratory, d_dirname)
-        d_parent_files = self.find_files(d_parent_folder.id)
+        return TransferEndpoints(
-        s_file = find_file(s_parent_files, s_basename)
+            laboratory=laboratory,
+            s_path=s_path,
+            s_basename=s_basename,
+            s_parent_folder=s_parent_folder,
+            s_parent_files=self.find_files(s_parent_folder.id),
+            d_path=d_path,
+            d_basename=d_basename,
+            d_parent_folder=d_parent_folder,
+            d_parent_files=self.find_files(d_parent_folder.id),
+        )
+
+    @staticmethod
+    def __check_file_destination(ends: TransferEndpoints) -> None:
+        """Refuse a destination that a file cannot take the place of."""
+        if find_file(ends.d_parent_files, ends.d_basename) is not None:
+            raise IllegalArgumentException(f"File `{ends.d_basename}` already exists.")
+        if ends.d_parent_folder.find_sub_folder(ends.d_basename) is not None:
+            raise IllegalArgumentException(
+                f"Cannot overwrite non-folder `{ends.d_basename}` with folder `{ends.d_path}`."
+            )
+
+    @staticmethod
+    def __check_folder_destination(ends: TransferEndpoints, s_folder: FolderSimple) -> None:
+        """Refuse a destination that a folder cannot take the place of."""
+        if find_file(ends.d_parent_files, ends.d_basename) is not None:
+            raise IllegalArgumentException(
+                f"Cannot overwrite non-folder `{ends.d_basename}` with folder `{ends.s_path}`."
+            )
+        d_folder = ends.d_parent_folder.find_sub_folder(ends.d_basename)
+        if d_folder is not None:
+            if d_folder.id == s_folder.id:
+                raise IllegalArgumentException(f"`{ends.s_path}` and `{ends.s_path}` are the same folder.")
+            raise IllegalArgumentException(f"Cannot move `{ends.s_path}` to `{ends.d_path}`: Folder not empty.")
+
+    @staticmethod
+    def __find_source_folder(ends: TransferEndpoints) -> FolderSimple:
+        s_folder = ends.s_parent_folder.find_sub_folder(ends.s_basename)
+        if s_folder is None:
+            raise IllegalArgumentException(f"File or folder `{ends.s_basename}` not found.")
+        return s_folder
+
+    def cp(self, src_path: str, dest_path: str, is_recursive: bool = False) -> None:
+        ends = self.__resolve_transfer(src_path, dest_path)
+        s_file = find_file(ends.s_parent_files, ends.s_basename)
         if s_file is not None:
-            d_file = find_file(d_parent_files, d_basename)
+            self.__check_file_destination(ends)
-            if d_file is not None:
+            if not ends.is_same_place():
-                raise IllegalArgumentException(f"File `{d_basename}` already exists.")
+                FilesApi(self.connection).copy(s_file, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
-            d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
+            return
-            if d_sub_folder is not None:
+        s_folder = self.__find_source_folder(ends)
-                raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
+        if not is_recursive:
-            file_api = FilesApi(self.connection)
+            raise IllegalArgumentException(f"Cannot copy `{ends.s_path}`: Is a folder.")
-            if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
+        self.__check_folder_destination(ends, s_folder)
-                file_api.copy(s_file, d_parent_folder.id, normalize("NFC", d_basename))
+        if not ends.is_same_place():
-        else:
+            FoldersApi(self.connection).copy(s_folder, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
-            s_folder = s_parent_folder.find_sub_folder(s_basename)
-            if s_folder is None:
-                raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
-            if not is_recursive:
-                raise IllegalArgumentException(f"Cannot copy `{s_path}`: Is a folder.")
-            if find_file(d_parent_files, d_basename) is not None:
-                raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
-            d_folder = d_parent_folder.find_sub_folder(d_basename)
-            if d_folder is not None:
-                if d_folder.id == s_folder.id:
-                    raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
-                raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
-            folder_api = FoldersApi(self.connection)
-            if s_parent_folder.id != d_parent_folder.id or s_basename != d_basename:
-                folder_api.copy(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
 
     def mv(self, src_path: str, dest_path: str) -> None:
-        s_remote, s_laboratory_name, s_path = self.parse_remote_host_with_path(src_path)
+        ends = self.__resolve_transfer(src_path, dest_path)
-        d_remote, d_laboratory_name, d_path = self.parse_remote_host_with_path(dest_path)
+        s_file = find_file(ends.s_parent_files, ends.s_basename)
-        if s_remote != d_remote:
-            raise IllegalArgumentException("Remote host mismatched.")
-        if s_laboratory_name != d_laboratory_name:
-            raise IllegalArgumentException("Laboratory mismatched.")
-        s_path = s_path.rstrip("/")
-        s_dirname = os.path.dirname(s_path)
-        s_basename = os.path.basename(s_path)
-        if d_path.endswith("/"):
-            d_dirname = d_path
-            d_basename = s_basename
-        else:
-            d_dirname = os.path.dirname(d_path)
-            d_basename = os.path.basename(d_path)
-        laboratory = self.find_laboratory(s_laboratory_name)
-        s_parent_folder = self.find_folder(laboratory, s_dirname)
-        s_parent_files = self.find_files(s_parent_folder.id)
-        d_parent_folder = self.find_folder(laboratory, d_dirname)
-        d_parent_files = self.find_files(d_parent_folder.id)
-        s_file = find_file(s_parent_files, s_basename)
         if s_file is not None:
-            d_file = find_file(d_parent_files, d_basename)
+            self.__check_file_destination(ends)
-            if d_file is not None:
+            if not ends.is_same_place():
-                raise IllegalArgumentException(f"File `{d_basename}` already exists.")
+                FilesApi(self.connection).move(s_file, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
-            d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
+            return
-            if d_sub_folder is not None:
+        s_folder = self.__find_source_folder(ends)
-                raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
+        self.__check_folder_destination(ends, s_folder)
-            file_api = FilesApi(self.connection)
+        if not ends.is_same_place():
-            if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
+            FoldersApi(self.connection).move(s_folder, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
-                file_api.move(s_file, d_parent_folder.id, normalize("NFC", d_basename))
-        else:
-            s_folder = s_parent_folder.find_sub_folder(s_basename)
-            if s_folder is None:
-                raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
-            if find_file(d_parent_files, d_basename) is not None:
-                raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
-            d_folder = d_parent_folder.find_sub_folder(d_basename)
-            if d_folder is not None:
-                if d_folder.id == s_folder.id:
-                    raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
-                raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
-            folder_api = FoldersApi(self.connection)
-            if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
-                folder_api.move(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
 
     def chacl(
         self, remote_path: str, access_level: int, is_recursive: bool = False, password: str | None = None

mdrsclient/commands/chacl.py

@@ -1,19 +1,28 @@
 from argparse import Namespace
-from typing import Any
+from typing import Any, Final
+
+import requests
 
-from mdrsclient.api import FoldersApi
 from mdrsclient.commands.base import BaseCommand
-from mdrsclient.exceptions import IllegalArgumentException
+from mdrsclient.exceptions import IllegalArgumentException, MDRSException
 from mdrsclient.models import FolderAccessLevel
 
 
 class ChaclCommand(BaseCommand):
+    ACCESS_LEVEL_HELP: Final[str] = (
+        "access level: private, pw_open, cbs_open, project<n>_open, "
+        "or several open levels joined by `_or_` (e.g. cbs_or_pw_open). "
+        "<n> is the project's bit position; an invalid level lists the projects you belong to"
+    )
+
     @classmethod
     def register(cls, parsers: Any) -> None:
         chacl_parser = parsers.add_parser("chacl", help="change the folder access level")
-        chacl_parser.add_argument("access_level", help="access level (private, cbs_open, pw_open)")
+        chacl_parser.add_argument("access_level", help=cls.ACCESS_LEVEL_HELP)
         chacl_parser.add_argument("-r", "--recursive", help="change access levels recursively", action="store_true")
-        chacl_parser.add_argument("-p", "--password", help="password to set when access level is `pw_open`")
+        chacl_parser.add_argument(
+            "-p", "--password", help="password to set when the access level includes `pw` (e.g. pw_open)"
+        )
         chacl_parser.add_argument("remote_path", help="remote folder path (remote:/lab/path/)")
         chacl_parser.set_defaults(func=cls.func)
 
@@ -22,9 +31,10 @@ class ChaclCommand(BaseCommand):
         remote_path = str(args.remote_path)
         access_level = FolderAccessLevel.key2id(str(args.access_level))
         if access_level is None:
-            raise IllegalArgumentException(
+            raise IllegalArgumentException(cls._invalid_level_message(remote_path))
-                "Invalid `access_level` parameter. must be `private`, `cbs_open` or `pw_open`."
+        if access_level in (FolderAccessLevel.STORAGE, FolderAccessLevel.PUBLIC):
-            )
+            # The server refuses both; saying so here spares a round trip and a bare 400.
+            raise IllegalArgumentException(f"`{args.access_level}` cannot be set as an access level.")
         password = str(args.password) if args.password else None
         is_recursive = bool(args.recursive)
         cls.chacl(remote_path, access_level, is_recursive, password)
@@ -36,3 +46,24 @@ class ChaclCommand(BaseCommand):
 
         client = MdrsClient.from_remote(remote)
         client.chacl(remote_path, access_level, is_recursive, password)
+
+    @classmethod
+    def _invalid_level_message(cls, remote_path: str) -> str:
+        message = (
+            "Invalid `access_level` parameter. must be `private`, `pw_open`, `cbs_open`, `project<n>_open` "
+            "or open levels joined by `_or_` (e.g. `cbs_or_pw_open`)."
+        )
+        remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
+        from mdrsclient.client import MdrsClient
+
+        # The project numbers are known only to the server, and only for the user's own
+        # projects. Not reaching it leaves the message as it is: the error being reported
+        # is the level, not the connection.
+        try:
+            projects = MdrsClient.from_remote(remote).project_names()
+        except (MDRSException, requests.RequestException):
+            return message
+        if not projects:
+            return message
+        lines = [f"  project{bit}_open: {name}" for bit, name in sorted(projects.items())]
+        return "\n".join([message, "Projects you belong to:", *lines])

mdrsclient/commands/cp.py

@@ -1,12 +1,7 @@
-import os
 from argparse import Namespace
 from typing import Any
-from unicodedata import normalize
 
-from mdrsclient.api import FilesApi, FoldersApi
 from mdrsclient.commands.base import BaseCommand
-from mdrsclient.exceptions import IllegalArgumentException
-from mdrsclient.models.file import find_file
 
 
 class CpCommand(BaseCommand):

mdrsclient/commands/file_metadata.py

@@ -1,12 +1,8 @@
 import json
-import os
 from argparse import Namespace
 from typing import Any
 
-from mdrsclient.api import FilesApi
 from mdrsclient.commands.base import BaseCommand
-from mdrsclient.exceptions import IllegalArgumentException
-from mdrsclient.models.file import find_file
 
 
 class FileMetadataCommand(BaseCommand):

mdrsclient/commands/labs.py

@@ -1,7 +1,6 @@
 from argparse import Namespace
 from typing import Any
 
-from mdrsclient.api import LaboratoriesApi
 from mdrsclient.commands.base import BaseCommand
 
 

mdrsclient/commands/ls.py

@@ -1,12 +1,14 @@
 import json
 from argparse import Namespace
-from typing import Any
+from dataclasses import field
+from typing import Any, Final
 
 from pydantic.dataclasses import dataclass
 
-from mdrsclient.api import FilesApi, FoldersApi
+from mdrsclient.api import FoldersApi
 from mdrsclient.client import MdrsClient
 from mdrsclient.commands.base import BaseCommand
+from mdrsclient.config import build_download_url
 from mdrsclient.exceptions import UnauthorizedException
 from mdrsclient.models import File, Folder, FolderSimple, Laboratory
 
@@ -25,6 +27,7 @@ class LsCommandContext:
     is_json: bool
     is_quiet: bool
     is_recursive: bool
+    projects: dict[int, str] = field(default_factory=dict)
 
 
 class LsCommand(BaseCommand):
@@ -84,6 +87,7 @@ class LsCommand(BaseCommand):
             is_json,
             is_quiet,
             is_recursive,
+            client.project_names(),
         )
         if context.is_json:
             cls._ls_json(context, folder, files)
@@ -94,33 +98,53 @@ class LsCommand(BaseCommand):
     def _ls_json(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
         print(json.dumps(cls._folder2dict(context, folder, files), ensure_ascii=False))
 
+    LABELS: Final[dict[str, str]] = {
+        "type": "Type",
+        "acl": "Access",
+        "laboratory": "Laboratory",
+        "size": "Size",
+        "date": "Date",
+        "name": "Name",
+    }
+
     @classmethod
-    def _ls_plain(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
+    def _column_widths(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> dict[str, int]:
-        label = {
+        """Width of each column: the widest of its heading and everything printed under it."""
-            "type": "Type",
+        length = {key: len(label) if not context.is_quiet else 0 for key, label in cls.LABELS.items()}
-            "acl": "Access",
-            "laboratory": "Laboratory",
-            "size": "Size",
-            "date": "Date",
-            "name": "Name",
-        }
-        length: dict[str, int] = {}
-        for key in label.keys():
-            length[key] = len(label[key]) if not context.is_quiet else 0
         for sub_folder in folder.sub_folders:
-            sub_laboratory = context.client.connection.laboratories.find_by_id(sub_folder.laboratory_id)
+            sub_laboratory_name = cls._laboratory_name(context, sub_folder.laboratory_id)
-            sub_laboratory_name = sub_laboratory.name if sub_laboratory is not None else "(invalid)"
+            length["acl"] = max(length["acl"], len(sub_folder.access_level_label(context.projects)))
-            length["acl"] = max(length["acl"], len(sub_folder.access_level_name))
             length["laboratory"] = max(length["laboratory"], len(sub_laboratory_name))
-            length["size"] = max(length["size"], len(str(folder.size)))
+            length["size"] = max(length["size"], len(str(sub_folder.size)))
             length["date"] = max(length["date"], len(sub_folder.updated_at_name))
             length["name"] = max(length["name"], len(sub_folder.name))
         for file in files:
             length["size"] = max(length["size"], len(str(file.size)))
             length["date"] = max(length["date"], len(file.updated_at_name))
             length["name"] = max(length["name"], len(file.name))
-        length["acl"] = max(length["acl"], len(folder.access_level_name))
+        length["acl"] = max(length["acl"], len(folder.access_level_label(context.projects)))
         length["laboratory"] = max(length["laboratory"], len(context.laboratory.name))
+        return length
+
+    @classmethod
+    def _ls_plain_children(cls, context: LsCommandContext, folder: Folder) -> None:
+        """List each sub-folder in turn, passing over the ones the caller cannot open."""
+        print("")
+        folder_api = FoldersApi(context.client.connection)
+        for sub_folder in sorted(folder.sub_folders, key=lambda x: x.name):
+            try:
+                if sub_folder.lock:
+                    folder_api.auth(sub_folder.id, context.password)
+                sub_detail = folder_api.retrieve(sub_folder.id)
+                sub_files = context.client.find_files(sub_folder.id)
+            except UnauthorizedException:
+                continue
+            cls._ls_plain(context, sub_detail, sub_files)
+
+    @classmethod
+    def _ls_plain(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
+        label = cls.LABELS
+        length = cls._column_widths(context, folder, files)
         header = (
             f"{label['type']:{length['type']}}\t{label['acl']:{length['acl']}}\t"
             f"{label['laboratory']:{length['laboratory']}}\t{label['size']:{length['size']}}\t"
@@ -138,30 +162,21 @@ class LsCommand(BaseCommand):
         for sub_folder in sorted(folder.sub_folders, key=lambda x: x.name):
             sub_laboratory_name = cls._laboratory_name(context, sub_folder.laboratory_id)
             sub_folder_type = "[d]" if sub_folder.lock is False else "[l]"
+            sub_folder_acl = sub_folder.access_level_label(context.projects)
             print(
-                f"{sub_folder_type:{length['type']}}\t{sub_folder.access_level_name:{length['acl']}}\t"
+                f"{sub_folder_type:{length['type']}}\t{sub_folder_acl:{length['acl']}}\t"
                 f"{sub_laboratory_name:{length['laboratory']}}\t{sub_folder.size:{length['size']}}\t"
                 f"{sub_folder.updated_at_name:{length['date']}}\t{sub_folder.name:{length['name']}}"
             )
         for file in sorted(files, key=lambda x: x.name):
             print(
-                f"{'[f]':{length['type']}}\t{folder.access_level_name:{length['acl']}}\t"
+                f"{'[f]':{length['type']}}\t{folder.access_level_label(context.projects):{length['acl']}}\t"
                 f"{context.laboratory.name:{length['laboratory']}}\t{file.size:{length['size']}}\t"
                 f"{file.updated_at_name:{length['date']}}\t{file.name:{length['name']}}"
             )
 
         if context.is_recursive:
-            print("")
+            cls._ls_plain_children(context, folder)
-            for sub_folder in sorted(folder.sub_folders, key=lambda x: x.name):
-                folder_api = FoldersApi(context.client.connection)
-                try:
-                    if sub_folder.lock:
-                        folder_api.auth(sub_folder.id, context.password)
-                    folder = folder_api.retrieve(sub_folder.id)
-                    files = context.client.find_files(sub_folder.id)
-                    cls._ls_plain(context, folder, files)
-                except UnauthorizedException:
-                    pass
 
     @classmethod
     def _folder2dict(
@@ -172,7 +187,7 @@ class LsCommand(BaseCommand):
             "pid": folder.pid,
             "name": folder.name,
             "size": folder.size,
-            "access_level": folder.access_level_name,
+            "access_level": folder.access_level_label(context.projects),
             "lock": folder.lock,
             "laboratory": cls._laboratory_name(context, folder.laboratory_id),
             "description": folder.description,
@@ -211,7 +226,7 @@ class LsCommand(BaseCommand):
             # "thumbnail": file.thumbnail,
             "description": file.description,
             "metadata": file.metadata,
-            "download_url": f"{context.client.connection.url}/{file.download_url}",
+            "download_url": build_download_url(context.client.connection.url, file.download_url),
             "created_at": file.created_at,
             "updated_at": file.updated_at,
         }

mdrsclient/commands/metadata.py

@@ -2,7 +2,6 @@ import json
 from argparse import Namespace
 from typing import Any
 
-from mdrsclient.api import FoldersApi
 from mdrsclient.commands.base import BaseCommand
 
 

mdrsclient/commands/mkdir.py

@@ -1,12 +1,7 @@
-import os
 from argparse import Namespace
 from typing import Any
-from unicodedata import normalize
 
-from mdrsclient.api import FoldersApi
 from mdrsclient.commands.base import BaseCommand
-from mdrsclient.exceptions import IllegalArgumentException
-from mdrsclient.models.file import find_file
 
 
 class MkdirCommand(BaseCommand):

mdrsclient/commands/mv.py

@@ -1,12 +1,7 @@
-import os
 from argparse import Namespace
 from typing import Any
-from unicodedata import normalize
 
-from mdrsclient.api import FilesApi, FoldersApi
 from mdrsclient.commands.base import BaseCommand
-from mdrsclient.exceptions import IllegalArgumentException
-from mdrsclient.models.file import find_file
 
 
 class MvCommand(BaseCommand):

mdrsclient/commands/rm.py

@@ -1,11 +1,7 @@
-import os
 from argparse import Namespace
 from typing import Any
 
-from mdrsclient.api import FilesApi, FoldersApi
 from mdrsclient.commands.base import BaseCommand
-from mdrsclient.exceptions import IllegalArgumentException
-from mdrsclient.models.file import find_file
 
 
 class RmCommand(BaseCommand):

mdrsclient/commands/version.py

@@ -1,7 +1,6 @@
 from argparse import Namespace
 from typing import Any
 
-from mdrsclient.__version__ import __version__
 from mdrsclient.commands.base import BaseCommand
 
 

mdrsclient/config.py

@@ -1,7 +1,7 @@
 import configparser
 import os
 import threading
-from typing import Final, Protocol, runtime_checkable
+from typing import TYPE_CHECKING, Final, Protocol, runtime_checkable
 
 import validators
 
@@ -10,10 +10,43 @@ from mdrsclient.settings import CONFIG_DIRNAME
 from mdrsclient.utils import FileLock
 
 
+def normalize_url(url: str) -> str:
+    """
+    Check a remote URL and put it in the one form every client agrees on.
+
+    `simple_host` is what lets a bare hostname through, so a development server on
+    `localhost` is as acceptable as a deployment behind a domain name. The trailing
+    slash goes because the URL is joined with a path that brings its own.
+    """
+    if not validators.url(url, simple_host=True, validate_scheme=lambda scheme: scheme in ("http", "https")):
+        raise IllegalArgumentException("malformed URI sequence")
+    return url.rstrip("/")
+
+
+def build_download_url(base_url: str | None, path: str) -> str:
+    """
+    Join a remote base URL with a path the API returned.
+
+    The API answers with a relative path and no leading separator, and a configuration
+    written before the URL was normalised may still carry a trailing one.
+    """
+    if path.startswith(("http://", "https://")):
+        return path
+    return f"{(base_url or '').rstrip('/')}/{path.lstrip('/')}"
+
+
 @runtime_checkable
 class ConfigInterface(Protocol):
     remote: str
 
+    if TYPE_CHECKING:
+        # Declared for the type checker, because the class itself is passed around and
+        # called: without this it describes something that takes no arguments to build.
+        # Kept out of the running class, because a protocol that defines `__init__` loses
+        # the guard that stops it being instantiated, and hands that empty `__init__` to
+        # any implementation that does not write its own.
+        def __init__(self, remote: str) -> None: ...
+
     def list(self) -> list[tuple[str, str]]: ...
     @property
     def url(self) -> str | None: ...
@@ -42,8 +75,7 @@ class InMemoryConfig(ConfigInterface):
 
     @url.setter
     def url(self, url: str) -> None:
-        if not validators.url(url):
+        url = normalize_url(url)
-            raise IllegalArgumentException("malformed URI sequence")
         with self.__lock:
             self.__configs[self.remote] = url
 
@@ -91,8 +123,7 @@ class ConfigFile(ConfigInterface):
 
     @url.setter
     def url(self, url: str) -> None:
-        if not validators.url(url):
+        url = normalize_url(url)
-            raise IllegalArgumentException("malformed URI sequence")
         self.__load()
         if self.__config.has_section(self.remote):
             self.__config.remove_section(self.remote)

mdrsclient/connection.py

@@ -1,5 +1,6 @@
 import platform
 import threading
+from contextlib import AbstractContextManager
 from typing import TypedDict
 
 from requests import Response, Session
@@ -23,6 +24,7 @@ class _KwArgsMDRSConnectionPost(TypedDict, total=False):
     params: dict[str, str | int]
     data: dict[str, str | int] | MultipartEncoder
     headers: dict[str, str]
+    timeout: float | tuple[float, float]
 
 
 class _KwArgsMDRSConnectionPut(TypedDict, total=False):
@@ -61,6 +63,14 @@ class MDRSConnection:
     def delete(self, url: str, **kwargs: Unpack[_KwArgsMDRSConnectionDelete]) -> Response:
         return self.session.delete(self.__build_url(url), **kwargs)
 
+    def cache_lock(self) -> AbstractContextManager[None]:
+        """Hold exclusive access to the login cache across every process using it."""
+        return self.__cache.lock()
+
+    def reload_cache(self) -> None:
+        """Re-read the login cache, discarding anything held from an earlier read."""
+        self.__cache.reload()
+
     def logout(self) -> None:
         del self.__cache.user
         del self.__cache.token
@@ -96,7 +106,9 @@ class MDRSConnection:
             return path
         if self.url == "":
             raise MissingConfigurationException("remote host is not configured")
-        return f"{self.url}/{path}"
+        # The path brings its own separator, and a configuration written before the URL
+        # was normalised may still carry a trailing slash of its own.
+        return f"{self.url.rstrip('/')}/{path}"
 
     def __prepare_headers(self) -> None:
         self.session.headers.update(

mdrsclient/exceptions.py

@@ -28,6 +28,24 @@ class UnauthorizedException(MDRSException):
     pass
 
 
+class FolderLockedException(UnauthorizedException):
+    """Thrown when a folder cannot be opened without its password"""
+
+    pass
+
+
+class TokenExpiredException(UnauthorizedException):
+    """Thrown when the access token was still valid when sent but had expired on arrival"""
+
+    pass
+
+
+class ServerBusyException(MDRSException):
+    """Thrown when the server did not start handling a request before its token lapsed"""
+
+    pass
+
+
 class ForbiddenException(MDRSException):
     """Thrown when the current user does not have enough privileges to access the resource"""
 

mdrsclient/models/folder.py

@@ -1,4 +1,5 @@
-from typing import Any, Final, NamedTuple
+import re
+from typing import Any, Final, Mapping
 from unicodedata import normalize
 
 from pydantic.dataclasses import dataclass
@@ -6,39 +7,99 @@ from pydantic.dataclasses import dataclass
 from mdrsclient.models.utils import iso8601_to_user_friendly
 
 
-class FolderAccessLevelItem(NamedTuple):
-    mask: int
-    key: str
-    label: str
-
-
 class FolderAccessLevel:
     # Bit Mask
     # - bit 0: Is Private
     # - bit 1: Is Public
     # - bit 2: With Password
     # - bit 3-7: (Reserved)
-    # - bit 8-15: Restricted Open
+    # - bit 8-15: Restricted Open, one bit per restrict network slot (slot 1 is bit 8)
-    ACCESS_LEVELS: Final[list[FolderAccessLevelItem]] = [
+    # - bit 16-: Project Open, one bit per project bit_position (bit_position 0 is bit 16)
-        FolderAccessLevelItem(0x0204, "5kikan_or_pw_open", "5Kikan or PW Open"),
+    STORAGE: Final[int] = 0x0000
-        FolderAccessLevelItem(0x0104, "cbs_or_pw_open", "CBS or PW Open"),
+    PRIVATE: Final[int] = 0x0001
-        FolderAccessLevelItem(0x0200, "5kikan_open", "5Kikan Open"),
+    PUBLIC: Final[int] = 0x0002
-        FolderAccessLevelItem(0x0100, "cbs_open", "CBS Open"),
+    PASSWORD: Final[int] = 0x0004
-        FolderAccessLevelItem(0x0004, "pw_open", "PW Open"),
+    RESERVED: Final[int] = 0x00F8
-        FolderAccessLevelItem(0x0002, "public", "Public"),
+    RESTRICT_SHIFT: Final[int] = 8
-        FolderAccessLevelItem(0x0001, "private", "Private"),
+    RESTRICT_SLOTS: Final[int] = 8
-        FolderAccessLevelItem(0x0000, "storage", "Storage"),
+    PROJECT_SHIFT: Final[int] = 16
-    ]
+    # The server keeps no names for its restrict networks, so the one in use is named here.
+    # Any other slot is named after its number.
+    RESTRICT_NAMES: Final[dict[int, str]] = {1: "CBS"}
+    LEVEL_NAMES: Final[dict[int, str]] = {STORAGE: "Storage", PRIVATE: "Private", PUBLIC: "Public"}
+
+    @staticmethod
+    def restrict_name(slot: int) -> str:
+        return FolderAccessLevel.RESTRICT_NAMES.get(slot, f"Restrict{slot}")
+
+    @staticmethod
+    def project_name(bit_position: int, projects: Mapping[int, str] | None = None) -> str:
+        name = (projects or {}).get(bit_position)
+        return name if name else f"Project{bit_position}"
+
+    @staticmethod
+    def id2label(id: int, projects: Mapping[int, str] | None = None) -> str | None:
+        """
+        Label an access level from its bits, e.g. 0x0104 is "CBS or PW Open".
+
+        `projects` names the projects by bit_position, as the server tells their members;
+        any other project gets a name made from its bit_position. None for a level that
+        has no meaning: Private or Public with any other bit, or a reserved bit.
+        """
+        if id in FolderAccessLevel.LEVEL_NAMES:
+            return FolderAccessLevel.LEVEL_NAMES[id]
+        if id < 0 or id & (FolderAccessLevel.PRIVATE | FolderAccessLevel.PUBLIC | FolderAccessLevel.RESERVED):
+            return None
+        names = [
+            FolderAccessLevel.restrict_name(slot)
+            for slot in range(1, FolderAccessLevel.RESTRICT_SLOTS + 1)
+            if id & (1 << (FolderAccessLevel.RESTRICT_SHIFT + slot - 1))
+        ]
+        project_bits = id >> FolderAccessLevel.PROJECT_SHIFT
+        names += [
+            FolderAccessLevel.project_name(bit, projects)
+            for bit in range(project_bits.bit_length())
+            if project_bits & (1 << bit)
+        ]
+        if id & FolderAccessLevel.PASSWORD:
+            names.append("PW")
+        return " or ".join(names) + " Open"
 
     @staticmethod
     def key2id(key: str) -> int | None:
-        acl = next((x for x in FolderAccessLevel.ACCESS_LEVELS if x.key == key), None)
+        """
-        return acl.mask if acl is not None else None
+        The access level a key names: `private`, `public`, `storage`, or the parts of an
+        open level joined by `_or_` and ending in `_open`, e.g. `cbs_or_pw_open`. A part
+        is `pw`, `cbs`, `restrict<slot>` or `project<bit_position>`.
+        """
+        fixed = {label.lower(): id for id, label in FolderAccessLevel.LEVEL_NAMES.items()}
+        if key in fixed:
+            return fixed[key]
+        if not key.endswith("_open"):
+            return None
+        id = 0
+        for part in key.removesuffix("_open").split("_or_"):
+            bit = FolderAccessLevel._part2bit(part)
+            if bit is None:
+                return None
+            id |= bit
+        return id
 
     @staticmethod
-    def id2label(id: int) -> str | None:
+    def _part2bit(part: str) -> int | None:
-        acl = next((x for x in FolderAccessLevel.ACCESS_LEVELS if (x.mask & id) == x.mask), None)
+        if part == "pw":
-        return acl.label if acl is not None else None
+            return FolderAccessLevel.PASSWORD
+        slots = {name.lower(): slot for slot, name in FolderAccessLevel.RESTRICT_NAMES.items()}
+        match = re.fullmatch(r"restrict([1-9][0-9]*)", part)
+        slot = slots.get(part, int(match.group(1)) if match else None)
+        if slot is not None:
+            if slot > FolderAccessLevel.RESTRICT_SLOTS:
+                return None
+            return 1 << (FolderAccessLevel.RESTRICT_SHIFT + slot - 1)
+        match = re.fullmatch(r"project(0|[1-9][0-9]*)", part)
+        if match:
+            return 1 << (FolderAccessLevel.PROJECT_SHIFT + int(match.group(1)))
+        return None
 
 
 @dataclass(frozen=True)
@@ -55,10 +116,14 @@ class FolderSimple:
     updated_at: str
     restrict_opened_at: str | None
 
+    def access_level_label(self, projects: Mapping[int, str] | None = None) -> str:
+        """The access level as shown to a user, naming the projects given by bit_position."""
+        label = FolderAccessLevel.id2label(self.access_level, projects)
+        return label if label is not None else f"Unknown (0x{self.access_level:x})"
+
     @property
     def access_level_name(self) -> str:
-        label = FolderAccessLevel.id2label(self.access_level)
+        return self.access_level_label()
-        return label if label is not None else ""
 
     @property
     def lock_name(self) -> str:

mdrsclient/services.py

@@ -1,16 +1,16 @@
 import os
 import re
-from typing import Any
 from unicodedata import normalize
 
 from mdrsclient.api import DoiApi, FilesApi, FoldersApi, LaboratoriesApi, UsersApi
+from mdrsclient.api.utils import token_check
 from mdrsclient.cache import CacheInterface
 from mdrsclient.config import ConfigFile, ConfigInterface
 from mdrsclient.connection import MDRSConnection
 from mdrsclient.exceptions import (
+    FolderLockedException,
     IllegalArgumentException,
     MissingConfigurationException,
-    UnauthorizedException,
     UnexpectedException,
 )
 from mdrsclient.models import File, Folder, Laboratory, Token, User
@@ -20,11 +20,30 @@ from mdrsclient.utils import page_num_from_url
 class MdrsService:
     config_class: type[ConfigInterface] = ConfigFile
 
-    def __init__(self, connection: MDRSConnection, config_class: type[ConfigInterface] | None = None):
+    def __init__(self, connection: MDRSConnection | None, config_class: type[ConfigInterface] | None = None):
-        self.connection = connection
+        self.__connection = connection
         if config_class is not None:
             self.config_class = config_class
 
+    @property
+    def connection(self) -> MDRSConnection:
+        """
+        The connection every remote operation goes through.
+
+        Optional to supply, because `config` and `version` do their work without a remote
+        and are reached through the same client. Asking for it when none was given is a
+        mistake in the caller, and says so rather than failing later on `None`.
+        """
+        if self.__connection is None:
+            raise MissingConfigurationException("This operation requires a remote host.")
+        return self.__connection
+
+    @connection.setter
+    def connection(self, connection: MDRSConnection | None) -> None:
+        # Assignable as it always was: this is a property to check for absence on read,
+        # not to make the connection fixed once the client is built.
+        self.__connection = connection
+
     @classmethod
     def create_connection(
         cls, remote: str, cache: CacheInterface | None = None, config: ConfigInterface | None = None
@@ -48,6 +67,17 @@ class MdrsService:
     def logout(self) -> None:
         self.connection.logout()
 
+    def project_names(self) -> dict[int, str]:
+        """
+        Names of the logged-in user's projects by bit_position, for labelling access levels.
+
+        Empty when nobody is logged in: the server names a project only to its members.
+        """
+        token_check(self.connection)
+        if self.connection.token is None:
+            return {}
+        return UsersApi(self.connection).project_names()
+
     def whoami(self) -> User:
         user_api = UsersApi(self.connection)
         return user_api.current()
@@ -73,7 +103,7 @@ class MdrsService:
             raise UnexpectedException(f"Folder `{path}` not found.")
         if folders[0].lock:
             if password is None:
-                raise UnauthorizedException(f"Folder `{path}` is locked.")
+                raise FolderLockedException(f"Folder `{path}` is locked.")
             folder_api.auth(folders[0].id, password)
         return folder_api.retrieve(folders[0].id)
 
@@ -177,12 +207,20 @@ class MdrsService:
             )
 
         folder_api = FoldersApi(self.connection)
-        folder = folder_api.retrieve(doi_resp.folder_id)
+        # The folder is found by its id, so whether it is locked is known only from what
+        # retrieve returns: a full record, or the short one of a locked folder.
+        folder: Folder | None
+        try:
+            folder = folder_api.retrieve(doi_resp.folder_id)
+        except FolderLockedException:
+            folder = None
 
-        if folder.lock:
+        if folder is None or folder.lock:
             if password is None:
-                raise UnauthorizedException(f"Folder for DOI `{doi_clean}` is locked.")
+                raise FolderLockedException(f"Folder for DOI `{doi_clean}` is locked.")
-            folder_api.auth(doi_resp.folder.id, password)
+            folder_api.auth(doi_resp.folder_id, password)
+            if folder is None:
+                folder = folder_api.retrieve(doi_resp.folder_id)
 
         lab_api = LaboratoriesApi(self.connection)
         labs = lab_api.list()

mdrsclient/transfer.py

@@ -6,7 +6,7 @@ from unicodedata import normalize
 from pydantic.dataclasses import dataclass
 
 from mdrsclient.api import FilesApi, FoldersApi
-from mdrsclient.exceptions import IllegalArgumentException, MDRSException, UnexpectedException
+from mdrsclient.exceptions import IllegalArgumentException, UnexpectedException
 from mdrsclient.models import File, Folder, Laboratory
 from mdrsclient.models.file import find_file
 from mdrsclient.settings import CONCURRENT
@@ -27,7 +27,6 @@ class DownloadFileInfo:
 
 @dataclass
 class DownloadContext:
-    hasError: bool
     isSkipIfExists: bool
     files: list[DownloadFileInfo]
 
@@ -47,54 +46,67 @@ class Uploader:
         laboratory = self.client.find_laboratory(laboratory_name)
         folder = self.client.find_folder(laboratory, r_path)
         files = self.client.find_files(folder.id)
-        infos: list[UploadFileInfo] = []
         if os.path.isdir(l_path):
             if not is_recursive:
                 raise IllegalArgumentException(f"Cannot upload `{local_path}`: Is a directory.")
-            folder_api = FoldersApi(self.client.connection)
+            infos = self.__collect_directory_uploads(laboratory, r_path, l_path, folder, files)
-            folder_map: dict[str, Folder] = {}
-            folder_map[r_path] = folder
-            files_map: dict[str, list[File]] = {}
-            files_map[r_path] = files
-            l_basename = os.path.basename(l_path)
-            for dirpath, _, filenames in os.walk(l_path, followlinks=True):
-                sub = l_basename if dirpath == l_path else os.path.join(l_basename, os.path.relpath(dirpath, l_path))
-                d_dirname = os.path.join(r_path, sub)
-                d_basename = os.path.basename(d_dirname)
-                # prepare destination parent path
-                d_parent_dirname = os.path.dirname(d_dirname)
-                if folder_map.get(d_parent_dirname) is None:
-                    parent_folder = self.client.find_folder(laboratory, d_parent_dirname)
-                    folder_map[d_parent_dirname] = parent_folder
-                    parent_files = self.client.find_files(parent_folder.id)
-                    files_map[d_parent_dirname] = parent_files
-                # prepare destination path
-                if folder_map.get(d_dirname) is None:
-                    d_folder = folder_map[d_parent_dirname].find_sub_folder(d_basename)
-                    if d_folder is None:
-                        d_folder_id = folder_api.create(normalize("NFC", d_basename), folder_map[d_parent_dirname].id)
-                    else:
-                        d_folder_id = d_folder.id
-                    print(d_dirname)
-                    folder_map[d_dirname] = folder_api.retrieve(d_folder_id)
-                    files_map[d_dirname] = self.client.find_files(d_folder_id)
-                    if d_folder is None:
-                        folder_map[d_parent_dirname].sub_folders.append(folder_map[d_dirname])
-                # register upload file list
-                for filename in filenames:
-                    infos.append(
-                        UploadFileInfo(folder_map[d_dirname], files_map[d_dirname], os.path.join(dirpath, filename))
-                    )
         else:
-            infos.append(UploadFileInfo(folder, files, l_path))
+            infos = [UploadFileInfo(folder, files, l_path)]
-        self.__multiple_upload(infos, is_skip_if_exists)
+        if not self.__multiple_upload(infos, is_skip_if_exists):
+            # One file failing is worth reporting on its own line, and worth the caller
+            # hearing about: a batch that lost files is not a batch that succeeded.
+            raise UnexpectedException("Some files failed to upload.")
 
-    def __multiple_upload(self, infos: list[UploadFileInfo], is_skip_if_exists: bool) -> None:
+    def __collect_directory_uploads(
+        self, laboratory: Laboratory, r_path: str, l_path: str, folder: Folder, files: list[File]
+    ) -> list[UploadFileInfo]:
+        """Mirror a local directory tree on the remote, and list the files to send into it."""
+        infos: list[UploadFileInfo] = []
+        folder_api = FoldersApi(self.client.connection)
+        folder_map: dict[str, Folder] = {}
+        folder_map[r_path] = folder
+        files_map: dict[str, list[File]] = {}
+        files_map[r_path] = files
+        l_basename = os.path.basename(l_path)
+        for dirpath, _, filenames in os.walk(l_path, followlinks=True):
+            sub = l_basename if dirpath == l_path else os.path.join(l_basename, os.path.relpath(dirpath, l_path))
+            d_dirname = os.path.join(r_path, sub)
+            d_basename = os.path.basename(d_dirname)
+            # prepare destination parent path
+            d_parent_dirname = os.path.dirname(d_dirname)
+            if folder_map.get(d_parent_dirname) is None:
+                parent_folder = self.client.find_folder(laboratory, d_parent_dirname)
+                folder_map[d_parent_dirname] = parent_folder
+                parent_files = self.client.find_files(parent_folder.id)
+                files_map[d_parent_dirname] = parent_files
+            # prepare destination path
+            if folder_map.get(d_dirname) is None:
+                d_folder = folder_map[d_parent_dirname].find_sub_folder(d_basename)
+                if d_folder is None:
+                    d_folder_id = folder_api.create(normalize("NFC", d_basename), folder_map[d_parent_dirname].id)
+                else:
+                    d_folder_id = d_folder.id
+                print(d_dirname)
+                folder_map[d_dirname] = folder_api.retrieve(d_folder_id)
+                files_map[d_dirname] = self.client.find_files(d_folder_id)
+                if d_folder is None:
+                    folder_map[d_parent_dirname].sub_folders.append(folder_map[d_dirname])
+            # register upload file list
+            for filename in filenames:
+                infos.append(
+                    UploadFileInfo(folder_map[d_dirname], files_map[d_dirname], os.path.join(dirpath, filename))
+                )
+        return infos
+
+    def __multiple_upload(self, infos: list[UploadFileInfo], is_skip_if_exists: bool) -> bool:
+        """Send every file, and report whether all of them arrived."""
         file_api = FilesApi(self.client.connection)
         with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
-            pool.map(lambda x: self.__multiple_upload_worker(file_api, x, is_skip_if_exists), infos)
+            results = pool.map(lambda x: self.__multiple_upload_worker(file_api, x, is_skip_if_exists), infos)
+            # Consumed inside the block: the results are what carry each worker's verdict.
+            return all(list(results))
 
-    def __multiple_upload_worker(self, file_api: FilesApi, info: UploadFileInfo, is_skip_if_exists: bool) -> None:
+    def __multiple_upload_worker(self, file_api: FilesApi, info: UploadFileInfo, is_skip_if_exists: bool) -> bool:
         basename = os.path.basename(info.path)
         file = find_file(info.files, basename)
         try:
@@ -103,8 +115,14 @@ class Uploader:
             elif not is_skip_if_exists or file.size != os.path.getsize(info.path):
                 file_api.update(file, info.path)
             print(os.path.join(info.folder.path, basename))
-        except MDRSException as e:
+        except Exception as e:
-            print(f"Error: {e}")
+            # Everything, not just the exceptions the API layer raises: the batch verdict
+            # is read now that the results are consumed, and a file vanishing between the
+            # walk and the upload would otherwise end the whole run with a traceback and
+            # throw away what every other file did.
+            print(f"Failed: {info.path}: {e}")
+            return False
+        return True
 
 
 class Downloader:
@@ -120,90 +138,117 @@ class Downloader:
         password: str | None = None,
         excludes: list[str] | None = None,
     ) -> None:
+        if not self.__download(remote_path, local_path, is_recursive, is_skip_if_exists, password, excludes):
+            # Every failure has already been printed against the file it belongs to.
+            # This is what makes the command as a whole end in failure.
+            raise UnexpectedException("Some files failed to download.")
+
+    def __download(
+        self,
+        remote_path: str,
+        local_path: str,
+        is_recursive: bool,
+        is_skip_if_exists: bool,
+        password: str | None,
+        excludes: list[str] | None,
+    ) -> bool:
+        """Fetch what the remote path names, and report whether every file arrived."""
         excludes_clean = excludes or []
-        # Detect DOI path: "remote:10.xxxx/prefix.ID[/optional/sub/path]"
-        path_component = remote_path.split(":", 1)[1] if ":" in remote_path else ""
-        if self.client.is_doi(path_component):
-            remote, doi, subpath = self.client.parse_doi_remote_host(remote_path)
-
-            l_dirname = os.path.realpath(local_path)
-            if not os.path.isdir(l_dirname):
-                raise IllegalArgumentException(f"Local directory `{local_path}` not found.")
-            doi_folder, laboratory = self.client.find_folder_by_doi(doi, password)
-
-            subpath_clean = subpath.rstrip("/")
-            if not subpath_clean:
-                folder = doi_folder
-                is_folder = True
-            else:
-                r_dirname = os.path.dirname(subpath_clean)
-                r_basename = os.path.basename(subpath_clean)
-                abs_path = doi_folder.path.rstrip("/") + r_dirname
-                r_parent_folder = self.client.find_folder(laboratory, abs_path, password)
-                r_parent_files = self.client.find_files(r_parent_folder.id)
-                file = find_file(r_parent_files, r_basename)
-                if file is not None:
-                    if self.__check_excludes(excludes_clean, laboratory, r_parent_folder, file):
-                        return
-                    context = DownloadContext(False, is_skip_if_exists, [])
-                    l_path = os.path.join(l_dirname, r_basename)
-                    context.files.append(DownloadFileInfo(file, l_path))
-                    self.__multiple_download(context)
-                    return
-                else:
-                    folder_simple = r_parent_folder.find_sub_folder(r_basename)
-                    if folder_simple is None:
-                        raise IllegalArgumentException(f"File or folder `{subpath_clean}` not found.")
-                    folder = FoldersApi(self.client.connection).retrieve(folder_simple.id)
-                    is_folder = True
-
-            # For a DOI target the whole folder is the download target.
-            if not is_recursive:
-                # Non-recursive: download only the files at the top level of the DOI folder.
-                files = self.client.find_files(folder.id)
-                context = DownloadContext(False, is_skip_if_exists, [])
-                for file in files:
-                    if self.__check_excludes(excludes_clean, laboratory, folder, file):
-                        continue
-                    l_path = os.path.join(l_dirname, file.name)
-                    context.files.append(DownloadFileInfo(file, l_path))
-                self.__multiple_download(context)
-                return
-            folder_api = FoldersApi(self.client.connection)
-            self.__multiple_download_pickup_recursive_files(
-                folder_api, laboratory, folder.id, l_dirname, excludes_clean, is_skip_if_exists
-            )
-            return
-
-        remote, laboratory_name, r_path = self.client.parse_remote_host_with_path(remote_path)
-        r_path = r_path.rstrip("/")
-        r_dirname = os.path.dirname(r_path)
-        r_basename = os.path.basename(r_path)
-
         l_dirname = os.path.realpath(local_path)
         if not os.path.isdir(l_dirname):
             raise IllegalArgumentException(f"Local directory `{local_path}` not found.")
+
+        # "remote:10.xxxx/prefix.ID[/optional/sub/path]" names a published dataset rather
+        # than a path within a laboratory, and is resolved through the DOI instead.
+        path_component = remote_path.split(":", 1)[1] if ":" in remote_path else ""
+        if self.client.is_doi(path_component):
+            return self.__download_doi(
+                remote_path, l_dirname, is_recursive, is_skip_if_exists, password, excludes_clean
+            )
+
+        remote, laboratory_name, r_path = self.client.parse_remote_host_with_path(remote_path)
+        r_path = r_path.rstrip("/")
+        r_basename = os.path.basename(r_path)
         laboratory = self.client.find_laboratory(laboratory_name)
-        r_parent_folder = self.client.find_folder(laboratory, r_dirname, password)
+        r_parent_folder = self.client.find_folder(laboratory, os.path.dirname(r_path), password)
         r_parent_files = self.client.find_files(r_parent_folder.id)
         file = find_file(r_parent_files, r_basename)
         if file is not None:
-            if self.__check_excludes(excludes_clean, laboratory, r_parent_folder, file):
+            return self.__download_one(
-                return
+                excludes_clean, laboratory, r_parent_folder, file, l_dirname, r_basename, is_skip_if_exists
-            context = DownloadContext(False, is_skip_if_exists, [])
-            l_path = os.path.join(l_dirname, r_basename)
-            context.files.append(DownloadFileInfo(file, l_path))
-            self.__multiple_download(context)
-        else:
-            folder = r_parent_folder.find_sub_folder(r_basename)
-            if folder is None:
-                raise IllegalArgumentException(f"File or folder `{r_path}` not found.")
-            if not is_recursive:
-                raise IllegalArgumentException(f"Cannot download `{r_path}`: Is a folder.")
-            folder_api = FoldersApi(self.client.connection)
-            self.__multiple_download_pickup_recursive_files(
-                folder_api, laboratory, folder.id, l_dirname, excludes_clean, is_skip_if_exists
             )
+        folder = r_parent_folder.find_sub_folder(r_basename)
+        if folder is None:
+            raise IllegalArgumentException(f"File or folder `{r_path}` not found.")
+        if not is_recursive:
+            raise IllegalArgumentException(f"Cannot download `{r_path}`: Is a folder.")
+        return self.__multiple_download_pickup_recursive_files(
+            FoldersApi(self.client.connection), laboratory, folder.id, l_dirname, excludes_clean, is_skip_if_exists
+        )
+
+    def __download_doi(
+        self,
+        remote_path: str,
+        l_dirname: str,
+        is_recursive: bool,
+        is_skip_if_exists: bool,
+        password: str | None,
+        excludes: list[str],
+    ) -> bool:
+        """Fetch what a DOI names: the dataset's folder, or something inside it."""
+        remote, doi, subpath = self.client.parse_doi_remote_host(remote_path)
+        doi_folder, laboratory = self.client.find_folder_by_doi(doi, password)
+
+        subpath_clean = subpath.rstrip("/")
+        if not subpath_clean:
+            folder = doi_folder
+        else:
+            r_basename = os.path.basename(subpath_clean)
+            abs_path = doi_folder.path.rstrip("/") + os.path.dirname(subpath_clean)
+            r_parent_folder = self.client.find_folder(laboratory, abs_path, password)
+            file = find_file(self.client.find_files(r_parent_folder.id), r_basename)
+            if file is not None:
+                return self.__download_one(
+                    excludes, laboratory, r_parent_folder, file, l_dirname, r_basename, is_skip_if_exists
+                )
+            folder_simple = r_parent_folder.find_sub_folder(r_basename)
+            if folder_simple is None:
+                raise IllegalArgumentException(f"File or folder `{subpath_clean}` not found.")
+            folder = FoldersApi(self.client.connection).retrieve(folder_simple.id)
+
+        if is_recursive:
+            return self.__multiple_download_pickup_recursive_files(
+                FoldersApi(self.client.connection), laboratory, folder.id, l_dirname, excludes, is_skip_if_exists
+            )
+        # Without -r the dataset's own files are fetched, and its sub-folders are not.
+        context = DownloadContext(is_skip_if_exists, [])
+        for file in self.client.find_files(folder.id):
+            if self.__check_excludes(excludes, laboratory, folder, file):
+                continue
+            context.files.append(DownloadFileInfo(file, os.path.join(l_dirname, file.name)))
+        return self.__multiple_download(context)
+
+    def __download_one(
+        self,
+        excludes: list[str],
+        laboratory: Laboratory,
+        folder: Folder,
+        file: File,
+        l_dirname: str,
+        local_name: str,
+        is_skip_if_exists: bool,
+    ) -> bool:
+        """
+        Fetch a single named file into the local directory.
+
+        Saved under the name the caller asked for rather than the one the server holds:
+        the two are matched case-insensitively, so they need not be spelled alike.
+        """
+        if self.__check_excludes(excludes, laboratory, folder, file):
+            return True
+        context = DownloadContext(is_skip_if_exists, [])
+        context.files.append(DownloadFileInfo(file, os.path.join(l_dirname, local_name)))
+        return self.__multiple_download(context)
 
     def __multiple_download_pickup_recursive_files(
         self,
@@ -213,47 +258,60 @@ class Downloader:
         basedir: str,
         excludes: list[str],
         is_skip_if_exists: bool,
-    ) -> None:
+    ) -> bool:
-        context = DownloadContext(False, is_skip_if_exists, [])
+        context = DownloadContext(is_skip_if_exists, [])
-        folder = folder_api.retrieve(folder_id)
+        try:
-        files = self.client.find_files(folder.id)
+            folder = folder_api.retrieve(folder_id)
+            files = self.client.find_files(folder.id)
+        except Exception as e:
+            print(f"Failed: {basedir}: {e}")
+            return False
         dirname = os.path.join(basedir, folder.name)
         if self.__check_excludes(excludes, laboratory, folder, None):
-            return
+            return True
-        if not os.path.exists(dirname):
+        try:
-            os.makedirs(dirname)
+            # `exist_ok` rather than a prior check: two workers can reach the same parent.
+            os.makedirs(dirname, exist_ok=True)
+        except OSError as e:
+            # One folder the client cannot make locally is not a reason to abandon its
+            # siblings, which is what this walk now promises.
+            print(f"Failed: {dirname}: {e}")
+            return False
         print(dirname)
         for file in files:
             if self.__check_excludes(excludes, laboratory, folder, file):
                 continue
             path = os.path.join(dirname, file.name)
             context.files.append(DownloadFileInfo(file, path))
-        self.__multiple_download(context)
+        succeeded = self.__multiple_download(context)
-        if context.hasError:
+        # A folder that lost a file is still a folder whose sub-folders the user asked
-            raise UnexpectedException("Some files failed to download.")
+        # for, so the walk carries on and the verdict is collected for the caller.
         for sub_folder in folder.sub_folders:
-            self.__multiple_download_pickup_recursive_files(
+            if not self.__multiple_download_pickup_recursive_files(
                 folder_api, laboratory, sub_folder.id, dirname, excludes, is_skip_if_exists
-            )
+            ):
+                succeeded = False
+        return succeeded
 
-    def __multiple_download(self, context: DownloadContext) -> None:
+    def __multiple_download(self, context: DownloadContext) -> bool:
+        """Fetch every file in the batch, and report whether all of them arrived."""
         file_api = FilesApi(self.client.connection)
         with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
             results = pool.map(
                 lambda x: self.__multiple_download_worker(file_api, x, context.isSkipIfExists), context.files
             )
-            hasError = next(filter(lambda x: x is False, results), None)
+            # Consumed inside the block, and in full: every worker's verdict counts, not
-            if hasError is not None:
+            # just the first refusal.
-                context.hasError = True
+            return all(list(results))
 
     def __multiple_download_worker(self, file_api: FilesApi, info: DownloadFileInfo, is_skip_if_exists: bool) -> bool:
         if not is_skip_if_exists or not os.path.exists(info.path) or info.file.size != os.path.getsize(info.path):
             try:
                 file_api.download(info.file, info.path)
-            except Exception:
+            except Exception as e:
-                print(f"Failed: {info.path}")
+                # Nothing to clear up: a failed transfer writes only to its own scratch
-                if os.path.isfile(info.path):
+                # file beside the destination, and removes that itself.
-                    os.remove(info.path)
+                print(f"Failed: {info.path}: {e}")
                 return False
         print(info.path)
         return True

mdrsclient/utils.py

@@ -1,5 +1,6 @@
 import os
-from typing import IO, Any
+import time
+from typing import IO, Any, Final
 from urllib.parse import parse_qs, urlparse
 
 if os.name == "nt":
@@ -9,10 +10,23 @@ elif os.name == "posix":
 
 
 class FileLock:
+    # Long enough to outlast a token refresh, which is what the lock is held across.
+    WAIT_SECONDS: Final[float] = 60.0
+
     @staticmethod
     def lock(file: IO[Any]) -> None:
         if os.name == "nt":
-            msvcrt.locking(file.fileno(), msvcrt.LK_LOCK, 1)
+            # msvcrt.LK_LOCK gives up after ten one-second attempts, which is shorter
+            # than the refresh it now has to wait for, so do the waiting here instead.
+            deadline = time.monotonic() + FileLock.WAIT_SECONDS
+            while True:
+                try:
+                    msvcrt.locking(file.fileno(), msvcrt.LK_NBLCK, 1)
+                    return
+                except OSError:
+                    if time.monotonic() >= deadline:
+                        raise
+                    time.sleep(0.1)
         elif os.name == "posix":
             fcntl.flock(file.fileno(), fcntl.LOCK_EX)
 

pyproject.toml

@@ -1,16 +1,25 @@
-[tool.poetry]
+[build-system]
+requires = ["flit_core>=4.0.2,<5.0.0"]
+build-backend = "flit_core.buildapi"
+
+[tool.flit.module]
+name = "mdrsclient"
+
+[project]
 name = "mdrs-client-python"
-version = "1.3.18"
+version = "1.3.21"
 description = "The mdrs-client-python is python library and a command-line client for up- and downloading files to and from MDRS based repository."
-authors = ["Yoshihiro OKUMURA <yoshihiro.okumura@riken.jp>"]
+authors = [
-license = "MIT"
+    { name = "Yoshihiro OKUMURA", email = "yoshihiro.okumura@riken.jp" }
+]
+license = { text = "MIT" }
 readme = "README.md"
-classifiers=[
+requires-python = ">=3.11"
+classifiers = [
     "Development Status :: 4 - Beta",
     "Environment :: Console",
     "Intended Audience :: Developers",
     "Intended Audience :: Science/Research",
-    "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
@@ -18,37 +27,32 @@ classifiers=[
     "OSI Approved :: MIT License",
     "Topic :: Utilities",
 ]
-packages = [
+
-    { include = "mdrsclient" }
+dependencies = [
+    "requests>=2.34.2",
+    "requests-toolbelt>=1.0.0",
+    "python-dotenv>=1.2.4",
+    "pydantic>=2.14.3",
+    "pydantic-settings>=2.16.2",
+    "PyJWT>=2.15.1",
+    "validators>=0.36.0",
 ]
 
-[tool.poetry.dependencies]
+[project.optional-dependencies]
-python = "^3.10"
+dev = [
-requests = "^2.34.2"
+    "black>=26.10.1",
-requests-toolbelt = "^1.0.0"
+    "flake8>=7.5.0",
-python-dotenv = "^1.1.0"
+    "Flake8-pyproject>=1.2.4",
-pydantic = "^2.13.4"
+    "isort>=9.0.2",
-pydantic-settings = "^2.14.2"
+    "pyright>=1.1.414",
-PyJWT = "^2.13.0"
+]
-validators = "^0.35.0"
 
-[tool.poetry.group.dev.dependencies]
+[project.scripts]
-black = "^26.5.1"
+mdrs = "mdrsclient.__main__:main"
-flake8 = "^7.2.0"
-Flake8-pyproject = "^1.2.3"
-isort = "^8.0.1"
-pyright = "^1.1.411"
-
-[tool.poetry.scripts]
-mdrs = 'mdrsclient.__main__:main'
-
-[build-system]
-requires = ["poetry-core"]
-build-backend = "poetry.core.masonry.api"
 
 [tool.black]
 line-length = 120
-target-version = ['py310']
+target-version = ['py311']
 include = '\.pyi?$'
 exclude = '''
 /(
@@ -63,6 +67,9 @@ exclude = '''
 exclude = ".git, .venv, __pycache__, dist"
 max-complexity = 10
 max-line-length = 120
+# E203 flags the space black puts before a slice colon. Black formats this project, so
+# its output is the standard and flake8 has to accept it.
+extend-ignore = ["E203"]
 
 [tool.isort]
 profile = "black"
@@ -72,6 +79,12 @@ line_length = 120
 
 [tool.pyright]
 typeCheckingMode = "basic"
+# This project keeps its environment in `./.venv`. Named explicitly because without it
+# pyright falls back to the system interpreter, cannot resolve pydantic, and reports every
+# model field as an unknown argument. An environment kept elsewhere gets that same fallback
+# and can override it with a local `pyrightconfig.json` or `pyright --pythonpath`.
+venvPath = "."
+venv = ".venv"
 exclude = ["**/__pycache__", "**/.*", "dist"]
 #reportUnknownMemberType = "warning"
 #reportUnknownVariableType = "warning"

tests/test_client.py

@@ -0,0 +1,165 @@
+import unittest
+from unittest.mock import MagicMock, patch
+
+from mdrsclient.client import MdrsClient
+from mdrsclient.exceptions import IllegalArgumentException
+from mdrsclient.models import File, Folder, FolderSimple, Laboratories, Laboratory
+
+TIMESTAMP = "2026-01-01T00:00:00+09:00"
+LABORATORY = Laboratory(id=1, name="mylab", pi_name="PI", full_name="My Laboratory")
+
+
+def make_file(id: str, name: str) -> File:
+    return File(
+        id=id,
+        name=name,
+        type="text/plain",
+        size=1,
+        thumbnail=None,
+        description="",
+        metadata={},
+        download_url=f"v3/files/{id}/download/",
+        created_at=TIMESTAMP,
+        updated_at=TIMESTAMP,
+    )
+
+
+def make_sub_folder(id: str, name: str) -> FolderSimple:
+    return FolderSimple(
+        id=id,
+        pid=None,
+        name=name,
+        access_level=1,
+        lock=False,
+        size=0,
+        laboratory_id=1,
+        description="",
+        created_at=TIMESTAMP,
+        updated_at=TIMESTAMP,
+        restrict_opened_at=None,
+    )
+
+
+def make_folder(id: str, name: str, path: str, sub_folders: list[FolderSimple] | None = None) -> Folder:
+    return Folder(
+        id=id,
+        pid=None,
+        name=name,
+        access_level=1,
+        lock=False,
+        size=0,
+        laboratory_id=1,
+        description="",
+        created_at=TIMESTAMP,
+        updated_at=TIMESTAMP,
+        restrict_opened_at=None,
+        metadata=[],
+        sub_folders=sub_folders if sub_folders is not None else [],
+        path=path,
+    )
+
+
+class TestLsColumnWidths(unittest.TestCase):
+    """Each column is as wide as the widest thing printed under it."""
+
+    def test_the_size_column_fits_a_sub_folder_larger_than_its_parent(self):
+        from unittest.mock import MagicMock as _MagicMock
+
+        from mdrsclient.commands.ls import LsCommand, LsCommandContext
+
+        laboratories = Laboratories()
+        laboratories.append(LABORATORY)
+        connection = _MagicMock()
+        connection.laboratories = laboratories
+        client = MdrsClient(connection)
+
+        big = make_sub_folder("s1", "big")
+        object.__setattr__(big, "size", 123456789)
+        parent = make_folder("f1", "root", "/root/", [big])
+        context = LsCommandContext("remote:/mylab", client, LABORATORY, "", False, False, False)
+
+        widths = LsCommand._column_widths(context, parent, [])
+
+        self.assertEqual(widths["size"], len("123456789"))
+
+
+class TestCopyAndMove(unittest.TestCase):
+    """`cp` and `mv` resolve both ends the same way and differ only in what they call."""
+
+    def make_client(self, folders_by_path: dict, files_by_folder: dict) -> MdrsClient:
+        client = MdrsClient(MagicMock())
+        client.find_laboratory = MagicMock(return_value=LABORATORY)
+        client.find_folder = MagicMock(side_effect=lambda lab, path, password=None: folders_by_path[path])
+        client.find_files = MagicMock(side_effect=lambda folder_id: files_by_folder.get(folder_id, []))
+        return client
+
+    def make_tree(self, source_has: str) -> tuple[MdrsClient, dict, dict]:
+        sub = make_sub_folder("fsub", "sub")
+        folder_a = make_folder("fa", "a", "/a/", [sub] if source_has == "folder" else [])
+        folder_b = make_folder("fb", "b", "/b/")
+        folders = {"/a": folder_a, "/b": folder_b, "/b/": folder_b}
+        files = {"fa": [make_file("x1", "data.txt")] if source_has == "file" else [], "fb": []}
+        return self.make_client(folders, files), folders, files
+
+    def test_a_file_is_copied_into_the_destination_folder(self):
+        client, _, files = self.make_tree("file")
+        with patch("mdrsclient.client.FilesApi") as files_api:
+            client.cp("myremote:/mylab/a/data.txt", "myremote:/mylab/b/data.txt")
+            files_api.return_value.copy.assert_called_once_with(files["fa"][0], "fb", "data.txt")
+
+    def test_a_file_is_moved_into_the_destination_folder(self):
+        client, _, files = self.make_tree("file")
+        with patch("mdrsclient.client.FilesApi") as files_api:
+            client.mv("myremote:/mylab/a/data.txt", "myremote:/mylab/b/data.txt")
+            files_api.return_value.move.assert_called_once_with(files["fa"][0], "fb", "data.txt")
+
+    def test_a_trailing_separator_keeps_the_source_name(self):
+        client, _, files = self.make_tree("file")
+        with patch("mdrsclient.client.FilesApi") as files_api:
+            client.mv("myremote:/mylab/a/data.txt", "myremote:/mylab/b/")
+            files_api.return_value.move.assert_called_once_with(files["fa"][0], "fb", "data.txt")
+
+    def test_copying_a_folder_needs_the_recursive_flag(self):
+        client, _, _ = self.make_tree("folder")
+        with patch("mdrsclient.client.FoldersApi") as folders_api:
+            with self.assertRaises(IllegalArgumentException) as caught:
+                client.cp("myremote:/mylab/a/sub", "myremote:/mylab/b/sub")
+            self.assertIn("Is a folder", str(caught.exception))
+            folders_api.return_value.copy.assert_not_called()
+
+    def test_a_folder_is_copied_when_recursive(self):
+        client, folders, _ = self.make_tree("folder")
+        with patch("mdrsclient.client.FoldersApi") as folders_api:
+            client.cp("myremote:/mylab/a/sub", "myremote:/mylab/b/sub", is_recursive=True)
+            folders_api.return_value.copy.assert_called_once_with(folders["/a"].sub_folders[0], "fb", "sub")
+
+    def test_a_folder_is_moved_without_the_recursive_flag(self):
+        client, folders, _ = self.make_tree("folder")
+        with patch("mdrsclient.client.FoldersApi") as folders_api:
+            client.mv("myremote:/mylab/a/sub", "myremote:/mylab/b/sub")
+            folders_api.return_value.move.assert_called_once_with(folders["/a"].sub_folders[0], "fb", "sub")
+
+    def test_a_destination_that_already_holds_the_name_is_refused(self):
+        sub = make_sub_folder("fsub", "sub")
+        folders = {"/a": make_folder("fa", "a", "/a/", [sub]), "/b": make_folder("fb", "b", "/b/")}
+        files = {"fa": [], "fb": [make_file("x2", "sub")]}
+        client = self.make_client(folders, files)
+        with self.assertRaises(IllegalArgumentException) as caught:
+            client.mv("myremote:/mylab/a/sub", "myremote:/mylab/b/sub")
+        self.assertIn("Cannot overwrite non-folder", str(caught.exception))
+
+    def test_a_transfer_across_laboratories_is_refused(self):
+        client, _, _ = self.make_tree("file")
+        with self.assertRaises(IllegalArgumentException) as caught:
+            client.cp("myremote:/mylab/a/data.txt", "myremote:/otherlab/b/data.txt")
+        self.assertIn("Laboratory mismatched", str(caught.exception))
+
+    def test_a_source_that_does_not_exist_is_refused(self):
+        client, _, _ = self.make_tree("file")
+        with self.assertRaises(IllegalArgumentException) as caught:
+            client.mv("myremote:/mylab/a/missing.txt", "myremote:/mylab/b/missing.txt")
+        self.assertIn("not found", str(caught.exception))
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_commands.py

@@ -23,6 +23,7 @@ from mdrsclient.commands import (
     VersionCommand,
     WhoamiCommand,
 )
+from mdrsclient.exceptions import IllegalArgumentException, MissingConfigurationException
 from mdrsclient.models import Folder, Laboratory
 
 
@@ -177,6 +178,7 @@ class TestCommands(unittest.TestCase):
             mock_lab = Laboratory(id=1, name="mylab", pi_name="pi_name", full_name="full_name")
 
             mock_client.resolve_folder.return_value = (mock_folder, mock_lab)
+            mock_client.project_names.return_value = {}
 
             mock_file = MagicMock()
             mock_file.name = "file.txt"
@@ -228,6 +230,7 @@ class TestCommands(unittest.TestCase):
             mock_lab = Laboratory(id=1, name="mylab", pi_name="pi_name", full_name="full_name")
 
             mock_client.resolve_folder.return_value = (mock_folder, mock_lab)
+            mock_client.project_names.return_value = {}
             mock_client.connection.laboratories.find_by_id.return_value = mock_lab
 
             mock_file = MagicMock()
@@ -316,6 +319,48 @@ class TestCommands(unittest.TestCase):
             mock_client_class.from_remote.assert_called_once_with("myremote")
             mock_client.chacl.assert_called_once_with("myremote:/mylab/", 1, True, "secret")
 
+    def test_chacl_command_project_level(self):
+        with patch("mdrsclient.client.MdrsClient") as mock_client_class:
+            mock_client = MagicMock()
+            mock_client_class.from_remote.return_value = mock_client
+
+            args = self.parse_args(ChaclCommand, ["chacl", "cbs_or_project0_open", "myremote:/mylab/"])
+            args.func(args)
+
+            mock_client.chacl.assert_called_once_with("myremote:/mylab/", 0x10100, False, None)
+
+    def test_chacl_command_invalid_level_lists_projects(self):
+        with patch("mdrsclient.client.MdrsClient") as mock_client_class:
+            mock_client = MagicMock()
+            mock_client_class.from_remote.return_value = mock_client
+            mock_client.project_names.return_value = {0: "Brain/MINDS 2.0"}
+
+            args = self.parse_args(ChaclCommand, ["chacl", "5kikan_open", "myremote:/mylab/"])
+            with self.assertRaises(IllegalArgumentException) as cm:
+                args.func(args)
+
+            self.assertIn("project<n>_open", str(cm.exception))
+            self.assertIn("project0_open: Brain/MINDS 2.0", str(cm.exception))
+            mock_client.chacl.assert_not_called()
+
+    def test_chacl_command_invalid_level_without_server(self):
+        with patch("mdrsclient.client.MdrsClient") as mock_client_class:
+            mock_client_class.from_remote.side_effect = MissingConfigurationException("no remote")
+
+            args = self.parse_args(ChaclCommand, ["chacl", "5kikan_open", "myremote:/mylab/"])
+            with self.assertRaises(IllegalArgumentException) as cm:
+                args.func(args)
+
+            self.assertNotIn("Projects you belong to", str(cm.exception))
+
+    def test_chacl_command_refuses_public_and_storage(self):
+        for level in ("public", "storage"):
+            with self.subTest(level=level), patch("mdrsclient.client.MdrsClient") as mock_client_class:
+                args = self.parse_args(ChaclCommand, ["chacl", level, "myremote:/mylab/"])
+                with self.assertRaises(IllegalArgumentException):
+                    args.func(args)
+                mock_client_class.from_remote.assert_not_called()
+
     def test_metadata_command(self):
         with patch("mdrsclient.client.MdrsClient") as mock_client_class:
             mock_client = MagicMock()

tests/test_folder_access_level.py

@@ -0,0 +1,154 @@
+import unittest
+from unittest.mock import MagicMock, patch
+
+from mdrsclient.api import UsersApi
+from mdrsclient.models import FolderSimple
+from mdrsclient.models.folder import FolderAccessLevel
+from mdrsclient.services import MdrsService
+
+TIMESTAMP = "2026-01-01T00:00:00+09:00"
+BM2 = {0: "Brain/MINDS 2.0"}
+
+
+def make_folder(access_level: int) -> FolderSimple:
+    return FolderSimple(
+        id="f1",
+        pid=None,
+        name="data",
+        access_level=access_level,
+        lock=False,
+        size=0,
+        laboratory_id=1,
+        description="",
+        created_at=TIMESTAMP,
+        updated_at=TIMESTAMP,
+        restrict_opened_at=None,
+    )
+
+
+def current_user(projects: list[dict]) -> dict:
+    return {
+        "id": 1,
+        "username": "user",
+        "first_name": "",
+        "last_name": "",
+        "email": "",
+        "orcid_id": "",
+        "laboratories": [],
+        "is_staff": False,
+        "is_active": True,
+        "is_superuser": False,
+        "is_reviewer": False,
+        "last_login": TIMESTAMP,
+        "date_joined": TIMESTAMP,
+        "projects": projects,
+    }
+
+
+class TestId2Label(unittest.TestCase):
+    def test_single_levels(self) -> None:
+        cases = {
+            0x0000: "Storage",
+            0x0001: "Private",
+            0x0002: "Public",
+            0x0004: "PW Open",
+            0x0100: "CBS Open",
+            0x0104: "CBS or PW Open",
+        }
+        for id, label in cases.items():
+            with self.subTest(id=hex(id)):
+                self.assertEqual(FolderAccessLevel.id2label(id), label)
+
+    def test_restrict_slot_without_a_name_is_numbered(self) -> None:
+        self.assertEqual(FolderAccessLevel.id2label(0x0200), "Restrict2 Open")
+        self.assertEqual(FolderAccessLevel.id2label(0x8000), "Restrict8 Open")
+
+    def test_project_is_named_from_the_server(self) -> None:
+        self.assertEqual(FolderAccessLevel.id2label(0x10000, BM2), "Brain/MINDS 2.0 Open")
+        self.assertEqual(FolderAccessLevel.id2label(0x10004, BM2), "Brain/MINDS 2.0 or PW Open")
+
+    def test_project_not_named_gets_its_bit_position(self) -> None:
+        self.assertEqual(FolderAccessLevel.id2label(0x10000), "Project0 Open")
+        self.assertEqual(FolderAccessLevel.id2label(0x20000, BM2), "Project1 Open")
+
+    def test_parts_are_restrict_then_project_then_password(self) -> None:
+        self.assertEqual(
+            FolderAccessLevel.id2label(0x30304, BM2), "CBS or Restrict2 or Brain/MINDS 2.0 or Project1 or PW Open"
+        )
+
+    def test_meaningless_levels_are_unlabelled(self) -> None:
+        # Private or Public mixed with another bit, a reserved bit, a negative value.
+        for id in (0x0003, 0x0006, 0x0102, 0x10001, 0x0008, 0x0080, -1):
+            with self.subTest(id=hex(id)):
+                self.assertIsNone(FolderAccessLevel.id2label(id))
+
+
+class TestKey2Id(unittest.TestCase):
+    def test_keys(self) -> None:
+        cases = {
+            "storage": 0x0000,
+            "private": 0x0001,
+            "public": 0x0002,
+            "pw_open": 0x0004,
+            "cbs_open": 0x0100,
+            "cbs_or_pw_open": 0x0104,
+            "restrict1_open": 0x0100,
+            "restrict2_open": 0x0200,
+            "project0_open": 0x10000,
+            "project1_or_pw_open": 0x20004,
+        }
+        for key, id in cases.items():
+            with self.subTest(key=key):
+                self.assertEqual(FolderAccessLevel.key2id(key), id)
+
+    def test_unknown_keys(self) -> None:
+        for key in ("5kikan_open", "restrict9_open", "restrict0_open", "project01_open", "cbs", "_open", "pw_or_open"):
+            with self.subTest(key=key):
+                self.assertIsNone(FolderAccessLevel.key2id(key))
+
+
+class TestAccessLevelLabel(unittest.TestCase):
+    def test_known_level(self) -> None:
+        self.assertEqual(make_folder(0x0100).access_level_name, "CBS Open")
+
+    def test_project_names_are_used(self) -> None:
+        self.assertEqual(make_folder(0x10000).access_level_label(BM2), "Brain/MINDS 2.0 Open")
+        self.assertEqual(make_folder(0x10000).access_level_name, "Project0 Open")
+
+    def test_meaningless_level_shows_its_value(self) -> None:
+        self.assertEqual(make_folder(0x0008).access_level_name, "Unknown (0x8)")
+
+
+class TestProjectNames(unittest.TestCase):
+    def test_anonymous_user_asks_nothing(self) -> None:
+        connection = MagicMock()
+        connection.token = None
+        with patch("mdrsclient.services.UsersApi") as users_api:
+            self.assertEqual(MdrsService(connection).project_names(), {})
+        users_api.assert_not_called()
+
+    def test_names_by_bit_position(self) -> None:
+        connection = MagicMock()
+        connection.get.return_value.json.return_value = current_user(
+            [
+                {"id": 1, "name": "Brain/MINDS 2.0", "bit_position": 0},
+                {"id": 2, "name": "B", "bit_position": 3},
+                {"id": 3, "name": "A", "bit_position": 3},
+                {"id": 4, "name": None, "bit_position": 5},
+            ]
+        )
+        with patch.object(UsersApi, "_raise_response_error"):
+            names = UsersApi(connection).project_names()
+        self.assertEqual(names, {0: "Brain/MINDS 2.0", 3: "A, B"})
+
+    def test_server_without_projects(self) -> None:
+        connection = MagicMock()
+        user = current_user([])
+        del user["projects"]
+        connection.get.return_value.json.return_value = user
+        with patch.object(UsersApi, "_raise_response_error"):
+            self.assertEqual(UsersApi(connection).project_names(), {})
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_services.py

@@ -0,0 +1,104 @@
+import unittest
+from unittest.mock import MagicMock, patch
+
+from mdrsclient.api import FoldersApi
+from mdrsclient.exceptions import FolderLockedException
+from mdrsclient.models import Doi, Folder, Laboratories, Laboratory
+from mdrsclient.services import MdrsService
+
+TIMESTAMP = "2026-01-01T00:00:00+09:00"
+DOI = "10.60178/cbs.20260429-001"
+LABORATORY = Laboratory(id=1, name="mylab", pi_name="PI", full_name="My Laboratory")
+
+
+def make_locked_folder() -> Folder:
+    return Folder(
+        id="f1",
+        pid=None,
+        name="root",
+        access_level=4,
+        lock=True,
+        size=0,
+        laboratory_id=1,
+        description="",
+        created_at=TIMESTAMP,
+        updated_at=TIMESTAMP,
+        restrict_opened_at=None,
+        metadata=[],
+        sub_folders=[],
+        path="/root/",
+    )
+
+
+# What the server returns for a locked folder not yet opened with its password.
+SHORT_RECORD = {
+    "id": "f1",
+    "pid": None,
+    "name": "root",
+    "access_level": 4,
+    "lock": True,
+    "size": 0,
+    "laboratory_id": 1,
+    "description": "",
+    "created_at": TIMESTAMP,
+    "updated_at": TIMESTAMP,
+    "restrict_opened_at": None,
+    "application": {"id": 0, "status": 0},
+}
+
+
+class TestRetrieveLockedFolder(unittest.TestCase):
+    def retrieve(self, data: dict) -> Folder:
+        connection = MagicMock()
+        connection.get.return_value.json.return_value = data
+        with patch.object(FoldersApi, "_raise_response_error"):
+            return FoldersApi(connection).retrieve("f1")
+
+    def test_short_record_means_locked(self):
+        with self.assertRaisesRegex(FolderLockedException, "Folder `root` is locked."):
+            self.retrieve(SHORT_RECORD)
+
+    def test_full_record_of_a_locked_folder_is_read(self):
+        full = dict(SHORT_RECORD, metadata=[], sub_folders=[], path="/root/")
+        self.assertEqual(self.retrieve(full).path, "/root/")
+
+
+class TestFindFolderByDoi(unittest.TestCase):
+    """The DOI response carries the folder id as a field, not as a nested object."""
+
+    def find(self, retrieved: list, password: str | None) -> tuple[MagicMock, tuple]:
+        laboratories = Laboratories()
+        laboratories.append(LABORATORY)
+
+        with (
+            patch("mdrsclient.services.DoiApi") as doi_api,
+            patch("mdrsclient.services.FoldersApi") as folders_api,
+            patch("mdrsclient.services.LaboratoriesApi") as laboratories_api,
+        ):
+            doi_api.return_value.retrieve.return_value = Doi(id="20260429-001", doi=DOI, folder_id="f1")
+            folders_api.return_value.retrieve.side_effect = retrieved
+            laboratories_api.return_value.list.return_value = laboratories
+            return folders_api.return_value, MdrsService(MagicMock()).find_folder_by_doi(DOI, password)
+
+    def test_a_locked_doi_folder_is_unlocked_with_its_folder_id(self):
+        api, (folder, laboratory) = self.find([FolderLockedException("locked"), make_locked_folder()], "secret")
+        api.auth.assert_called_once_with("f1", "secret")
+        self.assertEqual(api.retrieve.call_count, 2)
+        self.assertEqual(folder.id, "f1")
+        self.assertEqual(laboratory.name, "mylab")
+
+    def test_a_locked_doi_folder_without_a_password(self):
+        with self.assertRaisesRegex(FolderLockedException, f"Folder for DOI `{DOI}` is locked."):
+            self.find([FolderLockedException("locked")], None)
+
+    def test_a_full_record_still_asks_for_the_password(self):
+        # A locked folder whose record is readable anyway, e.g. its owner's.
+        api, (folder, _) = self.find([make_locked_folder()], "secret")
+        api.auth.assert_called_once_with("f1", "secret")
+        self.assertEqual(api.retrieve.call_count, 1)
+        with self.assertRaises(FolderLockedException):
+            self.find([make_locked_folder()], None)
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_token_cache.py

@@ -0,0 +1,198 @@
+import tempfile
+import time
+import unittest
+from unittest.mock import patch
+
+import jwt
+
+from mdrsclient.api.utils import retry_on_expired_token, token_check, token_recover
+from mdrsclient.cache import CacheFile
+from mdrsclient.config import normalize_url
+from mdrsclient.connection import MDRSConnection
+from mdrsclient.exceptions import IllegalArgumentException, ServerBusyException, TokenExpiredException
+from mdrsclient.models import Token
+
+REMOTE = "unittest"
+
+
+def make_token(access_offset: int, refresh_offset: int, label: str) -> Token:
+    now = int(time.time())
+
+    def encode(token_type: str, offset: int) -> str:
+        return jwt.encode(
+            {
+                "token_type": token_type,
+                "exp": now + offset,
+                "iat": now,
+                "jti": f"{label}-{token_type}",
+                "user_id": 1,
+            },
+            "unittest-signing-key-not-verified-anywhere",
+        )
+
+    return Token(access=encode("access", access_offset), refresh=encode("refresh", refresh_offset))
+
+
+class TestTokenCache(unittest.TestCase):
+    def setUp(self) -> None:
+        self.tmp = tempfile.TemporaryDirectory()
+        patcher = patch("mdrsclient.cache.CONFIG_DIRNAME", self.tmp.name)
+        patcher.start()
+        self.addCleanup(patcher.stop)
+        self.addCleanup(self.tmp.cleanup)
+
+    def test_reload_picks_up_a_write_from_another_holder(self) -> None:
+        """The cache is shared, so a value read earlier can already be out of date."""
+        reader = CacheFile(REMOTE)
+        writer = CacheFile(REMOTE)
+
+        first = make_token(3600, 86400, "first")
+        writer.token = first
+        self.assertEqual(reader.token, first)
+
+        second = make_token(3600, 86400, "second")
+        writer.token = second
+
+        reader.reload()
+        self.assertEqual(reader.token, second)
+
+    def test_token_check_uses_the_token_another_holder_just_wrote(self) -> None:
+        """A rotating provider drops the token it replaces, so the refresh must not
+        be sent again once someone else has already made the round trip."""
+        connection = MDRSConnection(REMOTE, "http://localhost:8000/api/")
+        connection.token = make_token(-60, 86400, "stale")
+
+        # Another process refreshes while this one is between requests.
+        other = CacheFile(REMOTE)
+        rotated = make_token(3600, 86400, "rotated")
+        other.token = rotated
+
+        with patch("mdrsclient.api.utils.UsersApi") as users_api:
+            token_check(connection)
+
+        users_api.assert_not_called()
+        self.assertEqual(connection.token, rotated)
+
+    def test_token_check_refreshes_when_nothing_else_has(self) -> None:
+        connection = MDRSConnection(REMOTE, "http://localhost:8000/api/")
+        connection.token = make_token(-60, 86400, "stale")
+        rotated = make_token(3600, 86400, "rotated")
+
+        with patch("mdrsclient.api.utils.UsersApi") as users_api:
+            users_api.return_value.tokenRefresh.return_value = rotated
+            token_check(connection)
+
+        users_api.return_value.tokenRefresh.assert_called_once()
+        self.assertEqual(connection.token, rotated)
+
+    def test_token_check_logs_out_once_the_refresh_token_expires(self) -> None:
+        connection = MDRSConnection(REMOTE, "http://localhost:8000/api/")
+        connection.token = make_token(-3600, -60, "dead")
+
+        with patch("mdrsclient.api.utils.UsersApi") as users_api:
+            token_check(connection)
+
+        users_api.assert_not_called()
+        self.assertIsNone(connection.token)
+
+
+class TestUrlNormalization(unittest.TestCase):
+    """Both clients share config.ini, so they have to agree on what a remote URL is."""
+
+    def test_trailing_slash_is_dropped(self) -> None:
+        self.assertEqual(normalize_url("http://127.0.0.1:8000/api/"), "http://127.0.0.1:8000/api")
+        self.assertEqual(normalize_url("https://neurodata.riken.jp/api/"), "https://neurodata.riken.jp/api")
+
+    def test_bare_hostname_is_accepted(self) -> None:
+        self.assertEqual(normalize_url("http://localhost:8000/api"), "http://localhost:8000/api")
+
+    def test_only_http_schemes_are_accepted(self) -> None:
+        for url in ("ftp://x.example.com/", "file:///etc/passwd", "not-a-url", "http://"):
+            with self.subTest(url=url), self.assertRaises(IllegalArgumentException):
+                normalize_url(url)
+
+
+class TestExpiredTokenRetry(unittest.TestCase):
+    """A request can wait in the server's queue for longer than its access token lives."""
+
+    def setUp(self) -> None:
+        self.tmp = tempfile.TemporaryDirectory()
+        patcher = patch("mdrsclient.cache.CONFIG_DIRNAME", self.tmp.name)
+        patcher.start()
+        self.addCleanup(patcher.stop)
+        self.addCleanup(self.tmp.cleanup)
+        self.connection = MDRSConnection(REMOTE, "http://localhost:8000/api")
+        self.connection.token = make_token(3600, 86400, "sent")
+
+    def test_a_token_someone_else_refreshed_is_reused_rather_than_replaced(self) -> None:
+        used = self.connection.token
+        rotated = make_token(3600, 86400, "rotated")
+        CacheFile(REMOTE).token = rotated  # another process, while the request waited
+
+        with patch("mdrsclient.api.utils.UsersApi") as users_api:
+            recovered = token_recover(self.connection, used)
+
+        self.assertTrue(recovered)
+        users_api.assert_not_called()
+        self.assertEqual(self.connection.token, rotated)
+        # The retry carries whatever the session header says, not what the cache holds,
+        # so reading the new token through is not enough on its own.
+        self.assertEqual(self.connection.session.headers["Authorization"], f"Bearer {rotated.access}")
+
+    def test_the_token_is_refreshed_when_nobody_else_has(self) -> None:
+        used = self.connection.token
+        rotated = make_token(3600, 86400, "rotated")
+
+        with patch("mdrsclient.api.utils.UsersApi") as users_api:
+            users_api.return_value.tokenRefresh.return_value = rotated
+            recovered = token_recover(self.connection, used)
+
+        self.assertTrue(recovered)
+        users_api.return_value.tokenRefresh.assert_called_once()
+        self.assertEqual(self.connection.token, rotated)
+
+    def test_the_request_is_sent_once_more_and_only_once(self) -> None:
+        calls: list[str] = []
+
+        class Api:
+            def __init__(self, connection: MDRSConnection) -> None:
+                self.connection = connection
+
+            @retry_on_expired_token
+            def send(self) -> str:
+                calls.append("sent")
+                if len(calls) == 1:
+                    raise TokenExpiredException("expired on arrival")
+                return "ok"
+
+        with patch("mdrsclient.api.utils.UsersApi") as users_api:
+            users_api.return_value.tokenRefresh.return_value = make_token(3600, 86400, "rotated")
+            self.assertEqual(Api(self.connection).send(), "ok")
+
+        self.assertEqual(len(calls), 2)
+
+    def test_a_second_refusal_is_reported_as_an_overloaded_server(self) -> None:
+        calls: list[str] = []
+
+        class Api:
+            def __init__(self, connection: MDRSConnection) -> None:
+                self.connection = connection
+
+            @retry_on_expired_token
+            def send(self) -> str:
+                calls.append("sent")
+                raise TokenExpiredException("expired on arrival")
+
+        with patch("mdrsclient.api.utils.UsersApi") as users_api:
+            users_api.return_value.tokenRefresh.return_value = make_token(3600, 86400, "rotated")
+            with self.assertRaises(ServerBusyException) as caught:
+                Api(self.connection).send()
+
+        # Sent twice and no more, and the message names the cause the user can act on
+        # rather than an authentication failure that would send them to the login form.
+        self.assertEqual(len(calls), 2)
+        self.assertIn("overloaded", str(caught.exception))
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_transfer.py

@@ -0,0 +1,297 @@
+import os
+import tempfile
+import unittest
+from io import StringIO
+from unittest.mock import MagicMock, patch
+
+from mdrsclient.api import FilesApi
+from mdrsclient.exceptions import MDRSException, UnexpectedException
+from mdrsclient.models import File, Folder, FolderSimple, Laboratory
+from mdrsclient.transfer import Downloader, Uploader
+
+TIMESTAMP = "2026-01-01T00:00:00+09:00"
+
+
+def make_file(id: str, name: str, size: int = 1) -> File:
+    return File(
+        id=id,
+        name=name,
+        type="text/plain",
+        size=size,
+        thumbnail=None,
+        description="",
+        metadata={},
+        download_url=f"v3/files/{id}/download/",
+        created_at=TIMESTAMP,
+        updated_at=TIMESTAMP,
+    )
+
+
+def make_folder_simple(id: str, name: str) -> FolderSimple:
+    return FolderSimple(
+        id=id,
+        pid=None,
+        name=name,
+        access_level=1,
+        lock=False,
+        size=0,
+        laboratory_id=1,
+        description="",
+        created_at=TIMESTAMP,
+        updated_at=TIMESTAMP,
+        restrict_opened_at=None,
+    )
+
+
+def make_folder(id: str, name: str, path: str, sub_folders: list[FolderSimple] | None = None) -> Folder:
+    return Folder(
+        id=id,
+        pid=None,
+        name=name,
+        access_level=1,
+        lock=False,
+        size=0,
+        laboratory_id=1,
+        description="",
+        created_at=TIMESTAMP,
+        updated_at=TIMESTAMP,
+        restrict_opened_at=None,
+        metadata=[],
+        sub_folders=sub_folders if sub_folders is not None else [],
+        path=path,
+    )
+
+
+LABORATORY = Laboratory(id=1, name="mylab", pi_name="PI", full_name="My Laboratory")
+
+
+class TestUploadReportsFailure(unittest.TestCase):
+    """A file that never reached the server must not leave the command reporting success."""
+
+    def make_client(self, folder: Folder, files: list[File]) -> MagicMock:
+        client = MagicMock()
+        client.parse_remote_host_with_path.return_value = ("myremote", "mylab", "/")
+        client.find_laboratory.return_value = LABORATORY
+        client.find_folder.return_value = folder
+        client.find_files.return_value = files
+        return client
+
+    def test_a_refused_upload_is_raised_to_the_caller(self):
+        client = self.make_client(make_folder("f1", "root", "/"), [])
+        with tempfile.TemporaryDirectory() as tmp:
+            local = os.path.join(tmp, "data.txt")
+            with open(local, "w") as f:
+                f.write("x")
+            with patch("mdrsclient.transfer.FilesApi") as files_api_class:
+                files_api_class.return_value.create.side_effect = MDRSException("Access is denied.")
+                with self.assertRaises(UnexpectedException):
+                    Uploader(client).upload(local, "myremote:/mylab/", False, False)
+
+    def test_an_error_the_api_layer_did_not_raise_is_still_counted(self):
+        """A file that vanished between the walk and the upload must not end the run."""
+        client = self.make_client(make_folder("f1", "root", "/"), [])
+        with tempfile.TemporaryDirectory() as tmp:
+            local = os.path.join(tmp, "data.txt")
+            with open(local, "w") as f:
+                f.write("x")
+            with patch("mdrsclient.transfer.FilesApi") as files_api_class:
+                files_api_class.return_value.create.side_effect = FileNotFoundError(local)
+                with self.assertRaises(UnexpectedException):
+                    Uploader(client).upload(local, "myremote:/mylab/", False, False)
+
+    def test_a_failed_upload_names_the_file(self):
+        client = self.make_client(make_folder("f1", "root", "/"), [])
+        with tempfile.TemporaryDirectory() as tmp:
+            local = os.path.join(tmp, "data.txt")
+            with open(local, "w") as f:
+                f.write("x")
+            with patch("mdrsclient.transfer.FilesApi") as files_api_class:
+                files_api_class.return_value.create.side_effect = MDRSException("Access is denied.")
+                with patch("sys.stdout", new=StringIO()) as fake_out:
+                    with self.assertRaises(UnexpectedException):
+                        Uploader(client).upload(local, "myremote:/mylab/", False, False)
+                    self.assertIn(local, fake_out.getvalue())
+
+    def test_an_upload_that_worked_stays_quiet(self):
+        client = self.make_client(make_folder("f1", "root", "/"), [])
+        with tempfile.TemporaryDirectory() as tmp:
+            local = os.path.join(tmp, "data.txt")
+            with open(local, "w") as f:
+                f.write("x")
+            with patch("mdrsclient.transfer.FilesApi") as files_api_class:
+                files_api_class.return_value.create.return_value = "new-id"
+                Uploader(client).upload(local, "myremote:/mylab/", False, False)
+                files_api_class.return_value.create.assert_called_once()
+
+
+class TestDownloadReportsFailure(unittest.TestCase):
+    def make_client(self, remote_path_parts, folder: Folder, files_by_folder: dict[str, list[File]]) -> MagicMock:
+        client = MagicMock()
+        client.is_doi.return_value = False
+        client.parse_remote_host_with_path.return_value = remote_path_parts
+        client.find_laboratory.return_value = LABORATORY
+        client.find_folder.return_value = folder
+        client.find_files.side_effect = lambda folder_id: files_by_folder.get(folder_id, [])
+        return client
+
+    def test_a_single_file_that_failed_is_raised_to_the_caller(self):
+        parent = make_folder("p1", "root", "/")
+        client = self.make_client(("myremote", "mylab", "/data.txt"), parent, {"p1": [make_file("x1", "data.txt")]})
+        with tempfile.TemporaryDirectory() as tmp:
+            with patch("mdrsclient.transfer.FilesApi") as files_api_class:
+                files_api_class.return_value.download.side_effect = OSError("Permission denied.")
+                with self.assertRaises(UnexpectedException):
+                    Downloader(client).download("myremote:/mylab/data.txt", tmp)
+
+    def test_a_single_file_that_arrived_stays_quiet(self):
+        parent = make_folder("p1", "root", "/")
+        client = self.make_client(("myremote", "mylab", "/data.txt"), parent, {"p1": [make_file("x1", "data.txt")]})
+        with tempfile.TemporaryDirectory() as tmp:
+            with patch("mdrsclient.transfer.FilesApi") as files_api_class:
+                files_api_class.return_value.download.return_value = True
+                Downloader(client).download("myremote:/mylab/data.txt", tmp)
+                files_api_class.return_value.download.assert_called_once()
+
+    def test_a_failed_file_does_not_abandon_the_remaining_sub_folders(self):
+        parent = make_folder("p1", "lab", "/", [make_folder_simple("f1", "root")])
+        folders = {
+            "f1": make_folder("f1", "root", "/root/", [make_folder_simple("fa", "a"), make_folder_simple("fb", "b")]),
+            "fa": make_folder("fa", "a", "/root/a/"),
+            "fb": make_folder("fb", "b", "/root/b/"),
+        }
+        files_by_folder = {
+            "p1": [],
+            "f1": [make_file("bad", "bad.txt")],
+            "fa": [make_file("good1", "good1.txt")],
+            "fb": [make_file("good2", "good2.txt")],
+        }
+        client = self.make_client(("myremote", "mylab", "/root"), parent, files_by_folder)
+        attempted: list[str] = []
+
+        def download(file: File, path: str) -> bool:
+            attempted.append(file.name)
+            if file.name == "bad.txt":
+                raise OSError("Permission denied.")
+            return True
+
+        with tempfile.TemporaryDirectory() as tmp:
+            with (
+                patch("mdrsclient.transfer.FoldersApi") as folders_api_class,
+                patch("mdrsclient.transfer.FilesApi") as files_api_class,
+            ):
+                folders_api_class.return_value.retrieve.side_effect = lambda folder_id: folders[folder_id]
+                files_api_class.return_value.download.side_effect = download
+                with self.assertRaises(UnexpectedException):
+                    Downloader(client).download("myremote:/mylab/root", tmp, is_recursive=True)
+
+        self.assertEqual(sorted(attempted), ["bad.txt", "good1.txt", "good2.txt"])
+
+
+class TestRecursiveDownloadResilience(unittest.TestCase):
+    """A folder the client cannot prepare locally must not abandon its siblings."""
+
+    def test_a_folder_that_cannot_be_created_does_not_stop_the_walk(self):
+        parent = make_folder("p1", "lab", "/", [make_folder_simple("f1", "root")])
+        folders = {
+            "f1": make_folder("f1", "root", "/root/", [make_folder_simple("fa", "a"), make_folder_simple("fb", "b")]),
+            "fa": make_folder("fa", "a", "/root/a/"),
+            "fb": make_folder("fb", "b", "/root/b/"),
+        }
+        files_by_folder = {"p1": [], "f1": [], "fa": [make_file("good1", "good1.txt")], "fb": []}
+        client = MagicMock()
+        client.is_doi.return_value = False
+        client.parse_remote_host_with_path.return_value = ("myremote", "mylab", "/root")
+        client.find_laboratory.return_value = LABORATORY
+        client.find_folder.return_value = parent
+        client.find_files.side_effect = lambda folder_id: files_by_folder.get(folder_id, [])
+
+        real_makedirs = os.makedirs
+
+        def makedirs(path, *args, **kwargs):
+            if os.path.basename(path) == "b":
+                raise PermissionError("Permission denied")
+            return real_makedirs(path, *args, **kwargs)
+
+        attempted: list[str] = []
+
+        with tempfile.TemporaryDirectory() as tmp:
+            with (
+                patch("mdrsclient.transfer.FoldersApi") as folders_api_class,
+                patch("mdrsclient.transfer.FilesApi") as files_api_class,
+                patch("mdrsclient.transfer.os.makedirs", side_effect=makedirs),
+            ):
+                folders_api_class.return_value.retrieve.side_effect = lambda folder_id: folders[folder_id]
+                files_api_class.return_value.download.side_effect = lambda file, path: attempted.append(file.name)
+                with self.assertRaises(UnexpectedException):
+                    Downloader(client).download("myremote:/mylab/root", tmp, is_recursive=True)
+
+        self.assertEqual(attempted, ["good1.txt"])
+
+
+class TestFileDownloadPermission(unittest.TestCase):
+    """A file the client could not write is a failure, not a line of successful output."""
+
+    def make_connection(self, chunks) -> MagicMock:
+        connection = MagicMock()
+        connection.token = None
+        response = MagicMock()
+        response.status_code = 200
+        response.iter_content.return_value = chunks
+        connection.get.return_value = response
+        return connection
+
+    def test_a_download_that_cannot_be_written_is_reported(self):
+        connection = self.make_connection([b"payload"])
+        with tempfile.TemporaryDirectory() as tmp:
+            path = os.path.join(tmp, "precious.dat")
+            with open(path, "wb") as f:
+                f.write(b"do not touch")
+            os.chmod(path, 0o444)
+            try:
+                with open(path, "r+b"):
+                    self.skipTest("running with rights that ignore the file mode")
+            except OSError:
+                pass
+
+            with self.assertRaises(UnexpectedException):
+                FilesApi(connection).download(make_file("x1", "precious.dat"), path)
+
+            os.chmod(path, 0o644)
+            with open(path, "rb") as f:
+                self.assertEqual(f.read(), b"do not touch")
+            connection.get.assert_not_called()
+
+    def test_an_interrupted_download_leaves_the_existing_file_intact(self):
+        def chunks():
+            yield b"half a file"
+            raise ConnectionError("connection reset")
+
+        connection = self.make_connection(chunks())
+        with tempfile.TemporaryDirectory() as tmp:
+            path = os.path.join(tmp, "existing.dat")
+            with open(path, "wb") as f:
+                f.write(b"the copy already here")
+
+            with self.assertRaises(ConnectionError):
+                FilesApi(connection).download(make_file("x1", "existing.dat"), path)
+
+            with open(path, "rb") as f:
+                self.assertEqual(f.read(), b"the copy already here")
+            self.assertEqual(os.listdir(tmp), ["existing.dat"])
+
+    def test_a_finished_download_replaces_the_destination(self):
+        connection = self.make_connection([b"new ", b"contents"])
+        with tempfile.TemporaryDirectory() as tmp:
+            path = os.path.join(tmp, "existing.dat")
+            with open(path, "wb") as f:
+                f.write(b"old")
+
+            FilesApi(connection).download(make_file("x1", "existing.dat"), path)
+
+            with open(path, "rb") as f:
+                self.assertEqual(f.read(), b"new contents")
+            self.assertEqual(os.listdir(tmp), ["existing.dat"])
+
+
+if __name__ == "__main__":
+    unittest.main()