docs: release 1.3.19 and document the environment variables

- Close the Unreleased section as 1.3.19, summarising it by what
  changed for the user rather than by what each commit touched.
- Add the environment variables the settings module reads, which the
  README did not mention.

CHANGELOG.md

@@ -2,6 +2,20 @@
 
 All notable changes to this project will be documented in this file.
 
+## [1.3.19] - 2026-09-04
+
+### Fixed
+- Reported failed transfers instead of ending in success, so a run that lost files exits non-zero.
+- Left an existing local file intact when a download fails part way through.
+- Opened password-protected DOI folders, which previously failed outright.
+- Kept a session working across concurrent runs and long server queues.
+- Other minor fixes to `ls` output and remote URL handling.
+
+### Changed
+- Moved packaging to PEP 621 metadata with flit as the build backend.
+- Updated dependencies.
+- **Breaking for embedders:** `CacheInterface` now requires `lock()` and `reload()`.
+
 ## [1.3.18] - 2026-07-02
 
 ### Added
@@ -37,36 +51,41 @@ All notable changes to this project will be documented in this file.
 
 ## [1.3.15] - 2026-05-01
 
-### Changed
+### Fixed
-- Bumped package version to 1.3.15.
+- Apply NFC normalization to filenames and folder names sent to the server.
 
 ## [1.3.14] - 2026-04-17
 
 ### Changed
 - Simplified `config list` command (removed `-l`/`--long` option, always display URL).
+- Renamed `--quick` option to `--quiet` for `ls` subcommand.
 
 ### Added
 - Added subcommand aliases for config commands (e.g. `ls` alias for list, `rm` alias for delete).
+- Added `version` command.
 
 ## [1.3.13] - 2025-07-02
 
-### Changed
+### Fixed
-- Bumped package version to 1.3.13.
+- Fixed pagination logic for the `file.list` API.
 
 ## [1.3.12] - 2025-05-20
 
-### Changed
+### Fixed
-- Bumped package version to 1.3.12.
+- Fixed bug where file downloading was skipped incorrectly when `-s`/`--skip-if-file-exists` option was present.
 
 ## [1.3.11] - 2025-01-21
 
-### Changed
+### Fixed
-- Bumped package version to 1.3.11.
+- Follow-up fixes for User API specification changes.
 
 ## [1.3.10] - 2024-12-23
 
+### Added
+- Delete broken files and show a summary when a file download fails.
+
 ### Changed
-- Bumped package version to 1.3.10.
+- Updated dependency libraries.
 
 ## [1.3.9] - 2024-10-23
 
@@ -75,8 +94,11 @@ All notable changes to this project will be documented in this file.
 
 ## [1.3.8] - 2024-09-18
 
-### Changed
+### Added
-- Bumped package version to 1.3.8.
+- Implemented `-s`/`--skip-if-file-exists` option for `download` command.
+
+### Fixed
+- Added exception handling for unexpected responses from the server.
 
 ## [1.3.7] - 2024-07-22
 
@@ -85,23 +107,29 @@ All notable changes to this project will be documented in this file.
 
 ## [1.3.6] - 2024-07-08
 
-### Changed
+### Added
-- Bumped package version to 1.3.6.
+- Support cancelling recursive downloads if downloading some files fails.
 
 ## [1.3.5] - 2024-07-08
 
-### Changed
+### Added
-- Bumped package version to 1.3.5.
+- Added authorization token validation checks for file download operations.
+
+### Removed
+- Removed unnecessary debug code.
 
 ## [1.3.4] - 2024-07-04
 
 ### Added
 - Added some aliases for config sub command.
 
+### Fixed
+- Fixed bug when uploading large files.
+
 ## [1.3.3] - 2024-02-13
 
-### Changed
+### Added
-- Bumped package version to 1.3.3.
+- Implemented `-s`/`--skip-if-file-exists` option for `upload` command.
 
 ## [1.3.2] - 2024-02-09
 

README.md

@@ -2,12 +2,53 @@
 
 The mdrs-client-python is python library and a command-line client for up- and downloading files to and from MDRS based repository.
 
-## Installing
+## Installation (Production)
+
+It is recommended to use a virtual environment (`venv`) to keep your Python environment isolated.
+
+### Option 1: Using standard `venv` + `pip`
 
 ```shell
-poetry install
+python3 -m venv .venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+pip install .
 ```
 
+### Option 2: Using `uv`
+
+```shell
+uv venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+uv pip install .
+```
+
+## Development Setup
+
+To set up a local development environment with development tools (testing, formatting, linting):
+
+### Option 1: Using standard `venv` + `pip`
+
+```shell
+python3 -m venv .venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+pip install -e ".[dev]"
+```
+
+### Option 2: Using `uv` (Recommended for developers)
+
+```shell
+uv sync
+```
+
+## Configuration
+
+The following environment variables can be set, either in the shell or via a `.env` file placed in the working directory.
+
+| Variable | Default | Description |
+|---|---|---|
+| `MDRS_CLIENT_CONFIG_DIRNAME` | `~/.mdrs-client` | Directory where config and login cache files are stored |
+| `MDRS_CLIENT_CONCURRENT` | `10` | Number of concurrent file transfers for upload and download |
+
 ## CLI Usage
 
 ### config create

mdrsclient/api/base.py

@@ -5,7 +5,13 @@ from pydantic import TypeAdapter
 from requests import Response
 
 from mdrsclient.connection import MDRSConnection
-from mdrsclient.exceptions import BadRequestException, ForbiddenException, UnauthorizedException, UnexpectedException
+from mdrsclient.exceptions import (
+    BadRequestException,
+    ForbiddenException,
+    TokenExpiredException,
+    UnauthorizedException,
+    UnexpectedException,
+)
 from mdrsclient.models.error import DRFStandardizedErrors
 
 
@@ -23,6 +29,11 @@ class BaseApi(ABC):
             if response.status_code == requests.codes.bad_request:
                 raise BadRequestException(errors.errors[0].detail)
             elif response.status_code == requests.codes.unauthorized:
+                # A request can wait in the server's queue for longer than the access
+                # token it was sent with lives, so an expired token here does not mean
+                # the session is over - it means this one request arrived too late.
+                if any(e.code == "token_not_valid" for e in errors.errors):
+                    raise TokenExpiredException("Access token expired before the request was served.")
                 raise UnauthorizedException("Login required.")
             elif response.status_code == requests.codes.forbidden:
                 raise ForbiddenException("You do not have enough permissions. Access is denied.")

mdrsclient/api/doi.py

@@ -4,7 +4,7 @@ from pydantic import TypeAdapter
 from pydantic.dataclasses import dataclass
 
 from mdrsclient.api.base import BaseApi
-from mdrsclient.api.utils import token_check
+from mdrsclient.api.utils import retry_on_expired_token, token_check
 from mdrsclient.models.doi import Doi
 
 
@@ -23,6 +23,7 @@ class DoiRetrieveResponse:
 class DoiApi(BaseApi):
     ENTRYPOINT: Final[str] = "v3/doi/"
 
+    @retry_on_expired_token
     def retrieve(self, doi_id: str) -> Doi:
         """Retrieve the folder associated with a DOI suffix ID (GET v3/doi/{id}/)."""
         url = self.ENTRYPOINT + doi_id + "/"

mdrsclient/api/files.py

@@ -1,5 +1,6 @@
 import mimetypes
 import os
+import threading
 from typing import Any, Final
 from unicodedata import normalize
 
@@ -8,8 +9,8 @@ from pydantic.dataclasses import dataclass
 from requests_toolbelt.multipart.encoder import MultipartEncoder
 
 from mdrsclient.api.base import BaseApi
-from mdrsclient.api.utils import token_check
+from mdrsclient.api.utils import retry_on_expired_token, token_check
-from mdrsclient.exceptions import UnexpectedException
+from mdrsclient.exceptions import MDRSException, UnexpectedException
 from mdrsclient.models import File
 
 
@@ -30,6 +31,7 @@ class FilesApi(BaseApi):
     ENTRYPOINT: Final[str] = "v3/files/"
     FALLBACK_MIMETYPE: Final[str] = "application/octet-stream"
 
+    @retry_on_expired_token
     def list(self, folder_id: str, page_num: int) -> FilesApiListResponse:
         url = self.ENTRYPOINT
         token_check(self.connection)
@@ -38,6 +40,7 @@ class FilesApi(BaseApi):
         self._raise_response_error(response)
         return TypeAdapter(FilesApiListResponse).validate_python(response.json())
 
+    @retry_on_expired_token
     def retrieve(self, id: str) -> File:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + id + "/"
@@ -46,6 +49,7 @@ class FilesApi(BaseApi):
         self._raise_response_error(response)
         return TypeAdapter(File).validate_python(response.json())
 
+    @retry_on_expired_token
     def create(self, folder_id: str, path: str) -> str:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT
@@ -66,10 +70,14 @@ class FilesApi(BaseApi):
             raise UnexpectedException(f"Could not open `{path}` file.")
         except MemoryError:
             raise UnexpectedException("Out of memory.")
+        except MDRSException:
+            # Already says what went wrong, and the caller may want to act on the kind.
+            raise
         except Exception as e:
             raise UnexpectedException("Unspecified error.") from e
         return ret.id
 
+    @retry_on_expired_token
     def update(self, file: File, path: str | None) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + file.id + "/"
@@ -96,6 +104,7 @@ class FilesApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def destroy(self, file: File) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + file.id + "/"
@@ -104,6 +113,7 @@ class FilesApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def move(self, file: File, folder_id: str, name: str) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + file.id + "/move/"
@@ -113,6 +123,7 @@ class FilesApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def copy(self, file: File, folder_id: str, name: str) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + file.id + "/copy/"
@@ -122,6 +133,7 @@ class FilesApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def metadata(self, file: File) -> dict[str, Any]:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + file.id + "/metadata/"
@@ -130,22 +142,58 @@ class FilesApi(BaseApi):
         self._raise_response_error(response)
         return response.json()
 
+    @retry_on_expired_token
     def download(self, file: File, path: str) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = file.download_url
         token_check(self.connection)
+        # Refused before anything is fetched. The finished file is moved into place, and a
+        # rename would replace a destination whose mode says it is protected.
+        if os.path.exists(path):
+            try:
+                with open(path, "r+b"):
+                    pass
+            except OSError as e:
+                raise UnexpectedException(f"Cannot write `{path}`: {e}")
         response = self.connection.get(url, stream=True)
         self._raise_response_error(response)
+        # Written beside the destination and moved in once the whole body has arrived, so
+        # a transfer that fails part way leaves whatever was already there untouched and
+        # never leaves a truncated file under the real name.
+        fd, tmp_path = self._open_partial(path)
         try:
-            with open(path, "wb") as f:
+            with os.fdopen(fd, "wb") as f:
                 for chunk in response.iter_content(chunk_size=4096):
                     if chunk:
                         f.write(chunk)
-                        f.flush()
+            os.replace(tmp_path, path)
-        except PermissionError:
+        except BaseException:
-            print(f"Cannot create file `{path}`: Permission denied.")
+            # Only the scratch file goes: anything at the destination was not written here.
+            if os.path.exists(tmp_path):
+                os.unlink(tmp_path)
+            raise
         return True
 
+    @staticmethod
+    def _open_partial(path: str) -> tuple[int, str]:
+        """
+        Create a scratch file beside `path` and return it open for writing.
+
+        Beside it, so moving the finished download into place is a rename within one
+        directory. `0o666` rather than a private mode because the umask is what decided
+        the permissions of a downloaded file before, and still should.
+        """
+        base = f"{path}.{os.getpid()}-{threading.get_ident()}"
+        for attempt in range(100):
+            tmp_path = f"{base}-{attempt}.mdrspart"
+            try:
+                return os.open(tmp_path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o666), tmp_path
+            except FileExistsError:
+                continue
+            except OSError as e:
+                raise UnexpectedException(f"Cannot write `{path}`: {e}")
+        raise UnexpectedException(f"Could not create a temporary file beside `{path}`.")
+
     def _get_mime_type(self, path: str) -> str:
         mt = mimetypes.guess_type(path)
         if mt:

mdrsclient/api/folders.py

@@ -5,7 +5,7 @@ from pydantic import TypeAdapter
 from pydantic.dataclasses import dataclass
 
 from mdrsclient.api.base import BaseApi
-from mdrsclient.api.utils import token_check
+from mdrsclient.api.utils import retry_on_expired_token, token_check
 from mdrsclient.exceptions import UnauthorizedException
 from mdrsclient.models import Folder, FolderSimple
 
@@ -18,6 +18,7 @@ class FoldersApiCreateResponse:
 class FoldersApi(BaseApi):
     ENTRYPOINT: Final[str] = "v3/folders/"
 
+    @retry_on_expired_token
     def list(self, laboratory_id: int, path: str) -> list[FolderSimple]:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT
@@ -30,6 +31,7 @@ class FoldersApi(BaseApi):
             ret.append(TypeAdapter(FolderSimple).validate_python(data))
         return ret
 
+    @retry_on_expired_token
     def retrieve(self, id: str) -> Folder:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + id + "/"
@@ -39,6 +41,7 @@ class FoldersApi(BaseApi):
         ret = TypeAdapter(Folder).validate_python(response.json())
         return ret
 
+    @retry_on_expired_token
     def create(self, name: str, parent_id: str) -> str:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT
@@ -49,6 +52,7 @@ class FoldersApi(BaseApi):
         ret = TypeAdapter(FoldersApiCreateResponse).validate_python(response.json())
         return ret.id
 
+    @retry_on_expired_token
     def update(self, folder: FolderSimple) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + folder.id + "/"
@@ -61,6 +65,7 @@ class FoldersApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def destroy(self, id: str, recursive: bool) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + id + "/"
@@ -70,6 +75,7 @@ class FoldersApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def auth(self, id: str, password: str) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + id + "/auth/"
@@ -81,6 +87,7 @@ class FoldersApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def acl(self, id: str, access_level: int, recursive: bool, password: str | None) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + id + "/acl/"
@@ -94,6 +101,7 @@ class FoldersApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def move(self, folder: FolderSimple, folder_id: str, name: str) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + folder.id + "/move/"
@@ -103,6 +111,7 @@ class FoldersApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def copy(self, folder: FolderSimple, folder_id: str, name: str) -> bool:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + folder.id + "/copy/"
@@ -112,6 +121,7 @@ class FoldersApi(BaseApi):
         self._raise_response_error(response)
         return True
 
+    @retry_on_expired_token
     def metadata(self, id: str) -> dict[str, Any]:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + id + "/metadata/"

mdrsclient/api/laboratories.py

@@ -3,13 +3,14 @@ from typing import Final
 from pydantic import TypeAdapter
 
 from mdrsclient.api.base import BaseApi
-from mdrsclient.api.utils import token_check
+from mdrsclient.api.utils import retry_on_expired_token, token_check
 from mdrsclient.models import Laboratories, Laboratory
 
 
 class LaboratoriesApi(BaseApi):
     ENTRYPOINT: Final[str] = "v3/laboratories/"
 
+    @retry_on_expired_token
     def list(self) -> Laboratories:
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT

mdrsclient/api/users.py

@@ -8,6 +8,10 @@ from mdrsclient.api.base import BaseApi
 from mdrsclient.exceptions import UnauthorizedException
 from mdrsclient.models import Token, User
 
+# (connect, read) seconds for the token refresh. Uploads are served by a separate
+# instance, so this one is not queued behind them and has no reason to be slow.
+TOKEN_REFRESH_TIMEOUT: Final[tuple[float, float]] = (5.0, 30.0)
+
 
 @dataclass(frozen=True)
 class UsersCurrentResponseLaboratory:
@@ -61,7 +65,10 @@ class UsersApi(BaseApi):
         # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
         url = self.ENTRYPOINT + "token/refresh/"
         data: dict[str, str | int] = {"refresh": token.refresh}
-        response = self.connection.post(url, data=data)
+        # Bounded on its own: the caller holds a lock that spans processes while this runs,
+        # so a provider that accepts the connection and then goes quiet would otherwise
+        # stall every other request on this machine rather than just this one.
+        response = self.connection.post(url, data=data, timeout=TOKEN_REFRESH_TIMEOUT)
         if response.status_code == requests.codes.unauthorized:
             raise UnauthorizedException("Token is invalid or expired.")
         self._raise_response_error(response)

mdrsclient/api/utils.py

@@ -1,19 +1,124 @@
+import functools
+from typing import Any, Callable, TypeVar, cast
+
 from mdrsclient.api.users import UsersApi
 from mdrsclient.connection import MDRSConnection
-from mdrsclient.exceptions import UnauthorizedException
+from mdrsclient.exceptions import (
+    MDRSException,
+    ServerBusyException,
+    TokenExpiredException,
+    UnauthorizedException,
+)
+from mdrsclient.models import Token
+
+F = TypeVar("F", bound=Callable[..., Any])
+
+SERVER_BUSY_MESSAGE = (
+    "The server took too long to start handling the request and may be overloaded. "
+    "Try again, or reduce the number of parallel transfers."
+)
 
 
 def token_check(connection: MDRSConnection) -> None:
-    try:
+    """
-        connection.lock.acquire()
+    Bring the access token up to date before a request goes out.
-        if connection.token is not None:
+
-            if connection.token.is_refresh_required:
+    Refreshing is a read-modify-write over a cache shared with every other client
-                user_api = UsersApi(connection)
+    process on this machine, and a rotating provider stops honouring the refresh token
-                try:
+    it replaces. Two processes reaching this at once would otherwise both send the same
-                    connection.token = user_api.tokenRefresh(connection.token)
+    token, and the loser would be left holding one the server no longer accepts, so the
-                except UnauthorizedException:
+    whole sequence runs under a lock that spans processes and the cache is re-read
-                    connection.logout()
+    inside it.
-            elif connection.token.is_expired:
+    """
+    with connection.lock:
+        token = connection.token
+        if token is None or not (token.is_refresh_required or token.is_expired):
+            # Nothing to do, which is the answer for almost every request. The lock below
+            # reaches across processes and is held for a round trip, so it is worth
+            # knowing that before taking it.
+            return
+        with connection.cache_lock():
+            connection.reload_cache()
+            token = connection.token
+            if token is None:
+                return
+            if token.is_expired:
                 connection.logout()
-    finally:
+                return
-        connection.lock.release()
+            if not token.is_refresh_required:
+                return
+            user_api = UsersApi(connection)
+            try:
+                connection.token = user_api.tokenRefresh(token)
+            except UnauthorizedException:
+                connection.logout()
+
+
+def token_recover(connection: MDRSConnection, used: Token | None) -> bool:
+    """
+    Get a usable access token after one was refused for having expired.
+
+    A request can sit in the server's queue for longer than its access token lives, and
+    while it waited another thread or process may already have refreshed. Prefer what
+    they left behind: refreshing again would spend a round trip, and with a provider
+    that rotates refresh tokens it would retire one that is still in use.
+
+    Returns whether the caller now holds a token worth retrying with.
+    """
+    with connection.lock, connection.cache_lock():
+        connection.reload_cache()
+        token = connection.token
+        if token is None:
+            return False
+        if used is None or token != used:
+            # Somebody else has been here since the request went out. Assign rather than
+            # just read: the setter is what rewrites the session's Authorization header,
+            # so a plain reload would leave the retry carrying the refused token.
+            connection.token = token
+            return True
+        user_api = UsersApi(connection)
+        try:
+            connection.token = user_api.tokenRefresh(token)
+        except UnauthorizedException:
+            connection.logout()
+            return False
+        return True
+
+
+def retry_on_expired_token(func: F) -> F:
+    """
+    Send a request again once when the server found its access token expired.
+
+    The token is checked before every request, so this only happens when the request was
+    held long enough on the way in for a valid token to lapse - a queue behind uploads
+    that take minutes, most often.
+    """
+
+    @functools.wraps(func)
+    def wrapper(self: Any, *args: Any, **kwargs: Any) -> Any:
+        connection: MDRSConnection = self.connection
+        # Settle the token first so `used` is what the request actually carries. The
+        # wrapped method checks it again, which costs nothing once it is already current.
+        token_check(connection)
+        used = connection.token
+        try:
+            return func(self, *args, **kwargs)
+        except TokenExpiredException:
+            try:
+                recovered = token_recover(connection, used)
+            except MDRSException:
+                raise
+            except Exception as e:
+                # A busy server queues the refresh too, and failing here would report an
+                # endpoint the user never asked for rather than the reason.
+                raise ServerBusyException(SERVER_BUSY_MESSAGE) from e
+            if not recovered:
+                raise
+        try:
+            return func(self, *args, **kwargs)
+        except TokenExpiredException as e:
+            # A second refusal, with a token that was current when it left. The session is
+            # fine; the server is not starting requests before their credentials lapse.
+            raise ServerBusyException(SERVER_BUSY_MESSAGE) from e
+
+    return cast(F, wrapper)

mdrsclient/cache.py

@@ -1,8 +1,12 @@
+import contextlib
 import dataclasses
 import hashlib
 import json
 import os
-from typing import Protocol, runtime_checkable
+import tempfile
+import time
+from contextlib import AbstractContextManager
+from typing import Iterator, Protocol, runtime_checkable
 
 from pydantic import TypeAdapter, ValidationError
 from pydantic.dataclasses import dataclass
@@ -46,6 +50,19 @@ class CacheData:
 
 @runtime_checkable
 class CacheInterface(Protocol):
+    def lock(self) -> AbstractContextManager[None]:
+        """
+        Hold exclusive access to the cache for the duration of the block.
+
+        Refreshing a token is a read-modify-write, and the cache is shared with every
+        other client process using the same remote.
+        """
+        ...
+
+    def reload(self) -> None:
+        """Re-read the cache, discarding anything held from an earlier read."""
+        ...
+
     @property
     def token(self) -> Token | None: ...
     @token.setter
@@ -70,6 +87,14 @@ class InMemoryCache(CacheInterface):
     def __init__(self) -> None:
         self.__data = CacheData()
 
+    @contextlib.contextmanager
+    def lock(self) -> Iterator[None]:
+        # Nothing else can reach this cache, so there is nothing to exclude.
+        yield
+
+    def reload(self) -> None:
+        pass
+
     @property
     def token(self) -> Token | None:
         return self.__data.token
@@ -109,14 +134,46 @@ class CacheFile(CacheInterface):
     __serial: int
     __cache_dir: str
     __cache_file: str
+    __lock_file: str
+    __lock_depth: int
     __data: CacheData
 
     def __init__(self, remote: str) -> None:
         self.__serial = -1
         self.__cache_dir = os.path.join(CONFIG_DIRNAME, "cache")
         self.__cache_file = os.path.join(self.__cache_dir, remote + ".json")
+        self.__lock_file = os.path.join(self.__cache_dir, remote + ".lock")
+        self.__lock_depth = 0
         self.__data = CacheData()
 
+    @contextlib.contextmanager
+    def lock(self) -> Iterator[None]:
+        # Re-entrant, because every write takes it and a refresh is a write made while
+        # already holding it. A second flock on the same file from the same process would
+        # wait for a release that cannot come.
+        if self.__lock_depth > 0:
+            self.__lock_depth += 1
+            try:
+                yield
+            finally:
+                self.__lock_depth -= 1
+            return
+        # A separate file, so that replacing the cache cannot disturb the lock holders.
+        self.__ensure_cache_dir()
+        with open(self.__lock_file, "a") as f:
+            FileLock.lock(f)
+            self.__lock_depth = 1
+            try:
+                yield
+            finally:
+                self.__lock_depth = 0
+                FileLock.unlock(f)
+        os.chmod(self.__lock_file, 0o600)
+
+    def reload(self) -> None:
+        self.__serial = -1
+        self.__load()
+
     @property
     def token(self) -> Token | None:
         self.__load()
@@ -124,9 +181,10 @@ class CacheFile(CacheInterface):
 
     @token.setter
     def token(self, token: Token) -> None:
-        self.__load()
+        with self.lock():
-        self.__data.token = token
+            self.reload()
-        self.__save()
+            self.__data.token = token
+            self.__save()
 
     @token.deleter
     def token(self) -> None:
@@ -139,9 +197,10 @@ class CacheFile(CacheInterface):
 
     @user.setter
     def user(self, user: User) -> None:
-        self.__load()
+        with self.lock():
-        self.__data.user = user
+            self.reload()
-        self.__save()
+            self.__data.user = user
+            self.__save()
 
     @user.deleter
     def user(self) -> None:
@@ -154,9 +213,10 @@ class CacheFile(CacheInterface):
 
     @laboratories.setter
     def laboratories(self, laboratories: Laboratories) -> None:
-        self.__load()
+        with self.lock():
-        self.__data.laboratories = laboratories
+            self.reload()
-        self.__save()
+            self.__data.laboratories = laboratories
+            self.__save()
 
     def __clear(self) -> None:
         self.__data.clear()
@@ -164,8 +224,7 @@ class CacheFile(CacheInterface):
 
     def __load(self) -> None:
         if os.path.isfile(self.__cache_file):
-            stat = os.stat(self.__cache_file)
+            serial = self.__stat_serial()
-            serial = hash((stat.st_uid, stat.st_gid, stat.st_mode, stat.st_size, stat.st_mtime))
             if self.__serial != serial:
                 try:
                     with open(self.__cache_file) as f:
@@ -185,15 +244,44 @@ class CacheFile(CacheInterface):
 
     def __save(self) -> None:
         self.__ensure_cache_dir()
-        with open(self.__cache_file, "w") as f:
+        self.__data.update_digest()
-            FileLock.lock(f)
+        payload = json.dumps(dataclasses.asdict(self.__data))
-            self.__data.update_digest()
+        # Written aside and moved into place: opening the cache for writing truncates it
-            f.write(json.dumps(dataclasses.asdict(self.__data)))
+        # first, and a reader landing in that window would find the file empty and treat
-            FileLock.unlock(f)
+        # the session as broken.
+        fd, tmp_file = tempfile.mkstemp(dir=self.__cache_dir, prefix=".tmp-")
+        try:
+            with os.fdopen(fd, "w") as f:
+                f.write(payload)
+            # ensure file is secure.
+            os.chmod(tmp_file, 0o600)
+            self.__replace(tmp_file, self.__cache_file)
+        except BaseException:
+            if os.path.exists(tmp_file):
+                os.unlink(tmp_file)
+            raise
+        self.__serial = self.__stat_serial()
+
+    @staticmethod
+    def __replace(source: str, destination: str) -> None:
+        # Windows refuses the rename while another process still has the destination
+        # open, which a reader briefly does, so give it a moment rather than failing the
+        # save outright. On POSIX the rename always succeeds and the loop ends at once.
+        deadline = time.monotonic() + 5.0
+        while True:
+            try:
+                os.replace(source, destination)
+                return
+            except PermissionError:
+                if time.monotonic() >= deadline:
+                    raise
+                time.sleep(0.05)
+
+    def __stat_serial(self) -> int:
         stat = os.stat(self.__cache_file)
-        self.__serial = hash((stat.st_uid, stat.st_gid, stat.st_mode, stat.st_size, stat.st_mtime))
+        # st_ino and st_mtime_ns both move when the file is replaced, which a refresh
-        # ensure file is secure.
+        # that happens to produce the same number of bytes otherwise would not show.
-        os.chmod(self.__cache_file, 0o600)
+        return hash((stat.st_ino, stat.st_uid, stat.st_gid, stat.st_mode, stat.st_size, stat.st_mtime_ns))
 
     def __ensure_cache_dir(self) -> None:
         if not os.path.exists(self.__cache_dir):

mdrsclient/client.py

@@ -1,21 +1,40 @@
 import os
-from typing import Any
+from dataclasses import dataclass
 from unicodedata import normalize
 
-from mdrsclient.api import DoiApi, FilesApi, FoldersApi, LaboratoriesApi, UsersApi
+from mdrsclient.api import FilesApi, FoldersApi
 from mdrsclient.cache import CacheInterface
 from mdrsclient.config import ConfigInterface
 from mdrsclient.connection import MDRSConnection
-from mdrsclient.exceptions import IllegalArgumentException, MDRSException, UnauthorizedException, UnexpectedException
+from mdrsclient.exceptions import IllegalArgumentException
-from mdrsclient.models import File, Folder, Laboratory, Token, User
+from mdrsclient.models import File, Folder, FolderSimple, Laboratory
 from mdrsclient.models.file import find_file
 from mdrsclient.services import MdrsService
 
 
+@dataclass(frozen=True)
+class TransferEndpoints:
+    """Both ends of a `cp` or `mv`, resolved: the parents, their contents, and the names."""
+
+    laboratory: Laboratory
+    s_path: str
+    s_basename: str
+    s_parent_folder: Folder
+    s_parent_files: list[File]
+    d_path: str
+    d_basename: str
+    d_parent_folder: Folder
+    d_parent_files: list[File]
+
+    def is_same_place(self) -> bool:
+        """Whether the destination names what the source already is, leaving nothing to do."""
+        return self.s_parent_folder.id == self.d_parent_folder.id and self.s_basename == self.d_basename
+
+
 class MdrsClient(MdrsService):
     """Service layer client for MDRS."""
 
-    def __init__(self, connection: MDRSConnection, config_class: type[ConfigInterface] | None = None):
+    def __init__(self, connection: MDRSConnection | None, config_class: type[ConfigInterface] | None = None):
         super().__init__(connection, config_class)
 
     @classmethod
@@ -63,7 +82,8 @@ class MdrsClient(MdrsService):
         files = self.find_files(folder.id)
         return folder, files
 
-    def cp(self, src_path: str, dest_path: str, is_recursive: bool = False) -> None:
+    def __resolve_transfer(self, src_path: str, dest_path: str) -> TransferEndpoints:
+        """Resolve both ends of a transfer, refusing one that crosses a remote or a laboratory."""
         s_remote, s_laboratory_name, s_path = self.parse_remote_host_with_path(src_path)
         d_remote, d_laboratory_name, d_path = self.parse_remote_host_with_path(dest_path)
         if s_remote != d_remote:
@@ -71,8 +91,9 @@ class MdrsClient(MdrsService):
         if s_laboratory_name != d_laboratory_name:
             raise IllegalArgumentException("Laboratory mismatched.")
         s_path = s_path.rstrip("/")
-        s_dirname = os.path.dirname(s_path)
         s_basename = os.path.basename(s_path)
+        # A destination ending in a separator names a folder to put the source into, and
+        # keeps the source's own name.
         if d_path.endswith("/"):
             d_dirname = d_path
             d_basename = s_basename
@@ -80,84 +101,77 @@ class MdrsClient(MdrsService):
             d_dirname = os.path.dirname(d_path)
             d_basename = os.path.basename(d_path)
         laboratory = self.find_laboratory(s_laboratory_name)
-        s_parent_folder = self.find_folder(laboratory, s_dirname)
+        s_parent_folder = self.find_folder(laboratory, os.path.dirname(s_path))
-        s_parent_files = self.find_files(s_parent_folder.id)
         d_parent_folder = self.find_folder(laboratory, d_dirname)
-        d_parent_files = self.find_files(d_parent_folder.id)
+        return TransferEndpoints(
-        s_file = find_file(s_parent_files, s_basename)
+            laboratory=laboratory,
+            s_path=s_path,
+            s_basename=s_basename,
+            s_parent_folder=s_parent_folder,
+            s_parent_files=self.find_files(s_parent_folder.id),
+            d_path=d_path,
+            d_basename=d_basename,
+            d_parent_folder=d_parent_folder,
+            d_parent_files=self.find_files(d_parent_folder.id),
+        )
+
+    @staticmethod
+    def __check_file_destination(ends: TransferEndpoints) -> None:
+        """Refuse a destination that a file cannot take the place of."""
+        if find_file(ends.d_parent_files, ends.d_basename) is not None:
+            raise IllegalArgumentException(f"File `{ends.d_basename}` already exists.")
+        if ends.d_parent_folder.find_sub_folder(ends.d_basename) is not None:
+            raise IllegalArgumentException(
+                f"Cannot overwrite non-folder `{ends.d_basename}` with folder `{ends.d_path}`."
+            )
+
+    @staticmethod
+    def __check_folder_destination(ends: TransferEndpoints, s_folder: FolderSimple) -> None:
+        """Refuse a destination that a folder cannot take the place of."""
+        if find_file(ends.d_parent_files, ends.d_basename) is not None:
+            raise IllegalArgumentException(
+                f"Cannot overwrite non-folder `{ends.d_basename}` with folder `{ends.s_path}`."
+            )
+        d_folder = ends.d_parent_folder.find_sub_folder(ends.d_basename)
+        if d_folder is not None:
+            if d_folder.id == s_folder.id:
+                raise IllegalArgumentException(f"`{ends.s_path}` and `{ends.s_path}` are the same folder.")
+            raise IllegalArgumentException(f"Cannot move `{ends.s_path}` to `{ends.d_path}`: Folder not empty.")
+
+    @staticmethod
+    def __find_source_folder(ends: TransferEndpoints) -> FolderSimple:
+        s_folder = ends.s_parent_folder.find_sub_folder(ends.s_basename)
+        if s_folder is None:
+            raise IllegalArgumentException(f"File or folder `{ends.s_basename}` not found.")
+        return s_folder
+
+    def cp(self, src_path: str, dest_path: str, is_recursive: bool = False) -> None:
+        ends = self.__resolve_transfer(src_path, dest_path)
+        s_file = find_file(ends.s_parent_files, ends.s_basename)
         if s_file is not None:
-            d_file = find_file(d_parent_files, d_basename)
+            self.__check_file_destination(ends)
-            if d_file is not None:
+            if not ends.is_same_place():
-                raise IllegalArgumentException(f"File `{d_basename}` already exists.")
+                FilesApi(self.connection).copy(s_file, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
-            d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
+            return
-            if d_sub_folder is not None:
+        s_folder = self.__find_source_folder(ends)
-                raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
+        if not is_recursive:
-            file_api = FilesApi(self.connection)
+            raise IllegalArgumentException(f"Cannot copy `{ends.s_path}`: Is a folder.")
-            if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
+        self.__check_folder_destination(ends, s_folder)
-                file_api.copy(s_file, d_parent_folder.id, normalize("NFC", d_basename))
+        if not ends.is_same_place():
-        else:
+            FoldersApi(self.connection).copy(s_folder, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
-            s_folder = s_parent_folder.find_sub_folder(s_basename)
-            if s_folder is None:
-                raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
-            if not is_recursive:
-                raise IllegalArgumentException(f"Cannot copy `{s_path}`: Is a folder.")
-            if find_file(d_parent_files, d_basename) is not None:
-                raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
-            d_folder = d_parent_folder.find_sub_folder(d_basename)
-            if d_folder is not None:
-                if d_folder.id == s_folder.id:
-                    raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
-                raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
-            folder_api = FoldersApi(self.connection)
-            if s_parent_folder.id != d_parent_folder.id or s_basename != d_basename:
-                folder_api.copy(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
 
     def mv(self, src_path: str, dest_path: str) -> None:
-        s_remote, s_laboratory_name, s_path = self.parse_remote_host_with_path(src_path)
+        ends = self.__resolve_transfer(src_path, dest_path)
-        d_remote, d_laboratory_name, d_path = self.parse_remote_host_with_path(dest_path)
+        s_file = find_file(ends.s_parent_files, ends.s_basename)
-        if s_remote != d_remote:
-            raise IllegalArgumentException("Remote host mismatched.")
-        if s_laboratory_name != d_laboratory_name:
-            raise IllegalArgumentException("Laboratory mismatched.")
-        s_path = s_path.rstrip("/")
-        s_dirname = os.path.dirname(s_path)
-        s_basename = os.path.basename(s_path)
-        if d_path.endswith("/"):
-            d_dirname = d_path
-            d_basename = s_basename
-        else:
-            d_dirname = os.path.dirname(d_path)
-            d_basename = os.path.basename(d_path)
-        laboratory = self.find_laboratory(s_laboratory_name)
-        s_parent_folder = self.find_folder(laboratory, s_dirname)
-        s_parent_files = self.find_files(s_parent_folder.id)
-        d_parent_folder = self.find_folder(laboratory, d_dirname)
-        d_parent_files = self.find_files(d_parent_folder.id)
-        s_file = find_file(s_parent_files, s_basename)
         if s_file is not None:
-            d_file = find_file(d_parent_files, d_basename)
+            self.__check_file_destination(ends)
-            if d_file is not None:
+            if not ends.is_same_place():
-                raise IllegalArgumentException(f"File `{d_basename}` already exists.")
+                FilesApi(self.connection).move(s_file, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
-            d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
+            return
-            if d_sub_folder is not None:
+        s_folder = self.__find_source_folder(ends)
-                raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
+        self.__check_folder_destination(ends, s_folder)
-            file_api = FilesApi(self.connection)
+        if not ends.is_same_place():
-            if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
+            FoldersApi(self.connection).move(s_folder, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
-                file_api.move(s_file, d_parent_folder.id, normalize("NFC", d_basename))
-        else:
-            s_folder = s_parent_folder.find_sub_folder(s_basename)
-            if s_folder is None:
-                raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
-            if find_file(d_parent_files, d_basename) is not None:
-                raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
-            d_folder = d_parent_folder.find_sub_folder(d_basename)
-            if d_folder is not None:
-                if d_folder.id == s_folder.id:
-                    raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
-                raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
-            folder_api = FoldersApi(self.connection)
-            if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
-                folder_api.move(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
 
     def chacl(
         self, remote_path: str, access_level: int, is_recursive: bool = False, password: str | None = None

mdrsclient/commands/chacl.py

@@ -1,7 +1,6 @@
 from argparse import Namespace
 from typing import Any
 
-from mdrsclient.api import FoldersApi
 from mdrsclient.commands.base import BaseCommand
 from mdrsclient.exceptions import IllegalArgumentException
 from mdrsclient.models import FolderAccessLevel

mdrsclient/commands/cp.py

@@ -1,12 +1,7 @@
-import os
 from argparse import Namespace
 from typing import Any
-from unicodedata import normalize
 
-from mdrsclient.api import FilesApi, FoldersApi
 from mdrsclient.commands.base import BaseCommand
-from mdrsclient.exceptions import IllegalArgumentException
-from mdrsclient.models.file import find_file
 
 
 class CpCommand(BaseCommand):

mdrsclient/commands/file_metadata.py

@@ -1,12 +1,8 @@
 import json
-import os
 from argparse import Namespace
 from typing import Any
 
-from mdrsclient.api import FilesApi
 from mdrsclient.commands.base import BaseCommand
-from mdrsclient.exceptions import IllegalArgumentException
-from mdrsclient.models.file import find_file
 
 
 class FileMetadataCommand(BaseCommand):

mdrsclient/commands/labs.py

@@ -1,7 +1,6 @@
 from argparse import Namespace
 from typing import Any
 
-from mdrsclient.api import LaboratoriesApi
 from mdrsclient.commands.base import BaseCommand
 
 

mdrsclient/commands/ls.py

@@ -1,12 +1,13 @@
 import json
 from argparse import Namespace
-from typing import Any
+from typing import Any, Final
 
 from pydantic.dataclasses import dataclass
 
-from mdrsclient.api import FilesApi, FoldersApi
+from mdrsclient.api import FoldersApi
 from mdrsclient.client import MdrsClient
 from mdrsclient.commands.base import BaseCommand
+from mdrsclient.config import build_download_url
 from mdrsclient.exceptions import UnauthorizedException
 from mdrsclient.models import File, Folder, FolderSimple, Laboratory
 
@@ -94,25 +95,24 @@ class LsCommand(BaseCommand):
     def _ls_json(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
         print(json.dumps(cls._folder2dict(context, folder, files), ensure_ascii=False))
 
+    LABELS: Final[dict[str, str]] = {
+        "type": "Type",
+        "acl": "Access",
+        "laboratory": "Laboratory",
+        "size": "Size",
+        "date": "Date",
+        "name": "Name",
+    }
+
     @classmethod
-    def _ls_plain(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
+    def _column_widths(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> dict[str, int]:
-        label = {
+        """Width of each column: the widest of its heading and everything printed under it."""
-            "type": "Type",
+        length = {key: len(label) if not context.is_quiet else 0 for key, label in cls.LABELS.items()}
-            "acl": "Access",
-            "laboratory": "Laboratory",
-            "size": "Size",
-            "date": "Date",
-            "name": "Name",
-        }
-        length: dict[str, int] = {}
-        for key in label.keys():
-            length[key] = len(label[key]) if not context.is_quiet else 0
         for sub_folder in folder.sub_folders:
-            sub_laboratory = context.client.connection.laboratories.find_by_id(sub_folder.laboratory_id)
+            sub_laboratory_name = cls._laboratory_name(context, sub_folder.laboratory_id)
-            sub_laboratory_name = sub_laboratory.name if sub_laboratory is not None else "(invalid)"
             length["acl"] = max(length["acl"], len(sub_folder.access_level_name))
             length["laboratory"] = max(length["laboratory"], len(sub_laboratory_name))
-            length["size"] = max(length["size"], len(str(folder.size)))
+            length["size"] = max(length["size"], len(str(sub_folder.size)))
             length["date"] = max(length["date"], len(sub_folder.updated_at_name))
             length["name"] = max(length["name"], len(sub_folder.name))
         for file in files:
@@ -121,6 +121,27 @@ class LsCommand(BaseCommand):
             length["name"] = max(length["name"], len(file.name))
         length["acl"] = max(length["acl"], len(folder.access_level_name))
         length["laboratory"] = max(length["laboratory"], len(context.laboratory.name))
+        return length
+
+    @classmethod
+    def _ls_plain_children(cls, context: LsCommandContext, folder: Folder) -> None:
+        """List each sub-folder in turn, passing over the ones the caller cannot open."""
+        print("")
+        folder_api = FoldersApi(context.client.connection)
+        for sub_folder in sorted(folder.sub_folders, key=lambda x: x.name):
+            try:
+                if sub_folder.lock:
+                    folder_api.auth(sub_folder.id, context.password)
+                sub_detail = folder_api.retrieve(sub_folder.id)
+                sub_files = context.client.find_files(sub_folder.id)
+            except UnauthorizedException:
+                continue
+            cls._ls_plain(context, sub_detail, sub_files)
+
+    @classmethod
+    def _ls_plain(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
+        label = cls.LABELS
+        length = cls._column_widths(context, folder, files)
         header = (
             f"{label['type']:{length['type']}}\t{label['acl']:{length['acl']}}\t"
             f"{label['laboratory']:{length['laboratory']}}\t{label['size']:{length['size']}}\t"
@@ -151,17 +172,7 @@ class LsCommand(BaseCommand):
             )
 
         if context.is_recursive:
-            print("")
+            cls._ls_plain_children(context, folder)
-            for sub_folder in sorted(folder.sub_folders, key=lambda x: x.name):
-                folder_api = FoldersApi(context.client.connection)
-                try:
-                    if sub_folder.lock:
-                        folder_api.auth(sub_folder.id, context.password)
-                    folder = folder_api.retrieve(sub_folder.id)
-                    files = context.client.find_files(sub_folder.id)
-                    cls._ls_plain(context, folder, files)
-                except UnauthorizedException:
-                    pass
 
     @classmethod
     def _folder2dict(
@@ -211,7 +222,7 @@ class LsCommand(BaseCommand):
             # "thumbnail": file.thumbnail,
             "description": file.description,
             "metadata": file.metadata,
-            "download_url": f"{context.client.connection.url}/{file.download_url}",
+            "download_url": build_download_url(context.client.connection.url, file.download_url),
             "created_at": file.created_at,
             "updated_at": file.updated_at,
         }

mdrsclient/commands/metadata.py

@@ -2,7 +2,6 @@ import json
 from argparse import Namespace
 from typing import Any
 
-from mdrsclient.api import FoldersApi
 from mdrsclient.commands.base import BaseCommand
 
 

mdrsclient/commands/mkdir.py

@@ -1,12 +1,7 @@
-import os
 from argparse import Namespace
 from typing import Any
-from unicodedata import normalize
 
-from mdrsclient.api import FoldersApi
 from mdrsclient.commands.base import BaseCommand
-from mdrsclient.exceptions import IllegalArgumentException
-from mdrsclient.models.file import find_file
 
 
 class MkdirCommand(BaseCommand):

mdrsclient/commands/mv.py

@@ -1,12 +1,7 @@
-import os
 from argparse import Namespace
 from typing import Any
-from unicodedata import normalize
 
-from mdrsclient.api import FilesApi, FoldersApi
 from mdrsclient.commands.base import BaseCommand
-from mdrsclient.exceptions import IllegalArgumentException
-from mdrsclient.models.file import find_file
 
 
 class MvCommand(BaseCommand):

mdrsclient/commands/rm.py

@@ -1,11 +1,7 @@
-import os
 from argparse import Namespace
 from typing import Any
 
-from mdrsclient.api import FilesApi, FoldersApi
 from mdrsclient.commands.base import BaseCommand
-from mdrsclient.exceptions import IllegalArgumentException
-from mdrsclient.models.file import find_file
 
 
 class RmCommand(BaseCommand):

mdrsclient/commands/version.py

@@ -1,7 +1,6 @@
 from argparse import Namespace
 from typing import Any
 
-from mdrsclient.__version__ import __version__
 from mdrsclient.commands.base import BaseCommand
 
 

mdrsclient/config.py

@@ -1,7 +1,7 @@
 import configparser
 import os
 import threading
-from typing import Final, Protocol, runtime_checkable
+from typing import TYPE_CHECKING, Final, Protocol, runtime_checkable
 
 import validators
 
@@ -10,10 +10,43 @@ from mdrsclient.settings import CONFIG_DIRNAME
 from mdrsclient.utils import FileLock
 
 
+def normalize_url(url: str) -> str:
+    """
+    Check a remote URL and put it in the one form every client agrees on.
+
+    `simple_host` is what lets a bare hostname through, so a development server on
+    `localhost` is as acceptable as a deployment behind a domain name. The trailing
+    slash goes because the URL is joined with a path that brings its own.
+    """
+    if not validators.url(url, simple_host=True, validate_scheme=lambda scheme: scheme in ("http", "https")):
+        raise IllegalArgumentException("malformed URI sequence")
+    return url.rstrip("/")
+
+
+def build_download_url(base_url: str | None, path: str) -> str:
+    """
+    Join a remote base URL with a path the API returned.
+
+    The API answers with a relative path and no leading separator, and a configuration
+    written before the URL was normalised may still carry a trailing one.
+    """
+    if path.startswith(("http://", "https://")):
+        return path
+    return f"{(base_url or '').rstrip('/')}/{path.lstrip('/')}"
+
+
 @runtime_checkable
 class ConfigInterface(Protocol):
     remote: str
 
+    if TYPE_CHECKING:
+        # Declared for the type checker, because the class itself is passed around and
+        # called: without this it describes something that takes no arguments to build.
+        # Kept out of the running class, because a protocol that defines `__init__` loses
+        # the guard that stops it being instantiated, and hands that empty `__init__` to
+        # any implementation that does not write its own.
+        def __init__(self, remote: str) -> None: ...
+
     def list(self) -> list[tuple[str, str]]: ...
     @property
     def url(self) -> str | None: ...
@@ -42,8 +75,7 @@ class InMemoryConfig(ConfigInterface):
 
     @url.setter
     def url(self, url: str) -> None:
-        if not validators.url(url):
+        url = normalize_url(url)
-            raise IllegalArgumentException("malformed URI sequence")
         with self.__lock:
             self.__configs[self.remote] = url
 
@@ -91,8 +123,7 @@ class ConfigFile(ConfigInterface):
 
     @url.setter
     def url(self, url: str) -> None:
-        if not validators.url(url):
+        url = normalize_url(url)
-            raise IllegalArgumentException("malformed URI sequence")
         self.__load()
         if self.__config.has_section(self.remote):
             self.__config.remove_section(self.remote)

mdrsclient/connection.py

@@ -1,5 +1,6 @@
 import platform
 import threading
+from contextlib import AbstractContextManager
 from typing import TypedDict
 
 from requests import Response, Session
@@ -23,6 +24,7 @@ class _KwArgsMDRSConnectionPost(TypedDict, total=False):
     params: dict[str, str | int]
     data: dict[str, str | int] | MultipartEncoder
     headers: dict[str, str]
+    timeout: float | tuple[float, float]
 
 
 class _KwArgsMDRSConnectionPut(TypedDict, total=False):
@@ -61,6 +63,14 @@ class MDRSConnection:
     def delete(self, url: str, **kwargs: Unpack[_KwArgsMDRSConnectionDelete]) -> Response:
         return self.session.delete(self.__build_url(url), **kwargs)
 
+    def cache_lock(self) -> AbstractContextManager[None]:
+        """Hold exclusive access to the login cache across every process using it."""
+        return self.__cache.lock()
+
+    def reload_cache(self) -> None:
+        """Re-read the login cache, discarding anything held from an earlier read."""
+        self.__cache.reload()
+
     def logout(self) -> None:
         del self.__cache.user
         del self.__cache.token
@@ -96,7 +106,9 @@ class MDRSConnection:
             return path
         if self.url == "":
             raise MissingConfigurationException("remote host is not configured")
-        return f"{self.url}/{path}"
+        # The path brings its own separator, and a configuration written before the URL
+        # was normalised may still carry a trailing slash of its own.
+        return f"{self.url.rstrip('/')}/{path}"
 
     def __prepare_headers(self) -> None:
         self.session.headers.update(

mdrsclient/exceptions.py

@@ -28,6 +28,18 @@ class UnauthorizedException(MDRSException):
     pass
 
 
+class TokenExpiredException(UnauthorizedException):
+    """Thrown when the access token was still valid when sent but had expired on arrival"""
+
+    pass
+
+
+class ServerBusyException(MDRSException):
+    """Thrown when the server did not start handling a request before its token lapsed"""
+
+    pass
+
+
 class ForbiddenException(MDRSException):
     """Thrown when the current user does not have enough privileges to access the resource"""
 

mdrsclient/services.py

@@ -1,6 +1,5 @@
 import os
 import re
-from typing import Any
 from unicodedata import normalize
 
 from mdrsclient.api import DoiApi, FilesApi, FoldersApi, LaboratoriesApi, UsersApi
@@ -20,11 +19,30 @@ from mdrsclient.utils import page_num_from_url
 class MdrsService:
     config_class: type[ConfigInterface] = ConfigFile
 
-    def __init__(self, connection: MDRSConnection, config_class: type[ConfigInterface] | None = None):
+    def __init__(self, connection: MDRSConnection | None, config_class: type[ConfigInterface] | None = None):
-        self.connection = connection
+        self.__connection = connection
         if config_class is not None:
             self.config_class = config_class
 
+    @property
+    def connection(self) -> MDRSConnection:
+        """
+        The connection every remote operation goes through.
+
+        Optional to supply, because `config` and `version` do their work without a remote
+        and are reached through the same client. Asking for it when none was given is a
+        mistake in the caller, and says so rather than failing later on `None`.
+        """
+        if self.__connection is None:
+            raise MissingConfigurationException("This operation requires a remote host.")
+        return self.__connection
+
+    @connection.setter
+    def connection(self, connection: MDRSConnection | None) -> None:
+        # Assignable as it always was: this is a property to check for absence on read,
+        # not to make the connection fixed once the client is built.
+        self.__connection = connection
+
     @classmethod
     def create_connection(
         cls, remote: str, cache: CacheInterface | None = None, config: ConfigInterface | None = None
@@ -182,7 +200,7 @@ class MdrsService:
         if folder.lock:
             if password is None:
                 raise UnauthorizedException(f"Folder for DOI `{doi_clean}` is locked.")
-            folder_api.auth(doi_resp.folder.id, password)
+            folder_api.auth(doi_resp.folder_id, password)
 
         lab_api = LaboratoriesApi(self.connection)
         labs = lab_api.list()

mdrsclient/transfer.py

@@ -6,7 +6,7 @@ from unicodedata import normalize
 from pydantic.dataclasses import dataclass
 
 from mdrsclient.api import FilesApi, FoldersApi
-from mdrsclient.exceptions import IllegalArgumentException, MDRSException, UnexpectedException
+from mdrsclient.exceptions import IllegalArgumentException, UnexpectedException
 from mdrsclient.models import File, Folder, Laboratory
 from mdrsclient.models.file import find_file
 from mdrsclient.settings import CONCURRENT
@@ -27,7 +27,6 @@ class DownloadFileInfo:
 
 @dataclass
 class DownloadContext:
-    hasError: bool
     isSkipIfExists: bool
     files: list[DownloadFileInfo]
 
@@ -47,54 +46,67 @@ class Uploader:
         laboratory = self.client.find_laboratory(laboratory_name)
         folder = self.client.find_folder(laboratory, r_path)
         files = self.client.find_files(folder.id)
-        infos: list[UploadFileInfo] = []
         if os.path.isdir(l_path):
             if not is_recursive:
                 raise IllegalArgumentException(f"Cannot upload `{local_path}`: Is a directory.")
-            folder_api = FoldersApi(self.client.connection)
+            infos = self.__collect_directory_uploads(laboratory, r_path, l_path, folder, files)
-            folder_map: dict[str, Folder] = {}
-            folder_map[r_path] = folder
-            files_map: dict[str, list[File]] = {}
-            files_map[r_path] = files
-            l_basename = os.path.basename(l_path)
-            for dirpath, _, filenames in os.walk(l_path, followlinks=True):
-                sub = l_basename if dirpath == l_path else os.path.join(l_basename, os.path.relpath(dirpath, l_path))
-                d_dirname = os.path.join(r_path, sub)
-                d_basename = os.path.basename(d_dirname)
-                # prepare destination parent path
-                d_parent_dirname = os.path.dirname(d_dirname)
-                if folder_map.get(d_parent_dirname) is None:
-                    parent_folder = self.client.find_folder(laboratory, d_parent_dirname)
-                    folder_map[d_parent_dirname] = parent_folder
-                    parent_files = self.client.find_files(parent_folder.id)
-                    files_map[d_parent_dirname] = parent_files
-                # prepare destination path
-                if folder_map.get(d_dirname) is None:
-                    d_folder = folder_map[d_parent_dirname].find_sub_folder(d_basename)
-                    if d_folder is None:
-                        d_folder_id = folder_api.create(normalize("NFC", d_basename), folder_map[d_parent_dirname].id)
-                    else:
-                        d_folder_id = d_folder.id
-                    print(d_dirname)
-                    folder_map[d_dirname] = folder_api.retrieve(d_folder_id)
-                    files_map[d_dirname] = self.client.find_files(d_folder_id)
-                    if d_folder is None:
-                        folder_map[d_parent_dirname].sub_folders.append(folder_map[d_dirname])
-                # register upload file list
-                for filename in filenames:
-                    infos.append(
-                        UploadFileInfo(folder_map[d_dirname], files_map[d_dirname], os.path.join(dirpath, filename))
-                    )
         else:
-            infos.append(UploadFileInfo(folder, files, l_path))
+            infos = [UploadFileInfo(folder, files, l_path)]
-        self.__multiple_upload(infos, is_skip_if_exists)
+        if not self.__multiple_upload(infos, is_skip_if_exists):
+            # One file failing is worth reporting on its own line, and worth the caller
+            # hearing about: a batch that lost files is not a batch that succeeded.
+            raise UnexpectedException("Some files failed to upload.")
 
-    def __multiple_upload(self, infos: list[UploadFileInfo], is_skip_if_exists: bool) -> None:
+    def __collect_directory_uploads(
+        self, laboratory: Laboratory, r_path: str, l_path: str, folder: Folder, files: list[File]
+    ) -> list[UploadFileInfo]:
+        """Mirror a local directory tree on the remote, and list the files to send into it."""
+        infos: list[UploadFileInfo] = []
+        folder_api = FoldersApi(self.client.connection)
+        folder_map: dict[str, Folder] = {}
+        folder_map[r_path] = folder
+        files_map: dict[str, list[File]] = {}
+        files_map[r_path] = files
+        l_basename = os.path.basename(l_path)
+        for dirpath, _, filenames in os.walk(l_path, followlinks=True):
+            sub = l_basename if dirpath == l_path else os.path.join(l_basename, os.path.relpath(dirpath, l_path))
+            d_dirname = os.path.join(r_path, sub)
+            d_basename = os.path.basename(d_dirname)
+            # prepare destination parent path
+            d_parent_dirname = os.path.dirname(d_dirname)
+            if folder_map.get(d_parent_dirname) is None:
+                parent_folder = self.client.find_folder(laboratory, d_parent_dirname)
+                folder_map[d_parent_dirname] = parent_folder
+                parent_files = self.client.find_files(parent_folder.id)
+                files_map[d_parent_dirname] = parent_files
+            # prepare destination path
+            if folder_map.get(d_dirname) is None:
+                d_folder = folder_map[d_parent_dirname].find_sub_folder(d_basename)
+                if d_folder is None:
+                    d_folder_id = folder_api.create(normalize("NFC", d_basename), folder_map[d_parent_dirname].id)
+                else:
+                    d_folder_id = d_folder.id
+                print(d_dirname)
+                folder_map[d_dirname] = folder_api.retrieve(d_folder_id)
+                files_map[d_dirname] = self.client.find_files(d_folder_id)
+                if d_folder is None:
+                    folder_map[d_parent_dirname].sub_folders.append(folder_map[d_dirname])
+            # register upload file list
+            for filename in filenames:
+                infos.append(
+                    UploadFileInfo(folder_map[d_dirname], files_map[d_dirname], os.path.join(dirpath, filename))
+                )
+        return infos
+
+    def __multiple_upload(self, infos: list[UploadFileInfo], is_skip_if_exists: bool) -> bool:
+        """Send every file, and report whether all of them arrived."""
         file_api = FilesApi(self.client.connection)
         with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
-            pool.map(lambda x: self.__multiple_upload_worker(file_api, x, is_skip_if_exists), infos)
+            results = pool.map(lambda x: self.__multiple_upload_worker(file_api, x, is_skip_if_exists), infos)
+            # Consumed inside the block: the results are what carry each worker's verdict.
+            return all(list(results))
 
-    def __multiple_upload_worker(self, file_api: FilesApi, info: UploadFileInfo, is_skip_if_exists: bool) -> None:
+    def __multiple_upload_worker(self, file_api: FilesApi, info: UploadFileInfo, is_skip_if_exists: bool) -> bool:
         basename = os.path.basename(info.path)
         file = find_file(info.files, basename)
         try:
@@ -103,8 +115,14 @@ class Uploader:
             elif not is_skip_if_exists or file.size != os.path.getsize(info.path):
                 file_api.update(file, info.path)
             print(os.path.join(info.folder.path, basename))
-        except MDRSException as e:
+        except Exception as e:
-            print(f"Error: {e}")
+            # Everything, not just the exceptions the API layer raises: the batch verdict
+            # is read now that the results are consumed, and a file vanishing between the
+            # walk and the upload would otherwise end the whole run with a traceback and
+            # throw away what every other file did.
+            print(f"Failed: {info.path}: {e}")
+            return False
+        return True
 
 
 class Downloader:
@@ -120,90 +138,117 @@ class Downloader:
         password: str | None = None,
         excludes: list[str] | None = None,
     ) -> None:
+        if not self.__download(remote_path, local_path, is_recursive, is_skip_if_exists, password, excludes):
+            # Every failure has already been printed against the file it belongs to.
+            # This is what makes the command as a whole end in failure.
+            raise UnexpectedException("Some files failed to download.")
+
+    def __download(
+        self,
+        remote_path: str,
+        local_path: str,
+        is_recursive: bool,
+        is_skip_if_exists: bool,
+        password: str | None,
+        excludes: list[str] | None,
+    ) -> bool:
+        """Fetch what the remote path names, and report whether every file arrived."""
         excludes_clean = excludes or []
-        # Detect DOI path: "remote:10.xxxx/prefix.ID[/optional/sub/path]"
-        path_component = remote_path.split(":", 1)[1] if ":" in remote_path else ""
-        if self.client.is_doi(path_component):
-            remote, doi, subpath = self.client.parse_doi_remote_host(remote_path)
-
-            l_dirname = os.path.realpath(local_path)
-            if not os.path.isdir(l_dirname):
-                raise IllegalArgumentException(f"Local directory `{local_path}` not found.")
-            doi_folder, laboratory = self.client.find_folder_by_doi(doi, password)
-
-            subpath_clean = subpath.rstrip("/")
-            if not subpath_clean:
-                folder = doi_folder
-                is_folder = True
-            else:
-                r_dirname = os.path.dirname(subpath_clean)
-                r_basename = os.path.basename(subpath_clean)
-                abs_path = doi_folder.path.rstrip("/") + r_dirname
-                r_parent_folder = self.client.find_folder(laboratory, abs_path, password)
-                r_parent_files = self.client.find_files(r_parent_folder.id)
-                file = find_file(r_parent_files, r_basename)
-                if file is not None:
-                    if self.__check_excludes(excludes_clean, laboratory, r_parent_folder, file):
-                        return
-                    context = DownloadContext(False, is_skip_if_exists, [])
-                    l_path = os.path.join(l_dirname, r_basename)
-                    context.files.append(DownloadFileInfo(file, l_path))
-                    self.__multiple_download(context)
-                    return
-                else:
-                    folder_simple = r_parent_folder.find_sub_folder(r_basename)
-                    if folder_simple is None:
-                        raise IllegalArgumentException(f"File or folder `{subpath_clean}` not found.")
-                    folder = FoldersApi(self.client.connection).retrieve(folder_simple.id)
-                    is_folder = True
-
-            # For a DOI target the whole folder is the download target.
-            if not is_recursive:
-                # Non-recursive: download only the files at the top level of the DOI folder.
-                files = self.client.find_files(folder.id)
-                context = DownloadContext(False, is_skip_if_exists, [])
-                for file in files:
-                    if self.__check_excludes(excludes_clean, laboratory, folder, file):
-                        continue
-                    l_path = os.path.join(l_dirname, file.name)
-                    context.files.append(DownloadFileInfo(file, l_path))
-                self.__multiple_download(context)
-                return
-            folder_api = FoldersApi(self.client.connection)
-            self.__multiple_download_pickup_recursive_files(
-                folder_api, laboratory, folder.id, l_dirname, excludes_clean, is_skip_if_exists
-            )
-            return
-
-        remote, laboratory_name, r_path = self.client.parse_remote_host_with_path(remote_path)
-        r_path = r_path.rstrip("/")
-        r_dirname = os.path.dirname(r_path)
-        r_basename = os.path.basename(r_path)
-
         l_dirname = os.path.realpath(local_path)
         if not os.path.isdir(l_dirname):
             raise IllegalArgumentException(f"Local directory `{local_path}` not found.")
+
+        # "remote:10.xxxx/prefix.ID[/optional/sub/path]" names a published dataset rather
+        # than a path within a laboratory, and is resolved through the DOI instead.
+        path_component = remote_path.split(":", 1)[1] if ":" in remote_path else ""
+        if self.client.is_doi(path_component):
+            return self.__download_doi(
+                remote_path, l_dirname, is_recursive, is_skip_if_exists, password, excludes_clean
+            )
+
+        remote, laboratory_name, r_path = self.client.parse_remote_host_with_path(remote_path)
+        r_path = r_path.rstrip("/")
+        r_basename = os.path.basename(r_path)
         laboratory = self.client.find_laboratory(laboratory_name)
-        r_parent_folder = self.client.find_folder(laboratory, r_dirname, password)
+        r_parent_folder = self.client.find_folder(laboratory, os.path.dirname(r_path), password)
         r_parent_files = self.client.find_files(r_parent_folder.id)
         file = find_file(r_parent_files, r_basename)
         if file is not None:
-            if self.__check_excludes(excludes_clean, laboratory, r_parent_folder, file):
+            return self.__download_one(
-                return
+                excludes_clean, laboratory, r_parent_folder, file, l_dirname, r_basename, is_skip_if_exists
-            context = DownloadContext(False, is_skip_if_exists, [])
-            l_path = os.path.join(l_dirname, r_basename)
-            context.files.append(DownloadFileInfo(file, l_path))
-            self.__multiple_download(context)
-        else:
-            folder = r_parent_folder.find_sub_folder(r_basename)
-            if folder is None:
-                raise IllegalArgumentException(f"File or folder `{r_path}` not found.")
-            if not is_recursive:
-                raise IllegalArgumentException(f"Cannot download `{r_path}`: Is a folder.")
-            folder_api = FoldersApi(self.client.connection)
-            self.__multiple_download_pickup_recursive_files(
-                folder_api, laboratory, folder.id, l_dirname, excludes_clean, is_skip_if_exists
             )
+        folder = r_parent_folder.find_sub_folder(r_basename)
+        if folder is None:
+            raise IllegalArgumentException(f"File or folder `{r_path}` not found.")
+        if not is_recursive:
+            raise IllegalArgumentException(f"Cannot download `{r_path}`: Is a folder.")
+        return self.__multiple_download_pickup_recursive_files(
+            FoldersApi(self.client.connection), laboratory, folder.id, l_dirname, excludes_clean, is_skip_if_exists
+        )
+
+    def __download_doi(
+        self,
+        remote_path: str,
+        l_dirname: str,
+        is_recursive: bool,
+        is_skip_if_exists: bool,
+        password: str | None,
+        excludes: list[str],
+    ) -> bool:
+        """Fetch what a DOI names: the dataset's folder, or something inside it."""
+        remote, doi, subpath = self.client.parse_doi_remote_host(remote_path)
+        doi_folder, laboratory = self.client.find_folder_by_doi(doi, password)
+
+        subpath_clean = subpath.rstrip("/")
+        if not subpath_clean:
+            folder = doi_folder
+        else:
+            r_basename = os.path.basename(subpath_clean)
+            abs_path = doi_folder.path.rstrip("/") + os.path.dirname(subpath_clean)
+            r_parent_folder = self.client.find_folder(laboratory, abs_path, password)
+            file = find_file(self.client.find_files(r_parent_folder.id), r_basename)
+            if file is not None:
+                return self.__download_one(
+                    excludes, laboratory, r_parent_folder, file, l_dirname, r_basename, is_skip_if_exists
+                )
+            folder_simple = r_parent_folder.find_sub_folder(r_basename)
+            if folder_simple is None:
+                raise IllegalArgumentException(f"File or folder `{subpath_clean}` not found.")
+            folder = FoldersApi(self.client.connection).retrieve(folder_simple.id)
+
+        if is_recursive:
+            return self.__multiple_download_pickup_recursive_files(
+                FoldersApi(self.client.connection), laboratory, folder.id, l_dirname, excludes, is_skip_if_exists
+            )
+        # Without -r the dataset's own files are fetched, and its sub-folders are not.
+        context = DownloadContext(is_skip_if_exists, [])
+        for file in self.client.find_files(folder.id):
+            if self.__check_excludes(excludes, laboratory, folder, file):
+                continue
+            context.files.append(DownloadFileInfo(file, os.path.join(l_dirname, file.name)))
+        return self.__multiple_download(context)
+
+    def __download_one(
+        self,
+        excludes: list[str],
+        laboratory: Laboratory,
+        folder: Folder,
+        file: File,
+        l_dirname: str,
+        local_name: str,
+        is_skip_if_exists: bool,
+    ) -> bool:
+        """
+        Fetch a single named file into the local directory.
+
+        Saved under the name the caller asked for rather than the one the server holds:
+        the two are matched case-insensitively, so they need not be spelled alike.
+        """
+        if self.__check_excludes(excludes, laboratory, folder, file):
+            return True
+        context = DownloadContext(is_skip_if_exists, [])
+        context.files.append(DownloadFileInfo(file, os.path.join(l_dirname, local_name)))
+        return self.__multiple_download(context)
 
     def __multiple_download_pickup_recursive_files(
         self,
@@ -213,47 +258,60 @@ class Downloader:
         basedir: str,
         excludes: list[str],
         is_skip_if_exists: bool,
-    ) -> None:
+    ) -> bool:
-        context = DownloadContext(False, is_skip_if_exists, [])
+        context = DownloadContext(is_skip_if_exists, [])
-        folder = folder_api.retrieve(folder_id)
+        try:
-        files = self.client.find_files(folder.id)
+            folder = folder_api.retrieve(folder_id)
+            files = self.client.find_files(folder.id)
+        except Exception as e:
+            print(f"Failed: {basedir}: {e}")
+            return False
         dirname = os.path.join(basedir, folder.name)
         if self.__check_excludes(excludes, laboratory, folder, None):
-            return
+            return True
-        if not os.path.exists(dirname):
+        try:
-            os.makedirs(dirname)
+            # `exist_ok` rather than a prior check: two workers can reach the same parent.
+            os.makedirs(dirname, exist_ok=True)
+        except OSError as e:
+            # One folder the client cannot make locally is not a reason to abandon its
+            # siblings, which is what this walk now promises.
+            print(f"Failed: {dirname}: {e}")
+            return False
         print(dirname)
         for file in files:
             if self.__check_excludes(excludes, laboratory, folder, file):
                 continue
             path = os.path.join(dirname, file.name)
             context.files.append(DownloadFileInfo(file, path))
-        self.__multiple_download(context)
+        succeeded = self.__multiple_download(context)
-        if context.hasError:
+        # A folder that lost a file is still a folder whose sub-folders the user asked
-            raise UnexpectedException("Some files failed to download.")
+        # for, so the walk carries on and the verdict is collected for the caller.
         for sub_folder in folder.sub_folders:
-            self.__multiple_download_pickup_recursive_files(
+            if not self.__multiple_download_pickup_recursive_files(
                 folder_api, laboratory, sub_folder.id, dirname, excludes, is_skip_if_exists
-            )
+            ):
+                succeeded = False
+        return succeeded
 
-    def __multiple_download(self, context: DownloadContext) -> None:
+    def __multiple_download(self, context: DownloadContext) -> bool:
+        """Fetch every file in the batch, and report whether all of them arrived."""
         file_api = FilesApi(self.client.connection)
         with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
             results = pool.map(
                 lambda x: self.__multiple_download_worker(file_api, x, context.isSkipIfExists), context.files
             )
-            hasError = next(filter(lambda x: x is False, results), None)
+            # Consumed inside the block, and in full: every worker's verdict counts, not
-            if hasError is not None:
+            # just the first refusal.
-                context.hasError = True
+            return all(list(results))
 
     def __multiple_download_worker(self, file_api: FilesApi, info: DownloadFileInfo, is_skip_if_exists: bool) -> bool:
         if not is_skip_if_exists or not os.path.exists(info.path) or info.file.size != os.path.getsize(info.path):
             try:
                 file_api.download(info.file, info.path)
-            except Exception:
+            except Exception as e:
-                print(f"Failed: {info.path}")
+                # Nothing to clear up: a failed transfer writes only to its own scratch
-                if os.path.isfile(info.path):
+                # file beside the destination, and removes that itself.
-                    os.remove(info.path)
+                print(f"Failed: {info.path}: {e}")
                 return False
         print(info.path)
         return True

mdrsclient/utils.py

@@ -1,5 +1,6 @@
 import os
-from typing import IO, Any
+import time
+from typing import IO, Any, Final
 from urllib.parse import parse_qs, urlparse
 
 if os.name == "nt":
@@ -9,10 +10,23 @@ elif os.name == "posix":
 
 
 class FileLock:
+    # Long enough to outlast a token refresh, which is what the lock is held across.
+    WAIT_SECONDS: Final[float] = 60.0
+
     @staticmethod
     def lock(file: IO[Any]) -> None:
         if os.name == "nt":
-            msvcrt.locking(file.fileno(), msvcrt.LK_LOCK, 1)
+            # msvcrt.LK_LOCK gives up after ten one-second attempts, which is shorter
+            # than the refresh it now has to wait for, so do the waiting here instead.
+            deadline = time.monotonic() + FileLock.WAIT_SECONDS
+            while True:
+                try:
+                    msvcrt.locking(file.fileno(), msvcrt.LK_NBLCK, 1)
+                    return
+                except OSError:
+                    if time.monotonic() >= deadline:
+                        raise
+                    time.sleep(0.1)
         elif os.name == "posix":
             fcntl.flock(file.fileno(), fcntl.LOCK_EX)
 

pyproject.toml

@@ -1,11 +1,21 @@
-[tool.poetry]
+[build-system]
+requires = ["flit_core>=4.0.2,<5.0.0"]
+build-backend = "flit_core.buildapi"
+
+[tool.flit.module]
+name = "mdrsclient"
+
+[project]
 name = "mdrs-client-python"
-version = "1.3.18"
+version = "1.3.19"
 description = "The mdrs-client-python is python library and a command-line client for up- and downloading files to and from MDRS based repository."
-authors = ["Yoshihiro OKUMURA <yoshihiro.okumura@riken.jp>"]
+authors = [
-license = "MIT"
+    { name = "Yoshihiro OKUMURA", email = "yoshihiro.okumura@riken.jp" }
+]
+license = { text = "MIT" }
 readme = "README.md"
-classifiers=[
+requires-python = ">=3.10"
+classifiers = [
     "Development Status :: 4 - Beta",
     "Environment :: Console",
     "Intended Audience :: Developers",
@@ -18,33 +28,28 @@ classifiers=[
     "OSI Approved :: MIT License",
     "Topic :: Utilities",
 ]
-packages = [
+
-    { include = "mdrsclient" }
+dependencies = [
+    "requests>=2.34.2",
+    "requests-toolbelt>=1.0.0",
+    "python-dotenv>=1.2.3",
+    "pydantic>=2.13.5",
+    "pydantic-settings>=2.15.0",
+    "PyJWT>=2.13.0",
+    "validators>=0.35.0",
 ]
 
-[tool.poetry.dependencies]
+[project.optional-dependencies]
-python = "^3.10"
+dev = [
-requests = "^2.34.2"
+    "black>=26.5.1",
-requests-toolbelt = "^1.0.0"
+    "flake8>=7.3.0",
-python-dotenv = "^1.1.0"
+    "Flake8-pyproject>=1.2.4",
-pydantic = "^2.13.4"
+    "isort>=9.0.1",
-pydantic-settings = "^2.14.2"
+    "pyright>=1.1.411",
-PyJWT = "^2.13.0"
+]
-validators = "^0.35.0"
 
-[tool.poetry.group.dev.dependencies]
+[project.scripts]
-black = "^26.5.1"
+mdrs = "mdrsclient.__main__:main"
-flake8 = "^7.2.0"
-Flake8-pyproject = "^1.2.3"
-isort = "^8.0.1"
-pyright = "^1.1.411"
-
-[tool.poetry.scripts]
-mdrs = 'mdrsclient.__main__:main'
-
-[build-system]
-requires = ["poetry-core"]
-build-backend = "poetry.core.masonry.api"
 
 [tool.black]
 line-length = 120
@@ -63,6 +68,9 @@ exclude = '''
 exclude = ".git, .venv, __pycache__, dist"
 max-complexity = 10
 max-line-length = 120
+# E203 flags the space black puts before a slice colon. Black formats this project, so
+# its output is the standard and flake8 has to accept it.
+extend-ignore = ["E203"]
 
 [tool.isort]
 profile = "black"
@@ -72,6 +80,12 @@ line_length = 120
 
 [tool.pyright]
 typeCheckingMode = "basic"
+# This project keeps its environment in `./.venv`. Named explicitly because without it
+# pyright falls back to the system interpreter, cannot resolve pydantic, and reports every
+# model field as an unknown argument. An environment kept elsewhere gets that same fallback
+# and can override it with a local `pyrightconfig.json` or `pyright --pythonpath`.
+venvPath = "."
+venv = ".venv"
 exclude = ["**/__pycache__", "**/.*", "dist"]
 #reportUnknownMemberType = "warning"
 #reportUnknownVariableType = "warning"

tests/test_client.py

@@ -0,0 +1,165 @@
+import unittest
+from unittest.mock import MagicMock, patch
+
+from mdrsclient.client import MdrsClient
+from mdrsclient.exceptions import IllegalArgumentException
+from mdrsclient.models import File, Folder, FolderSimple, Laboratories, Laboratory
+
+TIMESTAMP = "2026-01-01T00:00:00+09:00"
+LABORATORY = Laboratory(id=1, name="mylab", pi_name="PI", full_name="My Laboratory")
+
+
+def make_file(id: str, name: str) -> File:
+    return File(
+        id=id,
+        name=name,
+        type="text/plain",
+        size=1,
+        thumbnail=None,
+        description="",
+        metadata={},
+        download_url=f"v3/files/{id}/download/",
+        created_at=TIMESTAMP,
+        updated_at=TIMESTAMP,
+    )
+
+
+def make_sub_folder(id: str, name: str) -> FolderSimple:
+    return FolderSimple(
+        id=id,
+        pid=None,
+        name=name,
+        access_level=1,
+        lock=False,
+        size=0,
+        laboratory_id=1,
+        description="",
+        created_at=TIMESTAMP,
+        updated_at=TIMESTAMP,
+        restrict_opened_at=None,
+    )
+
+
+def make_folder(id: str, name: str, path: str, sub_folders: list[FolderSimple] | None = None) -> Folder:
+    return Folder(
+        id=id,
+        pid=None,
+        name=name,
+        access_level=1,
+        lock=False,
+        size=0,
+        laboratory_id=1,
+        description="",
+        created_at=TIMESTAMP,
+        updated_at=TIMESTAMP,
+        restrict_opened_at=None,
+        metadata=[],
+        sub_folders=sub_folders if sub_folders is not None else [],
+        path=path,
+    )
+
+
+class TestLsColumnWidths(unittest.TestCase):
+    """Each column is as wide as the widest thing printed under it."""
+
+    def test_the_size_column_fits_a_sub_folder_larger_than_its_parent(self):
+        from unittest.mock import MagicMock as _MagicMock
+
+        from mdrsclient.commands.ls import LsCommand, LsCommandContext
+
+        laboratories = Laboratories()
+        laboratories.append(LABORATORY)
+        connection = _MagicMock()
+        connection.laboratories = laboratories
+        client = MdrsClient(connection)
+
+        big = make_sub_folder("s1", "big")
+        object.__setattr__(big, "size", 123456789)
+        parent = make_folder("f1", "root", "/root/", [big])
+        context = LsCommandContext("remote:/mylab", client, LABORATORY, "", False, False, False)
+
+        widths = LsCommand._column_widths(context, parent, [])
+
+        self.assertEqual(widths["size"], len("123456789"))
+
+
+class TestCopyAndMove(unittest.TestCase):
+    """`cp` and `mv` resolve both ends the same way and differ only in what they call."""
+
+    def make_client(self, folders_by_path: dict, files_by_folder: dict) -> MdrsClient:
+        client = MdrsClient(MagicMock())
+        client.find_laboratory = MagicMock(return_value=LABORATORY)
+        client.find_folder = MagicMock(side_effect=lambda lab, path, password=None: folders_by_path[path])
+        client.find_files = MagicMock(side_effect=lambda folder_id: files_by_folder.get(folder_id, []))
+        return client
+
+    def make_tree(self, source_has: str) -> tuple[MdrsClient, dict, dict]:
+        sub = make_sub_folder("fsub", "sub")
+        folder_a = make_folder("fa", "a", "/a/", [sub] if source_has == "folder" else [])
+        folder_b = make_folder("fb", "b", "/b/")
+        folders = {"/a": folder_a, "/b": folder_b, "/b/": folder_b}
+        files = {"fa": [make_file("x1", "data.txt")] if source_has == "file" else [], "fb": []}
+        return self.make_client(folders, files), folders, files
+
+    def test_a_file_is_copied_into_the_destination_folder(self):
+        client, _, files = self.make_tree("file")
+        with patch("mdrsclient.client.FilesApi") as files_api:
+            client.cp("myremote:/mylab/a/data.txt", "myremote:/mylab/b/data.txt")
+            files_api.return_value.copy.assert_called_once_with(files["fa"][0], "fb", "data.txt")
+
+    def test_a_file_is_moved_into_the_destination_folder(self):
+        client, _, files = self.make_tree("file")
+        with patch("mdrsclient.client.FilesApi") as files_api:
+            client.mv("myremote:/mylab/a/data.txt", "myremote:/mylab/b/data.txt")
+            files_api.return_value.move.assert_called_once_with(files["fa"][0], "fb", "data.txt")
+
+    def test_a_trailing_separator_keeps_the_source_name(self):
+        client, _, files = self.make_tree("file")
+        with patch("mdrsclient.client.FilesApi") as files_api:
+            client.mv("myremote:/mylab/a/data.txt", "myremote:/mylab/b/")
+            files_api.return_value.move.assert_called_once_with(files["fa"][0], "fb", "data.txt")
+
+    def test_copying_a_folder_needs_the_recursive_flag(self):
+        client, _, _ = self.make_tree("folder")
+        with patch("mdrsclient.client.FoldersApi") as folders_api:
+            with self.assertRaises(IllegalArgumentException) as caught:
+                client.cp("myremote:/mylab/a/sub", "myremote:/mylab/b/sub")
+            self.assertIn("Is a folder", str(caught.exception))
+            folders_api.return_value.copy.assert_not_called()
+
+    def test_a_folder_is_copied_when_recursive(self):
+        client, folders, _ = self.make_tree("folder")
+        with patch("mdrsclient.client.FoldersApi") as folders_api:
+            client.cp("myremote:/mylab/a/sub", "myremote:/mylab/b/sub", is_recursive=True)
+            folders_api.return_value.copy.assert_called_once_with(folders["/a"].sub_folders[0], "fb", "sub")
+
+    def test_a_folder_is_moved_without_the_recursive_flag(self):
+        client, folders, _ = self.make_tree("folder")
+        with patch("mdrsclient.client.FoldersApi") as folders_api:
+            client.mv("myremote:/mylab/a/sub", "myremote:/mylab/b/sub")
+            folders_api.return_value.move.assert_called_once_with(folders["/a"].sub_folders[0], "fb", "sub")
+
+    def test_a_destination_that_already_holds_the_name_is_refused(self):
+        sub = make_sub_folder("fsub", "sub")
+        folders = {"/a": make_folder("fa", "a", "/a/", [sub]), "/b": make_folder("fb", "b", "/b/")}
+        files = {"fa": [], "fb": [make_file("x2", "sub")]}
+        client = self.make_client(folders, files)
+        with self.assertRaises(IllegalArgumentException) as caught:
+            client.mv("myremote:/mylab/a/sub", "myremote:/mylab/b/sub")
+        self.assertIn("Cannot overwrite non-folder", str(caught.exception))
+
+    def test_a_transfer_across_laboratories_is_refused(self):
+        client, _, _ = self.make_tree("file")
+        with self.assertRaises(IllegalArgumentException) as caught:
+            client.cp("myremote:/mylab/a/data.txt", "myremote:/otherlab/b/data.txt")
+        self.assertIn("Laboratory mismatched", str(caught.exception))
+
+    def test_a_source_that_does_not_exist_is_refused(self):
+        client, _, _ = self.make_tree("file")
+        with self.assertRaises(IllegalArgumentException) as caught:
+            client.mv("myremote:/mylab/a/missing.txt", "myremote:/mylab/b/missing.txt")
+        self.assertIn("not found", str(caught.exception))
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_services.py

@@ -0,0 +1,55 @@
+import unittest
+from unittest.mock import MagicMock, patch
+
+from mdrsclient.models import Doi, Folder, Laboratories, Laboratory
+from mdrsclient.services import MdrsService
+
+TIMESTAMP = "2026-01-01T00:00:00+09:00"
+DOI = "10.60178/cbs.20260429-001"
+LABORATORY = Laboratory(id=1, name="mylab", pi_name="PI", full_name="My Laboratory")
+
+
+def make_locked_folder() -> Folder:
+    return Folder(
+        id="f1",
+        pid=None,
+        name="root",
+        access_level=4,
+        lock=True,
+        size=0,
+        laboratory_id=1,
+        description="",
+        created_at=TIMESTAMP,
+        updated_at=TIMESTAMP,
+        restrict_opened_at=None,
+        metadata=[],
+        sub_folders=[],
+        path="/root/",
+    )
+
+
+class TestFindFolderByDoi(unittest.TestCase):
+    """The DOI response carries the folder id as a field, not as a nested object."""
+
+    def test_a_locked_doi_folder_is_unlocked_with_its_folder_id(self):
+        laboratories = Laboratories()
+        laboratories.append(LABORATORY)
+
+        with (
+            patch("mdrsclient.services.DoiApi") as doi_api,
+            patch("mdrsclient.services.FoldersApi") as folders_api,
+            patch("mdrsclient.services.LaboratoriesApi") as laboratories_api,
+        ):
+            doi_api.return_value.retrieve.return_value = Doi(id="20260429-001", doi=DOI, folder_id="f1")
+            folders_api.return_value.retrieve.return_value = make_locked_folder()
+            laboratories_api.return_value.list.return_value = laboratories
+
+            folder, laboratory = MdrsService(MagicMock()).find_folder_by_doi(DOI, "secret")
+
+            folders_api.return_value.auth.assert_called_once_with("f1", "secret")
+            self.assertEqual(folder.id, "f1")
+            self.assertEqual(laboratory.name, "mylab")
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_token_cache.py

@@ -0,0 +1,198 @@
+import tempfile
+import time
+import unittest
+from unittest.mock import patch
+
+import jwt
+
+from mdrsclient.api.utils import retry_on_expired_token, token_check, token_recover
+from mdrsclient.cache import CacheFile
+from mdrsclient.config import normalize_url
+from mdrsclient.connection import MDRSConnection
+from mdrsclient.exceptions import IllegalArgumentException, ServerBusyException, TokenExpiredException
+from mdrsclient.models import Token
+
+REMOTE = "unittest"
+
+
+def make_token(access_offset: int, refresh_offset: int, label: str) -> Token:
+    now = int(time.time())
+
+    def encode(token_type: str, offset: int) -> str:
+        return jwt.encode(
+            {
+                "token_type": token_type,
+                "exp": now + offset,
+                "iat": now,
+                "jti": f"{label}-{token_type}",
+                "user_id": 1,
+            },
+            "unittest-signing-key-not-verified-anywhere",
+        )
+
+    return Token(access=encode("access", access_offset), refresh=encode("refresh", refresh_offset))
+
+
+class TestTokenCache(unittest.TestCase):
+    def setUp(self) -> None:
+        self.tmp = tempfile.TemporaryDirectory()
+        patcher = patch("mdrsclient.cache.CONFIG_DIRNAME", self.tmp.name)
+        patcher.start()
+        self.addCleanup(patcher.stop)
+        self.addCleanup(self.tmp.cleanup)
+
+    def test_reload_picks_up_a_write_from_another_holder(self) -> None:
+        """The cache is shared, so a value read earlier can already be out of date."""
+        reader = CacheFile(REMOTE)
+        writer = CacheFile(REMOTE)
+
+        first = make_token(3600, 86400, "first")
+        writer.token = first
+        self.assertEqual(reader.token, first)
+
+        second = make_token(3600, 86400, "second")
+        writer.token = second
+
+        reader.reload()
+        self.assertEqual(reader.token, second)
+
+    def test_token_check_uses_the_token_another_holder_just_wrote(self) -> None:
+        """A rotating provider drops the token it replaces, so the refresh must not
+        be sent again once someone else has already made the round trip."""
+        connection = MDRSConnection(REMOTE, "http://localhost:8000/api/")
+        connection.token = make_token(-60, 86400, "stale")
+
+        # Another process refreshes while this one is between requests.
+        other = CacheFile(REMOTE)
+        rotated = make_token(3600, 86400, "rotated")
+        other.token = rotated
+
+        with patch("mdrsclient.api.utils.UsersApi") as users_api:
+            token_check(connection)
+
+        users_api.assert_not_called()
+        self.assertEqual(connection.token, rotated)
+
+    def test_token_check_refreshes_when_nothing_else_has(self) -> None:
+        connection = MDRSConnection(REMOTE, "http://localhost:8000/api/")
+        connection.token = make_token(-60, 86400, "stale")
+        rotated = make_token(3600, 86400, "rotated")
+
+        with patch("mdrsclient.api.utils.UsersApi") as users_api:
+            users_api.return_value.tokenRefresh.return_value = rotated
+            token_check(connection)
+
+        users_api.return_value.tokenRefresh.assert_called_once()
+        self.assertEqual(connection.token, rotated)
+
+    def test_token_check_logs_out_once_the_refresh_token_expires(self) -> None:
+        connection = MDRSConnection(REMOTE, "http://localhost:8000/api/")
+        connection.token = make_token(-3600, -60, "dead")
+
+        with patch("mdrsclient.api.utils.UsersApi") as users_api:
+            token_check(connection)
+
+        users_api.assert_not_called()
+        self.assertIsNone(connection.token)
+
+
+class TestUrlNormalization(unittest.TestCase):
+    """Both clients share config.ini, so they have to agree on what a remote URL is."""
+
+    def test_trailing_slash_is_dropped(self) -> None:
+        self.assertEqual(normalize_url("http://127.0.0.1:8000/api/"), "http://127.0.0.1:8000/api")
+        self.assertEqual(normalize_url("https://neurodata.riken.jp/api/"), "https://neurodata.riken.jp/api")
+
+    def test_bare_hostname_is_accepted(self) -> None:
+        self.assertEqual(normalize_url("http://localhost:8000/api"), "http://localhost:8000/api")
+
+    def test_only_http_schemes_are_accepted(self) -> None:
+        for url in ("ftp://x.example.com/", "file:///etc/passwd", "not-a-url", "http://"):
+            with self.subTest(url=url), self.assertRaises(IllegalArgumentException):
+                normalize_url(url)
+
+
+class TestExpiredTokenRetry(unittest.TestCase):
+    """A request can wait in the server's queue for longer than its access token lives."""
+
+    def setUp(self) -> None:
+        self.tmp = tempfile.TemporaryDirectory()
+        patcher = patch("mdrsclient.cache.CONFIG_DIRNAME", self.tmp.name)
+        patcher.start()
+        self.addCleanup(patcher.stop)
+        self.addCleanup(self.tmp.cleanup)
+        self.connection = MDRSConnection(REMOTE, "http://localhost:8000/api")
+        self.connection.token = make_token(3600, 86400, "sent")
+
+    def test_a_token_someone_else_refreshed_is_reused_rather_than_replaced(self) -> None:
+        used = self.connection.token
+        rotated = make_token(3600, 86400, "rotated")
+        CacheFile(REMOTE).token = rotated  # another process, while the request waited
+
+        with patch("mdrsclient.api.utils.UsersApi") as users_api:
+            recovered = token_recover(self.connection, used)
+
+        self.assertTrue(recovered)
+        users_api.assert_not_called()
+        self.assertEqual(self.connection.token, rotated)
+        # The retry carries whatever the session header says, not what the cache holds,
+        # so reading the new token through is not enough on its own.
+        self.assertEqual(self.connection.session.headers["Authorization"], f"Bearer {rotated.access}")
+
+    def test_the_token_is_refreshed_when_nobody_else_has(self) -> None:
+        used = self.connection.token
+        rotated = make_token(3600, 86400, "rotated")
+
+        with patch("mdrsclient.api.utils.UsersApi") as users_api:
+            users_api.return_value.tokenRefresh.return_value = rotated
+            recovered = token_recover(self.connection, used)
+
+        self.assertTrue(recovered)
+        users_api.return_value.tokenRefresh.assert_called_once()
+        self.assertEqual(self.connection.token, rotated)
+
+    def test_the_request_is_sent_once_more_and_only_once(self) -> None:
+        calls: list[str] = []
+
+        class Api:
+            def __init__(self, connection: MDRSConnection) -> None:
+                self.connection = connection
+
+            @retry_on_expired_token
+            def send(self) -> str:
+                calls.append("sent")
+                if len(calls) == 1:
+                    raise TokenExpiredException("expired on arrival")
+                return "ok"
+
+        with patch("mdrsclient.api.utils.UsersApi") as users_api:
+            users_api.return_value.tokenRefresh.return_value = make_token(3600, 86400, "rotated")
+            self.assertEqual(Api(self.connection).send(), "ok")
+
+        self.assertEqual(len(calls), 2)
+
+    def test_a_second_refusal_is_reported_as_an_overloaded_server(self) -> None:
+        calls: list[str] = []
+
+        class Api:
+            def __init__(self, connection: MDRSConnection) -> None:
+                self.connection = connection
+
+            @retry_on_expired_token
+            def send(self) -> str:
+                calls.append("sent")
+                raise TokenExpiredException("expired on arrival")
+
+        with patch("mdrsclient.api.utils.UsersApi") as users_api:
+            users_api.return_value.tokenRefresh.return_value = make_token(3600, 86400, "rotated")
+            with self.assertRaises(ServerBusyException) as caught:
+                Api(self.connection).send()
+
+        # Sent twice and no more, and the message names the cause the user can act on
+        # rather than an authentication failure that would send them to the login form.
+        self.assertEqual(len(calls), 2)
+        self.assertIn("overloaded", str(caught.exception))
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_transfer.py

@@ -0,0 +1,297 @@
+import os
+import tempfile
+import unittest
+from io import StringIO
+from unittest.mock import MagicMock, patch
+
+from mdrsclient.api import FilesApi
+from mdrsclient.exceptions import MDRSException, UnexpectedException
+from mdrsclient.models import File, Folder, FolderSimple, Laboratory
+from mdrsclient.transfer import Downloader, Uploader
+
+TIMESTAMP = "2026-01-01T00:00:00+09:00"
+
+
+def make_file(id: str, name: str, size: int = 1) -> File:
+    return File(
+        id=id,
+        name=name,
+        type="text/plain",
+        size=size,
+        thumbnail=None,
+        description="",
+        metadata={},
+        download_url=f"v3/files/{id}/download/",
+        created_at=TIMESTAMP,
+        updated_at=TIMESTAMP,
+    )
+
+
+def make_folder_simple(id: str, name: str) -> FolderSimple:
+    return FolderSimple(
+        id=id,
+        pid=None,
+        name=name,
+        access_level=1,
+        lock=False,
+        size=0,
+        laboratory_id=1,
+        description="",
+        created_at=TIMESTAMP,
+        updated_at=TIMESTAMP,
+        restrict_opened_at=None,
+    )
+
+
+def make_folder(id: str, name: str, path: str, sub_folders: list[FolderSimple] | None = None) -> Folder:
+    return Folder(
+        id=id,
+        pid=None,
+        name=name,
+        access_level=1,
+        lock=False,
+        size=0,
+        laboratory_id=1,
+        description="",
+        created_at=TIMESTAMP,
+        updated_at=TIMESTAMP,
+        restrict_opened_at=None,
+        metadata=[],
+        sub_folders=sub_folders if sub_folders is not None else [],
+        path=path,
+    )
+
+
+LABORATORY = Laboratory(id=1, name="mylab", pi_name="PI", full_name="My Laboratory")
+
+
+class TestUploadReportsFailure(unittest.TestCase):
+    """A file that never reached the server must not leave the command reporting success."""
+
+    def make_client(self, folder: Folder, files: list[File]) -> MagicMock:
+        client = MagicMock()
+        client.parse_remote_host_with_path.return_value = ("myremote", "mylab", "/")
+        client.find_laboratory.return_value = LABORATORY
+        client.find_folder.return_value = folder
+        client.find_files.return_value = files
+        return client
+
+    def test_a_refused_upload_is_raised_to_the_caller(self):
+        client = self.make_client(make_folder("f1", "root", "/"), [])
+        with tempfile.TemporaryDirectory() as tmp:
+            local = os.path.join(tmp, "data.txt")
+            with open(local, "w") as f:
+                f.write("x")
+            with patch("mdrsclient.transfer.FilesApi") as files_api_class:
+                files_api_class.return_value.create.side_effect = MDRSException("Access is denied.")
+                with self.assertRaises(UnexpectedException):
+                    Uploader(client).upload(local, "myremote:/mylab/", False, False)
+
+    def test_an_error_the_api_layer_did_not_raise_is_still_counted(self):
+        """A file that vanished between the walk and the upload must not end the run."""
+        client = self.make_client(make_folder("f1", "root", "/"), [])
+        with tempfile.TemporaryDirectory() as tmp:
+            local = os.path.join(tmp, "data.txt")
+            with open(local, "w") as f:
+                f.write("x")
+            with patch("mdrsclient.transfer.FilesApi") as files_api_class:
+                files_api_class.return_value.create.side_effect = FileNotFoundError(local)
+                with self.assertRaises(UnexpectedException):
+                    Uploader(client).upload(local, "myremote:/mylab/", False, False)
+
+    def test_a_failed_upload_names_the_file(self):
+        client = self.make_client(make_folder("f1", "root", "/"), [])
+        with tempfile.TemporaryDirectory() as tmp:
+            local = os.path.join(tmp, "data.txt")
+            with open(local, "w") as f:
+                f.write("x")
+            with patch("mdrsclient.transfer.FilesApi") as files_api_class:
+                files_api_class.return_value.create.side_effect = MDRSException("Access is denied.")
+                with patch("sys.stdout", new=StringIO()) as fake_out:
+                    with self.assertRaises(UnexpectedException):
+                        Uploader(client).upload(local, "myremote:/mylab/", False, False)
+                    self.assertIn(local, fake_out.getvalue())
+
+    def test_an_upload_that_worked_stays_quiet(self):
+        client = self.make_client(make_folder("f1", "root", "/"), [])
+        with tempfile.TemporaryDirectory() as tmp:
+            local = os.path.join(tmp, "data.txt")
+            with open(local, "w") as f:
+                f.write("x")
+            with patch("mdrsclient.transfer.FilesApi") as files_api_class:
+                files_api_class.return_value.create.return_value = "new-id"
+                Uploader(client).upload(local, "myremote:/mylab/", False, False)
+                files_api_class.return_value.create.assert_called_once()
+
+
+class TestDownloadReportsFailure(unittest.TestCase):
+    def make_client(self, remote_path_parts, folder: Folder, files_by_folder: dict[str, list[File]]) -> MagicMock:
+        client = MagicMock()
+        client.is_doi.return_value = False
+        client.parse_remote_host_with_path.return_value = remote_path_parts
+        client.find_laboratory.return_value = LABORATORY
+        client.find_folder.return_value = folder
+        client.find_files.side_effect = lambda folder_id: files_by_folder.get(folder_id, [])
+        return client
+
+    def test_a_single_file_that_failed_is_raised_to_the_caller(self):
+        parent = make_folder("p1", "root", "/")
+        client = self.make_client(("myremote", "mylab", "/data.txt"), parent, {"p1": [make_file("x1", "data.txt")]})
+        with tempfile.TemporaryDirectory() as tmp:
+            with patch("mdrsclient.transfer.FilesApi") as files_api_class:
+                files_api_class.return_value.download.side_effect = OSError("Permission denied.")
+                with self.assertRaises(UnexpectedException):
+                    Downloader(client).download("myremote:/mylab/data.txt", tmp)
+
+    def test_a_single_file_that_arrived_stays_quiet(self):
+        parent = make_folder("p1", "root", "/")
+        client = self.make_client(("myremote", "mylab", "/data.txt"), parent, {"p1": [make_file("x1", "data.txt")]})
+        with tempfile.TemporaryDirectory() as tmp:
+            with patch("mdrsclient.transfer.FilesApi") as files_api_class:
+                files_api_class.return_value.download.return_value = True
+                Downloader(client).download("myremote:/mylab/data.txt", tmp)
+                files_api_class.return_value.download.assert_called_once()
+
+    def test_a_failed_file_does_not_abandon_the_remaining_sub_folders(self):
+        parent = make_folder("p1", "lab", "/", [make_folder_simple("f1", "root")])
+        folders = {
+            "f1": make_folder("f1", "root", "/root/", [make_folder_simple("fa", "a"), make_folder_simple("fb", "b")]),
+            "fa": make_folder("fa", "a", "/root/a/"),
+            "fb": make_folder("fb", "b", "/root/b/"),
+        }
+        files_by_folder = {
+            "p1": [],
+            "f1": [make_file("bad", "bad.txt")],
+            "fa": [make_file("good1", "good1.txt")],
+            "fb": [make_file("good2", "good2.txt")],
+        }
+        client = self.make_client(("myremote", "mylab", "/root"), parent, files_by_folder)
+        attempted: list[str] = []
+
+        def download(file: File, path: str) -> bool:
+            attempted.append(file.name)
+            if file.name == "bad.txt":
+                raise OSError("Permission denied.")
+            return True
+
+        with tempfile.TemporaryDirectory() as tmp:
+            with (
+                patch("mdrsclient.transfer.FoldersApi") as folders_api_class,
+                patch("mdrsclient.transfer.FilesApi") as files_api_class,
+            ):
+                folders_api_class.return_value.retrieve.side_effect = lambda folder_id: folders[folder_id]
+                files_api_class.return_value.download.side_effect = download
+                with self.assertRaises(UnexpectedException):
+                    Downloader(client).download("myremote:/mylab/root", tmp, is_recursive=True)
+
+        self.assertEqual(sorted(attempted), ["bad.txt", "good1.txt", "good2.txt"])
+
+
+class TestRecursiveDownloadResilience(unittest.TestCase):
+    """A folder the client cannot prepare locally must not abandon its siblings."""
+
+    def test_a_folder_that_cannot_be_created_does_not_stop_the_walk(self):
+        parent = make_folder("p1", "lab", "/", [make_folder_simple("f1", "root")])
+        folders = {
+            "f1": make_folder("f1", "root", "/root/", [make_folder_simple("fa", "a"), make_folder_simple("fb", "b")]),
+            "fa": make_folder("fa", "a", "/root/a/"),
+            "fb": make_folder("fb", "b", "/root/b/"),
+        }
+        files_by_folder = {"p1": [], "f1": [], "fa": [make_file("good1", "good1.txt")], "fb": []}
+        client = MagicMock()
+        client.is_doi.return_value = False
+        client.parse_remote_host_with_path.return_value = ("myremote", "mylab", "/root")
+        client.find_laboratory.return_value = LABORATORY
+        client.find_folder.return_value = parent
+        client.find_files.side_effect = lambda folder_id: files_by_folder.get(folder_id, [])
+
+        real_makedirs = os.makedirs
+
+        def makedirs(path, *args, **kwargs):
+            if os.path.basename(path) == "b":
+                raise PermissionError("Permission denied")
+            return real_makedirs(path, *args, **kwargs)
+
+        attempted: list[str] = []
+
+        with tempfile.TemporaryDirectory() as tmp:
+            with (
+                patch("mdrsclient.transfer.FoldersApi") as folders_api_class,
+                patch("mdrsclient.transfer.FilesApi") as files_api_class,
+                patch("mdrsclient.transfer.os.makedirs", side_effect=makedirs),
+            ):
+                folders_api_class.return_value.retrieve.side_effect = lambda folder_id: folders[folder_id]
+                files_api_class.return_value.download.side_effect = lambda file, path: attempted.append(file.name)
+                with self.assertRaises(UnexpectedException):
+                    Downloader(client).download("myremote:/mylab/root", tmp, is_recursive=True)
+
+        self.assertEqual(attempted, ["good1.txt"])
+
+
+class TestFileDownloadPermission(unittest.TestCase):
+    """A file the client could not write is a failure, not a line of successful output."""
+
+    def make_connection(self, chunks) -> MagicMock:
+        connection = MagicMock()
+        connection.token = None
+        response = MagicMock()
+        response.status_code = 200
+        response.iter_content.return_value = chunks
+        connection.get.return_value = response
+        return connection
+
+    def test_a_download_that_cannot_be_written_is_reported(self):
+        connection = self.make_connection([b"payload"])
+        with tempfile.TemporaryDirectory() as tmp:
+            path = os.path.join(tmp, "precious.dat")
+            with open(path, "wb") as f:
+                f.write(b"do not touch")
+            os.chmod(path, 0o444)
+            try:
+                with open(path, "r+b"):
+                    self.skipTest("running with rights that ignore the file mode")
+            except OSError:
+                pass
+
+            with self.assertRaises(UnexpectedException):
+                FilesApi(connection).download(make_file("x1", "precious.dat"), path)
+
+            os.chmod(path, 0o644)
+            with open(path, "rb") as f:
+                self.assertEqual(f.read(), b"do not touch")
+            connection.get.assert_not_called()
+
+    def test_an_interrupted_download_leaves_the_existing_file_intact(self):
+        def chunks():
+            yield b"half a file"
+            raise ConnectionError("connection reset")
+
+        connection = self.make_connection(chunks())
+        with tempfile.TemporaryDirectory() as tmp:
+            path = os.path.join(tmp, "existing.dat")
+            with open(path, "wb") as f:
+                f.write(b"the copy already here")
+
+            with self.assertRaises(ConnectionError):
+                FilesApi(connection).download(make_file("x1", "existing.dat"), path)
+
+            with open(path, "rb") as f:
+                self.assertEqual(f.read(), b"the copy already here")
+            self.assertEqual(os.listdir(tmp), ["existing.dat"])
+
+    def test_a_finished_download_replaces_the_destination(self):
+        connection = self.make_connection([b"new ", b"contents"])
+        with tempfile.TemporaryDirectory() as tmp:
+            path = os.path.join(tmp, "existing.dat")
+            with open(path, "wb") as f:
+                f.write(b"old")
+
+            FilesApi(connection).download(make_file("x1", "existing.dat"), path)
+
+            with open(path, "rb") as f:
+                self.assertEqual(f.read(), b"new contents")
+            self.assertEqual(os.listdir(tmp), ["existing.dat"])
+
+
+if __name__ == "__main__":
+    unittest.main()