Compare commits

..
21 Commits
Author SHA1 Message Date
orrisroot 0423913b3d docs: release 1.3.19 and document the environment variables
- Close the Unreleased section as 1.3.19, summarising it by what
  changed for the user rather than by what each commit touched.
- Add the environment variables the settings module reads, which the
  README did not mention.
2026-09-04 18:23:25 +09:00
orrisroot d60221de21 build: bump to 1.3.19 and update dependencies
- Raise the floors for pydantic, pydantic-settings, python-dotenv and
  isort to the releases they are tested against.
- Move the flit build backend to 4.x.
2026-09-04 18:23:24 +09:00
orrisroot 4c9954c1fd fix: clear the type checker and linter findings
flake8 reported 49 findings and pyright 10, and two of them were real
bugs rather than matters of style. The rest were unused imports and
four functions over the complexity limit.

- Unlock a DOI folder with the id the DOI response carries, rather
  than an attribute the model does not have, which raised
  `AttributeError` on every locked DOI folder.
- Size the `ls` Size column from the sub-folder on the row rather than
  from its parent, which pushed the later columns out of line.
- Share the path resolution and the destination checks between `cp`
  and `mv`, and split the recursive download and the `ls` row
  printing, bringing all four functions under the complexity limit.
- Accept a client built without a connection, which `config` and
  `version` rely on, and report the reason if one is then asked for.
- Declare the config protocol's constructor for the type checker
  alone, so the protocol keeps its guard against being instantiated.
- Remove 41 unused imports, and let flake8 accept black's spacing.
- Point pyright at the project's own environment, without which it
  resolved no dependency and reported 124 findings that were not real.
- Cover the two fixes and the shared `cp`/`mv` paths with new tests.
2026-09-04 18:23:17 +09:00
orrisroot 914dd729aa fix: report failed transfers instead of ending in success
Upload and download failures were printed and then forgotten: a batch
that lost files still exited 0, and a file the client could not write
was listed as if it had arrived. Nothing downstream could tell.

- Return a verdict from every transfer worker and raise once at the
  end, so a run that lost a file exits 2.
- Raise the error when a downloaded file cannot be written locally,
  instead of printing it and reporting the path as a success.
- Write a download beside its destination and move it into place once
  complete, and refuse a destination that cannot be written, so a
  failed transfer no longer leaves a truncated file behind.
- Carry on through the remaining sub-folders when a recursive download
  loses a file or cannot create a folder locally.
- Name the file and give the reason in every failure message.
- Count a failure the API layer did not raise, such as a file that
  disappeared between the directory walk and its turn to be sent.
- Extract the directory walk from `Uploader.upload` to keep it within
  the complexity limit.
- Cover all of the above in `tests/test_transfer.py`.
2026-09-04 16:31:24 +09:00
orrisroot e5c28835b8 feat(auth): send a request again when its token lapsed in the queue
A request can wait to be served for longer than the access token it was
sent with lives, and comes back refused for a token that was valid when
it left. Uploads that take minutes make that wait ordinary.

- tell an expired token apart from any other refusal by the code the
  API returns, and retry only that one, once
- prefer a token another process left behind over minting a second: it
  saves a round trip, and a rotating provider would retire one that is
  still in use. Assign it rather than read it, since the setter is what
  rewrites the session header the retry will carry
- settle the token before the request so the one compared afterwards is
  the one that was actually sent
- report an overloaded server when the second attempt is refused too,
  and let API errors out of the upload path with their own type
2026-08-14 18:37:41 +09:00
orrisroot 0cac30ccf8 fix(auth): serialise the token refresh across processes
The login cache is shared by every mdrs process, but the refresh was
guarded by a lock that only reaches inside one. Concurrent runs each
sent the same refresh token, and a provider that rotates them accepts
the first and refuses the rest.

- hold a lock that spans processes across the whole read-refresh-
  write, checking cheaply first so ordinary requests never take it
- write the cache through a temporary file: opening it for writing
  truncates it, and a reader landing in that window found it empty
  and threw the session away
- take the lock for every write, not just the refresh, so a login
  running beside one cannot be silently reverted
- wait for a busy lock on Windows rather than giving up after the ten
  attempts msvcrt allows, and retry the rename it refuses while a
  reader still holds the file open
- bound the refresh request on its own, so a provider that goes quiet
  cannot hold the lock indefinitely
- accept bare hostnames such as localhost, store URLs without the
  trailing slash, and join download paths through one helper
2026-08-14 16:36:50 +09:00
orrisroot 1a4023ba47 refactor: migrate package management from poetry to pep 621 standard
Migrate package metadata and dependency configuration from Poetry to
PEP 621 standard using flit_core backend. Update documentation with
production and development setup instructions using venv, pip, or uv.

- update pyproject.toml to PEP 621 format with flit_core build-system
- bump minimum dependency versions to latest PyPI releases
- revise README.md with production and development setup guides
2026-07-25 19:41:17 +09:00
orrisroot 8e03f7a7f4 docs(changelog): update older release histories with actual changes
Historically, several versions in the changelog only had generic
version bump messages. This update fills in the missing details of
actual features and bug fixes by referencing the Git commit log.

- Add specific change logs for versions v1.3.3 through v1.3.15
- Include missing details for features like new command-line options
- Document bug fixes for API pagination, normalization, and downloads
2026-07-03 01:29:32 +09:00
orrisroot d59a150b4f chore(release): bump version to 1.3.18
Bump the package version to 1.3.18, upgrade dependencies,
consolidate module exports, add a unit test suite, and document
all changes.

- Bump package version to 1.3.18 in pyproject.toml
- Upgrade pydantic-settings to 2.14.2 and pyright to 1.1.411
- Consolidate package exports in mdrsclient/__init__.py
- Add a comprehensive unit test suite in tests/test_commands.py
- Document testing execution and add full history in CHANGELOG.md
2026-07-02 23:47:57 +09:00
orrisroot 7f6d496654 docs(readme): document custom configuration usage in library API
Update the Python API Usage section in README.md to demonstrate how to
initialize MdrsClient with InMemoryConfig to avoid creating local
config files.
2026-07-02 23:31:08 +09:00
orrisroot b95fc0cd7d refactor(config): abstract config storage and enable dependency injection
Abstract the configuration storage mechanism to allow using custom
configurations, such as in-memory setups, when using the tool as
a library. This aligns the configuration architecture with the
session cache abstraction.

- Define ConfigInterface protocol and InMemoryConfig class
- Make CacheFile, InMemoryCache, ConfigFile, and InMemoryConfig
  explicitly inherit their interfaces
- Update MdrsService and MdrsClient to accept customizable
  config_class and config instances
- Add validation to check remote parameter consistency in
  create_connection
- Remove unused imports across command files
2026-07-02 23:30:33 +09:00
orrisroot 8ce9e09e69 refactor: use services layer and modularize transfer operations
Decouple CLI commands from internal helper logic and consolidate the
core file transfer operations in the service layer to improve library
portability.

- Make MdrsClient subclass MdrsService to inherit resource resolution.
- Remove all deprecated helper methods from BaseCommand.
- Move core upload and download logic to a new transfer module.
- Refactor all CLI commands to route actions through MdrsClient.
- Eliminate circular imports between client and CLI command modules.
2026-07-02 23:16:53 +09:00
orrisroot 36cad6db52 refactor: extract MdrsClient service layer for library portability
To improve the tool's portability as a Python library, the core logic
has been decoupled from the CLI interface. This allows developers to
programmatically interact with MDRS without relying on CLI-specific
argument parsing or local file-based caches.

- Introduce `MdrsClient` service layer to handle core operations.
- Abstract authentication state using `CacheInterface` and `InMemoryCache`.
- Migrate all CLI commands to utilize `MdrsClient` for execution.
- Separate `Doi` data model from API responses and move to `models/doi.py`.
- Update `README.md` to include Python API usage examples.
- Bump package version to 1.3.17.
2026-07-02 13:07:18 +09:00
orrisroot 809140dfbc fix(download): retrieve full folder to resolve type error
Retrieve the full Folder object from FoldersApi instead of using the
FolderSimple returned by find_sub_folder when resolving DOI subfolders.
This fixes a type checker error under the upgraded pyright, and avoids
a potential AttributeError at runtime due to FolderSimple lacking the
path attribute.

Also upgrade dependencies and bump version to 1.3.16 in pyproject.toml.
2026-06-12 10:25:03 +09:00
orrisroot 5bdf837941 feat(doi): add DOI-based path access for commands
Support accessing repositories using DOI strings with optional subpaths
across ls, download, metadata, and file-metadata commands.

- Implement GET v3/doi/{id}/ API model and client calls
- Parse and resolve DOI paths into respective folder and files
- Extract common folder and file resolution logic to shared helpers
- Update README with example DOI-based shell commands
2026-06-12 01:28:40 +09:00
orrisroot 04c0003a61 fix(connection): support absolute download urls for public data 2026-06-11 21:04:28 +09:00
orrisrootandCopilot 428be1289c chore(version): bump package version to 1.3.15
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-01 11:11:22 +09:00
orrisrootandCopilot 4283481695 fix: apply NFC normalization to filenames and folder names sent to server
On macOS, local filenames and directory names may be in NFD encoding
(decomposed Unicode). Without normalization, files and folders are
created on the server with NFD names, inconsistent with the server's
NFC convention.

Apply normalize("NFC", ...) before sending names to the server in:
- FilesApi.create(): filename in multipart upload
- FilesApi.update(): filename in multipart upload
- UploadCommand: directory name in FoldersApi.create()

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-20 12:25:59 +09:00
orrisrootandCopilot ddb4300d85 feat(config): simplify list command and add subcommand aliases
- config list: remove -l/--long option, always display URL
- config list: add ls alias (already existed, kept)
- config delete: add rm alias (alongside existing remove alias)
- README: add config update, config list, config delete sections

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-17 18:59:07 +09:00
orrisrootandCopilot 68670a6588 fix(ls): rename --quick to --quiet; add version command; bump to 1.3.14
- Fix ls -q long option name: --quick → --quiet (typo fix)
- Remove mdrsclient/VERSION file; read version via importlib.metadata
- Bump version 1.3.13 → 1.3.14
- Add Python 3.14 to supported classifiers; promote to Development Status 4 - Beta
- Add `version` subcommand (prints "mdrs <version>")
- Document `version` command in README

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-17 17:41:37 +09:00
orrisroot 6d8fd0a598 update version to 1.3.13 2025-10-28 11:10:09 +09:00
49 changed files with 3058 additions and 701 deletions
+157
View File
@@ -0,0 +1,157 @@
# Changelog
All notable changes to this project will be documented in this file.
## [1.3.19] - 2026-09-04
### Fixed
- Reported failed transfers instead of ending in success, so a run that lost files exits non-zero.
- Left an existing local file intact when a download fails part way through.
- Opened password-protected DOI folders, which previously failed outright.
- Kept a session working across concurrent runs and long server queues.
- Other minor fixes to `ls` output and remote URL handling.
### Changed
- Moved packaging to PEP 621 metadata with flit as the build backend.
- Updated dependencies.
- **Breaking for embedders:** `CacheInterface` now requires `lock()` and `reload()`.
## [1.3.18] - 2026-07-02
### Added
- Added a comprehensive unit test suite in `tests/test_commands.py` checking registration, parsing, and execution flow of all 16 commands.
### Refactored
- Abstracted configuration storage (introducing `ConfigInterface`, `InMemoryConfig`, and updating `ConfigFile`) to enable dependency injection.
- Modularized transfer operations (upload and download) to decouple them from the service layer.
- Decoupled commands from direct file system configurations and migrated all subcommands to use abstract config classes.
### Changed
- Upgraded dependencies including `pydantic-settings` to `2.14.2` and `pyright` to `1.1.411`.
### Fixed
- Fixed duplicate `__all__` definitions in package initialization file `mdrsclient/__init__.py` that caused `__version__` export to be overwritten.
## [1.3.17] - 2026-07-02
### Refactored
- Decoupled core logic from CLI interface and introduced `MdrsClient` service layer to improve library portability.
- Migrated all CLI commands to utilize `MdrsClient` for execution.
### Added
- Abstract authentication state using `CacheInterface` and `InMemoryCache`.
## [1.3.16] - 2026-06-12
### Fixed
- Retrieve the full `Folder` object from `FoldersApi` instead of using the `FolderSimple` returned by `find_sub_folder` when resolving DOI subfolders. This fixes a type checker error under the upgraded pyright and avoids a potential AttributeError at runtime due to `FolderSimple` lacking the `path` attribute.
### Changed
- Upgraded dependencies and bumped version to 1.3.16 in pyproject.toml.
## [1.3.15] - 2026-05-01
### Fixed
- Apply NFC normalization to filenames and folder names sent to the server.
## [1.3.14] - 2026-04-17
### Changed
- Simplified `config list` command (removed `-l`/`--long` option, always display URL).
- Renamed `--quick` option to `--quiet` for `ls` subcommand.
### Added
- Added subcommand aliases for config commands (e.g. `ls` alias for list, `rm` alias for delete).
- Added `version` command.
## [1.3.13] - 2025-07-02
### Fixed
- Fixed pagination logic for the `file.list` API.
## [1.3.12] - 2025-05-20
### Fixed
- Fixed bug where file downloading was skipped incorrectly when `-s`/`--skip-if-file-exists` option was present.
## [1.3.11] - 2025-01-21
### Fixed
- Follow-up fixes for User API specification changes.
## [1.3.10] - 2024-12-23
### Added
- Delete broken files and show a summary when a file download fails.
### Changed
- Updated dependency libraries.
## [1.3.9] - 2024-10-23
### Fixed
- Fixed compatibility with Python 3.10.
## [1.3.8] - 2024-09-18
### Added
- Implemented `-s`/`--skip-if-file-exists` option for `download` command.
### Fixed
- Added exception handling for unexpected responses from the server.
## [1.3.7] - 2024-07-22
### Added
- Implemented `--exclude` argument for download subcommand.
## [1.3.6] - 2024-07-08
### Added
- Support cancelling recursive downloads if downloading some files fails.
## [1.3.5] - 2024-07-08
### Added
- Added authorization token validation checks for file download operations.
### Removed
- Removed unnecessary debug code.
## [1.3.4] - 2024-07-04
### Added
- Added some aliases for config sub command.
### Fixed
- Fixed bug when uploading large files.
## [1.3.3] - 2024-02-13
### Added
- Implemented `-s`/`--skip-if-file-exists` option for `upload` command.
## [1.3.2] - 2024-02-09
### Added
- Added `-u` and `-p` options to login command.
## [1.3.1] - 2023-12-20
### Fixed
- Fixed bug to resolve local files for recursive file upload.
## [1.3.0] - 2023-12-18
### Changed
- Removed debug comments.
## [1.2.0] - 2023-10-04
### Changed
- Follow-up recent specification changes about folder access level.
## [1.1.1] - 2023-07-26
### Changed
- Set destination folder name using name attribute of folder copy API.
+139 -7
View File
@@ -2,13 +2,54 @@
The mdrs-client-python is python library and a command-line client for up- and downloading files to and from MDRS based repository.
## Installing
## Installation (Production)
It is recommended to use a virtual environment (`venv`) to keep your Python environment isolated.
### Option 1: Using standard `venv` + `pip`
```shell
poetry install
python3 -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install .
```
## Example Usage
### Option 2: Using `uv`
```shell
uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
uv pip install .
```
## Development Setup
To set up a local development environment with development tools (testing, formatting, linting):
### Option 1: Using standard `venv` + `pip`
```shell
python3 -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install -e ".[dev]"
```
### Option 2: Using `uv` (Recommended for developers)
```shell
uv sync
```
## Configuration
The following environment variables can be set, either in the shell or via a `.env` file placed in the working directory.
| Variable | Default | Description |
|---|---|---|
| `MDRS_CLIENT_CONFIG_DIRNAME` | `~/.mdrs-client` | Directory where config and login cache files are stored |
| `MDRS_CLIENT_CONCURRENT` | `10` | Number of concurrent file transfers for upload and download |
## CLI Usage
### config create
@@ -18,6 +59,32 @@ Create remote host configuration
mdrs config create neurodata https://neurodata.riken.jp/api
```
### config update
Update the URL of a registered remote host.
```shell
mdrs config update neurodata https://neurodata.riken.jp/api
```
### config list
List registered remote hosts.
```shell
mdrs config list
mdrs config ls
```
### config delete
Remove a registered remote host.
```shell
mdrs config delete neurodata
mdrs config rm neurodata
```
### login
Login to remote host
@@ -56,13 +123,17 @@ mdrs labs neurodata:
### ls
List the folder contents
List the folder contents. You can also specify a DOI path in the form `remote:10.xxxx/yyy.ID[/optional/subpath]`.
```shell
mdrs ls neurodata:/NIU/Repository/
mdrs ls -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/
mdrs ls -r neurodata:/NIU/Repository/Dataset1/
mdrs ls -J -r neurodata:/NIU/Repository/Dataset1/
# DOI access examples:
mdrs ls neurodata:10.60178/cbs.20260429-001
mdrs ls "neurodata:10.60178/cbs.20260429-001/Figure 1"
```
### mkdir
@@ -85,7 +156,7 @@ mdrs upload -r --skip-if-exists ./dataset neurodata:/NIU/Repository/TEST/
### download
Download the file or folder
Download the file or folder. You can also specify a DOI path.
```shell
mdrs download neurodata:/NIU/Repository/TEST/sample.dat ./
@@ -93,6 +164,10 @@ mdrs download -r neurodata:/NIU/Repository/TEST/dataset/ ./
mdrs download -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/Readme.dat ./
mdrs download -r --exclude /NIU/Repository/TEST/dataset/skip neurodata:/NIU/Repository/TEST/dataset/ ./
mdrs download -r --skip-if-exists neurodata:/NIU/Repository/TEST/dataset/ ./
# DOI access examples:
mdrs download neurodata:10.60178/cbs.20260429-001/README_NeuroData.md ./
mdrs download -r "neurodata:10.60178/cbs.20260429-001/Figure 1" ./
```
### mv
@@ -134,20 +209,34 @@ mdrs chacl pw_open -r -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open
### metadata
Get a folder metadata
Get a folder metadata. You can also specify a DOI path.
```shell
mdrs metadata neurodata:/NIU/Repository/TEST/
mdrs metadata -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/
# DOI access examples:
mdrs metadata neurodata:10.60178/cbs.20260429-001
```
### file-metadata
Get the file metadata
Get the file metadata. You can also specify a DOI path.
```shell
mdrs file-metadata neurodata:/NIU/Repository/TEST/dataset/sample.dat
mdrs file-metadata -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/Readme.txt
# DOI access examples:
mdrs file-metadata "neurodata:10.60178/cbs.20260429-001/Figure 1/Figure1v3.pdf"
```
### version
Show the tool name and version number
```shell
mdrs version
```
### help
@@ -157,3 +246,46 @@ Show the help message and exit
```shell
mdrs -h
```
## Python API Usage
You can also use this package as a Python library to programmatically interact with MDRS repositories.
```python
from mdrsclient.client import MdrsClient
from mdrsclient.cache import InMemoryCache
from mdrsclient.config import InMemoryConfig
# 1. Setup in-memory configuration and cache to avoid local state files (e.g., config.ini, cache/*.json)
config = InMemoryConfig("neurodata")
config.url = "https://neurodata.riken.jp/api"
cache = InMemoryCache()
# 2. Initialize client with custom configuration and cache
client = MdrsClient.from_remote("neurodata", cache=cache, config=config)
# 3. Login to the remote server
client.login("username", "password")
# 4. Use service methods
labs = client.get_laboratories()
metadata = client.metadata("neurodata:/NIU/Repository/")
# Transfer files programmatically
client.upload("/path/to/local/data", "neurodata:/NIU/Repository/TEST/", is_recursive=True)
client.download("neurodata:/NIU/Repository/TEST/data", "/path/to/local", is_recursive=True)
```
## Testing
You can run the unit test suite using the standard library `unittest` discover runner:
```shell
.venv/bin/python -m unittest discover tests
```
## Changelog
See [CHANGELOG.md](./CHANGELOG.md) for the full change history.
-1
View File
@@ -1 +0,0 @@
1.3.13
+2 -1
View File
@@ -1,3 +1,4 @@
from mdrsclient.__version__ import __version__
from mdrsclient.client import MdrsClient
__all__ = ["__version__"]
__all__ = ["__version__", "MdrsClient"]
+2
View File
@@ -17,6 +17,7 @@ from mdrsclient.commands import (
MvCommand,
RmCommand,
UploadCommand,
VersionCommand,
WhoamiCommand,
)
from mdrsclient.exceptions import MDRSException
@@ -29,6 +30,7 @@ def main() -> None:
parsers = parser.add_subparsers(title="subcommands")
ConfigCommand.register(parsers)
VersionCommand.register(parsers)
LoginCommand.register(parsers)
LogoutCommand.register(parsers)
WhoamiCommand.register(parsers)
+5 -4
View File
@@ -1,8 +1,9 @@
import os
import importlib.metadata
here = os.path.realpath(os.path.dirname(__file__))
try:
__version__ = importlib.metadata.version("mdrs-client-python")
except importlib.metadata.PackageNotFoundError:
__version__ = "0.0.0-dev"
with open(os.path.join(here, "VERSION")) as version_file:
__version__ = version_file.read().strip()
__all__ = ["__version__"]
+2
View File
@@ -1,9 +1,11 @@
from mdrsclient.api.doi import DoiApi
from mdrsclient.api.files import FilesApi
from mdrsclient.api.folders import FoldersApi
from mdrsclient.api.laboratories import LaboratoriesApi
from mdrsclient.api.users import UsersApi
__all__ = [
"DoiApi",
"FilesApi",
"FoldersApi",
"LaboratoriesApi",
+6
View File
@@ -8,6 +8,7 @@ from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import (
BadRequestException,
ForbiddenException,
TokenExpiredException,
UnauthorizedException,
UnexpectedException,
)
@@ -28,6 +29,11 @@ class BaseApi(ABC):
if response.status_code == requests.codes.bad_request:
raise BadRequestException(errors.errors[0].detail)
elif response.status_code == requests.codes.unauthorized:
# A request can wait in the server's queue for longer than the access
# token it was sent with lives, so an expired token here does not mean
# the session is over - it means this one request arrived too late.
if any(e.code == "token_not_valid" for e in errors.errors):
raise TokenExpiredException("Access token expired before the request was served.")
raise UnauthorizedException("Login required.")
elif response.status_code == requests.codes.forbidden:
raise ForbiddenException("You do not have enough permissions. Access is denied.")
+34
View File
@@ -0,0 +1,34 @@
from typing import Final
from pydantic import TypeAdapter
from pydantic.dataclasses import dataclass
from mdrsclient.api.base import BaseApi
from mdrsclient.api.utils import retry_on_expired_token, token_check
from mdrsclient.models.doi import Doi
@dataclass(frozen=True)
class DoiRetrieveFolderRef:
id: str
@dataclass(frozen=True)
class DoiRetrieveResponse:
id: str
doi: str
folder: DoiRetrieveFolderRef
class DoiApi(BaseApi):
ENTRYPOINT: Final[str] = "v3/doi/"
@retry_on_expired_token
def retrieve(self, doi_id: str) -> Doi:
"""Retrieve the folder associated with a DOI suffix ID (GET v3/doi/{id}/)."""
url = self.ENTRYPOINT + doi_id + "/"
token_check(self.connection)
response = self.connection.get(url)
self._raise_response_error(response)
api_resp = TypeAdapter(DoiRetrieveResponse).validate_python(response.json())
return Doi(id=api_resp.id, doi=api_resp.doi, folder_id=api_resp.folder.id)
+62 -8
View File
@@ -1,14 +1,16 @@
import mimetypes
import os
import threading
from typing import Any, Final
from unicodedata import normalize
from pydantic import TypeAdapter
from pydantic.dataclasses import dataclass
from requests_toolbelt.multipart.encoder import MultipartEncoder
from mdrsclient.api.base import BaseApi
from mdrsclient.api.utils import token_check
from mdrsclient.exceptions import UnexpectedException
from mdrsclient.api.utils import retry_on_expired_token, token_check
from mdrsclient.exceptions import MDRSException, UnexpectedException
from mdrsclient.models import File
@@ -29,6 +31,7 @@ class FilesApi(BaseApi):
ENTRYPOINT: Final[str] = "v3/files/"
FALLBACK_MIMETYPE: Final[str] = "application/octet-stream"
@retry_on_expired_token
def list(self, folder_id: str, page_num: int) -> FilesApiListResponse:
url = self.ENTRYPOINT
token_check(self.connection)
@@ -37,6 +40,7 @@ class FilesApi(BaseApi):
self._raise_response_error(response)
return TypeAdapter(FilesApiListResponse).validate_python(response.json())
@retry_on_expired_token
def retrieve(self, id: str) -> File:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + id + "/"
@@ -45,6 +49,7 @@ class FilesApi(BaseApi):
self._raise_response_error(response)
return TypeAdapter(File).validate_python(response.json())
@retry_on_expired_token
def create(self, folder_id: str, path: str) -> str:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT
@@ -53,7 +58,10 @@ class FilesApi(BaseApi):
try:
with open(os.path.realpath(path), mode="rb") as fp:
data = MultipartEncoder(
fields={"folder_id": folder_id, "file": (os.path.basename(path), fp, self._get_mime_type(path))}
fields={
"folder_id": folder_id,
"file": (normalize("NFC", os.path.basename(path)), fp, self._get_mime_type(path)),
}
)
response = self.connection.post(url, data=data, headers={"Content-Type": data.content_type})
self._raise_response_error(response)
@@ -62,10 +70,14 @@ class FilesApi(BaseApi):
raise UnexpectedException(f"Could not open `{path}` file.")
except MemoryError:
raise UnexpectedException("Out of memory.")
except MDRSException:
# Already says what went wrong, and the caller may want to act on the kind.
raise
except Exception as e:
raise UnexpectedException("Unspecified error.") from e
return ret.id
@retry_on_expired_token
def update(self, file: File, path: str | None) -> bool:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + file.id + "/"
@@ -75,7 +87,9 @@ class FilesApi(BaseApi):
# update file body
try:
with open(os.path.realpath(path), mode="rb") as fp:
data = MultipartEncoder(fields={"file": (os.path.basename(path), fp, self._get_mime_type(path))})
data = MultipartEncoder(
fields={"file": (normalize("NFC", os.path.basename(path)), fp, self._get_mime_type(path))}
)
response = self.connection.put(url, data=data, headers={"Content-Type": data.content_type})
except OSError:
raise UnexpectedException(f"Could not open `{path}` file.")
@@ -90,6 +104,7 @@ class FilesApi(BaseApi):
self._raise_response_error(response)
return True
@retry_on_expired_token
def destroy(self, file: File) -> bool:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + file.id + "/"
@@ -98,6 +113,7 @@ class FilesApi(BaseApi):
self._raise_response_error(response)
return True
@retry_on_expired_token
def move(self, file: File, folder_id: str, name: str) -> bool:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + file.id + "/move/"
@@ -107,6 +123,7 @@ class FilesApi(BaseApi):
self._raise_response_error(response)
return True
@retry_on_expired_token
def copy(self, file: File, folder_id: str, name: str) -> bool:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + file.id + "/copy/"
@@ -116,6 +133,7 @@ class FilesApi(BaseApi):
self._raise_response_error(response)
return True
@retry_on_expired_token
def metadata(self, file: File) -> dict[str, Any]:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + file.id + "/metadata/"
@@ -124,22 +142,58 @@ class FilesApi(BaseApi):
self._raise_response_error(response)
return response.json()
@retry_on_expired_token
def download(self, file: File, path: str) -> bool:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = file.download_url
token_check(self.connection)
# Refused before anything is fetched. The finished file is moved into place, and a
# rename would replace a destination whose mode says it is protected.
if os.path.exists(path):
try:
with open(path, "r+b"):
pass
except OSError as e:
raise UnexpectedException(f"Cannot write `{path}`: {e}")
response = self.connection.get(url, stream=True)
self._raise_response_error(response)
# Written beside the destination and moved in once the whole body has arrived, so
# a transfer that fails part way leaves whatever was already there untouched and
# never leaves a truncated file under the real name.
fd, tmp_path = self._open_partial(path)
try:
with open(path, "wb") as f:
with os.fdopen(fd, "wb") as f:
for chunk in response.iter_content(chunk_size=4096):
if chunk:
f.write(chunk)
f.flush()
except PermissionError:
print(f"Cannot create file `{path}`: Permission denied.")
os.replace(tmp_path, path)
except BaseException:
# Only the scratch file goes: anything at the destination was not written here.
if os.path.exists(tmp_path):
os.unlink(tmp_path)
raise
return True
@staticmethod
def _open_partial(path: str) -> tuple[int, str]:
"""
Create a scratch file beside `path` and return it open for writing.
Beside it, so moving the finished download into place is a rename within one
directory. `0o666` rather than a private mode because the umask is what decided
the permissions of a downloaded file before, and still should.
"""
base = f"{path}.{os.getpid()}-{threading.get_ident()}"
for attempt in range(100):
tmp_path = f"{base}-{attempt}.mdrspart"
try:
return os.open(tmp_path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o666), tmp_path
except FileExistsError:
continue
except OSError as e:
raise UnexpectedException(f"Cannot write `{path}`: {e}")
raise UnexpectedException(f"Could not create a temporary file beside `{path}`.")
def _get_mime_type(self, path: str) -> str:
mt = mimetypes.guess_type(path)
if mt:
+11 -1
View File
@@ -5,7 +5,7 @@ from pydantic import TypeAdapter
from pydantic.dataclasses import dataclass
from mdrsclient.api.base import BaseApi
from mdrsclient.api.utils import token_check
from mdrsclient.api.utils import retry_on_expired_token, token_check
from mdrsclient.exceptions import UnauthorizedException
from mdrsclient.models import Folder, FolderSimple
@@ -18,6 +18,7 @@ class FoldersApiCreateResponse:
class FoldersApi(BaseApi):
ENTRYPOINT: Final[str] = "v3/folders/"
@retry_on_expired_token
def list(self, laboratory_id: int, path: str) -> list[FolderSimple]:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT
@@ -30,6 +31,7 @@ class FoldersApi(BaseApi):
ret.append(TypeAdapter(FolderSimple).validate_python(data))
return ret
@retry_on_expired_token
def retrieve(self, id: str) -> Folder:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + id + "/"
@@ -39,6 +41,7 @@ class FoldersApi(BaseApi):
ret = TypeAdapter(Folder).validate_python(response.json())
return ret
@retry_on_expired_token
def create(self, name: str, parent_id: str) -> str:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT
@@ -49,6 +52,7 @@ class FoldersApi(BaseApi):
ret = TypeAdapter(FoldersApiCreateResponse).validate_python(response.json())
return ret.id
@retry_on_expired_token
def update(self, folder: FolderSimple) -> bool:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + folder.id + "/"
@@ -61,6 +65,7 @@ class FoldersApi(BaseApi):
self._raise_response_error(response)
return True
@retry_on_expired_token
def destroy(self, id: str, recursive: bool) -> bool:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + id + "/"
@@ -70,6 +75,7 @@ class FoldersApi(BaseApi):
self._raise_response_error(response)
return True
@retry_on_expired_token
def auth(self, id: str, password: str) -> bool:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + id + "/auth/"
@@ -81,6 +87,7 @@ class FoldersApi(BaseApi):
self._raise_response_error(response)
return True
@retry_on_expired_token
def acl(self, id: str, access_level: int, recursive: bool, password: str | None) -> bool:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + id + "/acl/"
@@ -94,6 +101,7 @@ class FoldersApi(BaseApi):
self._raise_response_error(response)
return True
@retry_on_expired_token
def move(self, folder: FolderSimple, folder_id: str, name: str) -> bool:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + folder.id + "/move/"
@@ -103,6 +111,7 @@ class FoldersApi(BaseApi):
self._raise_response_error(response)
return True
@retry_on_expired_token
def copy(self, folder: FolderSimple, folder_id: str, name: str) -> bool:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + folder.id + "/copy/"
@@ -112,6 +121,7 @@ class FoldersApi(BaseApi):
self._raise_response_error(response)
return True
@retry_on_expired_token
def metadata(self, id: str) -> dict[str, Any]:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + id + "/metadata/"
+2 -1
View File
@@ -3,13 +3,14 @@ from typing import Final
from pydantic import TypeAdapter
from mdrsclient.api.base import BaseApi
from mdrsclient.api.utils import token_check
from mdrsclient.api.utils import retry_on_expired_token, token_check
from mdrsclient.models import Laboratories, Laboratory
class LaboratoriesApi(BaseApi):
ENTRYPOINT: Final[str] = "v3/laboratories/"
@retry_on_expired_token
def list(self) -> Laboratories:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT
+8 -1
View File
@@ -8,6 +8,10 @@ from mdrsclient.api.base import BaseApi
from mdrsclient.exceptions import UnauthorizedException
from mdrsclient.models import Token, User
# (connect, read) seconds for the token refresh. Uploads are served by a separate
# instance, so this one is not queued behind them and has no reason to be slow.
TOKEN_REFRESH_TIMEOUT: Final[tuple[float, float]] = (5.0, 30.0)
@dataclass(frozen=True)
class UsersCurrentResponseLaboratory:
@@ -61,7 +65,10 @@ class UsersApi(BaseApi):
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + "token/refresh/"
data: dict[str, str | int] = {"refresh": token.refresh}
response = self.connection.post(url, data=data)
# Bounded on its own: the caller holds a lock that spans processes while this runs,
# so a provider that accepts the connection and then goes quiet would otherwise
# stall every other request on this machine rather than just this one.
response = self.connection.post(url, data=data, timeout=TOKEN_REFRESH_TIMEOUT)
if response.status_code == requests.codes.unauthorized:
raise UnauthorizedException("Token is invalid or expired.")
self._raise_response_error(response)
+118 -13
View File
@@ -1,19 +1,124 @@
import functools
from typing import Any, Callable, TypeVar, cast
from mdrsclient.api.users import UsersApi
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import UnauthorizedException
from mdrsclient.exceptions import (
MDRSException,
ServerBusyException,
TokenExpiredException,
UnauthorizedException,
)
from mdrsclient.models import Token
F = TypeVar("F", bound=Callable[..., Any])
SERVER_BUSY_MESSAGE = (
"The server took too long to start handling the request and may be overloaded. "
"Try again, or reduce the number of parallel transfers."
)
def token_check(connection: MDRSConnection) -> None:
try:
connection.lock.acquire()
if connection.token is not None:
if connection.token.is_refresh_required:
user_api = UsersApi(connection)
try:
connection.token = user_api.tokenRefresh(connection.token)
except UnauthorizedException:
connection.logout()
elif connection.token.is_expired:
"""
Bring the access token up to date before a request goes out.
Refreshing is a read-modify-write over a cache shared with every other client
process on this machine, and a rotating provider stops honouring the refresh token
it replaces. Two processes reaching this at once would otherwise both send the same
token, and the loser would be left holding one the server no longer accepts, so the
whole sequence runs under a lock that spans processes and the cache is re-read
inside it.
"""
with connection.lock:
token = connection.token
if token is None or not (token.is_refresh_required or token.is_expired):
# Nothing to do, which is the answer for almost every request. The lock below
# reaches across processes and is held for a round trip, so it is worth
# knowing that before taking it.
return
with connection.cache_lock():
connection.reload_cache()
token = connection.token
if token is None:
return
if token.is_expired:
connection.logout()
finally:
connection.lock.release()
return
if not token.is_refresh_required:
return
user_api = UsersApi(connection)
try:
connection.token = user_api.tokenRefresh(token)
except UnauthorizedException:
connection.logout()
def token_recover(connection: MDRSConnection, used: Token | None) -> bool:
"""
Get a usable access token after one was refused for having expired.
A request can sit in the server's queue for longer than its access token lives, and
while it waited another thread or process may already have refreshed. Prefer what
they left behind: refreshing again would spend a round trip, and with a provider
that rotates refresh tokens it would retire one that is still in use.
Returns whether the caller now holds a token worth retrying with.
"""
with connection.lock, connection.cache_lock():
connection.reload_cache()
token = connection.token
if token is None:
return False
if used is None or token != used:
# Somebody else has been here since the request went out. Assign rather than
# just read: the setter is what rewrites the session's Authorization header,
# so a plain reload would leave the retry carrying the refused token.
connection.token = token
return True
user_api = UsersApi(connection)
try:
connection.token = user_api.tokenRefresh(token)
except UnauthorizedException:
connection.logout()
return False
return True
def retry_on_expired_token(func: F) -> F:
"""
Send a request again once when the server found its access token expired.
The token is checked before every request, so this only happens when the request was
held long enough on the way in for a valid token to lapse - a queue behind uploads
that take minutes, most often.
"""
@functools.wraps(func)
def wrapper(self: Any, *args: Any, **kwargs: Any) -> Any:
connection: MDRSConnection = self.connection
# Settle the token first so `used` is what the request actually carries. The
# wrapped method checks it again, which costs nothing once it is already current.
token_check(connection)
used = connection.token
try:
return func(self, *args, **kwargs)
except TokenExpiredException:
try:
recovered = token_recover(connection, used)
except MDRSException:
raise
except Exception as e:
# A busy server queues the refresh too, and failing here would report an
# endpoint the user never asked for rather than the reason.
raise ServerBusyException(SERVER_BUSY_MESSAGE) from e
if not recovered:
raise
try:
return func(self, *args, **kwargs)
except TokenExpiredException as e:
# A second refusal, with a token that was current when it left. The session is
# fine; the server is not starting requests before their credentials lapse.
raise ServerBusyException(SERVER_BUSY_MESSAGE) from e
return cast(F, wrapper)
+171 -21
View File
@@ -1,7 +1,12 @@
import contextlib
import dataclasses
import hashlib
import json
import os
import tempfile
import time
from contextlib import AbstractContextManager
from typing import Iterator, Protocol, runtime_checkable
from pydantic import TypeAdapter, ValidationError
from pydantic.dataclasses import dataclass
@@ -16,7 +21,7 @@ from mdrsclient.utils import FileLock
class CacheData:
user: User | None = None
token: Token | None = None
laboratories: Laboratories = Laboratories()
laboratories: Laboratories = dataclasses.field(default_factory=Laboratories)
digest: str = ""
def clear(self) -> None:
@@ -43,18 +48,132 @@ class CacheData:
).hexdigest()
class CacheFile:
@runtime_checkable
class CacheInterface(Protocol):
def lock(self) -> AbstractContextManager[None]:
"""
Hold exclusive access to the cache for the duration of the block.
Refreshing a token is a read-modify-write, and the cache is shared with every
other client process using the same remote.
"""
...
def reload(self) -> None:
"""Re-read the cache, discarding anything held from an earlier read."""
...
@property
def token(self) -> Token | None: ...
@token.setter
def token(self, token: Token) -> None: ...
@token.deleter
def token(self) -> None: ...
@property
def user(self) -> User | None: ...
@user.setter
def user(self, user: User) -> None: ...
@user.deleter
def user(self) -> None: ...
@property
def laboratories(self) -> Laboratories: ...
@laboratories.setter
def laboratories(self, laboratories: Laboratories) -> None: ...
class InMemoryCache(CacheInterface):
def __init__(self) -> None:
self.__data = CacheData()
@contextlib.contextmanager
def lock(self) -> Iterator[None]:
# Nothing else can reach this cache, so there is nothing to exclude.
yield
def reload(self) -> None:
pass
@property
def token(self) -> Token | None:
return self.__data.token
@token.setter
def token(self, token: Token) -> None:
self.__data.token = token
@token.deleter
def token(self) -> None:
if self.__data.token is not None:
self.__data.token = None
@property
def user(self) -> User | None:
return self.__data.user
@user.setter
def user(self, user: User) -> None:
self.__data.user = user
@user.deleter
def user(self) -> None:
if self.__data.user is not None:
self.__data.user = None
@property
def laboratories(self) -> Laboratories:
return self.__data.laboratories
@laboratories.setter
def laboratories(self, laboratories: Laboratories) -> None:
self.__data.laboratories = laboratories
class CacheFile(CacheInterface):
__serial: int
__cache_dir: str
__cache_file: str
__lock_file: str
__lock_depth: int
__data: CacheData
def __init__(self, remote: str) -> None:
self.__serial = -1
self.__cache_dir = os.path.join(CONFIG_DIRNAME, "cache")
self.__cache_file = os.path.join(self.__cache_dir, remote + ".json")
self.__lock_file = os.path.join(self.__cache_dir, remote + ".lock")
self.__lock_depth = 0
self.__data = CacheData()
@contextlib.contextmanager
def lock(self) -> Iterator[None]:
# Re-entrant, because every write takes it and a refresh is a write made while
# already holding it. A second flock on the same file from the same process would
# wait for a release that cannot come.
if self.__lock_depth > 0:
self.__lock_depth += 1
try:
yield
finally:
self.__lock_depth -= 1
return
# A separate file, so that replacing the cache cannot disturb the lock holders.
self.__ensure_cache_dir()
with open(self.__lock_file, "a") as f:
FileLock.lock(f)
self.__lock_depth = 1
try:
yield
finally:
self.__lock_depth = 0
FileLock.unlock(f)
os.chmod(self.__lock_file, 0o600)
def reload(self) -> None:
self.__serial = -1
self.__load()
@property
def token(self) -> Token | None:
self.__load()
@@ -62,9 +181,10 @@ class CacheFile:
@token.setter
def token(self, token: Token) -> None:
self.__load()
self.__data.token = token
self.__save()
with self.lock():
self.reload()
self.__data.token = token
self.__save()
@token.deleter
def token(self) -> None:
@@ -77,9 +197,10 @@ class CacheFile:
@user.setter
def user(self, user: User) -> None:
self.__load()
self.__data.user = user
self.__save()
with self.lock():
self.reload()
self.__data.user = user
self.__save()
@user.deleter
def user(self) -> None:
@@ -92,9 +213,10 @@ class CacheFile:
@laboratories.setter
def laboratories(self, laboratories: Laboratories) -> None:
self.__load()
self.__data.laboratories = laboratories
self.__save()
with self.lock():
self.reload()
self.__data.laboratories = laboratories
self.__save()
def __clear(self) -> None:
self.__data.clear()
@@ -102,8 +224,7 @@ class CacheFile:
def __load(self) -> None:
if os.path.isfile(self.__cache_file):
stat = os.stat(self.__cache_file)
serial = hash((stat.st_uid, stat.st_gid, stat.st_mode, stat.st_size, stat.st_mtime))
serial = self.__stat_serial()
if self.__serial != serial:
try:
with open(self.__cache_file) as f:
@@ -123,15 +244,44 @@ class CacheFile:
def __save(self) -> None:
self.__ensure_cache_dir()
with open(self.__cache_file, "w") as f:
FileLock.lock(f)
self.__data.update_digest()
f.write(json.dumps(dataclasses.asdict(self.__data)))
FileLock.unlock(f)
self.__data.update_digest()
payload = json.dumps(dataclasses.asdict(self.__data))
# Written aside and moved into place: opening the cache for writing truncates it
# first, and a reader landing in that window would find the file empty and treat
# the session as broken.
fd, tmp_file = tempfile.mkstemp(dir=self.__cache_dir, prefix=".tmp-")
try:
with os.fdopen(fd, "w") as f:
f.write(payload)
# ensure file is secure.
os.chmod(tmp_file, 0o600)
self.__replace(tmp_file, self.__cache_file)
except BaseException:
if os.path.exists(tmp_file):
os.unlink(tmp_file)
raise
self.__serial = self.__stat_serial()
@staticmethod
def __replace(source: str, destination: str) -> None:
# Windows refuses the rename while another process still has the destination
# open, which a reader briefly does, so give it a moment rather than failing the
# save outright. On POSIX the rename always succeeds and the loop ends at once.
deadline = time.monotonic() + 5.0
while True:
try:
os.replace(source, destination)
return
except PermissionError:
if time.monotonic() >= deadline:
raise
time.sleep(0.05)
def __stat_serial(self) -> int:
stat = os.stat(self.__cache_file)
self.__serial = hash((stat.st_uid, stat.st_gid, stat.st_mode, stat.st_size, stat.st_mtime))
# ensure file is secure.
os.chmod(self.__cache_file, 0o600)
# st_ino and st_mtime_ns both move when the file is replaced, which a refresh
# that happens to produce the same number of bytes otherwise would not show.
return hash((stat.st_ino, stat.st_uid, stat.st_gid, stat.st_mode, stat.st_size, stat.st_mtime_ns))
def __ensure_cache_dir(self) -> None:
if not os.path.exists(self.__cache_dir):
+253
View File
@@ -0,0 +1,253 @@
import os
from dataclasses import dataclass
from unicodedata import normalize
from mdrsclient.api import FilesApi, FoldersApi
from mdrsclient.cache import CacheInterface
from mdrsclient.config import ConfigInterface
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import IllegalArgumentException
from mdrsclient.models import File, Folder, FolderSimple, Laboratory
from mdrsclient.models.file import find_file
from mdrsclient.services import MdrsService
@dataclass(frozen=True)
class TransferEndpoints:
"""Both ends of a `cp` or `mv`, resolved: the parents, their contents, and the names."""
laboratory: Laboratory
s_path: str
s_basename: str
s_parent_folder: Folder
s_parent_files: list[File]
d_path: str
d_basename: str
d_parent_folder: Folder
d_parent_files: list[File]
def is_same_place(self) -> bool:
"""Whether the destination names what the source already is, leaving nothing to do."""
return self.s_parent_folder.id == self.d_parent_folder.id and self.s_basename == self.d_basename
class MdrsClient(MdrsService):
"""Service layer client for MDRS."""
def __init__(self, connection: MDRSConnection | None, config_class: type[ConfigInterface] | None = None):
super().__init__(connection, config_class)
@classmethod
def from_remote(
cls, remote: str, cache: CacheInterface | None = None, config: ConfigInterface | None = None
) -> "MdrsClient":
return cls(cls.create_connection(remote, cache, config))
def mkdir(self, remote_path: str) -> None:
remote, laboratory_name, r_path = self.parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
r_dirname = os.path.dirname(r_path)
r_basename = os.path.basename(r_path)
laboratory = self.find_laboratory(laboratory_name)
parent_folder = self.find_folder(laboratory, r_dirname)
files = self.find_files(parent_folder.id)
if parent_folder.find_sub_folder(r_basename) is not None or find_file(files, r_basename) is not None:
raise IllegalArgumentException(f"Cannot create folder `{r_path}`: File exists.")
folder_api = FoldersApi(self.connection)
folder_api.create(normalize("NFC", r_basename), parent_folder.id)
def rm(self, remote_path: str, is_recursive: bool = False) -> None:
remote, laboratory_name, r_path = self.parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
r_dirname = os.path.dirname(r_path)
r_basename = os.path.basename(r_path)
laboratory = self.find_laboratory(laboratory_name)
parent_folder = self.find_folder(laboratory, r_dirname)
parent_files = self.find_files(parent_folder.id)
file = find_file(parent_files, r_basename)
if file is not None:
file_api = FilesApi(self.connection)
file_api.destroy(file)
else:
folder = parent_folder.find_sub_folder(r_basename)
if folder is None:
raise IllegalArgumentException(f"Cannot remove `{r_path}`: No such file or folder.")
if not is_recursive:
raise IllegalArgumentException(f"Cannot remove `{r_path}`: Is a folder.")
folder_api = FoldersApi(self.connection)
folder_api.destroy(folder.id, True)
def ls(self, remote_path: str, password: str | None = None) -> tuple[Folder, list[File]]:
folder, laboratory = self.resolve_folder(remote_path, password)
files = self.find_files(folder.id)
return folder, files
def __resolve_transfer(self, src_path: str, dest_path: str) -> TransferEndpoints:
"""Resolve both ends of a transfer, refusing one that crosses a remote or a laboratory."""
s_remote, s_laboratory_name, s_path = self.parse_remote_host_with_path(src_path)
d_remote, d_laboratory_name, d_path = self.parse_remote_host_with_path(dest_path)
if s_remote != d_remote:
raise IllegalArgumentException("Remote host mismatched.")
if s_laboratory_name != d_laboratory_name:
raise IllegalArgumentException("Laboratory mismatched.")
s_path = s_path.rstrip("/")
s_basename = os.path.basename(s_path)
# A destination ending in a separator names a folder to put the source into, and
# keeps the source's own name.
if d_path.endswith("/"):
d_dirname = d_path
d_basename = s_basename
else:
d_dirname = os.path.dirname(d_path)
d_basename = os.path.basename(d_path)
laboratory = self.find_laboratory(s_laboratory_name)
s_parent_folder = self.find_folder(laboratory, os.path.dirname(s_path))
d_parent_folder = self.find_folder(laboratory, d_dirname)
return TransferEndpoints(
laboratory=laboratory,
s_path=s_path,
s_basename=s_basename,
s_parent_folder=s_parent_folder,
s_parent_files=self.find_files(s_parent_folder.id),
d_path=d_path,
d_basename=d_basename,
d_parent_folder=d_parent_folder,
d_parent_files=self.find_files(d_parent_folder.id),
)
@staticmethod
def __check_file_destination(ends: TransferEndpoints) -> None:
"""Refuse a destination that a file cannot take the place of."""
if find_file(ends.d_parent_files, ends.d_basename) is not None:
raise IllegalArgumentException(f"File `{ends.d_basename}` already exists.")
if ends.d_parent_folder.find_sub_folder(ends.d_basename) is not None:
raise IllegalArgumentException(
f"Cannot overwrite non-folder `{ends.d_basename}` with folder `{ends.d_path}`."
)
@staticmethod
def __check_folder_destination(ends: TransferEndpoints, s_folder: FolderSimple) -> None:
"""Refuse a destination that a folder cannot take the place of."""
if find_file(ends.d_parent_files, ends.d_basename) is not None:
raise IllegalArgumentException(
f"Cannot overwrite non-folder `{ends.d_basename}` with folder `{ends.s_path}`."
)
d_folder = ends.d_parent_folder.find_sub_folder(ends.d_basename)
if d_folder is not None:
if d_folder.id == s_folder.id:
raise IllegalArgumentException(f"`{ends.s_path}` and `{ends.s_path}` are the same folder.")
raise IllegalArgumentException(f"Cannot move `{ends.s_path}` to `{ends.d_path}`: Folder not empty.")
@staticmethod
def __find_source_folder(ends: TransferEndpoints) -> FolderSimple:
s_folder = ends.s_parent_folder.find_sub_folder(ends.s_basename)
if s_folder is None:
raise IllegalArgumentException(f"File or folder `{ends.s_basename}` not found.")
return s_folder
def cp(self, src_path: str, dest_path: str, is_recursive: bool = False) -> None:
ends = self.__resolve_transfer(src_path, dest_path)
s_file = find_file(ends.s_parent_files, ends.s_basename)
if s_file is not None:
self.__check_file_destination(ends)
if not ends.is_same_place():
FilesApi(self.connection).copy(s_file, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
return
s_folder = self.__find_source_folder(ends)
if not is_recursive:
raise IllegalArgumentException(f"Cannot copy `{ends.s_path}`: Is a folder.")
self.__check_folder_destination(ends, s_folder)
if not ends.is_same_place():
FoldersApi(self.connection).copy(s_folder, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
def mv(self, src_path: str, dest_path: str) -> None:
ends = self.__resolve_transfer(src_path, dest_path)
s_file = find_file(ends.s_parent_files, ends.s_basename)
if s_file is not None:
self.__check_file_destination(ends)
if not ends.is_same_place():
FilesApi(self.connection).move(s_file, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
return
s_folder = self.__find_source_folder(ends)
self.__check_folder_destination(ends, s_folder)
if not ends.is_same_place():
FoldersApi(self.connection).move(s_folder, ends.d_parent_folder.id, normalize("NFC", ends.d_basename))
def chacl(
self, remote_path: str, access_level: int, is_recursive: bool = False, password: str | None = None
) -> None:
remote, laboratory_name, r_path = self.parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
laboratory = self.find_laboratory(laboratory_name)
folder = self.find_folder(laboratory, r_path)
folder_api = FoldersApi(self.connection)
folder_api.acl(folder.id, access_level, is_recursive, password)
def metadata(self, remote_path: str, password: str | None = None) -> dict:
folder, laboratory = self.resolve_folder(remote_path, password)
folder_api = FoldersApi(self.connection)
return folder_api.metadata(folder.id)
def file_metadata(self, remote_path: str, password: str | None = None) -> dict:
folder, laboratory, r_basename = self.resolve_file(remote_path, password)
files = self.find_files(folder.id)
file = find_file(files, r_basename)
if file is None:
raise IllegalArgumentException(f"File `{r_basename}` not found.")
file_api = FilesApi(self.connection)
return file_api.metadata(file)
def upload(
self, local_path: str, remote_path: str, is_recursive: bool = False, is_skip_if_exists: bool = False
) -> None:
from mdrsclient.transfer import Uploader
uploader = Uploader(self)
uploader.upload(local_path, remote_path, is_recursive, is_skip_if_exists)
def download(
self,
remote_path: str,
local_path: str,
is_recursive: bool = False,
is_skip_if_exists: bool = False,
password: str | None = None,
excludes: list[str] | None = None,
) -> None:
from mdrsclient.transfer import Downloader
downloader = Downloader(self)
downloader.download(remote_path, local_path, is_recursive, is_skip_if_exists, password, excludes)
def version(self) -> str:
from mdrsclient.__version__ import __version__
return f"mdrs {__version__}"
def config_create(self, remote: str, url: str) -> None:
remote = self.parse_remote_host(remote)
config = self.config_class(remote)
if config.url is not None:
raise IllegalArgumentException(f"Remote host `{remote}` is already exists.")
else:
config.url = url
def config_update(self, remote: str, url: str) -> None:
remote = self.parse_remote_host(remote)
config = self.config_class(remote)
if config.url is None:
raise IllegalArgumentException(f"Remote host `{remote}` is not exists.")
else:
config.url = url
def config_list(self) -> list:
config = self.config_class("")
return config.list()
def config_delete(self, remote: str) -> None:
remote = self.parse_remote_host(remote)
config = self.config_class(remote)
if config.url is None:
raise IllegalArgumentException(f"Remote host `{remote}` is not exists.")
else:
del config.url
+2
View File
@@ -12,6 +12,7 @@ from mdrsclient.commands.mkdir import MkdirCommand
from mdrsclient.commands.mv import MvCommand
from mdrsclient.commands.rm import RmCommand
from mdrsclient.commands.upload import UploadCommand
from mdrsclient.commands.version import VersionCommand
from mdrsclient.commands.whoami import WhoamiCommand
__all__ = [
@@ -29,5 +30,6 @@ __all__ = [
"MvCommand",
"RmCommand",
"UploadCommand",
"VersionCommand",
"WhoamiCommand",
]
+1 -87
View File
@@ -1,19 +1,7 @@
import re
from abc import ABC, abstractmethod
from typing import Any
from unicodedata import normalize
from mdrsclient.api import FilesApi, FoldersApi, LaboratoriesApi
from mdrsclient.config import ConfigFile
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import (
IllegalArgumentException,
MissingConfigurationException,
UnauthorizedException,
UnexpectedException,
)
from mdrsclient.models import File, Folder, Laboratory
from mdrsclient.utils import page_num_from_url
from mdrsclient.exceptions import UnexpectedException
class BaseCommand(ABC):
@@ -21,77 +9,3 @@ class BaseCommand(ABC):
@abstractmethod
def register(cls, parsers: Any) -> None:
raise UnexpectedException("Not implemented.")
@classmethod
def _create_connection(cls, remote: str) -> MDRSConnection:
config = ConfigFile(remote)
if config.url is None:
raise MissingConfigurationException(f"Remote host `{remote}` is not found.")
return MDRSConnection(config.remote, config.url)
@classmethod
def _find_laboratory(cls, connection: MDRSConnection, name: str) -> Laboratory:
if connection.laboratories.empty() or connection.token is not None and connection.token.is_expired:
laboratory_api = LaboratoriesApi(connection)
connection.laboratories = laboratory_api.list()
laboratory = connection.laboratories.find_by_name(name)
if laboratory is None:
raise IllegalArgumentException(f"Laboratory `{name}` not found.")
return laboratory
@classmethod
def _find_folder(
cls, connection: MDRSConnection, laboratory: Laboratory, path: str, password: str | None = None
) -> Folder:
folder_api = FoldersApi(connection)
folders = folder_api.list(laboratory.id, normalize("NFC", path))
if len(folders) != 1:
raise UnexpectedException(f"Folder `{path}` not found.")
if folders[0].lock:
if password is None:
raise UnauthorizedException(f"Folder `{path}` is locked.")
folder_api.auth(folders[0].id, password)
return folder_api.retrieve(folders[0].id)
@classmethod
def _find_files(cls, connection: MDRSConnection, folder_id: str) -> list[File]:
files_api = FilesApi(connection)
page = 1
results_file = []
while page:
result = files_api.list(folder_id, page)
results_file.extend(result.results)
page = 0
if result.next:
page = page_num_from_url(result.next)
return results_file
@classmethod
def _parse_remote_host(cls, path: str) -> str:
path_array = path.split(":")
remote_host = path_array[0]
if len(path_array) == 2 and path_array[1] != "" or len(path_array) > 2:
raise IllegalArgumentException("Invalid remote host")
return remote_host
@classmethod
def _parse_remote_host_with_path(cls, path: str) -> tuple[str, str, str]:
path = re.sub(r"//+|/\./+|/\.$", "/", path)
if re.search(r"/\.\./|/\.\.$", path) is not None:
raise IllegalArgumentException("Path traversal found.")
path_array = path.split(":")
if len(path_array) != 2:
raise IllegalArgumentException("Invalid remote host.")
remote_host = path_array[0]
folder_array = path_array[1].split("/")
is_absolute_path = folder_array[0] == ""
if not is_absolute_path:
raise IllegalArgumentException("Must be absolute paths.")
del folder_array[0]
if len(folder_array) == 0:
laboratory = ""
folder = ""
else:
laboratory = folder_array.pop(0)
folder = "/" + "/".join(folder_array)
return (remote_host, laboratory, folder)
+5 -8
View File
@@ -1,7 +1,6 @@
from argparse import Namespace
from typing import Any
from mdrsclient.api import FoldersApi
from mdrsclient.commands.base import BaseCommand
from mdrsclient.exceptions import IllegalArgumentException
from mdrsclient.models import FolderAccessLevel
@@ -31,10 +30,8 @@ class ChaclCommand(BaseCommand):
@classmethod
def chacl(cls, remote_path: str, access_level: int, is_recursive: bool, password: str | None) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
connection = cls._create_connection(remote)
laboratory = cls._find_laboratory(connection, laboratory_name)
folder = cls._find_folder(connection, laboratory, r_path)
folder_api = FoldersApi(connection)
folder_api.acl(folder.id, access_level, is_recursive, password)
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
from mdrsclient.client import MdrsClient
client = MdrsClient.from_remote(remote)
client.chacl(remote_path, access_level, is_recursive, password)
+16 -44
View File
@@ -2,8 +2,6 @@ from argparse import Namespace
from typing import Any, Callable
from mdrsclient.commands.base import BaseCommand
from mdrsclient.config import ConfigFile
from mdrsclient.exceptions import IllegalArgumentException
class ConfigCommand(BaseCommand):
@@ -26,10 +24,11 @@ class ConfigCommand(BaseCommand):
update_parser.set_defaults(func=cls.func_update)
# config list
list_parser = config_parsers.add_parser("list", help="list all the remote hosts", aliases=["ls"])
list_parser.add_argument("-l", "--long", help="show the api url", action="store_true")
list_parser.set_defaults(func=cls.func_list)
# config delete
delete_parser = config_parsers.add_parser("delete", help="delete an existing remote host", aliases=["remove"])
delete_parser = config_parsers.add_parser(
"delete", help="delete an existing remote host", aliases=["remove", "rm"]
)
delete_parser.add_argument("remote", help="label of remote host")
delete_parser.set_defaults(func=cls.func_delete)
@@ -37,56 +36,29 @@ class ConfigCommand(BaseCommand):
def func_create(cls, args: Namespace) -> None:
remote = str(args.remote)
url = str(args.url)
cls.create(remote, url)
from mdrsclient.client import MdrsClient
MdrsClient(None).config_create(remote, url)
@classmethod
def func_update(cls, args: Namespace) -> None:
remote = str(args.remote)
url = str(args.url)
cls.update(remote, url)
from mdrsclient.client import MdrsClient
MdrsClient(None).config_update(remote, url)
@classmethod
def func_list(cls, args: Namespace) -> None:
is_long = bool(args.long)
cls.list(is_long)
from mdrsclient.client import MdrsClient
client = MdrsClient(None)
for remote, url in client.config_list():
print(f"{remote}:\t{url}")
@classmethod
def func_delete(cls, args: Namespace) -> None:
remote = str(args.remote)
cls.delete(remote)
from mdrsclient.client import MdrsClient
@classmethod
def create(cls, remote: str, url: str) -> None:
remote = cls._parse_remote_host(remote)
config = ConfigFile(remote)
if config.url is not None:
raise IllegalArgumentException(f"Remote host `{remote}` is already exists.")
else:
config.url = url
@classmethod
def update(cls, remote: str, url: str) -> None:
remote = cls._parse_remote_host(remote)
config = ConfigFile(remote)
if config.url is None:
raise IllegalArgumentException(f"Remote host `{remote}` is not exists.")
else:
config.url = url
@classmethod
def list(cls, is_long: bool) -> None:
config = ConfigFile("")
for remote, url in config.list():
line = f"{remote}:"
if is_long:
line += f"\t{url}"
print(line)
@classmethod
def delete(cls, remote: str) -> None:
remote = cls._parse_remote_host(remote)
config = ConfigFile(remote)
if config.url is None:
raise IllegalArgumentException(f"Remote host `{remote}` is not exists.")
else:
del config.url
MdrsClient(None).config_delete(remote)
+5 -55
View File
@@ -1,12 +1,7 @@
import os
from argparse import Namespace
from typing import Any
from unicodedata import normalize
from mdrsclient.api import FilesApi, FoldersApi
from mdrsclient.commands.base import BaseCommand
from mdrsclient.exceptions import IllegalArgumentException
from mdrsclient.models.file import find_file
class CpCommand(BaseCommand):
@@ -29,53 +24,8 @@ class CpCommand(BaseCommand):
@classmethod
def cp(cls, src_path: str, dest_path: str, is_recursive: bool) -> None:
(s_remote, s_laboratory_name, s_path) = cls._parse_remote_host_with_path(src_path)
(d_remote, d_laboratory_name, d_path) = cls._parse_remote_host_with_path(dest_path)
if s_remote != d_remote:
raise IllegalArgumentException("Remote host mismatched.")
if s_laboratory_name != d_laboratory_name:
raise IllegalArgumentException("Laboratory mismatched.")
s_path = s_path.rstrip("/")
s_dirname = os.path.dirname(s_path)
s_basename = os.path.basename(s_path)
if d_path.endswith("/"):
d_dirname = d_path
d_basename = s_basename
else:
d_dirname = os.path.dirname(d_path)
d_basename = os.path.basename(d_path)
connection = cls._create_connection(s_remote)
laboratory = cls._find_laboratory(connection, s_laboratory_name)
s_parent_folder = cls._find_folder(connection, laboratory, s_dirname)
s_parent_files = cls._find_files(connection, s_parent_folder.id)
d_parent_folder = cls._find_folder(connection, laboratory, d_dirname)
d_parent_files = cls._find_files(connection, d_parent_folder.id)
s_file = find_file(s_parent_files, s_basename)
if s_file is not None:
# source is file
d_file = find_file(d_parent_files, d_basename)
if d_file is not None:
raise IllegalArgumentException(f"File `{d_basename}` already exists.")
d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
if d_sub_folder is not None:
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
file_api = FilesApi(connection)
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
file_api.copy(s_file, d_parent_folder.id, normalize("NFC", d_basename))
else:
s_folder = s_parent_folder.find_sub_folder(s_basename)
if s_folder is None:
raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
# source is folder
if not is_recursive:
raise IllegalArgumentException(f"Cannot copy `{s_path}`: Is a folder.")
if find_file(d_parent_files, d_basename) is not None:
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
d_folder = d_parent_folder.find_sub_folder(d_basename)
if d_folder is not None:
if d_folder.id == s_folder.id:
raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
folder_api = FoldersApi(connection)
if s_parent_folder.id != d_parent_folder.id or s_basename != d_basename:
folder_api.copy(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
remote = src_path.split(":", 1)[0] if ":" in src_path else ""
from mdrsclient.client import MdrsClient
client = MdrsClient.from_remote(remote)
client.cp(src_path, dest_path, is_recursive)
+6 -114
View File
@@ -1,30 +1,7 @@
import os
from argparse import Namespace
from concurrent.futures import ThreadPoolExecutor
from typing import Any
from pydantic.dataclasses import dataclass
from mdrsclient.api import FilesApi, FoldersApi
from mdrsclient.commands.base import BaseCommand
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import IllegalArgumentException, UnexpectedException
from mdrsclient.models import File, Folder, Laboratory
from mdrsclient.models.file import find_file
from mdrsclient.settings import CONCURRENT
@dataclass(frozen=True)
class DownloadFileInfo:
file: File
path: str
@dataclass
class DownloadContext:
hasError: bool
isSkipIfExists: bool
files: list[DownloadFileInfo]
class DownloadCommand(BaseCommand):
@@ -37,7 +14,7 @@ class DownloadCommand(BaseCommand):
download_parser.add_argument(
"-s",
"--skip-if-exists",
help="skip the download if file is already downloaded and file size is the same",
help="skip the download if file is already uploaded and file size is the same",
action="store_true",
)
download_parser.add_argument(
@@ -68,94 +45,9 @@ class DownloadCommand(BaseCommand):
password: str | None,
excludes: list[str],
) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
r_dirname = os.path.dirname(r_path)
r_basename = os.path.basename(r_path)
connection = cls._create_connection(remote)
l_dirname = os.path.realpath(local_path)
if not os.path.isdir(l_dirname):
raise IllegalArgumentException(f"Local directory `{local_path}` not found.")
laboratory = cls._find_laboratory(connection, laboratory_name)
r_parent_folder = cls._find_folder(connection, laboratory, r_dirname, password)
r_parent_files = cls._find_files(connection, r_parent_folder.id)
file = find_file(r_parent_files, r_basename)
if file is not None:
if cls.__check_excludes(excludes, laboratory, r_parent_folder, file):
return
context = DownloadContext(False, is_skip_if_exists, [])
l_path = os.path.join(l_dirname, r_basename)
context.files.append(DownloadFileInfo(file, l_path))
cls.__multiple_download(connection, context)
else:
folder = r_parent_folder.find_sub_folder(r_basename)
if folder is None:
raise IllegalArgumentException(f"File or folder `{r_path}` not found.")
if not is_recursive:
raise IllegalArgumentException(f"Cannot download `{r_path}`: Is a folder.")
folder_api = FoldersApi(connection)
cls.__multiple_download_pickup_recursive_files(
connection, folder_api, laboratory, folder.id, l_dirname, excludes, is_skip_if_exists
)
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
from mdrsclient.client import MdrsClient
@classmethod
def __multiple_download_pickup_recursive_files(
cls,
connection: MDRSConnection,
folder_api: FoldersApi,
laboratory: Laboratory,
folder_id: str,
basedir: str,
excludes: list[str],
is_skip_if_exists: bool,
) -> None:
context = DownloadContext(False, is_skip_if_exists, [])
folder = folder_api.retrieve(folder_id)
files = cls._find_files(connection, folder.id)
dirname = os.path.join(basedir, folder.name)
if cls.__check_excludes(excludes, laboratory, folder, None):
return
if not os.path.exists(dirname):
os.makedirs(dirname)
print(dirname)
for file in files:
if cls.__check_excludes(excludes, laboratory, folder, file):
continue
path = os.path.join(dirname, file.name)
context.files.append(DownloadFileInfo(file, path))
cls.__multiple_download(connection, context)
if context.hasError:
raise UnexpectedException("Some files failed to download.")
for sub_folder in folder.sub_folders:
cls.__multiple_download_pickup_recursive_files(
connection, folder_api, laboratory, sub_folder.id, dirname, excludes, is_skip_if_exists
)
@classmethod
def __multiple_download(cls, connection: MDRSConnection, context: DownloadContext) -> None:
file_api = FilesApi(connection)
with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
results = pool.map(
lambda x: cls.__multiple_download_worker(file_api, x, context.isSkipIfExists), context.files
)
hasError = next(filter(lambda x: x is False, results), None)
if hasError is not None:
context.hasError = True
@classmethod
def __multiple_download_worker(cls, file_api: FilesApi, info: DownloadFileInfo, is_skip_if_exists: bool) -> bool:
if not is_skip_if_exists or not os.path.exists(info.path) or info.file.size != os.path.getsize(info.path):
try:
file_api.download(info.file, info.path)
except Exception:
print(f"Failed: ${info.path}")
if os.path.isfile(info.path):
os.remove(info.path)
return False
print(info.path)
return True
@classmethod
def __check_excludes(cls, excludes: list[str], laboratory: Laboratory, folder: Folder, file: File | None) -> bool:
path = f"/{laboratory.name}{folder.path}{file.name if file is not None else ''}".rstrip("/").lower()
return path in excludes
client = MdrsClient.from_remote(remote)
client.download(remote_path, local_path, is_recursive, is_skip_if_exists, password, excludes)
return
+5 -17
View File
@@ -1,12 +1,8 @@
import json
import os
from argparse import Namespace
from typing import Any
from mdrsclient.api import FilesApi
from mdrsclient.commands.base import BaseCommand
from mdrsclient.exceptions import IllegalArgumentException
from mdrsclient.models.file import find_file
class FileMetadataCommand(BaseCommand):
@@ -25,17 +21,9 @@ class FileMetadataCommand(BaseCommand):
@classmethod
def file_metadata(cls, remote_path: str, password: str | None) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
r_dirname = os.path.dirname(r_path)
r_basename = os.path.basename(r_path)
connection = cls._create_connection(remote)
laboratory = cls._find_laboratory(connection, laboratory_name)
folder = cls._find_folder(connection, laboratory, r_dirname, password)
files = cls._find_files(connection, folder.id)
file = find_file(files, r_basename)
if file is None:
raise IllegalArgumentException(f"File `{r_basename}` not found.")
file_api = FilesApi(connection)
metadata = file_api.metadata(file)
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
from mdrsclient.client import MdrsClient
client = MdrsClient.from_remote(remote)
metadata = client.file_metadata(remote_path, password)
print(json.dumps(metadata, ensure_ascii=False))
+5 -8
View File
@@ -1,7 +1,6 @@
from argparse import Namespace
from typing import Any
from mdrsclient.api import LaboratoriesApi
from mdrsclient.commands.base import BaseCommand
@@ -19,11 +18,11 @@ class LabsCommand(BaseCommand):
@classmethod
def labs(cls, remote: str) -> None:
remote = cls._parse_remote_host(remote)
connection = cls._create_connection(remote)
laboratory_api = LaboratoriesApi(connection)
laboratories = laboratory_api.list()
connection.laboratories = laboratories
from mdrsclient.client import MdrsClient
remote_host = MdrsClient.parse_remote_host(remote)
client = MdrsClient.from_remote(remote_host)
laboratories = client.get_laboratories()
label = {"id": "ID", "name": "Name", "pi_name": "PI", "full_name": "Laboratory"}
length: dict[str, int] = {}
for key in label.keys():
@@ -34,7 +33,6 @@ class LabsCommand(BaseCommand):
length["pi_name"] = max(length["pi_name"], len(laboratory.pi_name))
length["full_name"] = max(length["full_name"], len(laboratory.full_name))
header = (
# f"{label['id']:{length['id']}}\t{label['name']:{length['name']}}\t"
f"{label['name']:{length['name']}}\t"
f"{label['pi_name']:{length['pi_name']}}\t{label['full_name']:{length['full_name']}}"
)
@@ -42,7 +40,6 @@ class LabsCommand(BaseCommand):
print("-" * len(header.expandtabs()))
for laboratory in laboratories:
print(
# f"{laboratory.id:{length['id']}}\t{laboratory.name:{length['name']}}\t"
f"{laboratory.name:{length['name']}}\t"
f"{laboratory.pi_name:{length['pi_name']}}\t{laboratory.full_name:{length['full_name']}}"
)
+6 -15
View File
@@ -2,11 +2,7 @@ import getpass
from argparse import Namespace
from typing import Any
from mdrsclient.api import UsersApi
from mdrsclient.commands.base import BaseCommand
from mdrsclient.config import ConfigFile
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import MissingConfigurationException
class LoginCommand(BaseCommand):
@@ -21,20 +17,15 @@ class LoginCommand(BaseCommand):
@classmethod
def func(cls, args: Namespace) -> None:
remote = str(args.remote)
username = str(args.username) if args.password else input("Username: ").strip()
username = str(args.username) if args.username else input("Username: ").strip()
password = str(args.password) if args.password else getpass.getpass("Password: ").strip()
cls.login(remote, username, password)
@classmethod
def login(cls, remote: str, username: str, password: str) -> None:
remote = cls._parse_remote_host(remote)
config = ConfigFile(remote)
if config.url is None:
raise MissingConfigurationException(f"Remote host `{remote}` is not found.")
connection = MDRSConnection(config.remote, config.url)
user_api = UsersApi(connection)
token = user_api.token(username, password)
connection.token = token
user = user_api.current()
connection.user = user
from mdrsclient.client import MdrsClient
remote_host = MdrsClient.parse_remote_host(remote)
client = MdrsClient.from_remote(remote_host)
client.login(username, password)
print("Login Successful")
+5 -9
View File
@@ -2,9 +2,6 @@ from argparse import Namespace
from typing import Any
from mdrsclient.commands.base import BaseCommand
from mdrsclient.config import ConfigFile
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import MissingConfigurationException
class LogoutCommand(BaseCommand):
@@ -21,9 +18,8 @@ class LogoutCommand(BaseCommand):
@classmethod
def logout(cls, remote: str) -> None:
remote = cls._parse_remote_host(remote)
config = ConfigFile(remote)
if config.url is None:
raise MissingConfigurationException(f"Remote host `{remote}` is not found.")
connection = MDRSConnection(config.remote, config.url)
connection.logout()
from mdrsclient.client import MdrsClient
remote_host = MdrsClient.parse_remote_host(remote)
client = MdrsClient.from_remote(remote_host)
client.logout()
+75 -47
View File
@@ -1,12 +1,13 @@
import json
from argparse import Namespace
from typing import Any
from typing import Any, Final
from pydantic.dataclasses import dataclass
from mdrsclient.api import FilesApi, FoldersApi
from mdrsclient.api import FoldersApi
from mdrsclient.client import MdrsClient
from mdrsclient.commands.base import BaseCommand
from mdrsclient.connection import MDRSConnection
from mdrsclient.config import build_download_url
from mdrsclient.exceptions import UnauthorizedException
from mdrsclient.models import File, Folder, FolderSimple, Laboratory
@@ -19,11 +20,11 @@ class Config:
@dataclass(config=Config)
class LsCommandContext:
prefix: str
connection: MDRSConnection
client: MdrsClient
laboratory: Laboratory
password: str
is_json: bool
is_quick: bool
is_quiet: bool
is_recursive: bool
@@ -35,7 +36,7 @@ class LsCommand(BaseCommand):
ls_parser.add_argument("-J", "--json", help="turn on json output", action="store_true")
ls_parser.add_argument(
"-q",
"--quick",
"--quiet",
help="don't output header row. this option is forced if the -r option is specified",
action="store_true",
)
@@ -49,25 +50,42 @@ class LsCommand(BaseCommand):
password = str(args.password) if args.password else None
is_json = bool(args.json)
is_recursive = bool(args.recursive)
is_quick = bool(args.quick) if not is_recursive else True
cls.ls(remote_path, password, is_json, is_recursive, is_quick)
is_quiet = bool(args.quiet) if not is_recursive else True
cls.ls(remote_path, password, is_json, is_recursive, is_quiet)
@classmethod
def ls(cls, remote_path: str, password: str | None, is_json: bool, is_recursive: bool, is_quick: bool) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
connection = cls._create_connection(remote)
laboratory = cls._find_laboratory(connection, laboratory_name)
def ls(cls, remote_path: str, password: str | None, is_json: bool, is_recursive: bool, is_quiet: bool) -> None:
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
from mdrsclient.client import MdrsClient
client = MdrsClient.from_remote(remote)
cls._ls_logic(client, remote_path, password, is_json, is_recursive, is_quiet)
return
@classmethod
def _ls_logic(
cls,
client: MdrsClient,
remote_path: str,
password: str | None,
is_json: bool,
is_recursive: bool,
is_quiet: bool,
) -> None:
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
folder, laboratory = client.resolve_folder(remote_path, password)
laboratory_name = laboratory.name
files = client.find_files(folder.id)
context = LsCommandContext(
f"{remote}:/{laboratory_name}",
connection,
client,
laboratory,
password if password is not None else "",
is_json,
is_quick,
is_quiet,
is_recursive,
)
folder = cls._find_folder(connection, laboratory, r_path, password)
files = cls._find_files(connection, folder.id)
if context.is_json:
cls._ls_json(context, folder, files)
else:
@@ -77,25 +95,24 @@ class LsCommand(BaseCommand):
def _ls_json(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
print(json.dumps(cls._folder2dict(context, folder, files), ensure_ascii=False))
LABELS: Final[dict[str, str]] = {
"type": "Type",
"acl": "Access",
"laboratory": "Laboratory",
"size": "Size",
"date": "Date",
"name": "Name",
}
@classmethod
def _ls_plain(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
label = {
"type": "Type",
"acl": "Access",
"laboratory": "Laboratory",
"size": "Size",
"date": "Date",
"name": "Name",
}
length: dict[str, int] = {}
for key in label.keys():
length[key] = len(label[key]) if not context.is_quick else 0
def _column_widths(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> dict[str, int]:
"""Width of each column: the widest of its heading and everything printed under it."""
length = {key: len(label) if not context.is_quiet else 0 for key, label in cls.LABELS.items()}
for sub_folder in folder.sub_folders:
sub_laboratory = context.connection.laboratories.find_by_id(sub_folder.laboratory_id)
sub_laboratory_name = sub_laboratory.name if sub_laboratory is not None else "(invalid)"
sub_laboratory_name = cls._laboratory_name(context, sub_folder.laboratory_id)
length["acl"] = max(length["acl"], len(sub_folder.access_level_name))
length["laboratory"] = max(length["laboratory"], len(sub_laboratory_name))
length["size"] = max(length["size"], len(str(folder.size)))
length["size"] = max(length["size"], len(str(sub_folder.size)))
length["date"] = max(length["date"], len(sub_folder.updated_at_name))
length["name"] = max(length["name"], len(sub_folder.name))
for file in files:
@@ -104,6 +121,27 @@ class LsCommand(BaseCommand):
length["name"] = max(length["name"], len(file.name))
length["acl"] = max(length["acl"], len(folder.access_level_name))
length["laboratory"] = max(length["laboratory"], len(context.laboratory.name))
return length
@classmethod
def _ls_plain_children(cls, context: LsCommandContext, folder: Folder) -> None:
"""List each sub-folder in turn, passing over the ones the caller cannot open."""
print("")
folder_api = FoldersApi(context.client.connection)
for sub_folder in sorted(folder.sub_folders, key=lambda x: x.name):
try:
if sub_folder.lock:
folder_api.auth(sub_folder.id, context.password)
sub_detail = folder_api.retrieve(sub_folder.id)
sub_files = context.client.find_files(sub_folder.id)
except UnauthorizedException:
continue
cls._ls_plain(context, sub_detail, sub_files)
@classmethod
def _ls_plain(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
label = cls.LABELS
length = cls._column_widths(context, folder, files)
header = (
f"{label['type']:{length['type']}}\t{label['acl']:{length['acl']}}\t"
f"{label['laboratory']:{length['laboratory']}}\t{label['size']:{length['size']}}\t"
@@ -114,7 +152,7 @@ class LsCommand(BaseCommand):
print(f"{context.prefix}{folder.path}:")
print(f"total {sum(f.size for f in files)}")
if not context.is_quick:
if not context.is_quiet:
print(header)
print("-" * len(header.expandtabs()))
@@ -134,17 +172,7 @@ class LsCommand(BaseCommand):
)
if context.is_recursive:
print("")
for sub_folder in sorted(folder.sub_folders, key=lambda x: x.name):
folder_api = FoldersApi(context.connection)
try:
if sub_folder.lock:
folder_api.auth(sub_folder.id, context.password)
folder = folder_api.retrieve(sub_folder.id)
files = cls._find_files(context.connection, sub_folder.id)
cls._ls_plain(context, folder, files)
except UnauthorizedException:
pass
cls._ls_plain_children(context, folder)
@classmethod
def _folder2dict(
@@ -163,7 +191,7 @@ class LsCommand(BaseCommand):
"updated_at": folder.updated_at,
}
if isinstance(folder, Folder):
folder_api = FoldersApi(context.connection)
folder_api = FoldersApi(context.client.connection)
data["metadata"] = folder_api.metadata(folder.id)
if context.is_recursive:
sub_folders: list[dict[str, Any]] = []
@@ -172,7 +200,7 @@ class LsCommand(BaseCommand):
if sub_folder.lock:
folder_api.auth(sub_folder.id, context.password)
folder2 = folder_api.retrieve(sub_folder.id)
files2 = cls._find_files(context.connection, sub_folder.id)
files2 = context.client.find_files(sub_folder.id)
sub_folders.append(cls._folder2dict(context, folder2, files2))
except UnauthorizedException:
pass
@@ -194,7 +222,7 @@ class LsCommand(BaseCommand):
# "thumbnail": file.thumbnail,
"description": file.description,
"metadata": file.metadata,
"download_url": f"{context.connection.url}/{file.download_url}",
"download_url": build_download_url(context.client.connection.url, file.download_url),
"created_at": file.created_at,
"updated_at": file.updated_at,
}
@@ -202,5 +230,5 @@ class LsCommand(BaseCommand):
@classmethod
def _laboratory_name(cls, context: LsCommandContext, laboratory_id: int) -> str:
laboratory = context.connection.laboratories.find_by_id(laboratory_id)
laboratory = context.client.connection.laboratories.find_by_id(laboratory_id)
return laboratory.name if laboratory is not None else "(invalid)"
+5 -7
View File
@@ -2,7 +2,6 @@ import json
from argparse import Namespace
from typing import Any
from mdrsclient.api import FoldersApi
from mdrsclient.commands.base import BaseCommand
@@ -22,10 +21,9 @@ class MetadataCommand(BaseCommand):
@classmethod
def metadata(cls, remote_path: str, password: str | None) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
connection = cls._create_connection(remote)
laboratory = cls._find_laboratory(connection, laboratory_name)
folder = cls._find_folder(connection, laboratory, r_path, password)
folder_api = FoldersApi(connection)
metadata = folder_api.metadata(folder.id)
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
from mdrsclient.client import MdrsClient
client = MdrsClient.from_remote(remote)
metadata = client.metadata(remote_path, password)
print(json.dumps(metadata, ensure_ascii=False))
+5 -17
View File
@@ -1,12 +1,7 @@
import os
from argparse import Namespace
from typing import Any
from unicodedata import normalize
from mdrsclient.api import FoldersApi
from mdrsclient.commands.base import BaseCommand
from mdrsclient.exceptions import IllegalArgumentException
from mdrsclient.models.file import find_file
class MkdirCommand(BaseCommand):
@@ -23,15 +18,8 @@ class MkdirCommand(BaseCommand):
@classmethod
def mkdir(cls, remote_path: str) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
r_dirname = os.path.dirname(r_path)
r_basename = os.path.basename(r_path)
connection = cls._create_connection(remote)
laboratory = cls._find_laboratory(connection, laboratory_name)
parent_folder = cls._find_folder(connection, laboratory, r_dirname)
files = cls._find_files(connection, parent_folder.id)
if parent_folder.find_sub_folder(r_basename) is not None or find_file(files, r_basename) is not None:
raise IllegalArgumentException(f"Cannot create folder `{r_path}`: File exists.")
folder_api = FoldersApi(connection)
folder_api.create(normalize("NFC", r_basename), parent_folder.id)
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
from mdrsclient.client import MdrsClient
client = MdrsClient.from_remote(remote)
client.mkdir(remote_path)
+5 -53
View File
@@ -1,12 +1,7 @@
import os
from argparse import Namespace
from typing import Any
from unicodedata import normalize
from mdrsclient.api import FilesApi, FoldersApi
from mdrsclient.commands.base import BaseCommand
from mdrsclient.exceptions import IllegalArgumentException
from mdrsclient.models.file import find_file
class MvCommand(BaseCommand):
@@ -25,51 +20,8 @@ class MvCommand(BaseCommand):
@classmethod
def mv(cls, src_path: str, dest_path: str) -> None:
(s_remote, s_laboratory_name, s_path) = cls._parse_remote_host_with_path(src_path)
(d_remote, d_laboratory_name, d_path) = cls._parse_remote_host_with_path(dest_path)
if s_remote != d_remote:
raise IllegalArgumentException("Remote host mismatched.")
if s_laboratory_name != d_laboratory_name:
raise IllegalArgumentException("Laboratory mismatched.")
s_path = s_path.rstrip("/")
s_dirname = os.path.dirname(s_path)
s_basename = os.path.basename(s_path)
if d_path.endswith("/"):
d_dirname = d_path
d_basename = s_basename
else:
d_dirname = os.path.dirname(d_path)
d_basename = os.path.basename(d_path)
connection = cls._create_connection(s_remote)
laboratory = cls._find_laboratory(connection, s_laboratory_name)
s_parent_folder = cls._find_folder(connection, laboratory, s_dirname)
s_parent_files = cls._find_files(connection, s_parent_folder.id)
d_parent_folder = cls._find_folder(connection, laboratory, d_dirname)
d_parent_files = cls._find_files(connection, d_parent_folder.id)
s_file = find_file(s_parent_files, s_basename)
if s_file is not None:
# source is file
d_file = find_file(d_parent_files, d_basename)
if d_file is not None:
raise IllegalArgumentException(f"File `{d_basename}` already exists.")
d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
if d_sub_folder is not None:
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
file_api = FilesApi(connection)
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
file_api.move(s_file, d_parent_folder.id, normalize("NFC", d_basename))
else:
s_folder = s_parent_folder.find_sub_folder(s_basename)
if s_folder is None:
raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
# source is folder
if find_file(d_parent_files, d_basename) is not None:
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
d_folder = d_parent_folder.find_sub_folder(d_basename)
if d_folder is not None:
if d_folder.id == s_folder.id:
raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
folder_api = FoldersApi(connection)
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
folder_api.move(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
remote = src_path.split(":", 1)[0] if ":" in src_path else ""
from mdrsclient.client import MdrsClient
client = MdrsClient.from_remote(remote)
client.mv(src_path, dest_path)
+5 -24
View File
@@ -1,11 +1,7 @@
import os
from argparse import Namespace
from typing import Any
from mdrsclient.api import FilesApi, FoldersApi
from mdrsclient.commands.base import BaseCommand
from mdrsclient.exceptions import IllegalArgumentException
from mdrsclient.models.file import find_file
class RmCommand(BaseCommand):
@@ -26,23 +22,8 @@ class RmCommand(BaseCommand):
@classmethod
def rm(cls, remote_path: str, is_recursive: bool) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
r_dirname = os.path.dirname(r_path)
r_basename = os.path.basename(r_path)
connection = cls._create_connection(remote)
laboratory = cls._find_laboratory(connection, laboratory_name)
parent_folder = cls._find_folder(connection, laboratory, r_dirname)
parent_files = cls._find_files(connection, parent_folder.id)
file = find_file(parent_files, r_basename)
if file is not None:
file_api = FilesApi(connection)
file_api.destroy(file)
else:
folder = parent_folder.find_sub_folder(r_basename)
if folder is None:
raise IllegalArgumentException(f"Cannot remove `{r_path}`: No such file or folder.")
if not is_recursive:
raise IllegalArgumentException(f"Cannot remove `{r_path}`: Is a folder.")
folder_api = FoldersApi(connection)
folder_api.destroy(folder.id, True)
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
from mdrsclient.client import MdrsClient
client = MdrsClient.from_remote(remote)
client.rm(remote_path, is_recursive)
+5 -86
View File
@@ -1,24 +1,7 @@
import os
from argparse import Namespace
from concurrent.futures import ThreadPoolExecutor
from typing import Any
from pydantic.dataclasses import dataclass
from mdrsclient.api import FilesApi, FoldersApi
from mdrsclient.commands.base import BaseCommand
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import IllegalArgumentException, MDRSException
from mdrsclient.models import File, Folder
from mdrsclient.models.file import find_file
from mdrsclient.settings import CONCURRENT
@dataclass(frozen=True)
class UploadFileInfo:
folder: Folder
files: list[File]
path: str
class UploadCommand(BaseCommand):
@@ -48,73 +31,9 @@ class UploadCommand(BaseCommand):
@classmethod
def upload(cls, local_path: str, remote_path: str, is_recursive: bool, is_skip_if_exists: bool) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
l_path = os.path.abspath(local_path)
if not os.path.exists(l_path):
raise IllegalArgumentException(f"File or directory `{local_path}` not found.")
connection = cls._create_connection(remote)
laboratory = cls._find_laboratory(connection, laboratory_name)
folder = cls._find_folder(connection, laboratory, r_path)
files = cls._find_files(connection, folder.id)
infos: list[UploadFileInfo] = []
if os.path.isdir(l_path):
if not is_recursive:
raise IllegalArgumentException(f"Cannot upload `{local_path}`: Is a directory.")
folder_api = FoldersApi(connection)
folder_map: dict[str, Folder] = {}
folder_map[r_path] = folder
files_map: dict[str, list[File]] = {}
files_map[r_path] = files
l_basename = os.path.basename(l_path)
for dirpath, _, filenames in os.walk(l_path, followlinks=True):
sub = l_basename if dirpath == l_path else os.path.join(l_basename, os.path.relpath(dirpath, l_path))
d_dirname = os.path.join(r_path, sub)
d_basename = os.path.basename(d_dirname)
# prepare destination parent path
d_parent_dirname = os.path.dirname(d_dirname)
if folder_map.get(d_parent_dirname) is None:
parent_folder = cls._find_folder(connection, laboratory, d_parent_dirname)
folder_map[d_parent_dirname] = parent_folder
parent_files = cls._find_files(connection, parent_folder.id)
files_map[d_parent_dirname] = parent_files
# prepare destination path
if folder_map.get(d_dirname) is None:
d_folder = folder_map[d_parent_dirname].find_sub_folder(d_basename)
if d_folder is None:
d_folder_id = folder_api.create(d_basename, folder_map[d_parent_dirname].id)
else:
d_folder_id = d_folder.id
print(d_dirname)
folder_map[d_dirname] = folder_api.retrieve(d_folder_id)
files_map[d_dirname] = cls._find_files(connection, d_folder_id)
if d_folder is None:
folder_map[d_parent_dirname].sub_folders.append(folder_map[d_dirname])
# register upload file list
for filename in filenames:
infos.append(
UploadFileInfo(folder_map[d_dirname], files_map[d_dirname], os.path.join(dirpath, filename))
)
else:
infos.append(UploadFileInfo(folder, files, l_path))
cls.__multiple_upload(connection, infos, is_skip_if_exists)
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
from mdrsclient.client import MdrsClient
@classmethod
def __multiple_upload(
cls, connection: MDRSConnection, infos: list[UploadFileInfo], is_skip_if_exists: bool
) -> None:
file_api = FilesApi(connection)
with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
pool.map(lambda x: cls.__multiple_upload_worker(file_api, x, is_skip_if_exists), infos)
@classmethod
def __multiple_upload_worker(cls, file_api: FilesApi, info: UploadFileInfo, is_skip_if_exists: bool) -> None:
basename = os.path.basename(info.path)
file = find_file(info.files, basename)
try:
if file is None:
file_api.create(info.folder.id, info.path)
elif not is_skip_if_exists or file.size != os.path.getsize(info.path):
file_api.update(file, info.path)
print(os.path.join(info.folder.path, basename))
except MDRSException as e:
print(f"Error: {e}")
client = MdrsClient.from_remote(remote)
client.upload(local_path, remote_path, is_recursive, is_skip_if_exists)
return
+23
View File
@@ -0,0 +1,23 @@
from argparse import Namespace
from typing import Any
from mdrsclient.commands.base import BaseCommand
class VersionCommand(BaseCommand):
@classmethod
def register(cls, parsers: Any) -> None:
version_parser = parsers.add_parser("version", help="show the version of this tool")
version_parser.set_defaults(func=cls.func)
@classmethod
def func(cls, args: Namespace) -> None:
cls.version()
@classmethod
def version(cls) -> None:
from mdrsclient.client import MdrsClient
# Client initialization is not strictly needed for version, but for consistency:
client = MdrsClient(None)
print(client.version())
+11 -11
View File
@@ -2,9 +2,6 @@ from argparse import Namespace
from typing import Any, Final
from mdrsclient.commands.base import BaseCommand
from mdrsclient.config import ConfigFile
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import MissingConfigurationException
class WhoamiCommand(BaseCommand):
@@ -23,12 +20,15 @@ class WhoamiCommand(BaseCommand):
@classmethod
def whoami(cls, remote: str) -> None:
remote = cls._parse_remote_host(remote)
config = ConfigFile(remote)
if config.url is None:
raise MissingConfigurationException(f"Remote host `{remote}` is not found.")
connection = MDRSConnection(config.remote, config.url)
if connection.token is not None and connection.token.is_expired:
connection.logout()
username = connection.user.username if connection.user is not None else cls.ANONYMOUS_USERNAME
from mdrsclient.client import MdrsClient
remote_host = MdrsClient.parse_remote_host(remote)
client = MdrsClient.from_remote(remote_host)
if client.connection.token is not None and client.connection.token.is_expired:
client.logout()
try:
user = client.whoami()
username = user.username if user is not None else cls.ANONYMOUS_USERNAME
except Exception:
username = cls.ANONYMOUS_USERNAME
print(username)
+85 -4
View File
@@ -1,6 +1,7 @@
import configparser
import os
from typing import Final
import threading
from typing import TYPE_CHECKING, Final, Protocol, runtime_checkable
import validators
@@ -9,7 +10,88 @@ from mdrsclient.settings import CONFIG_DIRNAME
from mdrsclient.utils import FileLock
class ConfigFile:
def normalize_url(url: str) -> str:
"""
Check a remote URL and put it in the one form every client agrees on.
`simple_host` is what lets a bare hostname through, so a development server on
`localhost` is as acceptable as a deployment behind a domain name. The trailing
slash goes because the URL is joined with a path that brings its own.
"""
if not validators.url(url, simple_host=True, validate_scheme=lambda scheme: scheme in ("http", "https")):
raise IllegalArgumentException("malformed URI sequence")
return url.rstrip("/")
def build_download_url(base_url: str | None, path: str) -> str:
"""
Join a remote base URL with a path the API returned.
The API answers with a relative path and no leading separator, and a configuration
written before the URL was normalised may still carry a trailing one.
"""
if path.startswith(("http://", "https://")):
return path
return f"{(base_url or '').rstrip('/')}/{path.lstrip('/')}"
@runtime_checkable
class ConfigInterface(Protocol):
remote: str
if TYPE_CHECKING:
# Declared for the type checker, because the class itself is passed around and
# called: without this it describes something that takes no arguments to build.
# Kept out of the running class, because a protocol that defines `__init__` loses
# the guard that stops it being instantiated, and hands that empty `__init__` to
# any implementation that does not write its own.
def __init__(self, remote: str) -> None: ...
def list(self) -> list[tuple[str, str]]: ...
@property
def url(self) -> str | None: ...
@url.setter
def url(self, url: str) -> None: ...
@url.deleter
def url(self) -> None: ...
class InMemoryConfig(ConfigInterface):
__configs: dict[str, str] = {}
__lock: threading.Lock = threading.Lock()
remote: str
def __init__(self, remote: str) -> None:
self.remote = remote
def list(self) -> list[tuple[str, str]]:
with self.__lock:
return list(self.__configs.items())
@property
def url(self) -> str | None:
with self.__lock:
return self.__configs.get(self.remote)
@url.setter
def url(self, url: str) -> None:
url = normalize_url(url)
with self.__lock:
self.__configs[self.remote] = url
@url.deleter
def url(self) -> None:
with self.__lock:
if self.remote in self.__configs:
del self.__configs[self.remote]
@classmethod
def clear(cls) -> None:
with cls.__lock:
cls.__configs.clear()
class ConfigFile(ConfigInterface):
OPTION_URL: Final[str] = "url"
CONFIG_FILENAME: Final[str] = "config.ini"
remote: str
@@ -41,8 +123,7 @@ class ConfigFile:
@url.setter
def url(self, url: str) -> None:
if not validators.url(url):
raise IllegalArgumentException("malformed URI sequence")
url = normalize_url(url)
self.__load()
if self.__config.has_section(self.remote):
self.__config.remove_section(self.remote)
+19 -5
View File
@@ -1,5 +1,6 @@
import platform
import threading
from contextlib import AbstractContextManager
from typing import TypedDict
from requests import Response, Session
@@ -9,7 +10,7 @@ from requests_toolbelt.multipart.encoder import MultipartEncoder
from typing_extensions import Unpack
from mdrsclient.__version__ import __version__
from mdrsclient.cache import CacheFile
from mdrsclient.cache import CacheFile, CacheInterface
from mdrsclient.exceptions import MissingConfigurationException
from mdrsclient.models import Laboratories, Token, User
@@ -23,6 +24,7 @@ class _KwArgsMDRSConnectionPost(TypedDict, total=False):
params: dict[str, str | int]
data: dict[str, str | int] | MultipartEncoder
headers: dict[str, str]
timeout: float | tuple[float, float]
class _KwArgsMDRSConnectionPut(TypedDict, total=False):
@@ -39,14 +41,14 @@ class MDRSConnection:
url: str
session: Session
lock: threading.Lock
__cache: CacheFile
__cache: CacheInterface
def __init__(self, remote: str, url: str) -> None:
def __init__(self, remote: str, url: str, cache: CacheInterface | None = None) -> None:
super().__init__()
self.url = url
self.session = Session()
self.lock = threading.Lock()
self.__cache = CacheFile(remote)
self.__cache = cache if cache is not None else CacheFile(remote)
self.__prepare_headers()
def get(self, url: str, **kwargs: Unpack[_KwArgsMDRSConnectionGet]) -> Response:
@@ -61,6 +63,14 @@ class MDRSConnection:
def delete(self, url: str, **kwargs: Unpack[_KwArgsMDRSConnectionDelete]) -> Response:
return self.session.delete(self.__build_url(url), **kwargs)
def cache_lock(self) -> AbstractContextManager[None]:
"""Hold exclusive access to the login cache across every process using it."""
return self.__cache.lock()
def reload_cache(self) -> None:
"""Re-read the login cache, discarding anything held from an earlier read."""
self.__cache.reload()
def logout(self) -> None:
del self.__cache.user
del self.__cache.token
@@ -92,9 +102,13 @@ class MDRSConnection:
self.__cache.laboratories = laboratories
def __build_url(self, path: str) -> str:
if path.startswith("http://") or path.startswith("https://"):
return path
if self.url == "":
raise MissingConfigurationException("remote host is not configured")
return f"{self.url}/{path}"
# The path brings its own separator, and a configuration written before the URL
# was normalised may still carry a trailing slash of its own.
return f"{self.url.rstrip('/')}/{path}"
def __prepare_headers(self) -> None:
self.session.headers.update(
+12
View File
@@ -28,6 +28,18 @@ class UnauthorizedException(MDRSException):
pass
class TokenExpiredException(UnauthorizedException):
"""Thrown when the access token was still valid when sent but had expired on arrival"""
pass
class ServerBusyException(MDRSException):
"""Thrown when the server did not start handling a request before its token lapsed"""
pass
class ForbiddenException(MDRSException):
"""Thrown when the current user does not have enough privileges to access the resource"""
+2
View File
@@ -1,3 +1,4 @@
from mdrsclient.models.doi import Doi
from mdrsclient.models.error import DRFStandardizedErrors
from mdrsclient.models.file import File
from mdrsclient.models.folder import Folder, FolderAccessLevel, FolderSimple
@@ -6,6 +7,7 @@ from mdrsclient.models.user import Token, User
__all__ = [
"DRFStandardizedErrors",
"Doi",
"File",
"Folder",
"FolderAccessLevel",
+11
View File
@@ -0,0 +1,11 @@
from pydantic.dataclasses import dataclass
@dataclass(frozen=True)
class Doi:
"""Model representing a DOI entity (Response from GET v3/doi/{id}/)."""
# The internal DOI suffix ID returned as a string (e.g. "20260429-001").
id: str
doi: str
folder_id: str
+251
View File
@@ -0,0 +1,251 @@
import os
import re
from unicodedata import normalize
from mdrsclient.api import DoiApi, FilesApi, FoldersApi, LaboratoriesApi, UsersApi
from mdrsclient.cache import CacheInterface
from mdrsclient.config import ConfigFile, ConfigInterface
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import (
IllegalArgumentException,
MissingConfigurationException,
UnauthorizedException,
UnexpectedException,
)
from mdrsclient.models import File, Folder, Laboratory, Token, User
from mdrsclient.utils import page_num_from_url
class MdrsService:
config_class: type[ConfigInterface] = ConfigFile
def __init__(self, connection: MDRSConnection | None, config_class: type[ConfigInterface] | None = None):
self.__connection = connection
if config_class is not None:
self.config_class = config_class
@property
def connection(self) -> MDRSConnection:
"""
The connection every remote operation goes through.
Optional to supply, because `config` and `version` do their work without a remote
and are reached through the same client. Asking for it when none was given is a
mistake in the caller, and says so rather than failing later on `None`.
"""
if self.__connection is None:
raise MissingConfigurationException("This operation requires a remote host.")
return self.__connection
@connection.setter
def connection(self, connection: MDRSConnection | None) -> None:
# Assignable as it always was: this is a property to check for absence on read,
# not to make the connection fixed once the client is built.
self.__connection = connection
@classmethod
def create_connection(
cls, remote: str, cache: CacheInterface | None = None, config: ConfigInterface | None = None
) -> MDRSConnection:
if config is None:
config = ConfigFile(remote)
elif config.remote != remote:
raise IllegalArgumentException("Remote host parameter mismatch.")
if config.url is None:
raise MissingConfigurationException(f"Remote host `{remote}` is not found.")
return MDRSConnection(config.remote, config.url, cache=cache)
def login(self, username: str, password: str) -> tuple[Token, User]:
user_api = UsersApi(self.connection)
token = user_api.token(username, password)
self.connection.token = token
user = user_api.current()
self.connection.user = user
return token, user
def logout(self) -> None:
self.connection.logout()
def whoami(self) -> User:
user_api = UsersApi(self.connection)
return user_api.current()
def get_laboratories(self) -> list[Laboratory]:
laboratory_api = LaboratoriesApi(self.connection)
labs = laboratory_api.list()
self.connection.laboratories = labs
return list(labs)
def find_laboratory(self, name: str) -> Laboratory:
if self.connection.laboratories.empty() or (self.connection.token and self.connection.token.is_expired):
self.get_laboratories()
laboratory = self.connection.laboratories.find_by_name(name)
if laboratory is None:
raise IllegalArgumentException(f"Laboratory `{name}` not found.")
return laboratory
def find_folder(self, laboratory: Laboratory, path: str, password: str | None = None) -> Folder:
folder_api = FoldersApi(self.connection)
folders = folder_api.list(laboratory.id, normalize("NFC", path))
if len(folders) != 1:
raise UnexpectedException(f"Folder `{path}` not found.")
if folders[0].lock:
if password is None:
raise UnauthorizedException(f"Folder `{path}` is locked.")
folder_api.auth(folders[0].id, password)
return folder_api.retrieve(folders[0].id)
def find_files(self, folder_id: str) -> list[File]:
files_api = FilesApi(self.connection)
page = 1
results_file = []
while page:
result = files_api.list(folder_id, page)
results_file.extend(result.results)
page = 0
if result.next:
page = page_num_from_url(result.next)
return results_file
@staticmethod
def is_doi(path_component: str) -> bool:
return path_component.startswith("10.") and "/" in path_component
@staticmethod
def doi_suffix_id(doi: str) -> str:
doi = doi.rstrip("/")
slash_pos = doi.find("/")
if slash_pos == -1:
return doi
suffix = doi[slash_pos + 1 :]
dot_pos = suffix.rfind(".")
return suffix[dot_pos + 1 :] if dot_pos != -1 else suffix
@staticmethod
def split_doi_and_subpath(doi_with_path: str) -> tuple[str, str]:
first_slash = doi_with_path.find("/")
if first_slash != -1:
after_suffix_start = first_slash + 1
after_first = doi_with_path[after_suffix_start:]
second_slash = after_first.find("/")
if second_slash != -1:
doi_end = after_suffix_start + second_slash
doi = doi_with_path[:doi_end]
subpath = doi_with_path[doi_end:]
if subpath == "/":
return (doi, "")
else:
return (doi, subpath)
else:
return (doi_with_path, "")
else:
return (doi_with_path, "")
@classmethod
def parse_remote_host(cls, path: str) -> str:
path_array = path.split(":")
remote_host = path_array[0]
if len(path_array) == 2 and path_array[1] != "" or len(path_array) > 2:
raise IllegalArgumentException("Invalid remote host")
return remote_host
@classmethod
def parse_remote_host_with_path(cls, path: str) -> tuple[str, str, str]:
path = re.sub(r"//+|/\./+|/\.$", "/", path)
if re.search(r"/\.\./|/\.\.$", path) is not None:
raise IllegalArgumentException("Path traversal found.")
path_array = path.split(":")
if len(path_array) != 2:
raise IllegalArgumentException("Invalid remote host.")
remote_host = path_array[0]
folder_array = path_array[1].split("/")
is_absolute_path = folder_array[0] == ""
if not is_absolute_path:
raise IllegalArgumentException("Must be absolute paths.")
del folder_array[0]
if len(folder_array) == 0:
laboratory = ""
folder = ""
else:
laboratory = folder_array.pop(0)
folder = "/" + "/".join(folder_array)
return (remote_host, laboratory, folder)
@classmethod
def parse_doi_remote_host(cls, path: str) -> tuple[str, str, str]:
parts = path.split(":", 1)
if len(parts) != 2:
raise IllegalArgumentException("remote_path must be in the form 'remote:10.xxxx/prefix.ID'")
remote, doi_with_path = parts
if not cls.is_doi(doi_with_path):
raise IllegalArgumentException(f"Path `{doi_with_path}` does not look like a DOI.")
doi, subpath = cls.split_doi_and_subpath(doi_with_path)
return (remote, doi, subpath)
def find_folder_by_doi(self, doi: str, password: str | None = None) -> tuple[Folder, Laboratory]:
doi_clean = doi.rstrip("/")
doi_id = self.doi_suffix_id(doi_clean)
doi_api = DoiApi(self.connection)
doi_resp = doi_api.retrieve(doi_id)
returned_doi = doi_resp.doi.rstrip("/")
if returned_doi.lower() != doi_clean.lower():
raise IllegalArgumentException(
f"DOI mismatch: requested `{doi_clean}` but server returned `{returned_doi}`."
)
folder_api = FoldersApi(self.connection)
folder = folder_api.retrieve(doi_resp.folder_id)
if folder.lock:
if password is None:
raise UnauthorizedException(f"Folder for DOI `{doi_clean}` is locked.")
folder_api.auth(doi_resp.folder_id, password)
lab_api = LaboratoriesApi(self.connection)
labs = lab_api.list()
lab = labs.find_by_id(folder.laboratory_id)
if lab is None:
raise UnexpectedException(f"Laboratory with id {folder.laboratory_id} not found.")
self.connection.laboratories = labs
return (folder, lab)
def resolve_folder(self, remote_path: str, password: str | None = None) -> tuple[Folder, Laboratory]:
path_component = remote_path.split(":", 1)[1] if ":" in remote_path else ""
if self.is_doi(path_component):
remote, doi, subpath = self.parse_doi_remote_host(remote_path)
doi_folder, laboratory = self.find_folder_by_doi(doi, password)
if not subpath:
return (doi_folder, laboratory)
else:
abs_path = doi_folder.path.rstrip("/") + subpath
folder = self.find_folder(laboratory, abs_path, password)
return (folder, laboratory)
else:
remote, laboratory_name, r_path = self.parse_remote_host_with_path(remote_path)
laboratory = self.find_laboratory(laboratory_name)
folder = self.find_folder(laboratory, r_path, password)
return (folder, laboratory)
def resolve_file(self, remote_path: str, password: str | None = None) -> tuple[Folder, Laboratory, str]:
path_component = remote_path.split(":", 1)[1] if ":" in remote_path else ""
if self.is_doi(path_component):
remote, doi, subpath = self.parse_doi_remote_host(remote_path)
doi_folder, laboratory = self.find_folder_by_doi(doi, password)
subpath_clean = subpath.rstrip("/")
if not subpath_clean:
raise IllegalArgumentException("DOI path must point to a file, not a folder.")
r_dirname = os.path.dirname(subpath_clean)
r_basename = os.path.basename(subpath_clean)
abs_path = doi_folder.path.rstrip("/") + r_dirname
parent_folder = self.find_folder(laboratory, abs_path, password)
return (parent_folder, laboratory, r_basename)
else:
remote, laboratory_name, r_path = self.parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
r_dirname = os.path.dirname(r_path)
r_basename = os.path.basename(r_path)
laboratory = self.find_laboratory(laboratory_name)
parent_folder = self.find_folder(laboratory, r_dirname, password)
return (parent_folder, laboratory, r_basename)
+321
View File
@@ -0,0 +1,321 @@
import os
from concurrent.futures import ThreadPoolExecutor
from typing import Any
from unicodedata import normalize
from pydantic.dataclasses import dataclass
from mdrsclient.api import FilesApi, FoldersApi
from mdrsclient.exceptions import IllegalArgumentException, UnexpectedException
from mdrsclient.models import File, Folder, Laboratory
from mdrsclient.models.file import find_file
from mdrsclient.settings import CONCURRENT
@dataclass(frozen=True)
class UploadFileInfo:
folder: Folder
files: list[File]
path: str
@dataclass(frozen=True)
class DownloadFileInfo:
file: File
path: str
@dataclass
class DownloadContext:
isSkipIfExists: bool
files: list[DownloadFileInfo]
class Uploader:
def __init__(self, client: Any) -> None:
self.client = client
def upload(
self, local_path: str, remote_path: str, is_recursive: bool = False, is_skip_if_exists: bool = False
) -> None:
remote, laboratory_name, r_path = self.client.parse_remote_host_with_path(remote_path)
l_path = os.path.abspath(local_path)
if not os.path.exists(l_path):
raise IllegalArgumentException(f"File or directory `{local_path}` not found.")
laboratory = self.client.find_laboratory(laboratory_name)
folder = self.client.find_folder(laboratory, r_path)
files = self.client.find_files(folder.id)
if os.path.isdir(l_path):
if not is_recursive:
raise IllegalArgumentException(f"Cannot upload `{local_path}`: Is a directory.")
infos = self.__collect_directory_uploads(laboratory, r_path, l_path, folder, files)
else:
infos = [UploadFileInfo(folder, files, l_path)]
if not self.__multiple_upload(infos, is_skip_if_exists):
# One file failing is worth reporting on its own line, and worth the caller
# hearing about: a batch that lost files is not a batch that succeeded.
raise UnexpectedException("Some files failed to upload.")
def __collect_directory_uploads(
self, laboratory: Laboratory, r_path: str, l_path: str, folder: Folder, files: list[File]
) -> list[UploadFileInfo]:
"""Mirror a local directory tree on the remote, and list the files to send into it."""
infos: list[UploadFileInfo] = []
folder_api = FoldersApi(self.client.connection)
folder_map: dict[str, Folder] = {}
folder_map[r_path] = folder
files_map: dict[str, list[File]] = {}
files_map[r_path] = files
l_basename = os.path.basename(l_path)
for dirpath, _, filenames in os.walk(l_path, followlinks=True):
sub = l_basename if dirpath == l_path else os.path.join(l_basename, os.path.relpath(dirpath, l_path))
d_dirname = os.path.join(r_path, sub)
d_basename = os.path.basename(d_dirname)
# prepare destination parent path
d_parent_dirname = os.path.dirname(d_dirname)
if folder_map.get(d_parent_dirname) is None:
parent_folder = self.client.find_folder(laboratory, d_parent_dirname)
folder_map[d_parent_dirname] = parent_folder
parent_files = self.client.find_files(parent_folder.id)
files_map[d_parent_dirname] = parent_files
# prepare destination path
if folder_map.get(d_dirname) is None:
d_folder = folder_map[d_parent_dirname].find_sub_folder(d_basename)
if d_folder is None:
d_folder_id = folder_api.create(normalize("NFC", d_basename), folder_map[d_parent_dirname].id)
else:
d_folder_id = d_folder.id
print(d_dirname)
folder_map[d_dirname] = folder_api.retrieve(d_folder_id)
files_map[d_dirname] = self.client.find_files(d_folder_id)
if d_folder is None:
folder_map[d_parent_dirname].sub_folders.append(folder_map[d_dirname])
# register upload file list
for filename in filenames:
infos.append(
UploadFileInfo(folder_map[d_dirname], files_map[d_dirname], os.path.join(dirpath, filename))
)
return infos
def __multiple_upload(self, infos: list[UploadFileInfo], is_skip_if_exists: bool) -> bool:
"""Send every file, and report whether all of them arrived."""
file_api = FilesApi(self.client.connection)
with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
results = pool.map(lambda x: self.__multiple_upload_worker(file_api, x, is_skip_if_exists), infos)
# Consumed inside the block: the results are what carry each worker's verdict.
return all(list(results))
def __multiple_upload_worker(self, file_api: FilesApi, info: UploadFileInfo, is_skip_if_exists: bool) -> bool:
basename = os.path.basename(info.path)
file = find_file(info.files, basename)
try:
if file is None:
file_api.create(info.folder.id, info.path)
elif not is_skip_if_exists or file.size != os.path.getsize(info.path):
file_api.update(file, info.path)
print(os.path.join(info.folder.path, basename))
except Exception as e:
# Everything, not just the exceptions the API layer raises: the batch verdict
# is read now that the results are consumed, and a file vanishing between the
# walk and the upload would otherwise end the whole run with a traceback and
# throw away what every other file did.
print(f"Failed: {info.path}: {e}")
return False
return True
class Downloader:
def __init__(self, client: Any) -> None:
self.client = client
def download(
self,
remote_path: str,
local_path: str,
is_recursive: bool = False,
is_skip_if_exists: bool = False,
password: str | None = None,
excludes: list[str] | None = None,
) -> None:
if not self.__download(remote_path, local_path, is_recursive, is_skip_if_exists, password, excludes):
# Every failure has already been printed against the file it belongs to.
# This is what makes the command as a whole end in failure.
raise UnexpectedException("Some files failed to download.")
def __download(
self,
remote_path: str,
local_path: str,
is_recursive: bool,
is_skip_if_exists: bool,
password: str | None,
excludes: list[str] | None,
) -> bool:
"""Fetch what the remote path names, and report whether every file arrived."""
excludes_clean = excludes or []
l_dirname = os.path.realpath(local_path)
if not os.path.isdir(l_dirname):
raise IllegalArgumentException(f"Local directory `{local_path}` not found.")
# "remote:10.xxxx/prefix.ID[/optional/sub/path]" names a published dataset rather
# than a path within a laboratory, and is resolved through the DOI instead.
path_component = remote_path.split(":", 1)[1] if ":" in remote_path else ""
if self.client.is_doi(path_component):
return self.__download_doi(
remote_path, l_dirname, is_recursive, is_skip_if_exists, password, excludes_clean
)
remote, laboratory_name, r_path = self.client.parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
r_basename = os.path.basename(r_path)
laboratory = self.client.find_laboratory(laboratory_name)
r_parent_folder = self.client.find_folder(laboratory, os.path.dirname(r_path), password)
r_parent_files = self.client.find_files(r_parent_folder.id)
file = find_file(r_parent_files, r_basename)
if file is not None:
return self.__download_one(
excludes_clean, laboratory, r_parent_folder, file, l_dirname, r_basename, is_skip_if_exists
)
folder = r_parent_folder.find_sub_folder(r_basename)
if folder is None:
raise IllegalArgumentException(f"File or folder `{r_path}` not found.")
if not is_recursive:
raise IllegalArgumentException(f"Cannot download `{r_path}`: Is a folder.")
return self.__multiple_download_pickup_recursive_files(
FoldersApi(self.client.connection), laboratory, folder.id, l_dirname, excludes_clean, is_skip_if_exists
)
def __download_doi(
self,
remote_path: str,
l_dirname: str,
is_recursive: bool,
is_skip_if_exists: bool,
password: str | None,
excludes: list[str],
) -> bool:
"""Fetch what a DOI names: the dataset's folder, or something inside it."""
remote, doi, subpath = self.client.parse_doi_remote_host(remote_path)
doi_folder, laboratory = self.client.find_folder_by_doi(doi, password)
subpath_clean = subpath.rstrip("/")
if not subpath_clean:
folder = doi_folder
else:
r_basename = os.path.basename(subpath_clean)
abs_path = doi_folder.path.rstrip("/") + os.path.dirname(subpath_clean)
r_parent_folder = self.client.find_folder(laboratory, abs_path, password)
file = find_file(self.client.find_files(r_parent_folder.id), r_basename)
if file is not None:
return self.__download_one(
excludes, laboratory, r_parent_folder, file, l_dirname, r_basename, is_skip_if_exists
)
folder_simple = r_parent_folder.find_sub_folder(r_basename)
if folder_simple is None:
raise IllegalArgumentException(f"File or folder `{subpath_clean}` not found.")
folder = FoldersApi(self.client.connection).retrieve(folder_simple.id)
if is_recursive:
return self.__multiple_download_pickup_recursive_files(
FoldersApi(self.client.connection), laboratory, folder.id, l_dirname, excludes, is_skip_if_exists
)
# Without -r the dataset's own files are fetched, and its sub-folders are not.
context = DownloadContext(is_skip_if_exists, [])
for file in self.client.find_files(folder.id):
if self.__check_excludes(excludes, laboratory, folder, file):
continue
context.files.append(DownloadFileInfo(file, os.path.join(l_dirname, file.name)))
return self.__multiple_download(context)
def __download_one(
self,
excludes: list[str],
laboratory: Laboratory,
folder: Folder,
file: File,
l_dirname: str,
local_name: str,
is_skip_if_exists: bool,
) -> bool:
"""
Fetch a single named file into the local directory.
Saved under the name the caller asked for rather than the one the server holds:
the two are matched case-insensitively, so they need not be spelled alike.
"""
if self.__check_excludes(excludes, laboratory, folder, file):
return True
context = DownloadContext(is_skip_if_exists, [])
context.files.append(DownloadFileInfo(file, os.path.join(l_dirname, local_name)))
return self.__multiple_download(context)
def __multiple_download_pickup_recursive_files(
self,
folder_api: FoldersApi,
laboratory: Laboratory,
folder_id: str,
basedir: str,
excludes: list[str],
is_skip_if_exists: bool,
) -> bool:
context = DownloadContext(is_skip_if_exists, [])
try:
folder = folder_api.retrieve(folder_id)
files = self.client.find_files(folder.id)
except Exception as e:
print(f"Failed: {basedir}: {e}")
return False
dirname = os.path.join(basedir, folder.name)
if self.__check_excludes(excludes, laboratory, folder, None):
return True
try:
# `exist_ok` rather than a prior check: two workers can reach the same parent.
os.makedirs(dirname, exist_ok=True)
except OSError as e:
# One folder the client cannot make locally is not a reason to abandon its
# siblings, which is what this walk now promises.
print(f"Failed: {dirname}: {e}")
return False
print(dirname)
for file in files:
if self.__check_excludes(excludes, laboratory, folder, file):
continue
path = os.path.join(dirname, file.name)
context.files.append(DownloadFileInfo(file, path))
succeeded = self.__multiple_download(context)
# A folder that lost a file is still a folder whose sub-folders the user asked
# for, so the walk carries on and the verdict is collected for the caller.
for sub_folder in folder.sub_folders:
if not self.__multiple_download_pickup_recursive_files(
folder_api, laboratory, sub_folder.id, dirname, excludes, is_skip_if_exists
):
succeeded = False
return succeeded
def __multiple_download(self, context: DownloadContext) -> bool:
"""Fetch every file in the batch, and report whether all of them arrived."""
file_api = FilesApi(self.client.connection)
with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
results = pool.map(
lambda x: self.__multiple_download_worker(file_api, x, context.isSkipIfExists), context.files
)
# Consumed inside the block, and in full: every worker's verdict counts, not
# just the first refusal.
return all(list(results))
def __multiple_download_worker(self, file_api: FilesApi, info: DownloadFileInfo, is_skip_if_exists: bool) -> bool:
if not is_skip_if_exists or not os.path.exists(info.path) or info.file.size != os.path.getsize(info.path):
try:
file_api.download(info.file, info.path)
except Exception as e:
# Nothing to clear up: a failed transfer writes only to its own scratch
# file beside the destination, and removes that itself.
print(f"Failed: {info.path}: {e}")
return False
print(info.path)
return True
def __check_excludes(self, excludes: list[str], laboratory: Laboratory, folder: Folder, file: File | None) -> bool:
path = f"/{laboratory.name}{folder.path}{file.name if file is not None else ''}".rstrip("/").lower()
return path in excludes
+16 -2
View File
@@ -1,5 +1,6 @@
import os
from typing import IO, Any
import time
from typing import IO, Any, Final
from urllib.parse import parse_qs, urlparse
if os.name == "nt":
@@ -9,10 +10,23 @@ elif os.name == "posix":
class FileLock:
# Long enough to outlast a token refresh, which is what the lock is held across.
WAIT_SECONDS: Final[float] = 60.0
@staticmethod
def lock(file: IO[Any]) -> None:
if os.name == "nt":
msvcrt.locking(file.fileno(), msvcrt.LK_LOCK, 1)
# msvcrt.LK_LOCK gives up after ten one-second attempts, which is shorter
# than the refresh it now has to wait for, so do the waiting here instead.
deadline = time.monotonic() + FileLock.WAIT_SECONDS
while True:
try:
msvcrt.locking(file.fileno(), msvcrt.LK_NBLCK, 1)
return
except OSError:
if time.monotonic() >= deadline:
raise
time.sleep(0.1)
elif os.name == "posix":
fcntl.flock(file.fileno(), fcntl.LOCK_EX)
+45 -30
View File
@@ -1,12 +1,22 @@
[tool.poetry]
[build-system]
requires = ["flit_core>=4.0.2,<5.0.0"]
build-backend = "flit_core.buildapi"
[tool.flit.module]
name = "mdrsclient"
[project]
name = "mdrs-client-python"
version = "1.3.13"
version = "1.3.19"
description = "The mdrs-client-python is python library and a command-line client for up- and downloading files to and from MDRS based repository."
authors = ["Yoshihiro OKUMURA <yoshihiro.okumura@riken.jp>"]
license = "MIT"
authors = [
{ name = "Yoshihiro OKUMURA", email = "yoshihiro.okumura@riken.jp" }
]
license = { text = "MIT" }
readme = "README.md"
classifiers=[
"Development Status :: 3 - Alpha",
requires-python = ">=3.10"
classifiers = [
"Development Status :: 4 - Beta",
"Environment :: Console",
"Intended Audience :: Developers",
"Intended Audience :: Science/Research",
@@ -14,36 +24,32 @@ classifiers=[
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"OSI Approved :: MIT License",
"Topic :: Utilities",
]
packages = [
{ include = "mdrsclient" }
dependencies = [
"requests>=2.34.2",
"requests-toolbelt>=1.0.0",
"python-dotenv>=1.2.3",
"pydantic>=2.13.5",
"pydantic-settings>=2.15.0",
"PyJWT>=2.13.0",
"validators>=0.35.0",
]
[tool.poetry.dependencies]
python = "^3.10"
requests = "^2.32.3"
requests-toolbelt = "^1.0.0"
python-dotenv = "^1.1.0"
pydantic = "^2.11.4"
pydantic-settings = "^2.9.1"
PyJWT = "^2.10.1"
validators = "^0.34.0"
[project.optional-dependencies]
dev = [
"black>=26.5.1",
"flake8>=7.3.0",
"Flake8-pyproject>=1.2.4",
"isort>=9.0.1",
"pyright>=1.1.411",
]
[tool.poetry.group.dev.dependencies]
black = "^24.10.0"
flake8 = "^7.2.0"
Flake8-pyproject = "^1.2.3"
isort = "^5.13.2"
pyright = "^1.1.400"
[tool.poetry.scripts]
mdrs = 'mdrsclient.__main__:main'
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[project.scripts]
mdrs = "mdrsclient.__main__:main"
[tool.black]
line-length = 120
@@ -62,6 +68,9 @@ exclude = '''
exclude = ".git, .venv, __pycache__, dist"
max-complexity = 10
max-line-length = 120
# E203 flags the space black puts before a slice colon. Black formats this project, so
# its output is the standard and flake8 has to accept it.
extend-ignore = ["E203"]
[tool.isort]
profile = "black"
@@ -71,6 +80,12 @@ line_length = 120
[tool.pyright]
typeCheckingMode = "basic"
# This project keeps its environment in `./.venv`. Named explicitly because without it
# pyright falls back to the system interpreter, cannot resolve pydantic, and reports every
# model field as an unknown argument. An environment kept elsewhere gets that same fallback
# and can override it with a local `pyrightconfig.json` or `pyright --pythonpath`.
venvPath = "."
venv = ".venv"
exclude = ["**/__pycache__", "**/.*", "dist"]
#reportUnknownMemberType = "warning"
#reportUnknownVariableType = "warning"
+1
View File
@@ -0,0 +1 @@
# Mark tests directory as a Python package
+165
View File
@@ -0,0 +1,165 @@
import unittest
from unittest.mock import MagicMock, patch
from mdrsclient.client import MdrsClient
from mdrsclient.exceptions import IllegalArgumentException
from mdrsclient.models import File, Folder, FolderSimple, Laboratories, Laboratory
TIMESTAMP = "2026-01-01T00:00:00+09:00"
LABORATORY = Laboratory(id=1, name="mylab", pi_name="PI", full_name="My Laboratory")
def make_file(id: str, name: str) -> File:
return File(
id=id,
name=name,
type="text/plain",
size=1,
thumbnail=None,
description="",
metadata={},
download_url=f"v3/files/{id}/download/",
created_at=TIMESTAMP,
updated_at=TIMESTAMP,
)
def make_sub_folder(id: str, name: str) -> FolderSimple:
return FolderSimple(
id=id,
pid=None,
name=name,
access_level=1,
lock=False,
size=0,
laboratory_id=1,
description="",
created_at=TIMESTAMP,
updated_at=TIMESTAMP,
restrict_opened_at=None,
)
def make_folder(id: str, name: str, path: str, sub_folders: list[FolderSimple] | None = None) -> Folder:
return Folder(
id=id,
pid=None,
name=name,
access_level=1,
lock=False,
size=0,
laboratory_id=1,
description="",
created_at=TIMESTAMP,
updated_at=TIMESTAMP,
restrict_opened_at=None,
metadata=[],
sub_folders=sub_folders if sub_folders is not None else [],
path=path,
)
class TestLsColumnWidths(unittest.TestCase):
"""Each column is as wide as the widest thing printed under it."""
def test_the_size_column_fits_a_sub_folder_larger_than_its_parent(self):
from unittest.mock import MagicMock as _MagicMock
from mdrsclient.commands.ls import LsCommand, LsCommandContext
laboratories = Laboratories()
laboratories.append(LABORATORY)
connection = _MagicMock()
connection.laboratories = laboratories
client = MdrsClient(connection)
big = make_sub_folder("s1", "big")
object.__setattr__(big, "size", 123456789)
parent = make_folder("f1", "root", "/root/", [big])
context = LsCommandContext("remote:/mylab", client, LABORATORY, "", False, False, False)
widths = LsCommand._column_widths(context, parent, [])
self.assertEqual(widths["size"], len("123456789"))
class TestCopyAndMove(unittest.TestCase):
"""`cp` and `mv` resolve both ends the same way and differ only in what they call."""
def make_client(self, folders_by_path: dict, files_by_folder: dict) -> MdrsClient:
client = MdrsClient(MagicMock())
client.find_laboratory = MagicMock(return_value=LABORATORY)
client.find_folder = MagicMock(side_effect=lambda lab, path, password=None: folders_by_path[path])
client.find_files = MagicMock(side_effect=lambda folder_id: files_by_folder.get(folder_id, []))
return client
def make_tree(self, source_has: str) -> tuple[MdrsClient, dict, dict]:
sub = make_sub_folder("fsub", "sub")
folder_a = make_folder("fa", "a", "/a/", [sub] if source_has == "folder" else [])
folder_b = make_folder("fb", "b", "/b/")
folders = {"/a": folder_a, "/b": folder_b, "/b/": folder_b}
files = {"fa": [make_file("x1", "data.txt")] if source_has == "file" else [], "fb": []}
return self.make_client(folders, files), folders, files
def test_a_file_is_copied_into_the_destination_folder(self):
client, _, files = self.make_tree("file")
with patch("mdrsclient.client.FilesApi") as files_api:
client.cp("myremote:/mylab/a/data.txt", "myremote:/mylab/b/data.txt")
files_api.return_value.copy.assert_called_once_with(files["fa"][0], "fb", "data.txt")
def test_a_file_is_moved_into_the_destination_folder(self):
client, _, files = self.make_tree("file")
with patch("mdrsclient.client.FilesApi") as files_api:
client.mv("myremote:/mylab/a/data.txt", "myremote:/mylab/b/data.txt")
files_api.return_value.move.assert_called_once_with(files["fa"][0], "fb", "data.txt")
def test_a_trailing_separator_keeps_the_source_name(self):
client, _, files = self.make_tree("file")
with patch("mdrsclient.client.FilesApi") as files_api:
client.mv("myremote:/mylab/a/data.txt", "myremote:/mylab/b/")
files_api.return_value.move.assert_called_once_with(files["fa"][0], "fb", "data.txt")
def test_copying_a_folder_needs_the_recursive_flag(self):
client, _, _ = self.make_tree("folder")
with patch("mdrsclient.client.FoldersApi") as folders_api:
with self.assertRaises(IllegalArgumentException) as caught:
client.cp("myremote:/mylab/a/sub", "myremote:/mylab/b/sub")
self.assertIn("Is a folder", str(caught.exception))
folders_api.return_value.copy.assert_not_called()
def test_a_folder_is_copied_when_recursive(self):
client, folders, _ = self.make_tree("folder")
with patch("mdrsclient.client.FoldersApi") as folders_api:
client.cp("myremote:/mylab/a/sub", "myremote:/mylab/b/sub", is_recursive=True)
folders_api.return_value.copy.assert_called_once_with(folders["/a"].sub_folders[0], "fb", "sub")
def test_a_folder_is_moved_without_the_recursive_flag(self):
client, folders, _ = self.make_tree("folder")
with patch("mdrsclient.client.FoldersApi") as folders_api:
client.mv("myremote:/mylab/a/sub", "myremote:/mylab/b/sub")
folders_api.return_value.move.assert_called_once_with(folders["/a"].sub_folders[0], "fb", "sub")
def test_a_destination_that_already_holds_the_name_is_refused(self):
sub = make_sub_folder("fsub", "sub")
folders = {"/a": make_folder("fa", "a", "/a/", [sub]), "/b": make_folder("fb", "b", "/b/")}
files = {"fa": [], "fb": [make_file("x2", "sub")]}
client = self.make_client(folders, files)
with self.assertRaises(IllegalArgumentException) as caught:
client.mv("myremote:/mylab/a/sub", "myremote:/mylab/b/sub")
self.assertIn("Cannot overwrite non-folder", str(caught.exception))
def test_a_transfer_across_laboratories_is_refused(self):
client, _, _ = self.make_tree("file")
with self.assertRaises(IllegalArgumentException) as caught:
client.cp("myremote:/mylab/a/data.txt", "myremote:/otherlab/b/data.txt")
self.assertIn("Laboratory mismatched", str(caught.exception))
def test_a_source_that_does_not_exist_is_refused(self):
client, _, _ = self.make_tree("file")
with self.assertRaises(IllegalArgumentException) as caught:
client.mv("myremote:/mylab/a/missing.txt", "myremote:/mylab/b/missing.txt")
self.assertIn("not found", str(caught.exception))
if __name__ == "__main__":
unittest.main()
+418
View File
@@ -0,0 +1,418 @@
import argparse
import json
import unittest
from io import StringIO
from unittest.mock import MagicMock, patch
from mdrsclient.client import MdrsClient
from mdrsclient.commands import (
ChaclCommand,
ConfigCommand,
CpCommand,
DownloadCommand,
FileMetadataCommand,
LabsCommand,
LoginCommand,
LogoutCommand,
LsCommand,
MetadataCommand,
MkdirCommand,
MvCommand,
RmCommand,
UploadCommand,
VersionCommand,
WhoamiCommand,
)
from mdrsclient.models import Folder, Laboratory
class TestCommands(unittest.TestCase):
def parse_args(self, cmd_class, args_list):
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(title="subcommands")
cmd_class.register(subparsers)
return parser.parse_args(args_list)
def test_version_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.return_value = mock_client
mock_client.version.return_value = "mdrs 1.3.17"
args = self.parse_args(VersionCommand, ["version"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
self.assertEqual(fake_out.getvalue().strip(), "mdrs 1.3.17")
mock_client_class.assert_called_once_with(None)
mock_client.version.assert_called_once()
def test_login_command_with_args(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client_class.parse_remote_host.return_value = "myremote"
args = self.parse_args(LoginCommand, ["login", "-u", "myuser", "-p", "mypass", "myremote"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
self.assertIn("Login Successful", fake_out.getvalue())
mock_client_class.parse_remote_host.assert_called_once_with("myremote")
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.login.assert_called_once_with("myuser", "mypass")
def test_login_command_interactive(self):
with (
patch("mdrsclient.client.MdrsClient") as mock_client_class,
patch("builtins.input", return_value="myuser_int") as mock_input,
patch("getpass.getpass", return_value="mypass_int") as mock_getpass,
):
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client_class.parse_remote_host.return_value = "myremote"
args = self.parse_args(LoginCommand, ["login", "myremote"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
self.assertIn("Login Successful", fake_out.getvalue())
mock_input.assert_called_once_with("Username: ")
mock_getpass.assert_called_once_with("Password: ")
mock_client.login.assert_called_once_with("myuser_int", "mypass_int")
def test_logout_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client_class.parse_remote_host.return_value = "myremote"
args = self.parse_args(LogoutCommand, ["logout", "myremote"])
args.func(args)
mock_client_class.parse_remote_host.assert_called_once_with("myremote")
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.logout.assert_called_once()
def test_whoami_command_logged_in(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client_class.parse_remote_host.return_value = "myremote"
mock_client.connection.token = None
mock_user = MagicMock()
mock_user.username = "test_user"
mock_client.whoami.return_value = mock_user
args = self.parse_args(WhoamiCommand, ["whoami", "myremote"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
self.assertEqual(fake_out.getvalue().strip(), "test_user")
mock_client_class.parse_remote_host.assert_called_once_with("myremote")
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.whoami.assert_called_once()
def test_whoami_command_anonymous(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client_class.parse_remote_host.return_value = "myremote"
mock_client.connection.token = None
mock_client.whoami.side_effect = Exception("Not logged in")
args = self.parse_args(WhoamiCommand, ["whoami", "myremote"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
self.assertEqual(fake_out.getvalue().strip(), "(Anonymous)")
def test_labs_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client_class.parse_remote_host.return_value = "myremote"
mock_lab = MagicMock()
mock_lab.id = 1
mock_lab.name = "lab_name"
mock_lab.pi_name = "pi_name"
mock_lab.full_name = "full_name"
mock_client.get_laboratories.return_value = [mock_lab]
args = self.parse_args(LabsCommand, ["labs", "myremote"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
output = fake_out.getvalue()
self.assertIn("Name", output)
self.assertIn("lab_name", output)
self.assertIn("pi_name", output)
mock_client_class.parse_remote_host.assert_called_once_with("myremote")
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.get_laboratories.assert_called_once()
def test_ls_command_plain(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock(spec=MdrsClient)
mock_client_class.from_remote.return_value = mock_client
mock_folder = Folder(
id="folder_id",
pid=None,
name="root",
access_level=1,
lock=False,
size=0,
laboratory_id=1,
description="",
created_at="2026-07-02T00:00:00Z",
updated_at="2026-07-02T00:00:00Z",
restrict_opened_at=None,
metadata=[],
sub_folders=[],
path="/root",
)
mock_lab = Laboratory(id=1, name="mylab", pi_name="pi_name", full_name="full_name")
mock_client.resolve_folder.return_value = (mock_folder, mock_lab)
mock_file = MagicMock()
mock_file.name = "file.txt"
mock_file.size = 100
mock_file.updated_at_name = "2026-07-02"
mock_client.find_files.return_value = [mock_file]
args = self.parse_args(LsCommand, ["ls", "myremote:/mylab/"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
output = fake_out.getvalue()
self.assertIn("Type", output)
self.assertIn("file.txt", output)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.resolve_folder.assert_called_once_with("myremote:/mylab/", None)
mock_client.find_files.assert_called_once_with(mock_folder.id)
def test_ls_command_json(self):
with (
patch("mdrsclient.client.MdrsClient") as mock_client_class,
patch("mdrsclient.commands.ls.FoldersApi") as mock_folders_api_class,
):
mock_folders_api = MagicMock()
mock_folders_api_class.return_value = mock_folders_api
mock_folders_api.metadata.return_value = {"folder_meta": "val"}
mock_client = MagicMock(spec=MdrsClient)
mock_client_class.from_remote.return_value = mock_client
mock_client.connection = MagicMock()
mock_folder = Folder(
id="folder_id",
pid="parent_id",
name="root",
access_level=1,
lock=False,
size=0,
laboratory_id=1,
description="Root folder",
created_at="2026-07-02T00:00:00Z",
updated_at="2026-07-02T00:00:00Z",
restrict_opened_at=None,
metadata=[],
sub_folders=[],
path="/root",
)
mock_lab = Laboratory(id=1, name="mylab", pi_name="pi_name", full_name="full_name")
mock_client.resolve_folder.return_value = (mock_folder, mock_lab)
mock_client.connection.laboratories.find_by_id.return_value = mock_lab
mock_file = MagicMock()
mock_file.id = "file_id"
mock_file.name = "file.txt"
mock_file.type = "text"
mock_file.size = 100
mock_file.description = "A file"
mock_file.metadata = {}
mock_file.download_url = "download/file"
mock_file.created_at = "2026-07-02T00:00:00Z"
mock_file.updated_at = "2026-07-02T00:00:00Z"
mock_client.find_files.return_value = [mock_file]
args = self.parse_args(LsCommand, ["ls", "-J", "myremote:/mylab/"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
output = fake_out.getvalue()
parsed_json = json.loads(output)
self.assertEqual(parsed_json["name"], "root")
self.assertEqual(parsed_json["files"][0]["name"], "file.txt")
def test_mkdir_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(MkdirCommand, ["mkdir", "myremote:/mylab/newfolder"])
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.mkdir.assert_called_once_with("myremote:/mylab/newfolder")
def test_rm_command_file(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(RmCommand, ["rm", "myremote:/mylab/file.txt"])
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.rm.assert_called_once_with("myremote:/mylab/file.txt", False)
def test_rm_command_recursive(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(RmCommand, ["rm", "-r", "myremote:/mylab/folder"])
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.rm.assert_called_once_with("myremote:/mylab/folder", True)
def test_cp_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(CpCommand, ["cp", "myremote:/mylab/src", "myremote:/mylab/dest"])
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.cp.assert_called_once_with("myremote:/mylab/src", "myremote:/mylab/dest", False)
def test_mv_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(MvCommand, ["mv", "myremote:/mylab/src", "myremote:/mylab/dest"])
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.mv.assert_called_once_with("myremote:/mylab/src", "myremote:/mylab/dest")
def test_chacl_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(ChaclCommand, ["chacl", "-r", "-p", "secret", "private", "myremote:/mylab/"])
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.chacl.assert_called_once_with("myremote:/mylab/", 1, True, "secret")
def test_metadata_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client.metadata.return_value = {"meta_key": "meta_val"}
args = self.parse_args(MetadataCommand, ["metadata", "-p", "secret", "myremote:/mylab/"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
parsed_json = json.loads(fake_out.getvalue())
self.assertEqual(parsed_json["meta_key"], "meta_val")
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.metadata.assert_called_once_with("myremote:/mylab/", "secret")
def test_file_metadata_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client.file_metadata.return_value = {"file_meta": "val"}
args = self.parse_args(FileMetadataCommand, ["file-metadata", "-p", "secret", "myremote:/mylab/file.txt"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
parsed_json = json.loads(fake_out.getvalue())
self.assertEqual(parsed_json["file_meta"], "val")
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.file_metadata.assert_called_once_with("myremote:/mylab/file.txt", "secret")
def test_upload_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(UploadCommand, ["upload", "-r", "-s", "local_file.txt", "myremote:/mylab/"])
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.upload.assert_called_once_with("local_file.txt", "myremote:/mylab/", True, True)
def test_download_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(
DownloadCommand, ["download", "-r", "-s", "-p", "pass", "-e", "ex1", "myremote:/mylab/", "local_dir"]
)
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.download.assert_called_once_with("myremote:/mylab/", "local_dir", True, True, "pass", ["ex1"])
def test_config_create(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.return_value = mock_client
args = self.parse_args(ConfigCommand, ["config", "create", "myremote", "http://example.com"])
args.func(args)
mock_client_class.assert_called_once_with(None)
mock_client.config_create.assert_called_once_with("myremote", "http://example.com")
def test_config_update(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.return_value = mock_client
args = self.parse_args(ConfigCommand, ["config", "update", "myremote", "http://example.com"])
args.func(args)
mock_client_class.assert_called_once_with(None)
mock_client.config_update.assert_called_once_with("myremote", "http://example.com")
def test_config_delete(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.return_value = mock_client
args = self.parse_args(ConfigCommand, ["config", "delete", "myremote"])
args.func(args)
mock_client_class.assert_called_once_with(None)
mock_client.config_delete.assert_called_once_with("myremote")
def test_config_list(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.return_value = mock_client
mock_client.config_list.return_value = [("remote1", "url1"), ("remote2", "url2")]
args = self.parse_args(ConfigCommand, ["config", "list"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
self.assertEqual(fake_out.getvalue(), "remote1:\turl1\nremote2:\turl2\n")
mock_client_class.assert_called_once_with(None)
mock_client.config_list.assert_called_once()
+55
View File
@@ -0,0 +1,55 @@
import unittest
from unittest.mock import MagicMock, patch
from mdrsclient.models import Doi, Folder, Laboratories, Laboratory
from mdrsclient.services import MdrsService
TIMESTAMP = "2026-01-01T00:00:00+09:00"
DOI = "10.60178/cbs.20260429-001"
LABORATORY = Laboratory(id=1, name="mylab", pi_name="PI", full_name="My Laboratory")
def make_locked_folder() -> Folder:
return Folder(
id="f1",
pid=None,
name="root",
access_level=4,
lock=True,
size=0,
laboratory_id=1,
description="",
created_at=TIMESTAMP,
updated_at=TIMESTAMP,
restrict_opened_at=None,
metadata=[],
sub_folders=[],
path="/root/",
)
class TestFindFolderByDoi(unittest.TestCase):
"""The DOI response carries the folder id as a field, not as a nested object."""
def test_a_locked_doi_folder_is_unlocked_with_its_folder_id(self):
laboratories = Laboratories()
laboratories.append(LABORATORY)
with (
patch("mdrsclient.services.DoiApi") as doi_api,
patch("mdrsclient.services.FoldersApi") as folders_api,
patch("mdrsclient.services.LaboratoriesApi") as laboratories_api,
):
doi_api.return_value.retrieve.return_value = Doi(id="20260429-001", doi=DOI, folder_id="f1")
folders_api.return_value.retrieve.return_value = make_locked_folder()
laboratories_api.return_value.list.return_value = laboratories
folder, laboratory = MdrsService(MagicMock()).find_folder_by_doi(DOI, "secret")
folders_api.return_value.auth.assert_called_once_with("f1", "secret")
self.assertEqual(folder.id, "f1")
self.assertEqual(laboratory.name, "mylab")
if __name__ == "__main__":
unittest.main()
+198
View File
@@ -0,0 +1,198 @@
import tempfile
import time
import unittest
from unittest.mock import patch
import jwt
from mdrsclient.api.utils import retry_on_expired_token, token_check, token_recover
from mdrsclient.cache import CacheFile
from mdrsclient.config import normalize_url
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import IllegalArgumentException, ServerBusyException, TokenExpiredException
from mdrsclient.models import Token
REMOTE = "unittest"
def make_token(access_offset: int, refresh_offset: int, label: str) -> Token:
now = int(time.time())
def encode(token_type: str, offset: int) -> str:
return jwt.encode(
{
"token_type": token_type,
"exp": now + offset,
"iat": now,
"jti": f"{label}-{token_type}",
"user_id": 1,
},
"unittest-signing-key-not-verified-anywhere",
)
return Token(access=encode("access", access_offset), refresh=encode("refresh", refresh_offset))
class TestTokenCache(unittest.TestCase):
def setUp(self) -> None:
self.tmp = tempfile.TemporaryDirectory()
patcher = patch("mdrsclient.cache.CONFIG_DIRNAME", self.tmp.name)
patcher.start()
self.addCleanup(patcher.stop)
self.addCleanup(self.tmp.cleanup)
def test_reload_picks_up_a_write_from_another_holder(self) -> None:
"""The cache is shared, so a value read earlier can already be out of date."""
reader = CacheFile(REMOTE)
writer = CacheFile(REMOTE)
first = make_token(3600, 86400, "first")
writer.token = first
self.assertEqual(reader.token, first)
second = make_token(3600, 86400, "second")
writer.token = second
reader.reload()
self.assertEqual(reader.token, second)
def test_token_check_uses_the_token_another_holder_just_wrote(self) -> None:
"""A rotating provider drops the token it replaces, so the refresh must not
be sent again once someone else has already made the round trip."""
connection = MDRSConnection(REMOTE, "http://localhost:8000/api/")
connection.token = make_token(-60, 86400, "stale")
# Another process refreshes while this one is between requests.
other = CacheFile(REMOTE)
rotated = make_token(3600, 86400, "rotated")
other.token = rotated
with patch("mdrsclient.api.utils.UsersApi") as users_api:
token_check(connection)
users_api.assert_not_called()
self.assertEqual(connection.token, rotated)
def test_token_check_refreshes_when_nothing_else_has(self) -> None:
connection = MDRSConnection(REMOTE, "http://localhost:8000/api/")
connection.token = make_token(-60, 86400, "stale")
rotated = make_token(3600, 86400, "rotated")
with patch("mdrsclient.api.utils.UsersApi") as users_api:
users_api.return_value.tokenRefresh.return_value = rotated
token_check(connection)
users_api.return_value.tokenRefresh.assert_called_once()
self.assertEqual(connection.token, rotated)
def test_token_check_logs_out_once_the_refresh_token_expires(self) -> None:
connection = MDRSConnection(REMOTE, "http://localhost:8000/api/")
connection.token = make_token(-3600, -60, "dead")
with patch("mdrsclient.api.utils.UsersApi") as users_api:
token_check(connection)
users_api.assert_not_called()
self.assertIsNone(connection.token)
class TestUrlNormalization(unittest.TestCase):
"""Both clients share config.ini, so they have to agree on what a remote URL is."""
def test_trailing_slash_is_dropped(self) -> None:
self.assertEqual(normalize_url("http://127.0.0.1:8000/api/"), "http://127.0.0.1:8000/api")
self.assertEqual(normalize_url("https://neurodata.riken.jp/api/"), "https://neurodata.riken.jp/api")
def test_bare_hostname_is_accepted(self) -> None:
self.assertEqual(normalize_url("http://localhost:8000/api"), "http://localhost:8000/api")
def test_only_http_schemes_are_accepted(self) -> None:
for url in ("ftp://x.example.com/", "file:///etc/passwd", "not-a-url", "http://"):
with self.subTest(url=url), self.assertRaises(IllegalArgumentException):
normalize_url(url)
class TestExpiredTokenRetry(unittest.TestCase):
"""A request can wait in the server's queue for longer than its access token lives."""
def setUp(self) -> None:
self.tmp = tempfile.TemporaryDirectory()
patcher = patch("mdrsclient.cache.CONFIG_DIRNAME", self.tmp.name)
patcher.start()
self.addCleanup(patcher.stop)
self.addCleanup(self.tmp.cleanup)
self.connection = MDRSConnection(REMOTE, "http://localhost:8000/api")
self.connection.token = make_token(3600, 86400, "sent")
def test_a_token_someone_else_refreshed_is_reused_rather_than_replaced(self) -> None:
used = self.connection.token
rotated = make_token(3600, 86400, "rotated")
CacheFile(REMOTE).token = rotated # another process, while the request waited
with patch("mdrsclient.api.utils.UsersApi") as users_api:
recovered = token_recover(self.connection, used)
self.assertTrue(recovered)
users_api.assert_not_called()
self.assertEqual(self.connection.token, rotated)
# The retry carries whatever the session header says, not what the cache holds,
# so reading the new token through is not enough on its own.
self.assertEqual(self.connection.session.headers["Authorization"], f"Bearer {rotated.access}")
def test_the_token_is_refreshed_when_nobody_else_has(self) -> None:
used = self.connection.token
rotated = make_token(3600, 86400, "rotated")
with patch("mdrsclient.api.utils.UsersApi") as users_api:
users_api.return_value.tokenRefresh.return_value = rotated
recovered = token_recover(self.connection, used)
self.assertTrue(recovered)
users_api.return_value.tokenRefresh.assert_called_once()
self.assertEqual(self.connection.token, rotated)
def test_the_request_is_sent_once_more_and_only_once(self) -> None:
calls: list[str] = []
class Api:
def __init__(self, connection: MDRSConnection) -> None:
self.connection = connection
@retry_on_expired_token
def send(self) -> str:
calls.append("sent")
if len(calls) == 1:
raise TokenExpiredException("expired on arrival")
return "ok"
with patch("mdrsclient.api.utils.UsersApi") as users_api:
users_api.return_value.tokenRefresh.return_value = make_token(3600, 86400, "rotated")
self.assertEqual(Api(self.connection).send(), "ok")
self.assertEqual(len(calls), 2)
def test_a_second_refusal_is_reported_as_an_overloaded_server(self) -> None:
calls: list[str] = []
class Api:
def __init__(self, connection: MDRSConnection) -> None:
self.connection = connection
@retry_on_expired_token
def send(self) -> str:
calls.append("sent")
raise TokenExpiredException("expired on arrival")
with patch("mdrsclient.api.utils.UsersApi") as users_api:
users_api.return_value.tokenRefresh.return_value = make_token(3600, 86400, "rotated")
with self.assertRaises(ServerBusyException) as caught:
Api(self.connection).send()
# Sent twice and no more, and the message names the cause the user can act on
# rather than an authentication failure that would send them to the login form.
self.assertEqual(len(calls), 2)
self.assertIn("overloaded", str(caught.exception))
if __name__ == "__main__":
unittest.main()
+297
View File
@@ -0,0 +1,297 @@
import os
import tempfile
import unittest
from io import StringIO
from unittest.mock import MagicMock, patch
from mdrsclient.api import FilesApi
from mdrsclient.exceptions import MDRSException, UnexpectedException
from mdrsclient.models import File, Folder, FolderSimple, Laboratory
from mdrsclient.transfer import Downloader, Uploader
TIMESTAMP = "2026-01-01T00:00:00+09:00"
def make_file(id: str, name: str, size: int = 1) -> File:
return File(
id=id,
name=name,
type="text/plain",
size=size,
thumbnail=None,
description="",
metadata={},
download_url=f"v3/files/{id}/download/",
created_at=TIMESTAMP,
updated_at=TIMESTAMP,
)
def make_folder_simple(id: str, name: str) -> FolderSimple:
return FolderSimple(
id=id,
pid=None,
name=name,
access_level=1,
lock=False,
size=0,
laboratory_id=1,
description="",
created_at=TIMESTAMP,
updated_at=TIMESTAMP,
restrict_opened_at=None,
)
def make_folder(id: str, name: str, path: str, sub_folders: list[FolderSimple] | None = None) -> Folder:
return Folder(
id=id,
pid=None,
name=name,
access_level=1,
lock=False,
size=0,
laboratory_id=1,
description="",
created_at=TIMESTAMP,
updated_at=TIMESTAMP,
restrict_opened_at=None,
metadata=[],
sub_folders=sub_folders if sub_folders is not None else [],
path=path,
)
LABORATORY = Laboratory(id=1, name="mylab", pi_name="PI", full_name="My Laboratory")
class TestUploadReportsFailure(unittest.TestCase):
"""A file that never reached the server must not leave the command reporting success."""
def make_client(self, folder: Folder, files: list[File]) -> MagicMock:
client = MagicMock()
client.parse_remote_host_with_path.return_value = ("myremote", "mylab", "/")
client.find_laboratory.return_value = LABORATORY
client.find_folder.return_value = folder
client.find_files.return_value = files
return client
def test_a_refused_upload_is_raised_to_the_caller(self):
client = self.make_client(make_folder("f1", "root", "/"), [])
with tempfile.TemporaryDirectory() as tmp:
local = os.path.join(tmp, "data.txt")
with open(local, "w") as f:
f.write("x")
with patch("mdrsclient.transfer.FilesApi") as files_api_class:
files_api_class.return_value.create.side_effect = MDRSException("Access is denied.")
with self.assertRaises(UnexpectedException):
Uploader(client).upload(local, "myremote:/mylab/", False, False)
def test_an_error_the_api_layer_did_not_raise_is_still_counted(self):
"""A file that vanished between the walk and the upload must not end the run."""
client = self.make_client(make_folder("f1", "root", "/"), [])
with tempfile.TemporaryDirectory() as tmp:
local = os.path.join(tmp, "data.txt")
with open(local, "w") as f:
f.write("x")
with patch("mdrsclient.transfer.FilesApi") as files_api_class:
files_api_class.return_value.create.side_effect = FileNotFoundError(local)
with self.assertRaises(UnexpectedException):
Uploader(client).upload(local, "myremote:/mylab/", False, False)
def test_a_failed_upload_names_the_file(self):
client = self.make_client(make_folder("f1", "root", "/"), [])
with tempfile.TemporaryDirectory() as tmp:
local = os.path.join(tmp, "data.txt")
with open(local, "w") as f:
f.write("x")
with patch("mdrsclient.transfer.FilesApi") as files_api_class:
files_api_class.return_value.create.side_effect = MDRSException("Access is denied.")
with patch("sys.stdout", new=StringIO()) as fake_out:
with self.assertRaises(UnexpectedException):
Uploader(client).upload(local, "myremote:/mylab/", False, False)
self.assertIn(local, fake_out.getvalue())
def test_an_upload_that_worked_stays_quiet(self):
client = self.make_client(make_folder("f1", "root", "/"), [])
with tempfile.TemporaryDirectory() as tmp:
local = os.path.join(tmp, "data.txt")
with open(local, "w") as f:
f.write("x")
with patch("mdrsclient.transfer.FilesApi") as files_api_class:
files_api_class.return_value.create.return_value = "new-id"
Uploader(client).upload(local, "myremote:/mylab/", False, False)
files_api_class.return_value.create.assert_called_once()
class TestDownloadReportsFailure(unittest.TestCase):
def make_client(self, remote_path_parts, folder: Folder, files_by_folder: dict[str, list[File]]) -> MagicMock:
client = MagicMock()
client.is_doi.return_value = False
client.parse_remote_host_with_path.return_value = remote_path_parts
client.find_laboratory.return_value = LABORATORY
client.find_folder.return_value = folder
client.find_files.side_effect = lambda folder_id: files_by_folder.get(folder_id, [])
return client
def test_a_single_file_that_failed_is_raised_to_the_caller(self):
parent = make_folder("p1", "root", "/")
client = self.make_client(("myremote", "mylab", "/data.txt"), parent, {"p1": [make_file("x1", "data.txt")]})
with tempfile.TemporaryDirectory() as tmp:
with patch("mdrsclient.transfer.FilesApi") as files_api_class:
files_api_class.return_value.download.side_effect = OSError("Permission denied.")
with self.assertRaises(UnexpectedException):
Downloader(client).download("myremote:/mylab/data.txt", tmp)
def test_a_single_file_that_arrived_stays_quiet(self):
parent = make_folder("p1", "root", "/")
client = self.make_client(("myremote", "mylab", "/data.txt"), parent, {"p1": [make_file("x1", "data.txt")]})
with tempfile.TemporaryDirectory() as tmp:
with patch("mdrsclient.transfer.FilesApi") as files_api_class:
files_api_class.return_value.download.return_value = True
Downloader(client).download("myremote:/mylab/data.txt", tmp)
files_api_class.return_value.download.assert_called_once()
def test_a_failed_file_does_not_abandon_the_remaining_sub_folders(self):
parent = make_folder("p1", "lab", "/", [make_folder_simple("f1", "root")])
folders = {
"f1": make_folder("f1", "root", "/root/", [make_folder_simple("fa", "a"), make_folder_simple("fb", "b")]),
"fa": make_folder("fa", "a", "/root/a/"),
"fb": make_folder("fb", "b", "/root/b/"),
}
files_by_folder = {
"p1": [],
"f1": [make_file("bad", "bad.txt")],
"fa": [make_file("good1", "good1.txt")],
"fb": [make_file("good2", "good2.txt")],
}
client = self.make_client(("myremote", "mylab", "/root"), parent, files_by_folder)
attempted: list[str] = []
def download(file: File, path: str) -> bool:
attempted.append(file.name)
if file.name == "bad.txt":
raise OSError("Permission denied.")
return True
with tempfile.TemporaryDirectory() as tmp:
with (
patch("mdrsclient.transfer.FoldersApi") as folders_api_class,
patch("mdrsclient.transfer.FilesApi") as files_api_class,
):
folders_api_class.return_value.retrieve.side_effect = lambda folder_id: folders[folder_id]
files_api_class.return_value.download.side_effect = download
with self.assertRaises(UnexpectedException):
Downloader(client).download("myremote:/mylab/root", tmp, is_recursive=True)
self.assertEqual(sorted(attempted), ["bad.txt", "good1.txt", "good2.txt"])
class TestRecursiveDownloadResilience(unittest.TestCase):
"""A folder the client cannot prepare locally must not abandon its siblings."""
def test_a_folder_that_cannot_be_created_does_not_stop_the_walk(self):
parent = make_folder("p1", "lab", "/", [make_folder_simple("f1", "root")])
folders = {
"f1": make_folder("f1", "root", "/root/", [make_folder_simple("fa", "a"), make_folder_simple("fb", "b")]),
"fa": make_folder("fa", "a", "/root/a/"),
"fb": make_folder("fb", "b", "/root/b/"),
}
files_by_folder = {"p1": [], "f1": [], "fa": [make_file("good1", "good1.txt")], "fb": []}
client = MagicMock()
client.is_doi.return_value = False
client.parse_remote_host_with_path.return_value = ("myremote", "mylab", "/root")
client.find_laboratory.return_value = LABORATORY
client.find_folder.return_value = parent
client.find_files.side_effect = lambda folder_id: files_by_folder.get(folder_id, [])
real_makedirs = os.makedirs
def makedirs(path, *args, **kwargs):
if os.path.basename(path) == "b":
raise PermissionError("Permission denied")
return real_makedirs(path, *args, **kwargs)
attempted: list[str] = []
with tempfile.TemporaryDirectory() as tmp:
with (
patch("mdrsclient.transfer.FoldersApi") as folders_api_class,
patch("mdrsclient.transfer.FilesApi") as files_api_class,
patch("mdrsclient.transfer.os.makedirs", side_effect=makedirs),
):
folders_api_class.return_value.retrieve.side_effect = lambda folder_id: folders[folder_id]
files_api_class.return_value.download.side_effect = lambda file, path: attempted.append(file.name)
with self.assertRaises(UnexpectedException):
Downloader(client).download("myremote:/mylab/root", tmp, is_recursive=True)
self.assertEqual(attempted, ["good1.txt"])
class TestFileDownloadPermission(unittest.TestCase):
"""A file the client could not write is a failure, not a line of successful output."""
def make_connection(self, chunks) -> MagicMock:
connection = MagicMock()
connection.token = None
response = MagicMock()
response.status_code = 200
response.iter_content.return_value = chunks
connection.get.return_value = response
return connection
def test_a_download_that_cannot_be_written_is_reported(self):
connection = self.make_connection([b"payload"])
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "precious.dat")
with open(path, "wb") as f:
f.write(b"do not touch")
os.chmod(path, 0o444)
try:
with open(path, "r+b"):
self.skipTest("running with rights that ignore the file mode")
except OSError:
pass
with self.assertRaises(UnexpectedException):
FilesApi(connection).download(make_file("x1", "precious.dat"), path)
os.chmod(path, 0o644)
with open(path, "rb") as f:
self.assertEqual(f.read(), b"do not touch")
connection.get.assert_not_called()
def test_an_interrupted_download_leaves_the_existing_file_intact(self):
def chunks():
yield b"half a file"
raise ConnectionError("connection reset")
connection = self.make_connection(chunks())
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "existing.dat")
with open(path, "wb") as f:
f.write(b"the copy already here")
with self.assertRaises(ConnectionError):
FilesApi(connection).download(make_file("x1", "existing.dat"), path)
with open(path, "rb") as f:
self.assertEqual(f.read(), b"the copy already here")
self.assertEqual(os.listdir(tmp), ["existing.dat"])
def test_a_finished_download_replaces_the_destination(self):
connection = self.make_connection([b"new ", b"contents"])
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "existing.dat")
with open(path, "wb") as f:
f.write(b"old")
FilesApi(connection).download(make_file("x1", "existing.dat"), path)
with open(path, "rb") as f:
self.assertEqual(f.read(), b"new contents")
self.assertEqual(os.listdir(tmp), ["existing.dat"])
if __name__ == "__main__":
unittest.main()