26 Commits

Author SHA1 Message Date
orrisroot d59a150b4f chore(release): bump version to 1.3.18
Bump the package version to 1.3.18, upgrade dependencies,
consolidate module exports, add a unit test suite, and document
all changes.

- Bump package version to 1.3.18 in pyproject.toml
- Upgrade pydantic-settings to 2.14.2 and pyright to 1.1.411
- Consolidate package exports in mdrsclient/__init__.py
- Add a comprehensive unit test suite in tests/test_commands.py
- Document testing execution and add full history in CHANGELOG.md
2026-07-02 23:47:57 +09:00
orrisroot 7f6d496654 docs(readme): document custom configuration usage in library API
Update the Python API Usage section in README.md to demonstrate how to
initialize MdrsClient with InMemoryConfig to avoid creating local
config files.
2026-07-02 23:31:08 +09:00
orrisroot b95fc0cd7d refactor(config): abstract config storage and enable dependency injection
Abstract the configuration storage mechanism to allow using custom
configurations, such as in-memory setups, when using the tool as
a library. This aligns the configuration architecture with the
session cache abstraction.

- Define ConfigInterface protocol and InMemoryConfig class
- Make CacheFile, InMemoryCache, ConfigFile, and InMemoryConfig
  explicitly inherit their interfaces
- Update MdrsService and MdrsClient to accept customizable
  config_class and config instances
- Add validation to check remote parameter consistency in
  create_connection
- Remove unused imports across command files
2026-07-02 23:30:33 +09:00
orrisroot 8ce9e09e69 refactor: use services layer and modularize transfer operations
Decouple CLI commands from internal helper logic and consolidate the
core file transfer operations in the service layer to improve library
portability.

- Make MdrsClient subclass MdrsService to inherit resource resolution.
- Remove all deprecated helper methods from BaseCommand.
- Move core upload and download logic to a new transfer module.
- Refactor all CLI commands to route actions through MdrsClient.
- Eliminate circular imports between client and CLI command modules.
2026-07-02 23:16:53 +09:00
orrisroot 36cad6db52 refactor: extract MdrsClient service layer for library portability
To improve the tool's portability as a Python library, the core logic
has been decoupled from the CLI interface. This allows developers to
programmatically interact with MDRS without relying on CLI-specific
argument parsing or local file-based caches.

- Introduce `MdrsClient` service layer to handle core operations.
- Abstract authentication state using `CacheInterface` and `InMemoryCache`.
- Migrate all CLI commands to utilize `MdrsClient` for execution.
- Separate `Doi` data model from API responses and move to `models/doi.py`.
- Update `README.md` to include Python API usage examples.
- Bump package version to 1.3.17.
2026-07-02 13:07:18 +09:00
orrisroot 809140dfbc fix(download): retrieve full folder to resolve type error
Retrieve the full Folder object from FoldersApi instead of using the
FolderSimple returned by find_sub_folder when resolving DOI subfolders.
This fixes a type checker error under the upgraded pyright, and avoids
a potential AttributeError at runtime due to FolderSimple lacking the
path attribute.

Also upgrade dependencies and bump version to 1.3.16 in pyproject.toml.
2026-06-12 10:25:03 +09:00
orrisroot 5bdf837941 feat(doi): add DOI-based path access for commands
Support accessing repositories using DOI strings with optional subpaths
across ls, download, metadata, and file-metadata commands.

- Implement GET v3/doi/{id}/ API model and client calls
- Parse and resolve DOI paths into respective folder and files
- Extract common folder and file resolution logic to shared helpers
- Update README with example DOI-based shell commands
2026-06-12 01:28:40 +09:00
orrisroot 04c0003a61 fix(connection): support absolute download urls for public data 2026-06-11 21:04:28 +09:00
orrisroot 428be1289c chore(version): bump package version to 1.3.15
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-01 11:11:22 +09:00
orrisroot 4283481695 fix: apply NFC normalization to filenames and folder names sent to server
On macOS, local filenames and directory names may be in NFD encoding
(decomposed Unicode). Without normalization, files and folders are
created on the server with NFD names, inconsistent with the server's
NFC convention.

Apply normalize("NFC", ...) before sending names to the server in:
- FilesApi.create(): filename in multipart upload
- FilesApi.update(): filename in multipart upload
- UploadCommand: directory name in FoldersApi.create()

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-20 12:25:59 +09:00
orrisroot ddb4300d85 feat(config): simplify list command and add subcommand aliases
- config list: remove -l/--long option, always display URL
- config list: add ls alias (already existed, kept)
- config delete: add rm alias (alongside existing remove alias)
- README: add config update, config list, config delete sections

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-17 18:59:07 +09:00
orrisroot 68670a6588 fix(ls): rename --quick to --quiet; add version command; bump to 1.3.14
- Fix ls -q long option name: --quick → --quiet (typo fix)
- Remove mdrsclient/VERSION file; read version via importlib.metadata
- Bump version 1.3.13 → 1.3.14
- Add Python 3.14 to supported classifiers; promote to Development Status 4 - Beta
- Add `version` subcommand (prints "mdrs <version>")
- Document `version` command in README

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-17 17:41:37 +09:00
orrisroot 6d8fd0a598 update version to 1.3.13 2025-10-28 11:10:09 +09:00
pw-serizawa 634b5f1a13 update version to 1.3.12 2025-07-02 18:34:11 +09:00
pw-serizawa 95f22ea5f9 fix for file.list api pagination 2025-06-26 17:13:06 +09:00
orrisroot fffb686023 update version to 1.3.12 2025-05-20 18:02:47 +09:00
orrisroot 893bc77128 fixed bug to download file if skip flag is present. 2025-05-20 17:50:11 +09:00
orrisroot 7b3f1f2d09 update version to 1.3.11 2025-01-21 18:33:41 +09:00
orrisroot 8a5e1b68b7 follow-up fixes due to recent User API specification changes. 2025-01-21 18:30:22 +09:00
orrisroot dd00973bea update version number to 1.3.10. 2024-12-23 20:43:57 +09:00
orrisroot 0e5685d5ea update libraries and version number. 2024-12-23 14:28:25 +09:00
orrisroot 6a2810f603 show summary when file download fails and delete broken files. 2024-12-23 13:58:14 +09:00
orrisroot d5ac5cd427 fixed compatibility with python 3.10. 2024-10-23 18:33:12 +09:00
orrisroot ab7cd1b885 update version to 1.3.8. 2024-09-18 11:24:06 +09:00
orrisroot f2f898c263 check the exception to unexpected responses from the server. 2024-09-18 11:22:03 +09:00
orrisroot 24172dc65c implemented -s --skip-if-file-exists option for download commnad. 2024-09-18 10:56:40 +09:00
45 changed files with 1824 additions and 576 deletions
+5 -1
View File
@@ -16,7 +16,11 @@
"mdrsclient", "mdrsclient",
"neurodata", "neurodata",
"Neuroinformatics", "Neuroinformatics",
"orcid",
"RIKEN" "RIKEN"
], ],
"ignorePaths": [".env", "__pycache__"] "ignorePaths": [
".env",
"__pycache__"
]
} }
+1
View File
@@ -161,3 +161,4 @@ cython_debug/
# mdrs-cli # mdrs-cli
.neurodatacli.config .neurodatacli.config
poetry.toml
+129
View File
@@ -0,0 +1,129 @@
# Changelog
All notable changes to this project will be documented in this file.
## [1.3.18] - 2026-07-02
### Added
- Added a comprehensive unit test suite in `tests/test_commands.py` checking registration, parsing, and execution flow of all 16 commands.
### Refactored
- Abstracted configuration storage (introducing `ConfigInterface`, `InMemoryConfig`, and updating `ConfigFile`) to enable dependency injection.
- Modularized transfer operations (upload and download) to decouple them from the service layer.
- Decoupled commands from direct file system configurations and migrated all subcommands to use abstract config classes.
### Changed
- Upgraded dependencies including `pydantic-settings` to `2.14.2` and `pyright` to `1.1.411`.
### Fixed
- Fixed duplicate `__all__` definitions in package initialization file `mdrsclient/__init__.py` that caused `__version__` export to be overwritten.
## [1.3.17] - 2026-07-02
### Refactored
- Decoupled core logic from CLI interface and introduced `MdrsClient` service layer to improve library portability.
- Migrated all CLI commands to utilize `MdrsClient` for execution.
### Added
- Abstract authentication state using `CacheInterface` and `InMemoryCache`.
## [1.3.16] - 2026-06-12
### Fixed
- Retrieve the full `Folder` object from `FoldersApi` instead of using the `FolderSimple` returned by `find_sub_folder` when resolving DOI subfolders. This fixes a type checker error under the upgraded pyright and avoids a potential AttributeError at runtime due to `FolderSimple` lacking the `path` attribute.
### Changed
- Upgraded dependencies and bumped version to 1.3.16 in pyproject.toml.
## [1.3.15] - 2026-05-01
### Changed
- Bumped package version to 1.3.15.
## [1.3.14] - 2026-04-17
### Changed
- Simplified `config list` command (removed `-l`/`--long` option, always display URL).
### Added
- Added subcommand aliases for config commands (e.g. `ls` alias for list, `rm` alias for delete).
## [1.3.13] - 2025-07-02
### Changed
- Bumped package version to 1.3.13.
## [1.3.12] - 2025-05-20
### Changed
- Bumped package version to 1.3.12.
## [1.3.11] - 2025-01-21
### Changed
- Bumped package version to 1.3.11.
## [1.3.10] - 2024-12-23
### Changed
- Bumped package version to 1.3.10.
## [1.3.9] - 2024-10-23
### Fixed
- Fixed compatibility with Python 3.10.
## [1.3.8] - 2024-09-18
### Changed
- Bumped package version to 1.3.8.
## [1.3.7] - 2024-07-22
### Added
- Implemented `--exclude` argument for download subcommand.
## [1.3.6] - 2024-07-08
### Changed
- Bumped package version to 1.3.6.
## [1.3.5] - 2024-07-08
### Changed
- Bumped package version to 1.3.5.
## [1.3.4] - 2024-07-04
### Added
- Added some aliases for config sub command.
## [1.3.3] - 2024-02-13
### Changed
- Bumped package version to 1.3.3.
## [1.3.2] - 2024-02-09
### Added
- Added `-u` and `-p` options to login command.
## [1.3.1] - 2023-12-20
### Fixed
- Fixed bug to resolve local files for recursive file upload.
## [1.3.0] - 2023-12-18
### Changed
- Removed debug comments.
## [1.2.0] - 2023-10-04
### Changed
- Follow-up recent specification changes about folder access level.
## [1.1.1] - 2023-07-26
### Changed
- Set destination folder name using name attribute of folder copy API.
+104 -12
View File
@@ -8,7 +8,7 @@ The mdrs-client-python is python library and a command-line client for up- and d
poetry install poetry install
``` ```
## Example Usage ## CLI Usage
### config create ### config create
@@ -18,6 +18,32 @@ Create remote host configuration
mdrs config create neurodata https://neurodata.riken.jp/api mdrs config create neurodata https://neurodata.riken.jp/api
``` ```
### config update
Update the URL of a registered remote host.
```shell
mdrs config update neurodata https://neurodata.riken.jp/api
```
### config list
List registered remote hosts.
```shell
mdrs config list
mdrs config ls
```
### config delete
Remove a registered remote host.
```shell
mdrs config delete neurodata
mdrs config rm neurodata
```
### login ### login
Login to remote host Login to remote host
@@ -56,13 +82,17 @@ mdrs labs neurodata:
### ls ### ls
List the folder contents List the folder contents. You can also specify a DOI path in the form `remote:10.xxxx/yyy.ID[/optional/subpath]`.
```shell ```shell
mdrs ls neurodata:/NIU/Repository/ mdrs ls neurodata:/NIU/Repository/
mdrs ls -p PW_OPEN_PASSWORD neurodata:/NIU/Repository/PW_Open/ mdrs ls -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/
mdrs ls -r neurodata:/NIU/Repository/Dataset1/ mdrs ls -r neurodata:/NIU/Repository/Dataset1/
mdrs ls -J -r neurodata:/NIU/Repository/Dataset1/ mdrs ls -J -r neurodata:/NIU/Repository/Dataset1/
# DOI access examples:
mdrs ls neurodata:10.60178/cbs.20260429-001
mdrs ls "neurodata:10.60178/cbs.20260429-001/Figure 1"
``` ```
### mkdir ### mkdir
@@ -80,18 +110,23 @@ Upload the file or directory
```shell ```shell
mdrs upload ./sample.dat neurodata:/NIU/Repository/TEST/ mdrs upload ./sample.dat neurodata:/NIU/Repository/TEST/
mdrs upload -r ./dataset neurodata:/NIU/Repository/TEST/ mdrs upload -r ./dataset neurodata:/NIU/Repository/TEST/
mdrs upload -r -s ./dataset neurodata:/NIU/Repository/TEST/ mdrs upload -r --skip-if-exists ./dataset neurodata:/NIU/Repository/TEST/
``` ```
### download ### download
Download the file or folder Download the file or folder. You can also specify a DOI path.
```shell ```shell
mdrs download neurodata:/NIU/Repository/TEST/sample.dat ./ mdrs download neurodata:/NIU/Repository/TEST/sample.dat ./
mdrs download -r neurodata:/NIU/Repository/TEST/dataset/ ./ mdrs download -r neurodata:/NIU/Repository/TEST/dataset/ ./
mdrs download -p PW_OPEN_PASSWORD neurodata:/NIU/Repository/PW_Open/Readme.dat ./ mdrs download -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/Readme.dat ./
mdrs download -r --exclude /NIU/Repository/TEST/dataset/skip neurodata:/NIU/Repository/TEST/dataset/ ./ mdrs download -r --exclude /NIU/Repository/TEST/dataset/skip neurodata:/NIU/Repository/TEST/dataset/ ./
mdrs download -r --skip-if-exists neurodata:/NIU/Repository/TEST/dataset/ ./
# DOI access examples:
mdrs download neurodata:10.60178/cbs.20260429-001/README_NeuroData.md ./
mdrs download -r "neurodata:10.60178/cbs.20260429-001/Figure 1" ./
``` ```
### mv ### mv
@@ -128,25 +163,39 @@ Change the folder access level
```shell ```shell
mdrs chacl private neurodata:/NIU/Repository/Private mdrs chacl private neurodata:/NIU/Repository/Private
mdrs chacl cbs_open -r neurodata:/NIU/Repository/CBS_Open mdrs chacl cbs_open -r neurodata:/NIU/Repository/CBS_Open
mdrs chacl pw_open -r -p PW_OPEN_PASSWORD neurodata:/NIU/Repository/PW_Open mdrs chacl pw_open -r -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open
``` ```
### metadata ### metadata
Get a folder metadata Get a folder metadata. You can also specify a DOI path.
```shell ```shell
mdrs metadata neurodata:/NIU/Repository/TEST/ mdrs metadata neurodata:/NIU/Repository/TEST/
mdrs metadata -p PW_OPEN_PASSWORD neurodata:/NIU/Repository/PW_Open/ mdrs metadata -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/
# DOI access examples:
mdrs metadata neurodata:10.60178/cbs.20260429-001
``` ```
### file-metadata ### file-metadata
Get the file metadata Get the file metadata. You can also specify a DOI path.
```shell ```shell
mdrs file-metadata neurodata:/NIU/Repository/TEST/dataset/sample.dat mdrs file-metadata neurodata:/NIU/Repository/TEST/dataset/sample.dat
mdrs file-metadata -p PW_OPEN_PASSWORD neurodata:/NIU/Repository/PW_Open/Readme.txt mdrs file-metadata -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/Readme.txt
# DOI access examples:
mdrs file-metadata "neurodata:10.60178/cbs.20260429-001/Figure 1/Figure1v3.pdf"
```
### version
Show the tool name and version number
```shell
mdrs version
``` ```
### help ### help
@@ -156,3 +205,46 @@ Show the help message and exit
```shell ```shell
mdrs -h mdrs -h
``` ```
## Python API Usage
You can also use this package as a Python library to programmatically interact with MDRS repositories.
```python
from mdrsclient.client import MdrsClient
from mdrsclient.cache import InMemoryCache
from mdrsclient.config import InMemoryConfig
# 1. Setup in-memory configuration and cache to avoid local state files (e.g., config.ini, cache/*.json)
config = InMemoryConfig("neurodata")
config.url = "https://neurodata.riken.jp/api"
cache = InMemoryCache()
# 2. Initialize client with custom configuration and cache
client = MdrsClient.from_remote("neurodata", cache=cache, config=config)
# 3. Login to the remote server
client.login("username", "password")
# 4. Use service methods
labs = client.get_laboratories()
metadata = client.metadata("neurodata:/NIU/Repository/")
# Transfer files programmatically
client.upload("/path/to/local/data", "neurodata:/NIU/Repository/TEST/", is_recursive=True)
client.download("neurodata:/NIU/Repository/TEST/data", "/path/to/local", is_recursive=True)
```
## Testing
You can run the unit test suite using the standard library `unittest` discover runner:
```shell
.venv/bin/python -m unittest discover tests
```
## Changelog
See [CHANGELOG.md](./CHANGELOG.md) for the full change history.
-1
View File
@@ -1 +0,0 @@
1.3.7
+2 -1
View File
@@ -1,3 +1,4 @@
from mdrsclient.__version__ import __version__ from mdrsclient.__version__ import __version__
from mdrsclient.client import MdrsClient
__all__ = ["__version__"] __all__ = ["__version__", "MdrsClient"]
+6
View File
@@ -1,5 +1,6 @@
import argparse import argparse
import sys import sys
from json import JSONDecodeError
from mdrsclient.commands import ( from mdrsclient.commands import (
ChaclCommand, ChaclCommand,
@@ -16,6 +17,7 @@ from mdrsclient.commands import (
MvCommand, MvCommand,
RmCommand, RmCommand,
UploadCommand, UploadCommand,
VersionCommand,
WhoamiCommand, WhoamiCommand,
) )
from mdrsclient.exceptions import MDRSException from mdrsclient.exceptions import MDRSException
@@ -28,6 +30,7 @@ def main() -> None:
parsers = parser.add_subparsers(title="subcommands") parsers = parser.add_subparsers(title="subcommands")
ConfigCommand.register(parsers) ConfigCommand.register(parsers)
VersionCommand.register(parsers)
LoginCommand.register(parsers) LoginCommand.register(parsers)
LogoutCommand.register(parsers) LogoutCommand.register(parsers)
WhoamiCommand.register(parsers) WhoamiCommand.register(parsers)
@@ -52,6 +55,9 @@ def main() -> None:
except MDRSException as e: except MDRSException as e:
print(f"Error: {e}") print(f"Error: {e}")
sys.exit(2) sys.exit(2)
except JSONDecodeError:
print("Unexpected response returned. Please check the configuration or the server's operational status.")
sys.exit(2)
except KeyboardInterrupt: except KeyboardInterrupt:
sys.exit(130) sys.exit(130)
+5 -4
View File
@@ -1,8 +1,9 @@
import os import importlib.metadata
here = os.path.realpath(os.path.dirname(__file__)) try:
__version__ = importlib.metadata.version("mdrs-client-python")
except importlib.metadata.PackageNotFoundError:
__version__ = "0.0.0-dev"
with open(os.path.join(here, "VERSION")) as version_file:
__version__ = version_file.read().strip()
__all__ = ["__version__"] __all__ = ["__version__"]
+2
View File
@@ -1,9 +1,11 @@
from mdrsclient.api.doi import DoiApi
from mdrsclient.api.files import FilesApi from mdrsclient.api.files import FilesApi
from mdrsclient.api.folders import FoldersApi from mdrsclient.api.folders import FoldersApi
from mdrsclient.api.laboratories import LaboratoriesApi from mdrsclient.api.laboratories import LaboratoriesApi
from mdrsclient.api.users import UsersApi from mdrsclient.api.users import UsersApi
__all__ = [ __all__ = [
"DoiApi",
"FilesApi", "FilesApi",
"FoldersApi", "FoldersApi",
"LaboratoriesApi", "LaboratoriesApi",
+1 -6
View File
@@ -5,12 +5,7 @@ from pydantic import TypeAdapter
from requests import Response from requests import Response
from mdrsclient.connection import MDRSConnection from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import ( from mdrsclient.exceptions import BadRequestException, ForbiddenException, UnauthorizedException, UnexpectedException
BadRequestException,
ForbiddenException,
UnauthorizedException,
UnexpectedException,
)
from mdrsclient.models.error import DRFStandardizedErrors from mdrsclient.models.error import DRFStandardizedErrors
+33
View File
@@ -0,0 +1,33 @@
from typing import Final
from pydantic import TypeAdapter
from pydantic.dataclasses import dataclass
from mdrsclient.api.base import BaseApi
from mdrsclient.api.utils import token_check
from mdrsclient.models.doi import Doi
@dataclass(frozen=True)
class DoiRetrieveFolderRef:
id: str
@dataclass(frozen=True)
class DoiRetrieveResponse:
id: str
doi: str
folder: DoiRetrieveFolderRef
class DoiApi(BaseApi):
ENTRYPOINT: Final[str] = "v3/doi/"
def retrieve(self, doi_id: str) -> Doi:
"""Retrieve the folder associated with a DOI suffix ID (GET v3/doi/{id}/)."""
url = self.ENTRYPOINT + doi_id + "/"
token_check(self.connection)
response = self.connection.get(url)
self._raise_response_error(response)
api_resp = TypeAdapter(DoiRetrieveResponse).validate_python(response.json())
return Doi(id=api_resp.id, doi=api_resp.doi, folder_id=api_resp.folder.id)
+24 -2
View File
@@ -1,6 +1,7 @@
import mimetypes import mimetypes
import os import os
from typing import Any, Final from typing import Any, Final
from unicodedata import normalize
from pydantic import TypeAdapter from pydantic import TypeAdapter
from pydantic.dataclasses import dataclass from pydantic.dataclasses import dataclass
@@ -17,10 +18,26 @@ class FilesApiCreateResponse:
id: str id: str
@dataclass(frozen=True)
class FilesApiListResponse:
count: int
next: str | None
previous: str | None
results: list[File]
class FilesApi(BaseApi): class FilesApi(BaseApi):
ENTRYPOINT: Final[str] = "v3/files/" ENTRYPOINT: Final[str] = "v3/files/"
FALLBACK_MIMETYPE: Final[str] = "application/octet-stream" FALLBACK_MIMETYPE: Final[str] = "application/octet-stream"
def list(self, folder_id: str, page_num: int) -> FilesApiListResponse:
url = self.ENTRYPOINT
token_check(self.connection)
params: dict[str, str | int] = {"folder_id": folder_id, "page": page_num}
response = self.connection.get(url, params=params)
self._raise_response_error(response)
return TypeAdapter(FilesApiListResponse).validate_python(response.json())
def retrieve(self, id: str) -> File: def retrieve(self, id: str) -> File:
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name) # print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
url = self.ENTRYPOINT + id + "/" url = self.ENTRYPOINT + id + "/"
@@ -37,7 +54,10 @@ class FilesApi(BaseApi):
try: try:
with open(os.path.realpath(path), mode="rb") as fp: with open(os.path.realpath(path), mode="rb") as fp:
data = MultipartEncoder( data = MultipartEncoder(
fields={"folder_id": folder_id, "file": (os.path.basename(path), fp, self._get_mime_type(path))} fields={
"folder_id": folder_id,
"file": (normalize("NFC", os.path.basename(path)), fp, self._get_mime_type(path)),
}
) )
response = self.connection.post(url, data=data, headers={"Content-Type": data.content_type}) response = self.connection.post(url, data=data, headers={"Content-Type": data.content_type})
self._raise_response_error(response) self._raise_response_error(response)
@@ -59,7 +79,9 @@ class FilesApi(BaseApi):
# update file body # update file body
try: try:
with open(os.path.realpath(path), mode="rb") as fp: with open(os.path.realpath(path), mode="rb") as fp:
data = MultipartEncoder(fields={"file": (os.path.basename(path), fp, self._get_mime_type(path))}) data = MultipartEncoder(
fields={"file": (normalize("NFC", os.path.basename(path)), fp, self._get_mime_type(path))}
)
response = self.connection.put(url, data=data, headers={"Content-Type": data.content_type}) response = self.connection.put(url, data=data, headers={"Content-Type": data.content_type})
except OSError: except OSError:
raise UnexpectedException(f"Could not open `{path}` file.") raise UnexpectedException(f"Could not open `{path}` file.")
+3 -1
View File
@@ -20,8 +20,10 @@ class UsersCurrentResponseLaboratory:
class UsersApiCurrentResponse: class UsersApiCurrentResponse:
id: int id: int
username: str username: str
full_name: str first_name: str
last_name: str
email: str email: str
orcid_id: str
laboratories: list[UsersCurrentResponseLaboratory] laboratories: list[UsersCurrentResponseLaboratory]
is_staff: bool is_staff: bool
is_active: bool is_active: bool
+64 -2
View File
@@ -2,6 +2,7 @@ import dataclasses
import hashlib import hashlib
import json import json
import os import os
from typing import Protocol, runtime_checkable
from pydantic import TypeAdapter, ValidationError from pydantic import TypeAdapter, ValidationError
from pydantic.dataclasses import dataclass from pydantic.dataclasses import dataclass
@@ -16,7 +17,7 @@ from mdrsclient.utils import FileLock
class CacheData: class CacheData:
user: User | None = None user: User | None = None
token: Token | None = None token: Token | None = None
laboratories: Laboratories = Laboratories() laboratories: Laboratories = dataclasses.field(default_factory=Laboratories)
digest: str = "" digest: str = ""
def clear(self) -> None: def clear(self) -> None:
@@ -43,7 +44,68 @@ class CacheData:
).hexdigest() ).hexdigest()
class CacheFile: @runtime_checkable
class CacheInterface(Protocol):
@property
def token(self) -> Token | None: ...
@token.setter
def token(self, token: Token) -> None: ...
@token.deleter
def token(self) -> None: ...
@property
def user(self) -> User | None: ...
@user.setter
def user(self, user: User) -> None: ...
@user.deleter
def user(self) -> None: ...
@property
def laboratories(self) -> Laboratories: ...
@laboratories.setter
def laboratories(self, laboratories: Laboratories) -> None: ...
class InMemoryCache(CacheInterface):
def __init__(self) -> None:
self.__data = CacheData()
@property
def token(self) -> Token | None:
return self.__data.token
@token.setter
def token(self, token: Token) -> None:
self.__data.token = token
@token.deleter
def token(self) -> None:
if self.__data.token is not None:
self.__data.token = None
@property
def user(self) -> User | None:
return self.__data.user
@user.setter
def user(self, user: User) -> None:
self.__data.user = user
@user.deleter
def user(self) -> None:
if self.__data.user is not None:
self.__data.user = None
@property
def laboratories(self) -> Laboratories:
return self.__data.laboratories
@laboratories.setter
def laboratories(self, laboratories: Laboratories) -> None:
self.__data.laboratories = laboratories
class CacheFile(CacheInterface):
__serial: int __serial: int
__cache_dir: str __cache_dir: str
__cache_file: str __cache_file: str
+239
View File
@@ -0,0 +1,239 @@
import os
from typing import Any
from unicodedata import normalize
from mdrsclient.api import DoiApi, FilesApi, FoldersApi, LaboratoriesApi, UsersApi
from mdrsclient.cache import CacheInterface
from mdrsclient.config import ConfigInterface
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import IllegalArgumentException, MDRSException, UnauthorizedException, UnexpectedException
from mdrsclient.models import File, Folder, Laboratory, Token, User
from mdrsclient.models.file import find_file
from mdrsclient.services import MdrsService
class MdrsClient(MdrsService):
"""Service layer client for MDRS."""
def __init__(self, connection: MDRSConnection, config_class: type[ConfigInterface] | None = None):
super().__init__(connection, config_class)
@classmethod
def from_remote(
cls, remote: str, cache: CacheInterface | None = None, config: ConfigInterface | None = None
) -> "MdrsClient":
return cls(cls.create_connection(remote, cache, config))
def mkdir(self, remote_path: str) -> None:
remote, laboratory_name, r_path = self.parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
r_dirname = os.path.dirname(r_path)
r_basename = os.path.basename(r_path)
laboratory = self.find_laboratory(laboratory_name)
parent_folder = self.find_folder(laboratory, r_dirname)
files = self.find_files(parent_folder.id)
if parent_folder.find_sub_folder(r_basename) is not None or find_file(files, r_basename) is not None:
raise IllegalArgumentException(f"Cannot create folder `{r_path}`: File exists.")
folder_api = FoldersApi(self.connection)
folder_api.create(normalize("NFC", r_basename), parent_folder.id)
def rm(self, remote_path: str, is_recursive: bool = False) -> None:
remote, laboratory_name, r_path = self.parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
r_dirname = os.path.dirname(r_path)
r_basename = os.path.basename(r_path)
laboratory = self.find_laboratory(laboratory_name)
parent_folder = self.find_folder(laboratory, r_dirname)
parent_files = self.find_files(parent_folder.id)
file = find_file(parent_files, r_basename)
if file is not None:
file_api = FilesApi(self.connection)
file_api.destroy(file)
else:
folder = parent_folder.find_sub_folder(r_basename)
if folder is None:
raise IllegalArgumentException(f"Cannot remove `{r_path}`: No such file or folder.")
if not is_recursive:
raise IllegalArgumentException(f"Cannot remove `{r_path}`: Is a folder.")
folder_api = FoldersApi(self.connection)
folder_api.destroy(folder.id, True)
def ls(self, remote_path: str, password: str | None = None) -> tuple[Folder, list[File]]:
folder, laboratory = self.resolve_folder(remote_path, password)
files = self.find_files(folder.id)
return folder, files
def cp(self, src_path: str, dest_path: str, is_recursive: bool = False) -> None:
s_remote, s_laboratory_name, s_path = self.parse_remote_host_with_path(src_path)
d_remote, d_laboratory_name, d_path = self.parse_remote_host_with_path(dest_path)
if s_remote != d_remote:
raise IllegalArgumentException("Remote host mismatched.")
if s_laboratory_name != d_laboratory_name:
raise IllegalArgumentException("Laboratory mismatched.")
s_path = s_path.rstrip("/")
s_dirname = os.path.dirname(s_path)
s_basename = os.path.basename(s_path)
if d_path.endswith("/"):
d_dirname = d_path
d_basename = s_basename
else:
d_dirname = os.path.dirname(d_path)
d_basename = os.path.basename(d_path)
laboratory = self.find_laboratory(s_laboratory_name)
s_parent_folder = self.find_folder(laboratory, s_dirname)
s_parent_files = self.find_files(s_parent_folder.id)
d_parent_folder = self.find_folder(laboratory, d_dirname)
d_parent_files = self.find_files(d_parent_folder.id)
s_file = find_file(s_parent_files, s_basename)
if s_file is not None:
d_file = find_file(d_parent_files, d_basename)
if d_file is not None:
raise IllegalArgumentException(f"File `{d_basename}` already exists.")
d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
if d_sub_folder is not None:
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
file_api = FilesApi(self.connection)
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
file_api.copy(s_file, d_parent_folder.id, normalize("NFC", d_basename))
else:
s_folder = s_parent_folder.find_sub_folder(s_basename)
if s_folder is None:
raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
if not is_recursive:
raise IllegalArgumentException(f"Cannot copy `{s_path}`: Is a folder.")
if find_file(d_parent_files, d_basename) is not None:
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
d_folder = d_parent_folder.find_sub_folder(d_basename)
if d_folder is not None:
if d_folder.id == s_folder.id:
raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
folder_api = FoldersApi(self.connection)
if s_parent_folder.id != d_parent_folder.id or s_basename != d_basename:
folder_api.copy(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
def mv(self, src_path: str, dest_path: str) -> None:
s_remote, s_laboratory_name, s_path = self.parse_remote_host_with_path(src_path)
d_remote, d_laboratory_name, d_path = self.parse_remote_host_with_path(dest_path)
if s_remote != d_remote:
raise IllegalArgumentException("Remote host mismatched.")
if s_laboratory_name != d_laboratory_name:
raise IllegalArgumentException("Laboratory mismatched.")
s_path = s_path.rstrip("/")
s_dirname = os.path.dirname(s_path)
s_basename = os.path.basename(s_path)
if d_path.endswith("/"):
d_dirname = d_path
d_basename = s_basename
else:
d_dirname = os.path.dirname(d_path)
d_basename = os.path.basename(d_path)
laboratory = self.find_laboratory(s_laboratory_name)
s_parent_folder = self.find_folder(laboratory, s_dirname)
s_parent_files = self.find_files(s_parent_folder.id)
d_parent_folder = self.find_folder(laboratory, d_dirname)
d_parent_files = self.find_files(d_parent_folder.id)
s_file = find_file(s_parent_files, s_basename)
if s_file is not None:
d_file = find_file(d_parent_files, d_basename)
if d_file is not None:
raise IllegalArgumentException(f"File `{d_basename}` already exists.")
d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
if d_sub_folder is not None:
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
file_api = FilesApi(self.connection)
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
file_api.move(s_file, d_parent_folder.id, normalize("NFC", d_basename))
else:
s_folder = s_parent_folder.find_sub_folder(s_basename)
if s_folder is None:
raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
if find_file(d_parent_files, d_basename) is not None:
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
d_folder = d_parent_folder.find_sub_folder(d_basename)
if d_folder is not None:
if d_folder.id == s_folder.id:
raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
folder_api = FoldersApi(self.connection)
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
folder_api.move(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
def chacl(
self, remote_path: str, access_level: int, is_recursive: bool = False, password: str | None = None
) -> None:
remote, laboratory_name, r_path = self.parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
laboratory = self.find_laboratory(laboratory_name)
folder = self.find_folder(laboratory, r_path)
folder_api = FoldersApi(self.connection)
folder_api.acl(folder.id, access_level, is_recursive, password)
def metadata(self, remote_path: str, password: str | None = None) -> dict:
folder, laboratory = self.resolve_folder(remote_path, password)
folder_api = FoldersApi(self.connection)
return folder_api.metadata(folder.id)
def file_metadata(self, remote_path: str, password: str | None = None) -> dict:
folder, laboratory, r_basename = self.resolve_file(remote_path, password)
files = self.find_files(folder.id)
file = find_file(files, r_basename)
if file is None:
raise IllegalArgumentException(f"File `{r_basename}` not found.")
file_api = FilesApi(self.connection)
return file_api.metadata(file)
def upload(
self, local_path: str, remote_path: str, is_recursive: bool = False, is_skip_if_exists: bool = False
) -> None:
from mdrsclient.transfer import Uploader
uploader = Uploader(self)
uploader.upload(local_path, remote_path, is_recursive, is_skip_if_exists)
def download(
self,
remote_path: str,
local_path: str,
is_recursive: bool = False,
is_skip_if_exists: bool = False,
password: str | None = None,
excludes: list[str] | None = None,
) -> None:
from mdrsclient.transfer import Downloader
downloader = Downloader(self)
downloader.download(remote_path, local_path, is_recursive, is_skip_if_exists, password, excludes)
def version(self) -> str:
from mdrsclient.__version__ import __version__
return f"mdrs {__version__}"
def config_create(self, remote: str, url: str) -> None:
remote = self.parse_remote_host(remote)
config = self.config_class(remote)
if config.url is not None:
raise IllegalArgumentException(f"Remote host `{remote}` is already exists.")
else:
config.url = url
def config_update(self, remote: str, url: str) -> None:
remote = self.parse_remote_host(remote)
config = self.config_class(remote)
if config.url is None:
raise IllegalArgumentException(f"Remote host `{remote}` is not exists.")
else:
config.url = url
def config_list(self) -> list:
config = self.config_class("")
return config.list()
def config_delete(self, remote: str) -> None:
remote = self.parse_remote_host(remote)
config = self.config_class(remote)
if config.url is None:
raise IllegalArgumentException(f"Remote host `{remote}` is not exists.")
else:
del config.url
+2
View File
@@ -12,6 +12,7 @@ from mdrsclient.commands.mkdir import MkdirCommand
from mdrsclient.commands.mv import MvCommand from mdrsclient.commands.mv import MvCommand
from mdrsclient.commands.rm import RmCommand from mdrsclient.commands.rm import RmCommand
from mdrsclient.commands.upload import UploadCommand from mdrsclient.commands.upload import UploadCommand
from mdrsclient.commands.version import VersionCommand
from mdrsclient.commands.whoami import WhoamiCommand from mdrsclient.commands.whoami import WhoamiCommand
__all__ = [ __all__ = [
@@ -29,5 +30,6 @@ __all__ = [
"MvCommand", "MvCommand",
"RmCommand", "RmCommand",
"UploadCommand", "UploadCommand",
"VersionCommand",
"WhoamiCommand", "WhoamiCommand",
] ]
+1 -73
View File
@@ -1,18 +1,7 @@
import re
from abc import ABC, abstractmethod from abc import ABC, abstractmethod
from typing import Any from typing import Any
from unicodedata import normalize
from mdrsclient.api import FoldersApi, LaboratoriesApi from mdrsclient.exceptions import UnexpectedException
from mdrsclient.config import ConfigFile
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import (
IllegalArgumentException,
MissingConfigurationException,
UnauthorizedException,
UnexpectedException,
)
from mdrsclient.models import Folder, Laboratory
class BaseCommand(ABC): class BaseCommand(ABC):
@@ -20,64 +9,3 @@ class BaseCommand(ABC):
@abstractmethod @abstractmethod
def register(cls, parsers: Any) -> None: def register(cls, parsers: Any) -> None:
raise UnexpectedException("Not implemented.") raise UnexpectedException("Not implemented.")
@classmethod
def _create_connection(cls, remote: str) -> MDRSConnection:
config = ConfigFile(remote)
if config.url is None:
raise MissingConfigurationException(f"Remote host `{remote}` is not found.")
return MDRSConnection(config.remote, config.url)
@classmethod
def _find_laboratory(cls, connection: MDRSConnection, name: str) -> Laboratory:
if connection.laboratories.empty() or connection.token is not None and connection.token.is_expired:
laboratory_api = LaboratoriesApi(connection)
connection.laboratories = laboratory_api.list()
laboratory = connection.laboratories.find_by_name(name)
if laboratory is None:
raise IllegalArgumentException(f"Laboratory `{name}` not found.")
return laboratory
@classmethod
def _find_folder(
cls, connection: MDRSConnection, laboratory: Laboratory, path: str, password: str | None = None
) -> Folder:
folder_api = FoldersApi(connection)
folders = folder_api.list(laboratory.id, normalize("NFC", path))
if len(folders) != 1:
raise UnexpectedException(f"Folder `{path}` not found.")
if folders[0].lock:
if password is None:
raise UnauthorizedException(f"Folder `{path}` is locked.")
folder_api.auth(folders[0].id, password)
return folder_api.retrieve(folders[0].id)
@classmethod
def _parse_remote_host(cls, path: str) -> str:
path_array = path.split(":")
remote_host = path_array[0]
if len(path_array) == 2 and path_array[1] != "" or len(path_array) > 2:
raise IllegalArgumentException("Invalid remote host")
return remote_host
@classmethod
def _parse_remote_host_with_path(cls, path: str) -> tuple[str, str, str]:
path = re.sub(r"//+|/\./+|/\.$", "/", path)
if re.search(r"/\.\./|/\.\.$", path) is not None:
raise IllegalArgumentException("Path traversal found.")
path_array = path.split(":")
if len(path_array) != 2:
raise IllegalArgumentException("Invalid remote host.")
remote_host = path_array[0]
folder_array = path_array[1].split("/")
is_absolute_path = folder_array[0] == ""
if not is_absolute_path:
raise IllegalArgumentException("Must be absolute paths.")
del folder_array[0]
if len(folder_array) == 0:
laboratory = ""
folder = ""
else:
laboratory = folder_array.pop(0)
folder = "/" + "/".join(folder_array)
return (remote_host, laboratory, folder)
+5 -7
View File
@@ -31,10 +31,8 @@ class ChaclCommand(BaseCommand):
@classmethod @classmethod
def chacl(cls, remote_path: str, access_level: int, is_recursive: bool, password: str | None) -> None: def chacl(cls, remote_path: str, access_level: int, is_recursive: bool, password: str | None) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path) remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
r_path = r_path.rstrip("/") from mdrsclient.client import MdrsClient
connection = cls._create_connection(remote)
laboratory = cls._find_laboratory(connection, laboratory_name) client = MdrsClient.from_remote(remote)
folder = cls._find_folder(connection, laboratory, r_path) client.chacl(remote_path, access_level, is_recursive, password)
folder_api = FoldersApi(connection)
folder_api.acl(folder.id, access_level, is_recursive, password)
+16 -44
View File
@@ -2,8 +2,6 @@ from argparse import Namespace
from typing import Any, Callable from typing import Any, Callable
from mdrsclient.commands.base import BaseCommand from mdrsclient.commands.base import BaseCommand
from mdrsclient.config import ConfigFile
from mdrsclient.exceptions import IllegalArgumentException
class ConfigCommand(BaseCommand): class ConfigCommand(BaseCommand):
@@ -26,10 +24,11 @@ class ConfigCommand(BaseCommand):
update_parser.set_defaults(func=cls.func_update) update_parser.set_defaults(func=cls.func_update)
# config list # config list
list_parser = config_parsers.add_parser("list", help="list all the remote hosts", aliases=["ls"]) list_parser = config_parsers.add_parser("list", help="list all the remote hosts", aliases=["ls"])
list_parser.add_argument("-l", "--long", help="show the api url", action="store_true")
list_parser.set_defaults(func=cls.func_list) list_parser.set_defaults(func=cls.func_list)
# config delete # config delete
delete_parser = config_parsers.add_parser("delete", help="delete an existing remote host", aliases=["remove"]) delete_parser = config_parsers.add_parser(
"delete", help="delete an existing remote host", aliases=["remove", "rm"]
)
delete_parser.add_argument("remote", help="label of remote host") delete_parser.add_argument("remote", help="label of remote host")
delete_parser.set_defaults(func=cls.func_delete) delete_parser.set_defaults(func=cls.func_delete)
@@ -37,56 +36,29 @@ class ConfigCommand(BaseCommand):
def func_create(cls, args: Namespace) -> None: def func_create(cls, args: Namespace) -> None:
remote = str(args.remote) remote = str(args.remote)
url = str(args.url) url = str(args.url)
cls.create(remote, url) from mdrsclient.client import MdrsClient
MdrsClient(None).config_create(remote, url)
@classmethod @classmethod
def func_update(cls, args: Namespace) -> None: def func_update(cls, args: Namespace) -> None:
remote = str(args.remote) remote = str(args.remote)
url = str(args.url) url = str(args.url)
cls.update(remote, url) from mdrsclient.client import MdrsClient
MdrsClient(None).config_update(remote, url)
@classmethod @classmethod
def func_list(cls, args: Namespace) -> None: def func_list(cls, args: Namespace) -> None:
is_long = bool(args.long) from mdrsclient.client import MdrsClient
cls.list(is_long)
client = MdrsClient(None)
for remote, url in client.config_list():
print(f"{remote}:\t{url}")
@classmethod @classmethod
def func_delete(cls, args: Namespace) -> None: def func_delete(cls, args: Namespace) -> None:
remote = str(args.remote) remote = str(args.remote)
cls.delete(remote) from mdrsclient.client import MdrsClient
@classmethod MdrsClient(None).config_delete(remote)
def create(cls, remote: str, url: str) -> None:
remote = cls._parse_remote_host(remote)
config = ConfigFile(remote)
if config.url is not None:
raise IllegalArgumentException(f"Remote host `{remote}` is already exists.")
else:
config.url = url
@classmethod
def update(cls, remote: str, url: str) -> None:
remote = cls._parse_remote_host(remote)
config = ConfigFile(remote)
if config.url is None:
raise IllegalArgumentException(f"Remote host `{remote}` is not exists.")
else:
config.url = url
@classmethod
def list(cls, is_long: bool) -> None:
config = ConfigFile("")
for remote, url in config.list():
line = f"{remote}:"
if is_long:
line += f"\t{url}"
print(line)
@classmethod
def delete(cls, remote: str) -> None:
remote = cls._parse_remote_host(remote)
config = ConfigFile(remote)
if config.url is None:
raise IllegalArgumentException(f"Remote host `{remote}` is not exists.")
else:
del config.url
+6 -48
View File
@@ -6,6 +6,7 @@ from unicodedata import normalize
from mdrsclient.api import FilesApi, FoldersApi from mdrsclient.api import FilesApi, FoldersApi
from mdrsclient.commands.base import BaseCommand from mdrsclient.commands.base import BaseCommand
from mdrsclient.exceptions import IllegalArgumentException from mdrsclient.exceptions import IllegalArgumentException
from mdrsclient.models.file import find_file
class CpCommand(BaseCommand): class CpCommand(BaseCommand):
@@ -28,51 +29,8 @@ class CpCommand(BaseCommand):
@classmethod @classmethod
def cp(cls, src_path: str, dest_path: str, is_recursive: bool) -> None: def cp(cls, src_path: str, dest_path: str, is_recursive: bool) -> None:
(s_remote, s_laboratory_name, s_path) = cls._parse_remote_host_with_path(src_path) remote = src_path.split(":", 1)[0] if ":" in src_path else ""
(d_remote, d_laboratory_name, d_path) = cls._parse_remote_host_with_path(dest_path) from mdrsclient.client import MdrsClient
if s_remote != d_remote:
raise IllegalArgumentException("Remote host mismatched.") client = MdrsClient.from_remote(remote)
if s_laboratory_name != d_laboratory_name: client.cp(src_path, dest_path, is_recursive)
raise IllegalArgumentException("Laboratory mismatched.")
s_path = s_path.rstrip("/")
s_dirname = os.path.dirname(s_path)
s_basename = os.path.basename(s_path)
if d_path.endswith("/"):
d_dirname = d_path
d_basename = s_basename
else:
d_dirname = os.path.dirname(d_path)
d_basename = os.path.basename(d_path)
connection = cls._create_connection(s_remote)
laboratory = cls._find_laboratory(connection, s_laboratory_name)
s_parent_folder = cls._find_folder(connection, laboratory, s_dirname)
d_parent_folder = cls._find_folder(connection, laboratory, d_dirname)
s_file = s_parent_folder.find_file(s_basename)
if s_file is not None:
# source is file
d_file = d_parent_folder.find_file(d_basename)
if d_file is not None:
raise IllegalArgumentException(f"File `{d_basename}` already exists.")
d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
if d_sub_folder is not None:
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
file_api = FilesApi(connection)
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
file_api.copy(s_file, d_parent_folder.id, normalize("NFC", d_basename))
else:
s_folder = s_parent_folder.find_sub_folder(s_basename)
if s_folder is None:
raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
# source is folder
if not is_recursive:
raise IllegalArgumentException(f"Cannot copy `{s_path}`: Is a folder.")
if d_parent_folder.find_file(d_basename) is not None:
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
d_folder = d_parent_folder.find_sub_folder(d_basename)
if d_folder is not None:
if d_folder.id == s_folder.id:
raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
folder_api = FoldersApi(connection)
if s_parent_folder.id != d_parent_folder.id or s_basename != d_basename:
folder_api.copy(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
+18 -102
View File
@@ -1,28 +1,7 @@
import os
from argparse import Namespace from argparse import Namespace
from concurrent.futures import ThreadPoolExecutor
from typing import Any from typing import Any
from pydantic.dataclasses import dataclass
from mdrsclient.api import FilesApi, FoldersApi
from mdrsclient.commands.base import BaseCommand from mdrsclient.commands.base import BaseCommand
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import IllegalArgumentException, UnexpectedException
from mdrsclient.models import File, Folder, Laboratory
from mdrsclient.settings import CONCURRENT
@dataclass(frozen=True)
class DownloadFileInfo:
file: File
path: str
@dataclass
class DownloadContext:
hasError: bool
files: list[DownloadFileInfo]
class DownloadCommand(BaseCommand): class DownloadCommand(BaseCommand):
@@ -32,6 +11,12 @@ class DownloadCommand(BaseCommand):
download_parser.add_argument( download_parser.add_argument(
"-r", "--recursive", help="download folders and their contents recursive", action="store_true" "-r", "--recursive", help="download folders and their contents recursive", action="store_true"
) )
download_parser.add_argument(
"-s",
"--skip-if-exists",
help="skip the download if file is already uploaded and file size is the same",
action="store_true",
)
download_parser.add_argument( download_parser.add_argument(
"-e", "--exclude", help="exclude to download path matched file or folders", action="append" "-e", "--exclude", help="exclude to download path matched file or folders", action="append"
) )
@@ -45,93 +30,24 @@ class DownloadCommand(BaseCommand):
remote_path = str(args.remote_path) remote_path = str(args.remote_path)
local_path = str(args.local_path) local_path = str(args.local_path)
is_recursive = bool(args.recursive) is_recursive = bool(args.recursive)
is_skip_if_exists = bool(args.skip_if_exists)
password = str(args.password) if args.password else None password = str(args.password) if args.password else None
excludes = list(map(lambda x: str(x).rstrip("/").lower(), args.exclude)) if args.exclude is not None else [] excludes = list(map(lambda x: str(x).rstrip("/").lower(), args.exclude)) if args.exclude is not None else []
cls.download(remote_path, local_path, is_recursive, password, excludes) cls.download(remote_path, local_path, is_recursive, is_skip_if_exists, password, excludes)
@classmethod @classmethod
def download( def download(
cls, remote_path: str, local_path: str, is_recursive: bool, password: str | None, excludes: list[str]
) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
r_dirname = os.path.dirname(r_path)
r_basename = os.path.basename(r_path)
connection = cls._create_connection(remote)
l_dirname = os.path.realpath(local_path)
if not os.path.isdir(l_dirname):
raise IllegalArgumentException(f"Local directory `{local_path}` not found.")
laboratory = cls._find_laboratory(connection, laboratory_name)
r_parent_folder = cls._find_folder(connection, laboratory, r_dirname, password)
file = r_parent_folder.find_file(r_basename)
if file is not None:
if cls.__check_excludes(excludes, laboratory, r_parent_folder, file):
return
context = DownloadContext(False, [])
l_path = os.path.join(l_dirname, r_basename)
context.files.append(DownloadFileInfo(file, l_path))
cls.__multiple_download(connection, context)
else:
folder = r_parent_folder.find_sub_folder(r_basename)
if folder is None:
raise IllegalArgumentException(f"File or folder `{r_path}` not found.")
if not is_recursive:
raise IllegalArgumentException(f"Cannot download `{r_path}`: Is a folder.")
folder_api = FoldersApi(connection)
cls.__multiple_download_pickup_recursive_files(
connection, folder_api, laboratory, folder.id, l_dirname, excludes
)
@classmethod
def __multiple_download_pickup_recursive_files(
cls, cls,
connection: MDRSConnection, remote_path: str,
folder_api: FoldersApi, local_path: str,
laboratory: Laboratory, is_recursive: bool,
folder_id: str, is_skip_if_exists: bool,
basedir: str, password: str | None,
excludes: list[str], excludes: list[str],
) -> None: ) -> None:
context = DownloadContext(False, []) remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
folder = folder_api.retrieve(folder_id) from mdrsclient.client import MdrsClient
dirname = os.path.join(basedir, folder.name)
if cls.__check_excludes(excludes, laboratory, folder, None):
return
if not os.path.exists(dirname):
os.makedirs(dirname)
print(dirname)
for file in folder.files:
if cls.__check_excludes(excludes, laboratory, folder, file):
continue
path = os.path.join(dirname, file.name)
context.files.append(DownloadFileInfo(file, path))
cls.__multiple_download(connection, context)
if context.hasError:
raise UnexpectedException("Some files failed to download.")
for sub_folder in folder.sub_folders:
cls.__multiple_download_pickup_recursive_files(
connection, folder_api, laboratory, sub_folder.id, dirname, excludes
)
@classmethod client = MdrsClient.from_remote(remote)
def __multiple_download(cls, connection: MDRSConnection, context: DownloadContext) -> None: client.download(remote_path, local_path, is_recursive, is_skip_if_exists, password, excludes)
file_api = FilesApi(connection) return
with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
results = pool.map(lambda x: cls.__multiple_download_worker(file_api, x), context.files)
hasError = next(filter(lambda x: x is False, results), None)
if hasError is not None:
context.hasError = True
@classmethod
def __multiple_download_worker(cls, file_api: FilesApi, info: DownloadFileInfo) -> bool:
try:
file_api.download(info.file, info.path)
except Exception:
return False
print(info.path)
return True
@classmethod
def __check_excludes(cls, excludes: list[str], laboratory: Laboratory, folder: Folder, file: File | None) -> bool:
path = f"/{laboratory.name}{folder.path}{file.name if file is not None else ""}".rstrip("/").lower()
return path in excludes
+6 -12
View File
@@ -6,6 +6,7 @@ from typing import Any
from mdrsclient.api import FilesApi from mdrsclient.api import FilesApi
from mdrsclient.commands.base import BaseCommand from mdrsclient.commands.base import BaseCommand
from mdrsclient.exceptions import IllegalArgumentException from mdrsclient.exceptions import IllegalArgumentException
from mdrsclient.models.file import find_file
class FileMetadataCommand(BaseCommand): class FileMetadataCommand(BaseCommand):
@@ -24,16 +25,9 @@ class FileMetadataCommand(BaseCommand):
@classmethod @classmethod
def file_metadata(cls, remote_path: str, password: str | None) -> None: def file_metadata(cls, remote_path: str, password: str | None) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path) remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
r_path = r_path.rstrip("/") from mdrsclient.client import MdrsClient
r_dirname = os.path.dirname(r_path)
r_basename = os.path.basename(r_path) client = MdrsClient.from_remote(remote)
connection = cls._create_connection(remote) metadata = client.file_metadata(remote_path, password)
laboratory = cls._find_laboratory(connection, laboratory_name)
folder = cls._find_folder(connection, laboratory, r_dirname, password)
file = folder.find_file(r_basename)
if file is None:
raise IllegalArgumentException(f"File `{r_basename}` not found.")
file_api = FilesApi(connection)
metadata = file_api.metadata(file)
print(json.dumps(metadata, ensure_ascii=False)) print(json.dumps(metadata, ensure_ascii=False))
+5 -7
View File
@@ -19,11 +19,11 @@ class LabsCommand(BaseCommand):
@classmethod @classmethod
def labs(cls, remote: str) -> None: def labs(cls, remote: str) -> None:
remote = cls._parse_remote_host(remote) from mdrsclient.client import MdrsClient
connection = cls._create_connection(remote)
laboratory_api = LaboratoriesApi(connection) remote_host = MdrsClient.parse_remote_host(remote)
laboratories = laboratory_api.list() client = MdrsClient.from_remote(remote_host)
connection.laboratories = laboratories laboratories = client.get_laboratories()
label = {"id": "ID", "name": "Name", "pi_name": "PI", "full_name": "Laboratory"} label = {"id": "ID", "name": "Name", "pi_name": "PI", "full_name": "Laboratory"}
length: dict[str, int] = {} length: dict[str, int] = {}
for key in label.keys(): for key in label.keys():
@@ -34,7 +34,6 @@ class LabsCommand(BaseCommand):
length["pi_name"] = max(length["pi_name"], len(laboratory.pi_name)) length["pi_name"] = max(length["pi_name"], len(laboratory.pi_name))
length["full_name"] = max(length["full_name"], len(laboratory.full_name)) length["full_name"] = max(length["full_name"], len(laboratory.full_name))
header = ( header = (
# f"{label['id']:{length['id']}}\t{label['name']:{length['name']}}\t"
f"{label['name']:{length['name']}}\t" f"{label['name']:{length['name']}}\t"
f"{label['pi_name']:{length['pi_name']}}\t{label['full_name']:{length['full_name']}}" f"{label['pi_name']:{length['pi_name']}}\t{label['full_name']:{length['full_name']}}"
) )
@@ -42,7 +41,6 @@ class LabsCommand(BaseCommand):
print("-" * len(header.expandtabs())) print("-" * len(header.expandtabs()))
for laboratory in laboratories: for laboratory in laboratories:
print( print(
# f"{laboratory.id:{length['id']}}\t{laboratory.name:{length['name']}}\t"
f"{laboratory.name:{length['name']}}\t" f"{laboratory.name:{length['name']}}\t"
f"{laboratory.pi_name:{length['pi_name']}}\t{laboratory.full_name:{length['full_name']}}" f"{laboratory.pi_name:{length['pi_name']}}\t{laboratory.full_name:{length['full_name']}}"
) )
+6 -15
View File
@@ -2,11 +2,7 @@ import getpass
from argparse import Namespace from argparse import Namespace
from typing import Any from typing import Any
from mdrsclient.api import UsersApi
from mdrsclient.commands.base import BaseCommand from mdrsclient.commands.base import BaseCommand
from mdrsclient.config import ConfigFile
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import MissingConfigurationException
class LoginCommand(BaseCommand): class LoginCommand(BaseCommand):
@@ -21,20 +17,15 @@ class LoginCommand(BaseCommand):
@classmethod @classmethod
def func(cls, args: Namespace) -> None: def func(cls, args: Namespace) -> None:
remote = str(args.remote) remote = str(args.remote)
username = str(args.username) if args.password else input("Username: ").strip() username = str(args.username) if args.username else input("Username: ").strip()
password = str(args.password) if args.password else getpass.getpass("Password: ").strip() password = str(args.password) if args.password else getpass.getpass("Password: ").strip()
cls.login(remote, username, password) cls.login(remote, username, password)
@classmethod @classmethod
def login(cls, remote: str, username: str, password: str) -> None: def login(cls, remote: str, username: str, password: str) -> None:
remote = cls._parse_remote_host(remote) from mdrsclient.client import MdrsClient
config = ConfigFile(remote)
if config.url is None: remote_host = MdrsClient.parse_remote_host(remote)
raise MissingConfigurationException(f"Remote host `{remote}` is not found.") client = MdrsClient.from_remote(remote_host)
connection = MDRSConnection(config.remote, config.url) client.login(username, password)
user_api = UsersApi(connection)
token = user_api.token(username, password)
connection.token = token
user = user_api.current()
connection.user = user
print("Login Successful") print("Login Successful")
+5 -9
View File
@@ -2,9 +2,6 @@ from argparse import Namespace
from typing import Any from typing import Any
from mdrsclient.commands.base import BaseCommand from mdrsclient.commands.base import BaseCommand
from mdrsclient.config import ConfigFile
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import MissingConfigurationException
class LogoutCommand(BaseCommand): class LogoutCommand(BaseCommand):
@@ -21,9 +18,8 @@ class LogoutCommand(BaseCommand):
@classmethod @classmethod
def logout(cls, remote: str) -> None: def logout(cls, remote: str) -> None:
remote = cls._parse_remote_host(remote) from mdrsclient.client import MdrsClient
config = ConfigFile(remote)
if config.url is None: remote_host = MdrsClient.parse_remote_host(remote)
raise MissingConfigurationException(f"Remote host `{remote}` is not found.") client = MdrsClient.from_remote(remote_host)
connection = MDRSConnection(config.remote, config.url) client.logout()
connection.logout()
+56 -34
View File
@@ -4,9 +4,9 @@ from typing import Any
from pydantic.dataclasses import dataclass from pydantic.dataclasses import dataclass
from mdrsclient.api import FoldersApi from mdrsclient.api import FilesApi, FoldersApi
from mdrsclient.client import MdrsClient
from mdrsclient.commands.base import BaseCommand from mdrsclient.commands.base import BaseCommand
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import UnauthorizedException from mdrsclient.exceptions import UnauthorizedException
from mdrsclient.models import File, Folder, FolderSimple, Laboratory from mdrsclient.models import File, Folder, FolderSimple, Laboratory
@@ -19,11 +19,11 @@ class Config:
@dataclass(config=Config) @dataclass(config=Config)
class LsCommandContext: class LsCommandContext:
prefix: str prefix: str
connection: MDRSConnection client: MdrsClient
laboratory: Laboratory laboratory: Laboratory
password: str password: str
is_json: bool is_json: bool
is_quick: bool is_quiet: bool
is_recursive: bool is_recursive: bool
@@ -35,7 +35,7 @@ class LsCommand(BaseCommand):
ls_parser.add_argument("-J", "--json", help="turn on json output", action="store_true") ls_parser.add_argument("-J", "--json", help="turn on json output", action="store_true")
ls_parser.add_argument( ls_parser.add_argument(
"-q", "-q",
"--quick", "--quiet",
help="don't output header row. this option is forced if the -r option is specified", help="don't output header row. this option is forced if the -r option is specified",
action="store_true", action="store_true",
) )
@@ -49,35 +49,53 @@ class LsCommand(BaseCommand):
password = str(args.password) if args.password else None password = str(args.password) if args.password else None
is_json = bool(args.json) is_json = bool(args.json)
is_recursive = bool(args.recursive) is_recursive = bool(args.recursive)
is_quick = bool(args.quick) if not is_recursive else True is_quiet = bool(args.quiet) if not is_recursive else True
cls.ls(remote_path, password, is_json, is_recursive, is_quick) cls.ls(remote_path, password, is_json, is_recursive, is_quiet)
@classmethod @classmethod
def ls(cls, remote_path: str, password: str | None, is_json: bool, is_recursive: bool, is_quick: bool) -> None: def ls(cls, remote_path: str, password: str | None, is_json: bool, is_recursive: bool, is_quiet: bool) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path) remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
connection = cls._create_connection(remote) from mdrsclient.client import MdrsClient
laboratory = cls._find_laboratory(connection, laboratory_name)
client = MdrsClient.from_remote(remote)
cls._ls_logic(client, remote_path, password, is_json, is_recursive, is_quiet)
return
@classmethod
def _ls_logic(
cls,
client: MdrsClient,
remote_path: str,
password: str | None,
is_json: bool,
is_recursive: bool,
is_quiet: bool,
) -> None:
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
folder, laboratory = client.resolve_folder(remote_path, password)
laboratory_name = laboratory.name
files = client.find_files(folder.id)
context = LsCommandContext( context = LsCommandContext(
f"{remote}:/{laboratory_name}", f"{remote}:/{laboratory_name}",
connection, client,
laboratory, laboratory,
password if password is not None else "", password if password is not None else "",
is_json, is_json,
is_quick, is_quiet,
is_recursive, is_recursive,
) )
folder = cls._find_folder(connection, laboratory, r_path, password)
if context.is_json: if context.is_json:
cls._ls_json(context, folder) cls._ls_json(context, folder, files)
else: else:
cls._ls_plain(context, folder) cls._ls_plain(context, folder, files)
@classmethod @classmethod
def _ls_json(cls, context: LsCommandContext, folder: Folder) -> None: def _ls_json(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
print(json.dumps(cls._folder2dict(context, folder), ensure_ascii=False)) print(json.dumps(cls._folder2dict(context, folder, files), ensure_ascii=False))
@classmethod @classmethod
def _ls_plain(cls, context: LsCommandContext, folder: Folder) -> None: def _ls_plain(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
label = { label = {
"type": "Type", "type": "Type",
"acl": "Access", "acl": "Access",
@@ -88,16 +106,16 @@ class LsCommand(BaseCommand):
} }
length: dict[str, int] = {} length: dict[str, int] = {}
for key in label.keys(): for key in label.keys():
length[key] = len(label[key]) if not context.is_quick else 0 length[key] = len(label[key]) if not context.is_quiet else 0
for sub_folder in folder.sub_folders: for sub_folder in folder.sub_folders:
sub_laboratory = context.connection.laboratories.find_by_id(sub_folder.laboratory_id) sub_laboratory = context.client.connection.laboratories.find_by_id(sub_folder.laboratory_id)
sub_laboratory_name = sub_laboratory.name if sub_laboratory is not None else "(invalid)" sub_laboratory_name = sub_laboratory.name if sub_laboratory is not None else "(invalid)"
length["acl"] = max(length["acl"], len(sub_folder.access_level_name)) length["acl"] = max(length["acl"], len(sub_folder.access_level_name))
length["laboratory"] = max(length["laboratory"], len(sub_laboratory_name)) length["laboratory"] = max(length["laboratory"], len(sub_laboratory_name))
length["size"] = max(length["size"], len(str(folder.size))) length["size"] = max(length["size"], len(str(folder.size)))
length["date"] = max(length["date"], len(sub_folder.updated_at_name)) length["date"] = max(length["date"], len(sub_folder.updated_at_name))
length["name"] = max(length["name"], len(sub_folder.name)) length["name"] = max(length["name"], len(sub_folder.name))
for file in folder.files: for file in files:
length["size"] = max(length["size"], len(str(file.size))) length["size"] = max(length["size"], len(str(file.size)))
length["date"] = max(length["date"], len(file.updated_at_name)) length["date"] = max(length["date"], len(file.updated_at_name))
length["name"] = max(length["name"], len(file.name)) length["name"] = max(length["name"], len(file.name))
@@ -111,9 +129,9 @@ class LsCommand(BaseCommand):
if context.is_recursive: if context.is_recursive:
print(f"{context.prefix}{folder.path}:") print(f"{context.prefix}{folder.path}:")
print(f"total {sum(f.size for f in folder.files)}") print(f"total {sum(f.size for f in files)}")
if not context.is_quick: if not context.is_quiet:
print(header) print(header)
print("-" * len(header.expandtabs())) print("-" * len(header.expandtabs()))
@@ -125,7 +143,7 @@ class LsCommand(BaseCommand):
f"{sub_laboratory_name:{length['laboratory']}}\t{sub_folder.size:{length['size']}}\t" f"{sub_laboratory_name:{length['laboratory']}}\t{sub_folder.size:{length['size']}}\t"
f"{sub_folder.updated_at_name:{length['date']}}\t{sub_folder.name:{length['name']}}" f"{sub_folder.updated_at_name:{length['date']}}\t{sub_folder.name:{length['name']}}"
) )
for file in sorted(folder.files, key=lambda x: x.name): for file in sorted(files, key=lambda x: x.name):
print( print(
f"{'[f]':{length['type']}}\t{folder.access_level_name:{length['acl']}}\t" f"{'[f]':{length['type']}}\t{folder.access_level_name:{length['acl']}}\t"
f"{context.laboratory.name:{length['laboratory']}}\t{file.size:{length['size']}}\t" f"{context.laboratory.name:{length['laboratory']}}\t{file.size:{length['size']}}\t"
@@ -135,17 +153,20 @@ class LsCommand(BaseCommand):
if context.is_recursive: if context.is_recursive:
print("") print("")
for sub_folder in sorted(folder.sub_folders, key=lambda x: x.name): for sub_folder in sorted(folder.sub_folders, key=lambda x: x.name):
folder_api = FoldersApi(context.connection) folder_api = FoldersApi(context.client.connection)
try: try:
if sub_folder.lock: if sub_folder.lock:
folder_api.auth(sub_folder.id, context.password) folder_api.auth(sub_folder.id, context.password)
folder = folder_api.retrieve(sub_folder.id) folder = folder_api.retrieve(sub_folder.id)
cls._ls_plain(context, folder) files = context.client.find_files(sub_folder.id)
cls._ls_plain(context, folder, files)
except UnauthorizedException: except UnauthorizedException:
pass pass
@classmethod @classmethod
def _folder2dict(cls, context: LsCommandContext, folder: Folder | FolderSimple) -> dict[str, Any]: def _folder2dict(
cls, context: LsCommandContext, folder: Folder | FolderSimple, files: list[File]
) -> dict[str, Any]:
data: dict[str, Any] = { data: dict[str, Any] = {
"id": folder.id, "id": folder.id,
"pid": folder.pid, "pid": folder.pid,
@@ -159,7 +180,7 @@ class LsCommand(BaseCommand):
"updated_at": folder.updated_at, "updated_at": folder.updated_at,
} }
if isinstance(folder, Folder): if isinstance(folder, Folder):
folder_api = FoldersApi(context.connection) folder_api = FoldersApi(context.client.connection)
data["metadata"] = folder_api.metadata(folder.id) data["metadata"] = folder_api.metadata(folder.id)
if context.is_recursive: if context.is_recursive:
sub_folders: list[dict[str, Any]] = [] sub_folders: list[dict[str, Any]] = []
@@ -168,15 +189,16 @@ class LsCommand(BaseCommand):
if sub_folder.lock: if sub_folder.lock:
folder_api.auth(sub_folder.id, context.password) folder_api.auth(sub_folder.id, context.password)
folder2 = folder_api.retrieve(sub_folder.id) folder2 = folder_api.retrieve(sub_folder.id)
sub_folders.append(cls._folder2dict(context, folder2)) files2 = context.client.find_files(sub_folder.id)
sub_folders.append(cls._folder2dict(context, folder2, files2))
except UnauthorizedException: except UnauthorizedException:
pass pass
data["sub_folders"] = sub_folders data["sub_folders"] = sub_folders
else: else:
data["sub_folders"] = list( data["sub_folders"] = list(
map(lambda x: cls._folder2dict(context, x), sorted(folder.sub_folders, key=lambda x: x.name)) map(lambda x: cls._folder2dict(context, x, []), sorted(folder.sub_folders, key=lambda x: x.name))
) )
data["files"] = list(map(lambda x: cls._file2dict(context, x), sorted(folder.files, key=lambda x: x.name))) data["files"] = list(map(lambda x: cls._file2dict(context, x), sorted(files, key=lambda x: x.name)))
return data return data
@classmethod @classmethod
@@ -189,7 +211,7 @@ class LsCommand(BaseCommand):
# "thumbnail": file.thumbnail, # "thumbnail": file.thumbnail,
"description": file.description, "description": file.description,
"metadata": file.metadata, "metadata": file.metadata,
"download_url": f"{context.connection.url}/{file.download_url}", "download_url": f"{context.client.connection.url}/{file.download_url}",
"created_at": file.created_at, "created_at": file.created_at,
"updated_at": file.updated_at, "updated_at": file.updated_at,
} }
@@ -197,5 +219,5 @@ class LsCommand(BaseCommand):
@classmethod @classmethod
def _laboratory_name(cls, context: LsCommandContext, laboratory_id: int) -> str: def _laboratory_name(cls, context: LsCommandContext, laboratory_id: int) -> str:
laboratory = context.connection.laboratories.find_by_id(laboratory_id) laboratory = context.client.connection.laboratories.find_by_id(laboratory_id)
return laboratory.name if laboratory is not None else "(invalid)" return laboratory.name if laboratory is not None else "(invalid)"
+5 -6
View File
@@ -22,10 +22,9 @@ class MetadataCommand(BaseCommand):
@classmethod @classmethod
def metadata(cls, remote_path: str, password: str | None) -> None: def metadata(cls, remote_path: str, password: str | None) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path) remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
connection = cls._create_connection(remote) from mdrsclient.client import MdrsClient
laboratory = cls._find_laboratory(connection, laboratory_name)
folder = cls._find_folder(connection, laboratory, r_path, password) client = MdrsClient.from_remote(remote)
folder_api = FoldersApi(connection) metadata = client.metadata(remote_path, password)
metadata = folder_api.metadata(folder.id)
print(json.dumps(metadata, ensure_ascii=False)) print(json.dumps(metadata, ensure_ascii=False))
+6 -11
View File
@@ -6,6 +6,7 @@ from unicodedata import normalize
from mdrsclient.api import FoldersApi from mdrsclient.api import FoldersApi
from mdrsclient.commands.base import BaseCommand from mdrsclient.commands.base import BaseCommand
from mdrsclient.exceptions import IllegalArgumentException from mdrsclient.exceptions import IllegalArgumentException
from mdrsclient.models.file import find_file
class MkdirCommand(BaseCommand): class MkdirCommand(BaseCommand):
@@ -22,14 +23,8 @@ class MkdirCommand(BaseCommand):
@classmethod @classmethod
def mkdir(cls, remote_path: str) -> None: def mkdir(cls, remote_path: str) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path) remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
r_path = r_path.rstrip("/") from mdrsclient.client import MdrsClient
r_dirname = os.path.dirname(r_path)
r_basename = os.path.basename(r_path) client = MdrsClient.from_remote(remote)
connection = cls._create_connection(remote) client.mkdir(remote_path)
laboratory = cls._find_laboratory(connection, laboratory_name)
parent_folder = cls._find_folder(connection, laboratory, r_dirname)
if parent_folder.find_sub_folder(r_basename) is not None or parent_folder.find_file(r_basename) is not None:
raise IllegalArgumentException(f"Cannot create folder `{r_path}`: File exists.")
folder_api = FoldersApi(connection)
folder_api.create(normalize("NFC", r_basename), parent_folder.id)
+6 -46
View File
@@ -6,6 +6,7 @@ from unicodedata import normalize
from mdrsclient.api import FilesApi, FoldersApi from mdrsclient.api import FilesApi, FoldersApi
from mdrsclient.commands.base import BaseCommand from mdrsclient.commands.base import BaseCommand
from mdrsclient.exceptions import IllegalArgumentException from mdrsclient.exceptions import IllegalArgumentException
from mdrsclient.models.file import find_file
class MvCommand(BaseCommand): class MvCommand(BaseCommand):
@@ -24,49 +25,8 @@ class MvCommand(BaseCommand):
@classmethod @classmethod
def mv(cls, src_path: str, dest_path: str) -> None: def mv(cls, src_path: str, dest_path: str) -> None:
(s_remote, s_laboratory_name, s_path) = cls._parse_remote_host_with_path(src_path) remote = src_path.split(":", 1)[0] if ":" in src_path else ""
(d_remote, d_laboratory_name, d_path) = cls._parse_remote_host_with_path(dest_path) from mdrsclient.client import MdrsClient
if s_remote != d_remote:
raise IllegalArgumentException("Remote host mismatched.") client = MdrsClient.from_remote(remote)
if s_laboratory_name != d_laboratory_name: client.mv(src_path, dest_path)
raise IllegalArgumentException("Laboratory mismatched.")
s_path = s_path.rstrip("/")
s_dirname = os.path.dirname(s_path)
s_basename = os.path.basename(s_path)
if d_path.endswith("/"):
d_dirname = d_path
d_basename = s_basename
else:
d_dirname = os.path.dirname(d_path)
d_basename = os.path.basename(d_path)
connection = cls._create_connection(s_remote)
laboratory = cls._find_laboratory(connection, s_laboratory_name)
s_parent_folder = cls._find_folder(connection, laboratory, s_dirname)
d_parent_folder = cls._find_folder(connection, laboratory, d_dirname)
s_file = s_parent_folder.find_file(s_basename)
if s_file is not None:
# source is file
d_file = d_parent_folder.find_file(d_basename)
if d_file is not None:
raise IllegalArgumentException(f"File `{d_basename}` already exists.")
d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
if d_sub_folder is not None:
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
file_api = FilesApi(connection)
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
file_api.move(s_file, d_parent_folder.id, normalize("NFC", d_basename))
else:
s_folder = s_parent_folder.find_sub_folder(s_basename)
if s_folder is None:
raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
# source is folder
if d_parent_folder.find_file(d_basename) is not None:
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
d_folder = d_parent_folder.find_sub_folder(d_basename)
if d_folder is not None:
if d_folder.id == s_folder.id:
raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
folder_api = FoldersApi(connection)
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
folder_api.move(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
+6 -19
View File
@@ -5,6 +5,7 @@ from typing import Any
from mdrsclient.api import FilesApi, FoldersApi from mdrsclient.api import FilesApi, FoldersApi
from mdrsclient.commands.base import BaseCommand from mdrsclient.commands.base import BaseCommand
from mdrsclient.exceptions import IllegalArgumentException from mdrsclient.exceptions import IllegalArgumentException
from mdrsclient.models.file import find_file
class RmCommand(BaseCommand): class RmCommand(BaseCommand):
@@ -25,22 +26,8 @@ class RmCommand(BaseCommand):
@classmethod @classmethod
def rm(cls, remote_path: str, is_recursive: bool) -> None: def rm(cls, remote_path: str, is_recursive: bool) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path) remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
r_path = r_path.rstrip("/") from mdrsclient.client import MdrsClient
r_dirname = os.path.dirname(r_path)
r_basename = os.path.basename(r_path) client = MdrsClient.from_remote(remote)
connection = cls._create_connection(remote) client.rm(remote_path, is_recursive)
laboratory = cls._find_laboratory(connection, laboratory_name)
parent_folder = cls._find_folder(connection, laboratory, r_dirname)
file = parent_folder.find_file(r_basename)
if file is not None:
file_api = FilesApi(connection)
file_api.destroy(file)
else:
folder = parent_folder.find_sub_folder(r_basename)
if folder is None:
raise IllegalArgumentException(f"Cannot remove `{r_path}`: No such file or folder.")
if not is_recursive:
raise IllegalArgumentException(f"Cannot remove `{r_path}`: Is a folder.")
folder_api = FoldersApi(connection)
folder_api.destroy(folder.id, True)
+5 -75
View File
@@ -1,22 +1,7 @@
import os
from argparse import Namespace from argparse import Namespace
from concurrent.futures import ThreadPoolExecutor
from typing import Any from typing import Any
from pydantic.dataclasses import dataclass
from mdrsclient.api import FilesApi, FoldersApi
from mdrsclient.commands.base import BaseCommand from mdrsclient.commands.base import BaseCommand
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import IllegalArgumentException, MDRSException
from mdrsclient.models import Folder
from mdrsclient.settings import CONCURRENT
@dataclass(frozen=True)
class UploadFileInfo:
folder: Folder
path: str
class UploadCommand(BaseCommand): class UploadCommand(BaseCommand):
@@ -46,64 +31,9 @@ class UploadCommand(BaseCommand):
@classmethod @classmethod
def upload(cls, local_path: str, remote_path: str, is_recursive: bool, is_skip_if_exists: bool) -> None: def upload(cls, local_path: str, remote_path: str, is_recursive: bool, is_skip_if_exists: bool) -> None:
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path) remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
l_path = os.path.abspath(local_path) from mdrsclient.client import MdrsClient
if not os.path.exists(l_path):
raise IllegalArgumentException(f"File or directory `{local_path}` not found.")
connection = cls._create_connection(remote)
laboratory = cls._find_laboratory(connection, laboratory_name)
folder = cls._find_folder(connection, laboratory, r_path)
infos: list[UploadFileInfo] = []
if os.path.isdir(l_path):
if not is_recursive:
raise IllegalArgumentException(f"Cannot upload `{local_path}`: Is a directory.")
folder_api = FoldersApi(connection)
folder_map: dict[str, Folder] = {}
folder_map[r_path] = folder
l_basename = os.path.basename(l_path)
for dirpath, _, filenames in os.walk(l_path, followlinks=True):
sub = l_basename if dirpath == l_path else os.path.join(l_basename, os.path.relpath(dirpath, l_path))
d_dirname = os.path.join(r_path, sub)
d_basename = os.path.basename(d_dirname)
# prepare destination parent path
d_parent_dirname = os.path.dirname(d_dirname)
if folder_map.get(d_parent_dirname) is None:
folder_map[d_parent_dirname] = cls._find_folder(connection, laboratory, d_parent_dirname)
# prepare destination path
if folder_map.get(d_dirname) is None:
d_folder = folder_map[d_parent_dirname].find_sub_folder(d_basename)
if d_folder is None:
d_folder_id = folder_api.create(d_basename, folder_map[d_parent_dirname].id)
else:
d_folder_id = d_folder.id
print(d_dirname)
folder_map[d_dirname] = folder_api.retrieve(d_folder_id)
if d_folder is None:
folder_map[d_parent_dirname].sub_folders.append(folder_map[d_dirname])
# register upload file list
for filename in filenames:
infos.append(UploadFileInfo(folder_map[d_dirname], os.path.join(dirpath, filename)))
else:
infos.append(UploadFileInfo(folder, l_path))
cls.__multiple_upload(connection, infos, is_skip_if_exists)
@classmethod client = MdrsClient.from_remote(remote)
def __multiple_upload( client.upload(local_path, remote_path, is_recursive, is_skip_if_exists)
cls, connection: MDRSConnection, infos: list[UploadFileInfo], is_skip_if_exists: bool return
) -> None:
file_api = FilesApi(connection)
with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
pool.map(lambda x: cls.__multiple_upload_worker(file_api, x, is_skip_if_exists), infos)
@classmethod
def __multiple_upload_worker(cls, file_api: FilesApi, info: UploadFileInfo, is_skip_if_exists: bool) -> None:
basename = os.path.basename(info.path)
file = info.folder.find_file(basename)
try:
if file is None:
file_api.create(info.folder.id, info.path)
elif not is_skip_if_exists or file.size != os.path.getsize(info.path):
file_api.update(file, info.path)
print(os.path.join(info.folder.path, basename))
except MDRSException as e:
print(f"Error: {e}")
+24
View File
@@ -0,0 +1,24 @@
from argparse import Namespace
from typing import Any
from mdrsclient.__version__ import __version__
from mdrsclient.commands.base import BaseCommand
class VersionCommand(BaseCommand):
@classmethod
def register(cls, parsers: Any) -> None:
version_parser = parsers.add_parser("version", help="show the version of this tool")
version_parser.set_defaults(func=cls.func)
@classmethod
def func(cls, args: Namespace) -> None:
cls.version()
@classmethod
def version(cls) -> None:
from mdrsclient.client import MdrsClient
# Client initialization is not strictly needed for version, but for consistency:
client = MdrsClient(None)
print(client.version())
+11 -11
View File
@@ -2,9 +2,6 @@ from argparse import Namespace
from typing import Any, Final from typing import Any, Final
from mdrsclient.commands.base import BaseCommand from mdrsclient.commands.base import BaseCommand
from mdrsclient.config import ConfigFile
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import MissingConfigurationException
class WhoamiCommand(BaseCommand): class WhoamiCommand(BaseCommand):
@@ -23,12 +20,15 @@ class WhoamiCommand(BaseCommand):
@classmethod @classmethod
def whoami(cls, remote: str) -> None: def whoami(cls, remote: str) -> None:
remote = cls._parse_remote_host(remote) from mdrsclient.client import MdrsClient
config = ConfigFile(remote)
if config.url is None: remote_host = MdrsClient.parse_remote_host(remote)
raise MissingConfigurationException(f"Remote host `{remote}` is not found.") client = MdrsClient.from_remote(remote_host)
connection = MDRSConnection(config.remote, config.url) if client.connection.token is not None and client.connection.token.is_expired:
if connection.token is not None and connection.token.is_expired: client.logout()
connection.logout() try:
username = connection.user.username if connection.user is not None else cls.ANONYMOUS_USERNAME user = client.whoami()
username = user.username if user is not None else cls.ANONYMOUS_USERNAME
except Exception:
username = cls.ANONYMOUS_USERNAME
print(username) print(username)
+54 -4
View File
@@ -1,15 +1,65 @@
import configparser import configparser
import os import os
from typing import Final import threading
from typing import Final, Protocol, runtime_checkable
import validators # type: ignore import validators
from mdrsclient.exceptions import IllegalArgumentException from mdrsclient.exceptions import IllegalArgumentException
from mdrsclient.settings import CONFIG_DIRNAME from mdrsclient.settings import CONFIG_DIRNAME
from mdrsclient.utils import FileLock from mdrsclient.utils import FileLock
class ConfigFile: @runtime_checkable
class ConfigInterface(Protocol):
remote: str
def list(self) -> list[tuple[str, str]]: ...
@property
def url(self) -> str | None: ...
@url.setter
def url(self, url: str) -> None: ...
@url.deleter
def url(self) -> None: ...
class InMemoryConfig(ConfigInterface):
__configs: dict[str, str] = {}
__lock: threading.Lock = threading.Lock()
remote: str
def __init__(self, remote: str) -> None:
self.remote = remote
def list(self) -> list[tuple[str, str]]:
with self.__lock:
return list(self.__configs.items())
@property
def url(self) -> str | None:
with self.__lock:
return self.__configs.get(self.remote)
@url.setter
def url(self, url: str) -> None:
if not validators.url(url):
raise IllegalArgumentException("malformed URI sequence")
with self.__lock:
self.__configs[self.remote] = url
@url.deleter
def url(self) -> None:
with self.__lock:
if self.remote in self.__configs:
del self.__configs[self.remote]
@classmethod
def clear(cls) -> None:
with cls.__lock:
cls.__configs.clear()
class ConfigFile(ConfigInterface):
OPTION_URL: Final[str] = "url" OPTION_URL: Final[str] = "url"
CONFIG_FILENAME: Final[str] = "config.ini" CONFIG_FILENAME: Final[str] = "config.ini"
remote: str remote: str
@@ -41,7 +91,7 @@ class ConfigFile:
@url.setter @url.setter
def url(self, url: str) -> None: def url(self, url: str) -> None:
if not validators.url(url): # type: ignore if not validators.url(url):
raise IllegalArgumentException("malformed URI sequence") raise IllegalArgumentException("malformed URI sequence")
self.__load() self.__load()
if self.__config.has_section(self.remote): if self.__config.has_section(self.remote):
+6 -4
View File
@@ -9,7 +9,7 @@ from requests_toolbelt.multipart.encoder import MultipartEncoder
from typing_extensions import Unpack from typing_extensions import Unpack
from mdrsclient.__version__ import __version__ from mdrsclient.__version__ import __version__
from mdrsclient.cache import CacheFile from mdrsclient.cache import CacheFile, CacheInterface
from mdrsclient.exceptions import MissingConfigurationException from mdrsclient.exceptions import MissingConfigurationException
from mdrsclient.models import Laboratories, Token, User from mdrsclient.models import Laboratories, Token, User
@@ -39,14 +39,14 @@ class MDRSConnection:
url: str url: str
session: Session session: Session
lock: threading.Lock lock: threading.Lock
__cache: CacheFile __cache: CacheInterface
def __init__(self, remote: str, url: str) -> None: def __init__(self, remote: str, url: str, cache: CacheInterface | None = None) -> None:
super().__init__() super().__init__()
self.url = url self.url = url
self.session = Session() self.session = Session()
self.lock = threading.Lock() self.lock = threading.Lock()
self.__cache = CacheFile(remote) self.__cache = cache if cache is not None else CacheFile(remote)
self.__prepare_headers() self.__prepare_headers()
def get(self, url: str, **kwargs: Unpack[_KwArgsMDRSConnectionGet]) -> Response: def get(self, url: str, **kwargs: Unpack[_KwArgsMDRSConnectionGet]) -> Response:
@@ -92,6 +92,8 @@ class MDRSConnection:
self.__cache.laboratories = laboratories self.__cache.laboratories = laboratories
def __build_url(self, path: str) -> str: def __build_url(self, path: str) -> str:
if path.startswith("http://") or path.startswith("https://"):
return path
if self.url == "": if self.url == "":
raise MissingConfigurationException("remote host is not configured") raise MissingConfigurationException("remote host is not configured")
return f"{self.url}/{path}" return f"{self.url}/{path}"
+2
View File
@@ -1,3 +1,4 @@
from mdrsclient.models.doi import Doi
from mdrsclient.models.error import DRFStandardizedErrors from mdrsclient.models.error import DRFStandardizedErrors
from mdrsclient.models.file import File from mdrsclient.models.file import File
from mdrsclient.models.folder import Folder, FolderAccessLevel, FolderSimple from mdrsclient.models.folder import Folder, FolderAccessLevel, FolderSimple
@@ -6,6 +7,7 @@ from mdrsclient.models.user import Token, User
__all__ = [ __all__ = [
"DRFStandardizedErrors", "DRFStandardizedErrors",
"Doi",
"File", "File",
"Folder", "Folder",
"FolderAccessLevel", "FolderAccessLevel",
+11
View File
@@ -0,0 +1,11 @@
from pydantic.dataclasses import dataclass
@dataclass(frozen=True)
class Doi:
"""Model representing a DOI entity (Response from GET v3/doi/{id}/)."""
# The internal DOI suffix ID returned as a string (e.g. "20260429-001").
id: str
doi: str
folder_id: str
+6
View File
@@ -1,4 +1,5 @@
from typing import Any from typing import Any
from unicodedata import normalize
from pydantic.dataclasses import dataclass from pydantic.dataclasses import dataclass
@@ -25,3 +26,8 @@ class File:
@property @property
def updated_at_name(self) -> str: def updated_at_name(self) -> str:
return iso8601_to_user_friendly(self.updated_at) return iso8601_to_user_friendly(self.updated_at)
def find_file(files: list[File], name: str) -> File | None:
_name = normalize("NFC", name).lower()
return next((x for x in files if x.name.lower() == _name), None)
-6
View File
@@ -3,7 +3,6 @@ from unicodedata import normalize
from pydantic.dataclasses import dataclass from pydantic.dataclasses import dataclass
from mdrsclient.models.file import File
from mdrsclient.models.utils import iso8601_to_user_friendly from mdrsclient.models.utils import iso8601_to_user_friendly
@@ -78,13 +77,8 @@ class FolderSimple:
class Folder(FolderSimple): class Folder(FolderSimple):
metadata: list[dict[str, Any]] metadata: list[dict[str, Any]]
sub_folders: list[FolderSimple] sub_folders: list[FolderSimple]
files: list[File]
path: str path: str
def find_sub_folder(self, name: str) -> FolderSimple | None: def find_sub_folder(self, name: str) -> FolderSimple | None:
_name = normalize("NFC", name).lower() _name = normalize("NFC", name).lower()
return next((x for x in self.sub_folders if x.name.lower() == _name), None) return next((x for x in self.sub_folders if x.name.lower() == _name), None)
def find_file(self, name: str) -> File | None:
_name = normalize("NFC", name).lower()
return next((x for x in self.files if x.name.lower() == _name), None)
+233
View File
@@ -0,0 +1,233 @@
import os
import re
from typing import Any
from unicodedata import normalize
from mdrsclient.api import DoiApi, FilesApi, FoldersApi, LaboratoriesApi, UsersApi
from mdrsclient.cache import CacheInterface
from mdrsclient.config import ConfigFile, ConfigInterface
from mdrsclient.connection import MDRSConnection
from mdrsclient.exceptions import (
IllegalArgumentException,
MissingConfigurationException,
UnauthorizedException,
UnexpectedException,
)
from mdrsclient.models import File, Folder, Laboratory, Token, User
from mdrsclient.utils import page_num_from_url
class MdrsService:
config_class: type[ConfigInterface] = ConfigFile
def __init__(self, connection: MDRSConnection, config_class: type[ConfigInterface] | None = None):
self.connection = connection
if config_class is not None:
self.config_class = config_class
@classmethod
def create_connection(
cls, remote: str, cache: CacheInterface | None = None, config: ConfigInterface | None = None
) -> MDRSConnection:
if config is None:
config = ConfigFile(remote)
elif config.remote != remote:
raise IllegalArgumentException("Remote host parameter mismatch.")
if config.url is None:
raise MissingConfigurationException(f"Remote host `{remote}` is not found.")
return MDRSConnection(config.remote, config.url, cache=cache)
def login(self, username: str, password: str) -> tuple[Token, User]:
user_api = UsersApi(self.connection)
token = user_api.token(username, password)
self.connection.token = token
user = user_api.current()
self.connection.user = user
return token, user
def logout(self) -> None:
self.connection.logout()
def whoami(self) -> User:
user_api = UsersApi(self.connection)
return user_api.current()
def get_laboratories(self) -> list[Laboratory]:
laboratory_api = LaboratoriesApi(self.connection)
labs = laboratory_api.list()
self.connection.laboratories = labs
return list(labs)
def find_laboratory(self, name: str) -> Laboratory:
if self.connection.laboratories.empty() or (self.connection.token and self.connection.token.is_expired):
self.get_laboratories()
laboratory = self.connection.laboratories.find_by_name(name)
if laboratory is None:
raise IllegalArgumentException(f"Laboratory `{name}` not found.")
return laboratory
def find_folder(self, laboratory: Laboratory, path: str, password: str | None = None) -> Folder:
folder_api = FoldersApi(self.connection)
folders = folder_api.list(laboratory.id, normalize("NFC", path))
if len(folders) != 1:
raise UnexpectedException(f"Folder `{path}` not found.")
if folders[0].lock:
if password is None:
raise UnauthorizedException(f"Folder `{path}` is locked.")
folder_api.auth(folders[0].id, password)
return folder_api.retrieve(folders[0].id)
def find_files(self, folder_id: str) -> list[File]:
files_api = FilesApi(self.connection)
page = 1
results_file = []
while page:
result = files_api.list(folder_id, page)
results_file.extend(result.results)
page = 0
if result.next:
page = page_num_from_url(result.next)
return results_file
@staticmethod
def is_doi(path_component: str) -> bool:
return path_component.startswith("10.") and "/" in path_component
@staticmethod
def doi_suffix_id(doi: str) -> str:
doi = doi.rstrip("/")
slash_pos = doi.find("/")
if slash_pos == -1:
return doi
suffix = doi[slash_pos + 1 :]
dot_pos = suffix.rfind(".")
return suffix[dot_pos + 1 :] if dot_pos != -1 else suffix
@staticmethod
def split_doi_and_subpath(doi_with_path: str) -> tuple[str, str]:
first_slash = doi_with_path.find("/")
if first_slash != -1:
after_suffix_start = first_slash + 1
after_first = doi_with_path[after_suffix_start:]
second_slash = after_first.find("/")
if second_slash != -1:
doi_end = after_suffix_start + second_slash
doi = doi_with_path[:doi_end]
subpath = doi_with_path[doi_end:]
if subpath == "/":
return (doi, "")
else:
return (doi, subpath)
else:
return (doi_with_path, "")
else:
return (doi_with_path, "")
@classmethod
def parse_remote_host(cls, path: str) -> str:
path_array = path.split(":")
remote_host = path_array[0]
if len(path_array) == 2 and path_array[1] != "" or len(path_array) > 2:
raise IllegalArgumentException("Invalid remote host")
return remote_host
@classmethod
def parse_remote_host_with_path(cls, path: str) -> tuple[str, str, str]:
path = re.sub(r"//+|/\./+|/\.$", "/", path)
if re.search(r"/\.\./|/\.\.$", path) is not None:
raise IllegalArgumentException("Path traversal found.")
path_array = path.split(":")
if len(path_array) != 2:
raise IllegalArgumentException("Invalid remote host.")
remote_host = path_array[0]
folder_array = path_array[1].split("/")
is_absolute_path = folder_array[0] == ""
if not is_absolute_path:
raise IllegalArgumentException("Must be absolute paths.")
del folder_array[0]
if len(folder_array) == 0:
laboratory = ""
folder = ""
else:
laboratory = folder_array.pop(0)
folder = "/" + "/".join(folder_array)
return (remote_host, laboratory, folder)
@classmethod
def parse_doi_remote_host(cls, path: str) -> tuple[str, str, str]:
parts = path.split(":", 1)
if len(parts) != 2:
raise IllegalArgumentException("remote_path must be in the form 'remote:10.xxxx/prefix.ID'")
remote, doi_with_path = parts
if not cls.is_doi(doi_with_path):
raise IllegalArgumentException(f"Path `{doi_with_path}` does not look like a DOI.")
doi, subpath = cls.split_doi_and_subpath(doi_with_path)
return (remote, doi, subpath)
def find_folder_by_doi(self, doi: str, password: str | None = None) -> tuple[Folder, Laboratory]:
doi_clean = doi.rstrip("/")
doi_id = self.doi_suffix_id(doi_clean)
doi_api = DoiApi(self.connection)
doi_resp = doi_api.retrieve(doi_id)
returned_doi = doi_resp.doi.rstrip("/")
if returned_doi.lower() != doi_clean.lower():
raise IllegalArgumentException(
f"DOI mismatch: requested `{doi_clean}` but server returned `{returned_doi}`."
)
folder_api = FoldersApi(self.connection)
folder = folder_api.retrieve(doi_resp.folder_id)
if folder.lock:
if password is None:
raise UnauthorizedException(f"Folder for DOI `{doi_clean}` is locked.")
folder_api.auth(doi_resp.folder.id, password)
lab_api = LaboratoriesApi(self.connection)
labs = lab_api.list()
lab = labs.find_by_id(folder.laboratory_id)
if lab is None:
raise UnexpectedException(f"Laboratory with id {folder.laboratory_id} not found.")
self.connection.laboratories = labs
return (folder, lab)
def resolve_folder(self, remote_path: str, password: str | None = None) -> tuple[Folder, Laboratory]:
path_component = remote_path.split(":", 1)[1] if ":" in remote_path else ""
if self.is_doi(path_component):
remote, doi, subpath = self.parse_doi_remote_host(remote_path)
doi_folder, laboratory = self.find_folder_by_doi(doi, password)
if not subpath:
return (doi_folder, laboratory)
else:
abs_path = doi_folder.path.rstrip("/") + subpath
folder = self.find_folder(laboratory, abs_path, password)
return (folder, laboratory)
else:
remote, laboratory_name, r_path = self.parse_remote_host_with_path(remote_path)
laboratory = self.find_laboratory(laboratory_name)
folder = self.find_folder(laboratory, r_path, password)
return (folder, laboratory)
def resolve_file(self, remote_path: str, password: str | None = None) -> tuple[Folder, Laboratory, str]:
path_component = remote_path.split(":", 1)[1] if ":" in remote_path else ""
if self.is_doi(path_component):
remote, doi, subpath = self.parse_doi_remote_host(remote_path)
doi_folder, laboratory = self.find_folder_by_doi(doi, password)
subpath_clean = subpath.rstrip("/")
if not subpath_clean:
raise IllegalArgumentException("DOI path must point to a file, not a folder.")
r_dirname = os.path.dirname(subpath_clean)
r_basename = os.path.basename(subpath_clean)
abs_path = doi_folder.path.rstrip("/") + r_dirname
parent_folder = self.find_folder(laboratory, abs_path, password)
return (parent_folder, laboratory, r_basename)
else:
remote, laboratory_name, r_path = self.parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
r_dirname = os.path.dirname(r_path)
r_basename = os.path.basename(r_path)
laboratory = self.find_laboratory(laboratory_name)
parent_folder = self.find_folder(laboratory, r_dirname, password)
return (parent_folder, laboratory, r_basename)
+263
View File
@@ -0,0 +1,263 @@
import os
from concurrent.futures import ThreadPoolExecutor
from typing import Any
from unicodedata import normalize
from pydantic.dataclasses import dataclass
from mdrsclient.api import FilesApi, FoldersApi
from mdrsclient.exceptions import IllegalArgumentException, MDRSException, UnexpectedException
from mdrsclient.models import File, Folder, Laboratory
from mdrsclient.models.file import find_file
from mdrsclient.settings import CONCURRENT
@dataclass(frozen=True)
class UploadFileInfo:
folder: Folder
files: list[File]
path: str
@dataclass(frozen=True)
class DownloadFileInfo:
file: File
path: str
@dataclass
class DownloadContext:
hasError: bool
isSkipIfExists: bool
files: list[DownloadFileInfo]
class Uploader:
def __init__(self, client: Any) -> None:
self.client = client
def upload(
self, local_path: str, remote_path: str, is_recursive: bool = False, is_skip_if_exists: bool = False
) -> None:
remote, laboratory_name, r_path = self.client.parse_remote_host_with_path(remote_path)
l_path = os.path.abspath(local_path)
if not os.path.exists(l_path):
raise IllegalArgumentException(f"File or directory `{local_path}` not found.")
laboratory = self.client.find_laboratory(laboratory_name)
folder = self.client.find_folder(laboratory, r_path)
files = self.client.find_files(folder.id)
infos: list[UploadFileInfo] = []
if os.path.isdir(l_path):
if not is_recursive:
raise IllegalArgumentException(f"Cannot upload `{local_path}`: Is a directory.")
folder_api = FoldersApi(self.client.connection)
folder_map: dict[str, Folder] = {}
folder_map[r_path] = folder
files_map: dict[str, list[File]] = {}
files_map[r_path] = files
l_basename = os.path.basename(l_path)
for dirpath, _, filenames in os.walk(l_path, followlinks=True):
sub = l_basename if dirpath == l_path else os.path.join(l_basename, os.path.relpath(dirpath, l_path))
d_dirname = os.path.join(r_path, sub)
d_basename = os.path.basename(d_dirname)
# prepare destination parent path
d_parent_dirname = os.path.dirname(d_dirname)
if folder_map.get(d_parent_dirname) is None:
parent_folder = self.client.find_folder(laboratory, d_parent_dirname)
folder_map[d_parent_dirname] = parent_folder
parent_files = self.client.find_files(parent_folder.id)
files_map[d_parent_dirname] = parent_files
# prepare destination path
if folder_map.get(d_dirname) is None:
d_folder = folder_map[d_parent_dirname].find_sub_folder(d_basename)
if d_folder is None:
d_folder_id = folder_api.create(normalize("NFC", d_basename), folder_map[d_parent_dirname].id)
else:
d_folder_id = d_folder.id
print(d_dirname)
folder_map[d_dirname] = folder_api.retrieve(d_folder_id)
files_map[d_dirname] = self.client.find_files(d_folder_id)
if d_folder is None:
folder_map[d_parent_dirname].sub_folders.append(folder_map[d_dirname])
# register upload file list
for filename in filenames:
infos.append(
UploadFileInfo(folder_map[d_dirname], files_map[d_dirname], os.path.join(dirpath, filename))
)
else:
infos.append(UploadFileInfo(folder, files, l_path))
self.__multiple_upload(infos, is_skip_if_exists)
def __multiple_upload(self, infos: list[UploadFileInfo], is_skip_if_exists: bool) -> None:
file_api = FilesApi(self.client.connection)
with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
pool.map(lambda x: self.__multiple_upload_worker(file_api, x, is_skip_if_exists), infos)
def __multiple_upload_worker(self, file_api: FilesApi, info: UploadFileInfo, is_skip_if_exists: bool) -> None:
basename = os.path.basename(info.path)
file = find_file(info.files, basename)
try:
if file is None:
file_api.create(info.folder.id, info.path)
elif not is_skip_if_exists or file.size != os.path.getsize(info.path):
file_api.update(file, info.path)
print(os.path.join(info.folder.path, basename))
except MDRSException as e:
print(f"Error: {e}")
class Downloader:
def __init__(self, client: Any) -> None:
self.client = client
def download(
self,
remote_path: str,
local_path: str,
is_recursive: bool = False,
is_skip_if_exists: bool = False,
password: str | None = None,
excludes: list[str] | None = None,
) -> None:
excludes_clean = excludes or []
# Detect DOI path: "remote:10.xxxx/prefix.ID[/optional/sub/path]"
path_component = remote_path.split(":", 1)[1] if ":" in remote_path else ""
if self.client.is_doi(path_component):
remote, doi, subpath = self.client.parse_doi_remote_host(remote_path)
l_dirname = os.path.realpath(local_path)
if not os.path.isdir(l_dirname):
raise IllegalArgumentException(f"Local directory `{local_path}` not found.")
doi_folder, laboratory = self.client.find_folder_by_doi(doi, password)
subpath_clean = subpath.rstrip("/")
if not subpath_clean:
folder = doi_folder
is_folder = True
else:
r_dirname = os.path.dirname(subpath_clean)
r_basename = os.path.basename(subpath_clean)
abs_path = doi_folder.path.rstrip("/") + r_dirname
r_parent_folder = self.client.find_folder(laboratory, abs_path, password)
r_parent_files = self.client.find_files(r_parent_folder.id)
file = find_file(r_parent_files, r_basename)
if file is not None:
if self.__check_excludes(excludes_clean, laboratory, r_parent_folder, file):
return
context = DownloadContext(False, is_skip_if_exists, [])
l_path = os.path.join(l_dirname, r_basename)
context.files.append(DownloadFileInfo(file, l_path))
self.__multiple_download(context)
return
else:
folder_simple = r_parent_folder.find_sub_folder(r_basename)
if folder_simple is None:
raise IllegalArgumentException(f"File or folder `{subpath_clean}` not found.")
folder = FoldersApi(self.client.connection).retrieve(folder_simple.id)
is_folder = True
# For a DOI target the whole folder is the download target.
if not is_recursive:
# Non-recursive: download only the files at the top level of the DOI folder.
files = self.client.find_files(folder.id)
context = DownloadContext(False, is_skip_if_exists, [])
for file in files:
if self.__check_excludes(excludes_clean, laboratory, folder, file):
continue
l_path = os.path.join(l_dirname, file.name)
context.files.append(DownloadFileInfo(file, l_path))
self.__multiple_download(context)
return
folder_api = FoldersApi(self.client.connection)
self.__multiple_download_pickup_recursive_files(
folder_api, laboratory, folder.id, l_dirname, excludes_clean, is_skip_if_exists
)
return
remote, laboratory_name, r_path = self.client.parse_remote_host_with_path(remote_path)
r_path = r_path.rstrip("/")
r_dirname = os.path.dirname(r_path)
r_basename = os.path.basename(r_path)
l_dirname = os.path.realpath(local_path)
if not os.path.isdir(l_dirname):
raise IllegalArgumentException(f"Local directory `{local_path}` not found.")
laboratory = self.client.find_laboratory(laboratory_name)
r_parent_folder = self.client.find_folder(laboratory, r_dirname, password)
r_parent_files = self.client.find_files(r_parent_folder.id)
file = find_file(r_parent_files, r_basename)
if file is not None:
if self.__check_excludes(excludes_clean, laboratory, r_parent_folder, file):
return
context = DownloadContext(False, is_skip_if_exists, [])
l_path = os.path.join(l_dirname, r_basename)
context.files.append(DownloadFileInfo(file, l_path))
self.__multiple_download(context)
else:
folder = r_parent_folder.find_sub_folder(r_basename)
if folder is None:
raise IllegalArgumentException(f"File or folder `{r_path}` not found.")
if not is_recursive:
raise IllegalArgumentException(f"Cannot download `{r_path}`: Is a folder.")
folder_api = FoldersApi(self.client.connection)
self.__multiple_download_pickup_recursive_files(
folder_api, laboratory, folder.id, l_dirname, excludes_clean, is_skip_if_exists
)
def __multiple_download_pickup_recursive_files(
self,
folder_api: FoldersApi,
laboratory: Laboratory,
folder_id: str,
basedir: str,
excludes: list[str],
is_skip_if_exists: bool,
) -> None:
context = DownloadContext(False, is_skip_if_exists, [])
folder = folder_api.retrieve(folder_id)
files = self.client.find_files(folder.id)
dirname = os.path.join(basedir, folder.name)
if self.__check_excludes(excludes, laboratory, folder, None):
return
if not os.path.exists(dirname):
os.makedirs(dirname)
print(dirname)
for file in files:
if self.__check_excludes(excludes, laboratory, folder, file):
continue
path = os.path.join(dirname, file.name)
context.files.append(DownloadFileInfo(file, path))
self.__multiple_download(context)
if context.hasError:
raise UnexpectedException("Some files failed to download.")
for sub_folder in folder.sub_folders:
self.__multiple_download_pickup_recursive_files(
folder_api, laboratory, sub_folder.id, dirname, excludes, is_skip_if_exists
)
def __multiple_download(self, context: DownloadContext) -> None:
file_api = FilesApi(self.client.connection)
with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
results = pool.map(
lambda x: self.__multiple_download_worker(file_api, x, context.isSkipIfExists), context.files
)
hasError = next(filter(lambda x: x is False, results), None)
if hasError is not None:
context.hasError = True
def __multiple_download_worker(self, file_api: FilesApi, info: DownloadFileInfo, is_skip_if_exists: bool) -> bool:
if not is_skip_if_exists or not os.path.exists(info.path) or info.file.size != os.path.getsize(info.path):
try:
file_api.download(info.file, info.path)
except Exception:
print(f"Failed: {info.path}")
if os.path.isfile(info.path):
os.remove(info.path)
return False
print(info.path)
return True
def __check_excludes(self, excludes: list[str], laboratory: Laboratory, folder: Folder, file: File | None) -> bool:
path = f"/{laboratory.name}{folder.path}{file.name if file is not None else ''}".rstrip("/").lower()
return path in excludes
+8
View File
@@ -1,5 +1,6 @@
import os import os
from typing import IO, Any from typing import IO, Any
from urllib.parse import parse_qs, urlparse
if os.name == "nt": if os.name == "nt":
import msvcrt import msvcrt
@@ -21,3 +22,10 @@ class FileLock:
msvcrt.locking(file.fileno(), msvcrt.LK_UNLCK, 1) msvcrt.locking(file.fileno(), msvcrt.LK_UNLCK, 1)
elif os.name == "posix": elif os.name == "posix":
fcntl.flock(file.fileno(), fcntl.LOCK_UN) fcntl.flock(file.fileno(), fcntl.LOCK_UN)
def page_num_from_url(url: str) -> int | None:
parsed_url = urlparse(url)
params = parse_qs(parsed_url.query)
page = params.get("page", [None])[0]
return int(page) if page is not None else None
+14 -12
View File
@@ -1,18 +1,20 @@
[tool.poetry] [tool.poetry]
name = "mdrs-client-python" name = "mdrs-client-python"
version = "1.3.7" version = "1.3.18"
description = "The mdrs-client-python is python library and a command-line client for up- and downloading files to and from MDRS based repository." description = "The mdrs-client-python is python library and a command-line client for up- and downloading files to and from MDRS based repository."
authors = ["Yoshihiro OKUMURA <yoshihiro.okumura@riken.jp>"] authors = ["Yoshihiro OKUMURA <yoshihiro.okumura@riken.jp>"]
license = "MIT" license = "MIT"
readme = "README.md" readme = "README.md"
classifiers=[ classifiers=[
"Development Status :: 3 - Alpha", "Development Status :: 4 - Beta",
"Environment :: Console", "Environment :: Console",
"Intended Audience :: Developers", "Intended Audience :: Developers",
"Intended Audience :: Science/Research", "Intended Audience :: Science/Research",
"Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"OSI Approved :: MIT License", "OSI Approved :: MIT License",
"Topic :: Utilities", "Topic :: Utilities",
] ]
@@ -22,20 +24,20 @@ packages = [
[tool.poetry.dependencies] [tool.poetry.dependencies]
python = "^3.10" python = "^3.10"
requests = "^2.32.3" requests = "^2.34.2"
requests-toolbelt = "^1.0.0" requests-toolbelt = "^1.0.0"
python-dotenv = "^1.0.1" python-dotenv = "^1.1.0"
pydantic = "^2.8.2" pydantic = "^2.13.4"
pydantic-settings = "^2.3.4" pydantic-settings = "^2.14.2"
PyJWT = "^2.8.0" PyJWT = "^2.13.0"
validators = "^0.33.0" validators = "^0.35.0"
[tool.poetry.group.dev.dependencies] [tool.poetry.group.dev.dependencies]
black = "^24.2.2" black = "^26.5.1"
flake8 = "^7.1.0" flake8 = "^7.2.0"
Flake8-pyproject = "^1.2.3" Flake8-pyproject = "^1.2.3"
isort = "^5.13.2" isort = "^8.0.1"
pyright = "^1.1.372" pyright = "^1.1.411"
[tool.poetry.scripts] [tool.poetry.scripts]
mdrs = 'mdrsclient.__main__:main' mdrs = 'mdrsclient.__main__:main'
+1
View File
@@ -0,0 +1 @@
# Mark tests directory as a Python package
+418
View File
@@ -0,0 +1,418 @@
import argparse
import json
import unittest
from io import StringIO
from unittest.mock import MagicMock, patch
from mdrsclient.client import MdrsClient
from mdrsclient.commands import (
ChaclCommand,
ConfigCommand,
CpCommand,
DownloadCommand,
FileMetadataCommand,
LabsCommand,
LoginCommand,
LogoutCommand,
LsCommand,
MetadataCommand,
MkdirCommand,
MvCommand,
RmCommand,
UploadCommand,
VersionCommand,
WhoamiCommand,
)
from mdrsclient.models import Folder, Laboratory
class TestCommands(unittest.TestCase):
def parse_args(self, cmd_class, args_list):
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(title="subcommands")
cmd_class.register(subparsers)
return parser.parse_args(args_list)
def test_version_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.return_value = mock_client
mock_client.version.return_value = "mdrs 1.3.17"
args = self.parse_args(VersionCommand, ["version"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
self.assertEqual(fake_out.getvalue().strip(), "mdrs 1.3.17")
mock_client_class.assert_called_once_with(None)
mock_client.version.assert_called_once()
def test_login_command_with_args(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client_class.parse_remote_host.return_value = "myremote"
args = self.parse_args(LoginCommand, ["login", "-u", "myuser", "-p", "mypass", "myremote"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
self.assertIn("Login Successful", fake_out.getvalue())
mock_client_class.parse_remote_host.assert_called_once_with("myremote")
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.login.assert_called_once_with("myuser", "mypass")
def test_login_command_interactive(self):
with (
patch("mdrsclient.client.MdrsClient") as mock_client_class,
patch("builtins.input", return_value="myuser_int") as mock_input,
patch("getpass.getpass", return_value="mypass_int") as mock_getpass,
):
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client_class.parse_remote_host.return_value = "myremote"
args = self.parse_args(LoginCommand, ["login", "myremote"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
self.assertIn("Login Successful", fake_out.getvalue())
mock_input.assert_called_once_with("Username: ")
mock_getpass.assert_called_once_with("Password: ")
mock_client.login.assert_called_once_with("myuser_int", "mypass_int")
def test_logout_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client_class.parse_remote_host.return_value = "myremote"
args = self.parse_args(LogoutCommand, ["logout", "myremote"])
args.func(args)
mock_client_class.parse_remote_host.assert_called_once_with("myremote")
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.logout.assert_called_once()
def test_whoami_command_logged_in(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client_class.parse_remote_host.return_value = "myremote"
mock_client.connection.token = None
mock_user = MagicMock()
mock_user.username = "test_user"
mock_client.whoami.return_value = mock_user
args = self.parse_args(WhoamiCommand, ["whoami", "myremote"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
self.assertEqual(fake_out.getvalue().strip(), "test_user")
mock_client_class.parse_remote_host.assert_called_once_with("myremote")
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.whoami.assert_called_once()
def test_whoami_command_anonymous(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client_class.parse_remote_host.return_value = "myremote"
mock_client.connection.token = None
mock_client.whoami.side_effect = Exception("Not logged in")
args = self.parse_args(WhoamiCommand, ["whoami", "myremote"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
self.assertEqual(fake_out.getvalue().strip(), "(Anonymous)")
def test_labs_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client_class.parse_remote_host.return_value = "myremote"
mock_lab = MagicMock()
mock_lab.id = 1
mock_lab.name = "lab_name"
mock_lab.pi_name = "pi_name"
mock_lab.full_name = "full_name"
mock_client.get_laboratories.return_value = [mock_lab]
args = self.parse_args(LabsCommand, ["labs", "myremote"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
output = fake_out.getvalue()
self.assertIn("Name", output)
self.assertIn("lab_name", output)
self.assertIn("pi_name", output)
mock_client_class.parse_remote_host.assert_called_once_with("myremote")
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.get_laboratories.assert_called_once()
def test_ls_command_plain(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock(spec=MdrsClient)
mock_client_class.from_remote.return_value = mock_client
mock_folder = Folder(
id="folder_id",
pid=None,
name="root",
access_level=1,
lock=False,
size=0,
laboratory_id=1,
description="",
created_at="2026-07-02T00:00:00Z",
updated_at="2026-07-02T00:00:00Z",
restrict_opened_at=None,
metadata=[],
sub_folders=[],
path="/root",
)
mock_lab = Laboratory(id=1, name="mylab", pi_name="pi_name", full_name="full_name")
mock_client.resolve_folder.return_value = (mock_folder, mock_lab)
mock_file = MagicMock()
mock_file.name = "file.txt"
mock_file.size = 100
mock_file.updated_at_name = "2026-07-02"
mock_client.find_files.return_value = [mock_file]
args = self.parse_args(LsCommand, ["ls", "myremote:/mylab/"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
output = fake_out.getvalue()
self.assertIn("Type", output)
self.assertIn("file.txt", output)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.resolve_folder.assert_called_once_with("myremote:/mylab/", None)
mock_client.find_files.assert_called_once_with(mock_folder.id)
def test_ls_command_json(self):
with (
patch("mdrsclient.client.MdrsClient") as mock_client_class,
patch("mdrsclient.commands.ls.FoldersApi") as mock_folders_api_class,
):
mock_folders_api = MagicMock()
mock_folders_api_class.return_value = mock_folders_api
mock_folders_api.metadata.return_value = {"folder_meta": "val"}
mock_client = MagicMock(spec=MdrsClient)
mock_client_class.from_remote.return_value = mock_client
mock_client.connection = MagicMock()
mock_folder = Folder(
id="folder_id",
pid="parent_id",
name="root",
access_level=1,
lock=False,
size=0,
laboratory_id=1,
description="Root folder",
created_at="2026-07-02T00:00:00Z",
updated_at="2026-07-02T00:00:00Z",
restrict_opened_at=None,
metadata=[],
sub_folders=[],
path="/root",
)
mock_lab = Laboratory(id=1, name="mylab", pi_name="pi_name", full_name="full_name")
mock_client.resolve_folder.return_value = (mock_folder, mock_lab)
mock_client.connection.laboratories.find_by_id.return_value = mock_lab
mock_file = MagicMock()
mock_file.id = "file_id"
mock_file.name = "file.txt"
mock_file.type = "text"
mock_file.size = 100
mock_file.description = "A file"
mock_file.metadata = {}
mock_file.download_url = "download/file"
mock_file.created_at = "2026-07-02T00:00:00Z"
mock_file.updated_at = "2026-07-02T00:00:00Z"
mock_client.find_files.return_value = [mock_file]
args = self.parse_args(LsCommand, ["ls", "-J", "myremote:/mylab/"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
output = fake_out.getvalue()
parsed_json = json.loads(output)
self.assertEqual(parsed_json["name"], "root")
self.assertEqual(parsed_json["files"][0]["name"], "file.txt")
def test_mkdir_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(MkdirCommand, ["mkdir", "myremote:/mylab/newfolder"])
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.mkdir.assert_called_once_with("myremote:/mylab/newfolder")
def test_rm_command_file(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(RmCommand, ["rm", "myremote:/mylab/file.txt"])
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.rm.assert_called_once_with("myremote:/mylab/file.txt", False)
def test_rm_command_recursive(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(RmCommand, ["rm", "-r", "myremote:/mylab/folder"])
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.rm.assert_called_once_with("myremote:/mylab/folder", True)
def test_cp_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(CpCommand, ["cp", "myremote:/mylab/src", "myremote:/mylab/dest"])
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.cp.assert_called_once_with("myremote:/mylab/src", "myremote:/mylab/dest", False)
def test_mv_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(MvCommand, ["mv", "myremote:/mylab/src", "myremote:/mylab/dest"])
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.mv.assert_called_once_with("myremote:/mylab/src", "myremote:/mylab/dest")
def test_chacl_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(ChaclCommand, ["chacl", "-r", "-p", "secret", "private", "myremote:/mylab/"])
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.chacl.assert_called_once_with("myremote:/mylab/", 1, True, "secret")
def test_metadata_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client.metadata.return_value = {"meta_key": "meta_val"}
args = self.parse_args(MetadataCommand, ["metadata", "-p", "secret", "myremote:/mylab/"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
parsed_json = json.loads(fake_out.getvalue())
self.assertEqual(parsed_json["meta_key"], "meta_val")
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.metadata.assert_called_once_with("myremote:/mylab/", "secret")
def test_file_metadata_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
mock_client.file_metadata.return_value = {"file_meta": "val"}
args = self.parse_args(FileMetadataCommand, ["file-metadata", "-p", "secret", "myremote:/mylab/file.txt"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
parsed_json = json.loads(fake_out.getvalue())
self.assertEqual(parsed_json["file_meta"], "val")
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.file_metadata.assert_called_once_with("myremote:/mylab/file.txt", "secret")
def test_upload_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(UploadCommand, ["upload", "-r", "-s", "local_file.txt", "myremote:/mylab/"])
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.upload.assert_called_once_with("local_file.txt", "myremote:/mylab/", True, True)
def test_download_command(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.from_remote.return_value = mock_client
args = self.parse_args(
DownloadCommand, ["download", "-r", "-s", "-p", "pass", "-e", "ex1", "myremote:/mylab/", "local_dir"]
)
args.func(args)
mock_client_class.from_remote.assert_called_once_with("myremote")
mock_client.download.assert_called_once_with("myremote:/mylab/", "local_dir", True, True, "pass", ["ex1"])
def test_config_create(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.return_value = mock_client
args = self.parse_args(ConfigCommand, ["config", "create", "myremote", "http://example.com"])
args.func(args)
mock_client_class.assert_called_once_with(None)
mock_client.config_create.assert_called_once_with("myremote", "http://example.com")
def test_config_update(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.return_value = mock_client
args = self.parse_args(ConfigCommand, ["config", "update", "myremote", "http://example.com"])
args.func(args)
mock_client_class.assert_called_once_with(None)
mock_client.config_update.assert_called_once_with("myremote", "http://example.com")
def test_config_delete(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.return_value = mock_client
args = self.parse_args(ConfigCommand, ["config", "delete", "myremote"])
args.func(args)
mock_client_class.assert_called_once_with(None)
mock_client.config_delete.assert_called_once_with("myremote")
def test_config_list(self):
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
mock_client = MagicMock()
mock_client_class.return_value = mock_client
mock_client.config_list.return_value = [("remote1", "url1"), ("remote2", "url2")]
args = self.parse_args(ConfigCommand, ["config", "list"])
with patch("sys.stdout", new=StringIO()) as fake_out:
args.func(args)
self.assertEqual(fake_out.getvalue(), "remote1:\turl1\nremote2:\turl2\n")
mock_client_class.assert_called_once_with(None)
mock_client.config_list.assert_called_once()