Compare commits
27 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
d59a150b4f
|
|||
|
7f6d496654
|
|||
|
b95fc0cd7d
|
|||
|
8ce9e09e69
|
|||
|
36cad6db52
|
|||
|
809140dfbc
|
|||
|
5bdf837941
|
|||
|
04c0003a61
|
|||
|
428be1289c
|
|||
|
4283481695
|
|||
|
ddb4300d85
|
|||
|
68670a6588
|
|||
| 6d8fd0a598 | |||
| 634b5f1a13 | |||
| 95f22ea5f9 | |||
|
fffb686023
|
|||
|
893bc77128
|
|||
|
7b3f1f2d09
|
|||
|
8a5e1b68b7
|
|||
|
dd00973bea
|
|||
|
0e5685d5ea
|
|||
|
6a2810f603
|
|||
|
d5ac5cd427
|
|||
|
ab7cd1b885
|
|||
|
f2f898c263
|
|||
|
24172dc65c
|
|||
|
f2c5a06cb4
|
+5
-1
@@ -16,7 +16,11 @@
|
|||||||
"mdrsclient",
|
"mdrsclient",
|
||||||
"neurodata",
|
"neurodata",
|
||||||
"Neuroinformatics",
|
"Neuroinformatics",
|
||||||
|
"orcid",
|
||||||
"RIKEN"
|
"RIKEN"
|
||||||
],
|
],
|
||||||
"ignorePaths": [".env", "__pycache__"]
|
"ignorePaths": [
|
||||||
|
".env",
|
||||||
|
"__pycache__"
|
||||||
|
]
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -161,3 +161,4 @@ cython_debug/
|
|||||||
|
|
||||||
# mdrs-cli
|
# mdrs-cli
|
||||||
.neurodatacli.config
|
.neurodatacli.config
|
||||||
|
poetry.toml
|
||||||
+129
@@ -0,0 +1,129 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
All notable changes to this project will be documented in this file.
|
||||||
|
|
||||||
|
## [1.3.18] - 2026-07-02
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- Added a comprehensive unit test suite in `tests/test_commands.py` checking registration, parsing, and execution flow of all 16 commands.
|
||||||
|
|
||||||
|
### Refactored
|
||||||
|
- Abstracted configuration storage (introducing `ConfigInterface`, `InMemoryConfig`, and updating `ConfigFile`) to enable dependency injection.
|
||||||
|
- Modularized transfer operations (upload and download) to decouple them from the service layer.
|
||||||
|
- Decoupled commands from direct file system configurations and migrated all subcommands to use abstract config classes.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Upgraded dependencies including `pydantic-settings` to `2.14.2` and `pyright` to `1.1.411`.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- Fixed duplicate `__all__` definitions in package initialization file `mdrsclient/__init__.py` that caused `__version__` export to be overwritten.
|
||||||
|
|
||||||
|
## [1.3.17] - 2026-07-02
|
||||||
|
|
||||||
|
### Refactored
|
||||||
|
- Decoupled core logic from CLI interface and introduced `MdrsClient` service layer to improve library portability.
|
||||||
|
- Migrated all CLI commands to utilize `MdrsClient` for execution.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- Abstract authentication state using `CacheInterface` and `InMemoryCache`.
|
||||||
|
|
||||||
|
## [1.3.16] - 2026-06-12
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- Retrieve the full `Folder` object from `FoldersApi` instead of using the `FolderSimple` returned by `find_sub_folder` when resolving DOI subfolders. This fixes a type checker error under the upgraded pyright and avoids a potential AttributeError at runtime due to `FolderSimple` lacking the `path` attribute.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Upgraded dependencies and bumped version to 1.3.16 in pyproject.toml.
|
||||||
|
|
||||||
|
## [1.3.15] - 2026-05-01
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Bumped package version to 1.3.15.
|
||||||
|
|
||||||
|
## [1.3.14] - 2026-04-17
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Simplified `config list` command (removed `-l`/`--long` option, always display URL).
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- Added subcommand aliases for config commands (e.g. `ls` alias for list, `rm` alias for delete).
|
||||||
|
|
||||||
|
## [1.3.13] - 2025-07-02
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Bumped package version to 1.3.13.
|
||||||
|
|
||||||
|
## [1.3.12] - 2025-05-20
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Bumped package version to 1.3.12.
|
||||||
|
|
||||||
|
## [1.3.11] - 2025-01-21
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Bumped package version to 1.3.11.
|
||||||
|
|
||||||
|
## [1.3.10] - 2024-12-23
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Bumped package version to 1.3.10.
|
||||||
|
|
||||||
|
## [1.3.9] - 2024-10-23
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- Fixed compatibility with Python 3.10.
|
||||||
|
|
||||||
|
## [1.3.8] - 2024-09-18
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Bumped package version to 1.3.8.
|
||||||
|
|
||||||
|
## [1.3.7] - 2024-07-22
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- Implemented `--exclude` argument for download subcommand.
|
||||||
|
|
||||||
|
## [1.3.6] - 2024-07-08
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Bumped package version to 1.3.6.
|
||||||
|
|
||||||
|
## [1.3.5] - 2024-07-08
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Bumped package version to 1.3.5.
|
||||||
|
|
||||||
|
## [1.3.4] - 2024-07-04
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- Added some aliases for config sub command.
|
||||||
|
|
||||||
|
## [1.3.3] - 2024-02-13
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Bumped package version to 1.3.3.
|
||||||
|
|
||||||
|
## [1.3.2] - 2024-02-09
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- Added `-u` and `-p` options to login command.
|
||||||
|
|
||||||
|
## [1.3.1] - 2023-12-20
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- Fixed bug to resolve local files for recursive file upload.
|
||||||
|
|
||||||
|
## [1.3.0] - 2023-12-18
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Removed debug comments.
|
||||||
|
|
||||||
|
## [1.2.0] - 2023-10-04
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Follow-up recent specification changes about folder access level.
|
||||||
|
|
||||||
|
## [1.1.1] - 2023-07-26
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Set destination folder name using name attribute of folder copy API.
|
||||||
@@ -8,7 +8,7 @@ The mdrs-client-python is python library and a command-line client for up- and d
|
|||||||
poetry install
|
poetry install
|
||||||
```
|
```
|
||||||
|
|
||||||
## Example Usage
|
## CLI Usage
|
||||||
|
|
||||||
### config create
|
### config create
|
||||||
|
|
||||||
@@ -18,6 +18,32 @@ Create remote host configuration
|
|||||||
mdrs config create neurodata https://neurodata.riken.jp/api
|
mdrs config create neurodata https://neurodata.riken.jp/api
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### config update
|
||||||
|
|
||||||
|
Update the URL of a registered remote host.
|
||||||
|
|
||||||
|
```shell
|
||||||
|
mdrs config update neurodata https://neurodata.riken.jp/api
|
||||||
|
```
|
||||||
|
|
||||||
|
### config list
|
||||||
|
|
||||||
|
List registered remote hosts.
|
||||||
|
|
||||||
|
```shell
|
||||||
|
mdrs config list
|
||||||
|
mdrs config ls
|
||||||
|
```
|
||||||
|
|
||||||
|
### config delete
|
||||||
|
|
||||||
|
Remove a registered remote host.
|
||||||
|
|
||||||
|
```shell
|
||||||
|
mdrs config delete neurodata
|
||||||
|
mdrs config rm neurodata
|
||||||
|
```
|
||||||
|
|
||||||
### login
|
### login
|
||||||
|
|
||||||
Login to remote host
|
Login to remote host
|
||||||
@@ -56,13 +82,17 @@ mdrs labs neurodata:
|
|||||||
|
|
||||||
### ls
|
### ls
|
||||||
|
|
||||||
List the folder contents
|
List the folder contents. You can also specify a DOI path in the form `remote:10.xxxx/yyy.ID[/optional/subpath]`.
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
mdrs ls neurodata:/NIU/Repository/
|
mdrs ls neurodata:/NIU/Repository/
|
||||||
mdrs ls -p PW_OPEN_PASSWORD neurodata:/NIU/Repository/PW_Open/
|
mdrs ls -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/
|
||||||
mdrs ls -r neurodata:/NIU/Repository/Dataset1/
|
mdrs ls -r neurodata:/NIU/Repository/Dataset1/
|
||||||
mdrs ls -J -r neurodata:/NIU/Repository/Dataset1/
|
mdrs ls -J -r neurodata:/NIU/Repository/Dataset1/
|
||||||
|
|
||||||
|
# DOI access examples:
|
||||||
|
mdrs ls neurodata:10.60178/cbs.20260429-001
|
||||||
|
mdrs ls "neurodata:10.60178/cbs.20260429-001/Figure 1"
|
||||||
```
|
```
|
||||||
|
|
||||||
### mkdir
|
### mkdir
|
||||||
@@ -80,17 +110,23 @@ Upload the file or directory
|
|||||||
```shell
|
```shell
|
||||||
mdrs upload ./sample.dat neurodata:/NIU/Repository/TEST/
|
mdrs upload ./sample.dat neurodata:/NIU/Repository/TEST/
|
||||||
mdrs upload -r ./dataset neurodata:/NIU/Repository/TEST/
|
mdrs upload -r ./dataset neurodata:/NIU/Repository/TEST/
|
||||||
mdrs upload -r -s ./dataset neurodata:/NIU/Repository/TEST/
|
mdrs upload -r --skip-if-exists ./dataset neurodata:/NIU/Repository/TEST/
|
||||||
```
|
```
|
||||||
|
|
||||||
### download
|
### download
|
||||||
|
|
||||||
Download the file or folder
|
Download the file or folder. You can also specify a DOI path.
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
mdrs download neurodata:/NIU/Repository/TEST/sample.dat ./
|
mdrs download neurodata:/NIU/Repository/TEST/sample.dat ./
|
||||||
mdrs download -r neurodata:/NIU/Repository/TEST/dataset/ ./
|
mdrs download -r neurodata:/NIU/Repository/TEST/dataset/ ./
|
||||||
mdrs download -p PW_OPEN_PASSWORD neurodata:/NIU/Repository/PW_Open/Readme.dat ./
|
mdrs download -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/Readme.dat ./
|
||||||
|
mdrs download -r --exclude /NIU/Repository/TEST/dataset/skip neurodata:/NIU/Repository/TEST/dataset/ ./
|
||||||
|
mdrs download -r --skip-if-exists neurodata:/NIU/Repository/TEST/dataset/ ./
|
||||||
|
|
||||||
|
# DOI access examples:
|
||||||
|
mdrs download neurodata:10.60178/cbs.20260429-001/README_NeuroData.md ./
|
||||||
|
mdrs download -r "neurodata:10.60178/cbs.20260429-001/Figure 1" ./
|
||||||
```
|
```
|
||||||
|
|
||||||
### mv
|
### mv
|
||||||
@@ -127,25 +163,39 @@ Change the folder access level
|
|||||||
```shell
|
```shell
|
||||||
mdrs chacl private neurodata:/NIU/Repository/Private
|
mdrs chacl private neurodata:/NIU/Repository/Private
|
||||||
mdrs chacl cbs_open -r neurodata:/NIU/Repository/CBS_Open
|
mdrs chacl cbs_open -r neurodata:/NIU/Repository/CBS_Open
|
||||||
mdrs chacl pw_open -r -p PW_OPEN_PASSWORD neurodata:/NIU/Repository/PW_Open
|
mdrs chacl pw_open -r -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open
|
||||||
```
|
```
|
||||||
|
|
||||||
### metadata
|
### metadata
|
||||||
|
|
||||||
Get a folder metadata
|
Get a folder metadata. You can also specify a DOI path.
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
mdrs metadata neurodata:/NIU/Repository/TEST/
|
mdrs metadata neurodata:/NIU/Repository/TEST/
|
||||||
mdrs metadata -p PW_OPEN_PASSWORD neurodata:/NIU/Repository/PW_Open/
|
mdrs metadata -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/
|
||||||
|
|
||||||
|
# DOI access examples:
|
||||||
|
mdrs metadata neurodata:10.60178/cbs.20260429-001
|
||||||
```
|
```
|
||||||
|
|
||||||
### file-metadata
|
### file-metadata
|
||||||
|
|
||||||
Get the file metadata
|
Get the file metadata. You can also specify a DOI path.
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
mdrs file-metadata neurodata:/NIU/Repository/TEST/dataset/sample.dat
|
mdrs file-metadata neurodata:/NIU/Repository/TEST/dataset/sample.dat
|
||||||
mdrs file-metadata -p PW_OPEN_PASSWORD neurodata:/NIU/Repository/PW_Open/Readme.txt
|
mdrs file-metadata -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/Readme.txt
|
||||||
|
|
||||||
|
# DOI access examples:
|
||||||
|
mdrs file-metadata "neurodata:10.60178/cbs.20260429-001/Figure 1/Figure1v3.pdf"
|
||||||
|
```
|
||||||
|
|
||||||
|
### version
|
||||||
|
|
||||||
|
Show the tool name and version number
|
||||||
|
|
||||||
|
```shell
|
||||||
|
mdrs version
|
||||||
```
|
```
|
||||||
|
|
||||||
### help
|
### help
|
||||||
@@ -155,3 +205,46 @@ Show the help message and exit
|
|||||||
```shell
|
```shell
|
||||||
mdrs -h
|
mdrs -h
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Python API Usage
|
||||||
|
|
||||||
|
You can also use this package as a Python library to programmatically interact with MDRS repositories.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from mdrsclient.client import MdrsClient
|
||||||
|
from mdrsclient.cache import InMemoryCache
|
||||||
|
from mdrsclient.config import InMemoryConfig
|
||||||
|
|
||||||
|
# 1. Setup in-memory configuration and cache to avoid local state files (e.g., config.ini, cache/*.json)
|
||||||
|
config = InMemoryConfig("neurodata")
|
||||||
|
config.url = "https://neurodata.riken.jp/api"
|
||||||
|
|
||||||
|
cache = InMemoryCache()
|
||||||
|
|
||||||
|
# 2. Initialize client with custom configuration and cache
|
||||||
|
client = MdrsClient.from_remote("neurodata", cache=cache, config=config)
|
||||||
|
|
||||||
|
# 3. Login to the remote server
|
||||||
|
client.login("username", "password")
|
||||||
|
|
||||||
|
# 4. Use service methods
|
||||||
|
labs = client.get_laboratories()
|
||||||
|
metadata = client.metadata("neurodata:/NIU/Repository/")
|
||||||
|
|
||||||
|
# Transfer files programmatically
|
||||||
|
client.upload("/path/to/local/data", "neurodata:/NIU/Repository/TEST/", is_recursive=True)
|
||||||
|
client.download("neurodata:/NIU/Repository/TEST/data", "/path/to/local", is_recursive=True)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
You can run the unit test suite using the standard library `unittest` discover runner:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
.venv/bin/python -m unittest discover tests
|
||||||
|
```
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
See [CHANGELOG.md](./CHANGELOG.md) for the full change history.
|
||||||
|
|
||||||
|
|||||||
@@ -1 +0,0 @@
|
|||||||
1.3.6
|
|
||||||
@@ -1,3 +1,4 @@
|
|||||||
from mdrsclient.__version__ import __version__
|
from mdrsclient.__version__ import __version__
|
||||||
|
from mdrsclient.client import MdrsClient
|
||||||
|
|
||||||
__all__ = ["__version__"]
|
__all__ = ["__version__", "MdrsClient"]
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
import argparse
|
import argparse
|
||||||
import sys
|
import sys
|
||||||
|
from json import JSONDecodeError
|
||||||
|
|
||||||
from mdrsclient.commands import (
|
from mdrsclient.commands import (
|
||||||
ChaclCommand,
|
ChaclCommand,
|
||||||
@@ -16,6 +17,7 @@ from mdrsclient.commands import (
|
|||||||
MvCommand,
|
MvCommand,
|
||||||
RmCommand,
|
RmCommand,
|
||||||
UploadCommand,
|
UploadCommand,
|
||||||
|
VersionCommand,
|
||||||
WhoamiCommand,
|
WhoamiCommand,
|
||||||
)
|
)
|
||||||
from mdrsclient.exceptions import MDRSException
|
from mdrsclient.exceptions import MDRSException
|
||||||
@@ -28,6 +30,7 @@ def main() -> None:
|
|||||||
parsers = parser.add_subparsers(title="subcommands")
|
parsers = parser.add_subparsers(title="subcommands")
|
||||||
|
|
||||||
ConfigCommand.register(parsers)
|
ConfigCommand.register(parsers)
|
||||||
|
VersionCommand.register(parsers)
|
||||||
LoginCommand.register(parsers)
|
LoginCommand.register(parsers)
|
||||||
LogoutCommand.register(parsers)
|
LogoutCommand.register(parsers)
|
||||||
WhoamiCommand.register(parsers)
|
WhoamiCommand.register(parsers)
|
||||||
@@ -52,6 +55,9 @@ def main() -> None:
|
|||||||
except MDRSException as e:
|
except MDRSException as e:
|
||||||
print(f"Error: {e}")
|
print(f"Error: {e}")
|
||||||
sys.exit(2)
|
sys.exit(2)
|
||||||
|
except JSONDecodeError:
|
||||||
|
print("Unexpected response returned. Please check the configuration or the server's operational status.")
|
||||||
|
sys.exit(2)
|
||||||
except KeyboardInterrupt:
|
except KeyboardInterrupt:
|
||||||
sys.exit(130)
|
sys.exit(130)
|
||||||
|
|
||||||
|
|||||||
@@ -1,8 +1,9 @@
|
|||||||
import os
|
import importlib.metadata
|
||||||
|
|
||||||
here = os.path.realpath(os.path.dirname(__file__))
|
try:
|
||||||
|
__version__ = importlib.metadata.version("mdrs-client-python")
|
||||||
|
except importlib.metadata.PackageNotFoundError:
|
||||||
|
__version__ = "0.0.0-dev"
|
||||||
|
|
||||||
with open(os.path.join(here, "VERSION")) as version_file:
|
|
||||||
__version__ = version_file.read().strip()
|
|
||||||
|
|
||||||
__all__ = ["__version__"]
|
__all__ = ["__version__"]
|
||||||
|
|||||||
@@ -1,9 +1,11 @@
|
|||||||
|
from mdrsclient.api.doi import DoiApi
|
||||||
from mdrsclient.api.files import FilesApi
|
from mdrsclient.api.files import FilesApi
|
||||||
from mdrsclient.api.folders import FoldersApi
|
from mdrsclient.api.folders import FoldersApi
|
||||||
from mdrsclient.api.laboratories import LaboratoriesApi
|
from mdrsclient.api.laboratories import LaboratoriesApi
|
||||||
from mdrsclient.api.users import UsersApi
|
from mdrsclient.api.users import UsersApi
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
|
"DoiApi",
|
||||||
"FilesApi",
|
"FilesApi",
|
||||||
"FoldersApi",
|
"FoldersApi",
|
||||||
"LaboratoriesApi",
|
"LaboratoriesApi",
|
||||||
|
|||||||
@@ -5,12 +5,7 @@ from pydantic import TypeAdapter
|
|||||||
from requests import Response
|
from requests import Response
|
||||||
|
|
||||||
from mdrsclient.connection import MDRSConnection
|
from mdrsclient.connection import MDRSConnection
|
||||||
from mdrsclient.exceptions import (
|
from mdrsclient.exceptions import BadRequestException, ForbiddenException, UnauthorizedException, UnexpectedException
|
||||||
BadRequestException,
|
|
||||||
ForbiddenException,
|
|
||||||
UnauthorizedException,
|
|
||||||
UnexpectedException,
|
|
||||||
)
|
|
||||||
from mdrsclient.models.error import DRFStandardizedErrors
|
from mdrsclient.models.error import DRFStandardizedErrors
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,33 @@
|
|||||||
|
from typing import Final
|
||||||
|
|
||||||
|
from pydantic import TypeAdapter
|
||||||
|
from pydantic.dataclasses import dataclass
|
||||||
|
|
||||||
|
from mdrsclient.api.base import BaseApi
|
||||||
|
from mdrsclient.api.utils import token_check
|
||||||
|
from mdrsclient.models.doi import Doi
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class DoiRetrieveFolderRef:
|
||||||
|
id: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class DoiRetrieveResponse:
|
||||||
|
id: str
|
||||||
|
doi: str
|
||||||
|
folder: DoiRetrieveFolderRef
|
||||||
|
|
||||||
|
|
||||||
|
class DoiApi(BaseApi):
|
||||||
|
ENTRYPOINT: Final[str] = "v3/doi/"
|
||||||
|
|
||||||
|
def retrieve(self, doi_id: str) -> Doi:
|
||||||
|
"""Retrieve the folder associated with a DOI suffix ID (GET v3/doi/{id}/)."""
|
||||||
|
url = self.ENTRYPOINT + doi_id + "/"
|
||||||
|
token_check(self.connection)
|
||||||
|
response = self.connection.get(url)
|
||||||
|
self._raise_response_error(response)
|
||||||
|
api_resp = TypeAdapter(DoiRetrieveResponse).validate_python(response.json())
|
||||||
|
return Doi(id=api_resp.id, doi=api_resp.doi, folder_id=api_resp.folder.id)
|
||||||
+24
-2
@@ -1,6 +1,7 @@
|
|||||||
import mimetypes
|
import mimetypes
|
||||||
import os
|
import os
|
||||||
from typing import Any, Final
|
from typing import Any, Final
|
||||||
|
from unicodedata import normalize
|
||||||
|
|
||||||
from pydantic import TypeAdapter
|
from pydantic import TypeAdapter
|
||||||
from pydantic.dataclasses import dataclass
|
from pydantic.dataclasses import dataclass
|
||||||
@@ -17,10 +18,26 @@ class FilesApiCreateResponse:
|
|||||||
id: str
|
id: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class FilesApiListResponse:
|
||||||
|
count: int
|
||||||
|
next: str | None
|
||||||
|
previous: str | None
|
||||||
|
results: list[File]
|
||||||
|
|
||||||
|
|
||||||
class FilesApi(BaseApi):
|
class FilesApi(BaseApi):
|
||||||
ENTRYPOINT: Final[str] = "v3/files/"
|
ENTRYPOINT: Final[str] = "v3/files/"
|
||||||
FALLBACK_MIMETYPE: Final[str] = "application/octet-stream"
|
FALLBACK_MIMETYPE: Final[str] = "application/octet-stream"
|
||||||
|
|
||||||
|
def list(self, folder_id: str, page_num: int) -> FilesApiListResponse:
|
||||||
|
url = self.ENTRYPOINT
|
||||||
|
token_check(self.connection)
|
||||||
|
params: dict[str, str | int] = {"folder_id": folder_id, "page": page_num}
|
||||||
|
response = self.connection.get(url, params=params)
|
||||||
|
self._raise_response_error(response)
|
||||||
|
return TypeAdapter(FilesApiListResponse).validate_python(response.json())
|
||||||
|
|
||||||
def retrieve(self, id: str) -> File:
|
def retrieve(self, id: str) -> File:
|
||||||
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
|
# print(self.__class__.__name__ + "::" + sys._getframe().f_code.co_name)
|
||||||
url = self.ENTRYPOINT + id + "/"
|
url = self.ENTRYPOINT + id + "/"
|
||||||
@@ -37,7 +54,10 @@ class FilesApi(BaseApi):
|
|||||||
try:
|
try:
|
||||||
with open(os.path.realpath(path), mode="rb") as fp:
|
with open(os.path.realpath(path), mode="rb") as fp:
|
||||||
data = MultipartEncoder(
|
data = MultipartEncoder(
|
||||||
fields={"folder_id": folder_id, "file": (os.path.basename(path), fp, self._get_mime_type(path))}
|
fields={
|
||||||
|
"folder_id": folder_id,
|
||||||
|
"file": (normalize("NFC", os.path.basename(path)), fp, self._get_mime_type(path)),
|
||||||
|
}
|
||||||
)
|
)
|
||||||
response = self.connection.post(url, data=data, headers={"Content-Type": data.content_type})
|
response = self.connection.post(url, data=data, headers={"Content-Type": data.content_type})
|
||||||
self._raise_response_error(response)
|
self._raise_response_error(response)
|
||||||
@@ -59,7 +79,9 @@ class FilesApi(BaseApi):
|
|||||||
# update file body
|
# update file body
|
||||||
try:
|
try:
|
||||||
with open(os.path.realpath(path), mode="rb") as fp:
|
with open(os.path.realpath(path), mode="rb") as fp:
|
||||||
data = MultipartEncoder(fields={"file": (os.path.basename(path), fp, self._get_mime_type(path))})
|
data = MultipartEncoder(
|
||||||
|
fields={"file": (normalize("NFC", os.path.basename(path)), fp, self._get_mime_type(path))}
|
||||||
|
)
|
||||||
response = self.connection.put(url, data=data, headers={"Content-Type": data.content_type})
|
response = self.connection.put(url, data=data, headers={"Content-Type": data.content_type})
|
||||||
except OSError:
|
except OSError:
|
||||||
raise UnexpectedException(f"Could not open `{path}` file.")
|
raise UnexpectedException(f"Could not open `{path}` file.")
|
||||||
|
|||||||
@@ -20,8 +20,10 @@ class UsersCurrentResponseLaboratory:
|
|||||||
class UsersApiCurrentResponse:
|
class UsersApiCurrentResponse:
|
||||||
id: int
|
id: int
|
||||||
username: str
|
username: str
|
||||||
full_name: str
|
first_name: str
|
||||||
|
last_name: str
|
||||||
email: str
|
email: str
|
||||||
|
orcid_id: str
|
||||||
laboratories: list[UsersCurrentResponseLaboratory]
|
laboratories: list[UsersCurrentResponseLaboratory]
|
||||||
is_staff: bool
|
is_staff: bool
|
||||||
is_active: bool
|
is_active: bool
|
||||||
|
|||||||
+64
-2
@@ -2,6 +2,7 @@ import dataclasses
|
|||||||
import hashlib
|
import hashlib
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
from typing import Protocol, runtime_checkable
|
||||||
|
|
||||||
from pydantic import TypeAdapter, ValidationError
|
from pydantic import TypeAdapter, ValidationError
|
||||||
from pydantic.dataclasses import dataclass
|
from pydantic.dataclasses import dataclass
|
||||||
@@ -16,7 +17,7 @@ from mdrsclient.utils import FileLock
|
|||||||
class CacheData:
|
class CacheData:
|
||||||
user: User | None = None
|
user: User | None = None
|
||||||
token: Token | None = None
|
token: Token | None = None
|
||||||
laboratories: Laboratories = Laboratories()
|
laboratories: Laboratories = dataclasses.field(default_factory=Laboratories)
|
||||||
digest: str = ""
|
digest: str = ""
|
||||||
|
|
||||||
def clear(self) -> None:
|
def clear(self) -> None:
|
||||||
@@ -43,7 +44,68 @@ class CacheData:
|
|||||||
).hexdigest()
|
).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
class CacheFile:
|
@runtime_checkable
|
||||||
|
class CacheInterface(Protocol):
|
||||||
|
@property
|
||||||
|
def token(self) -> Token | None: ...
|
||||||
|
@token.setter
|
||||||
|
def token(self, token: Token) -> None: ...
|
||||||
|
@token.deleter
|
||||||
|
def token(self) -> None: ...
|
||||||
|
|
||||||
|
@property
|
||||||
|
def user(self) -> User | None: ...
|
||||||
|
@user.setter
|
||||||
|
def user(self, user: User) -> None: ...
|
||||||
|
@user.deleter
|
||||||
|
def user(self) -> None: ...
|
||||||
|
|
||||||
|
@property
|
||||||
|
def laboratories(self) -> Laboratories: ...
|
||||||
|
@laboratories.setter
|
||||||
|
def laboratories(self, laboratories: Laboratories) -> None: ...
|
||||||
|
|
||||||
|
|
||||||
|
class InMemoryCache(CacheInterface):
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.__data = CacheData()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def token(self) -> Token | None:
|
||||||
|
return self.__data.token
|
||||||
|
|
||||||
|
@token.setter
|
||||||
|
def token(self, token: Token) -> None:
|
||||||
|
self.__data.token = token
|
||||||
|
|
||||||
|
@token.deleter
|
||||||
|
def token(self) -> None:
|
||||||
|
if self.__data.token is not None:
|
||||||
|
self.__data.token = None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def user(self) -> User | None:
|
||||||
|
return self.__data.user
|
||||||
|
|
||||||
|
@user.setter
|
||||||
|
def user(self, user: User) -> None:
|
||||||
|
self.__data.user = user
|
||||||
|
|
||||||
|
@user.deleter
|
||||||
|
def user(self) -> None:
|
||||||
|
if self.__data.user is not None:
|
||||||
|
self.__data.user = None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def laboratories(self) -> Laboratories:
|
||||||
|
return self.__data.laboratories
|
||||||
|
|
||||||
|
@laboratories.setter
|
||||||
|
def laboratories(self, laboratories: Laboratories) -> None:
|
||||||
|
self.__data.laboratories = laboratories
|
||||||
|
|
||||||
|
|
||||||
|
class CacheFile(CacheInterface):
|
||||||
__serial: int
|
__serial: int
|
||||||
__cache_dir: str
|
__cache_dir: str
|
||||||
__cache_file: str
|
__cache_file: str
|
||||||
|
|||||||
@@ -0,0 +1,239 @@
|
|||||||
|
import os
|
||||||
|
from typing import Any
|
||||||
|
from unicodedata import normalize
|
||||||
|
|
||||||
|
from mdrsclient.api import DoiApi, FilesApi, FoldersApi, LaboratoriesApi, UsersApi
|
||||||
|
from mdrsclient.cache import CacheInterface
|
||||||
|
from mdrsclient.config import ConfigInterface
|
||||||
|
from mdrsclient.connection import MDRSConnection
|
||||||
|
from mdrsclient.exceptions import IllegalArgumentException, MDRSException, UnauthorizedException, UnexpectedException
|
||||||
|
from mdrsclient.models import File, Folder, Laboratory, Token, User
|
||||||
|
from mdrsclient.models.file import find_file
|
||||||
|
from mdrsclient.services import MdrsService
|
||||||
|
|
||||||
|
|
||||||
|
class MdrsClient(MdrsService):
|
||||||
|
"""Service layer client for MDRS."""
|
||||||
|
|
||||||
|
def __init__(self, connection: MDRSConnection, config_class: type[ConfigInterface] | None = None):
|
||||||
|
super().__init__(connection, config_class)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_remote(
|
||||||
|
cls, remote: str, cache: CacheInterface | None = None, config: ConfigInterface | None = None
|
||||||
|
) -> "MdrsClient":
|
||||||
|
return cls(cls.create_connection(remote, cache, config))
|
||||||
|
|
||||||
|
def mkdir(self, remote_path: str) -> None:
|
||||||
|
remote, laboratory_name, r_path = self.parse_remote_host_with_path(remote_path)
|
||||||
|
r_path = r_path.rstrip("/")
|
||||||
|
r_dirname = os.path.dirname(r_path)
|
||||||
|
r_basename = os.path.basename(r_path)
|
||||||
|
laboratory = self.find_laboratory(laboratory_name)
|
||||||
|
parent_folder = self.find_folder(laboratory, r_dirname)
|
||||||
|
files = self.find_files(parent_folder.id)
|
||||||
|
if parent_folder.find_sub_folder(r_basename) is not None or find_file(files, r_basename) is not None:
|
||||||
|
raise IllegalArgumentException(f"Cannot create folder `{r_path}`: File exists.")
|
||||||
|
folder_api = FoldersApi(self.connection)
|
||||||
|
folder_api.create(normalize("NFC", r_basename), parent_folder.id)
|
||||||
|
|
||||||
|
def rm(self, remote_path: str, is_recursive: bool = False) -> None:
|
||||||
|
remote, laboratory_name, r_path = self.parse_remote_host_with_path(remote_path)
|
||||||
|
r_path = r_path.rstrip("/")
|
||||||
|
r_dirname = os.path.dirname(r_path)
|
||||||
|
r_basename = os.path.basename(r_path)
|
||||||
|
laboratory = self.find_laboratory(laboratory_name)
|
||||||
|
parent_folder = self.find_folder(laboratory, r_dirname)
|
||||||
|
parent_files = self.find_files(parent_folder.id)
|
||||||
|
file = find_file(parent_files, r_basename)
|
||||||
|
if file is not None:
|
||||||
|
file_api = FilesApi(self.connection)
|
||||||
|
file_api.destroy(file)
|
||||||
|
else:
|
||||||
|
folder = parent_folder.find_sub_folder(r_basename)
|
||||||
|
if folder is None:
|
||||||
|
raise IllegalArgumentException(f"Cannot remove `{r_path}`: No such file or folder.")
|
||||||
|
if not is_recursive:
|
||||||
|
raise IllegalArgumentException(f"Cannot remove `{r_path}`: Is a folder.")
|
||||||
|
folder_api = FoldersApi(self.connection)
|
||||||
|
folder_api.destroy(folder.id, True)
|
||||||
|
|
||||||
|
def ls(self, remote_path: str, password: str | None = None) -> tuple[Folder, list[File]]:
|
||||||
|
folder, laboratory = self.resolve_folder(remote_path, password)
|
||||||
|
files = self.find_files(folder.id)
|
||||||
|
return folder, files
|
||||||
|
|
||||||
|
def cp(self, src_path: str, dest_path: str, is_recursive: bool = False) -> None:
|
||||||
|
s_remote, s_laboratory_name, s_path = self.parse_remote_host_with_path(src_path)
|
||||||
|
d_remote, d_laboratory_name, d_path = self.parse_remote_host_with_path(dest_path)
|
||||||
|
if s_remote != d_remote:
|
||||||
|
raise IllegalArgumentException("Remote host mismatched.")
|
||||||
|
if s_laboratory_name != d_laboratory_name:
|
||||||
|
raise IllegalArgumentException("Laboratory mismatched.")
|
||||||
|
s_path = s_path.rstrip("/")
|
||||||
|
s_dirname = os.path.dirname(s_path)
|
||||||
|
s_basename = os.path.basename(s_path)
|
||||||
|
if d_path.endswith("/"):
|
||||||
|
d_dirname = d_path
|
||||||
|
d_basename = s_basename
|
||||||
|
else:
|
||||||
|
d_dirname = os.path.dirname(d_path)
|
||||||
|
d_basename = os.path.basename(d_path)
|
||||||
|
laboratory = self.find_laboratory(s_laboratory_name)
|
||||||
|
s_parent_folder = self.find_folder(laboratory, s_dirname)
|
||||||
|
s_parent_files = self.find_files(s_parent_folder.id)
|
||||||
|
d_parent_folder = self.find_folder(laboratory, d_dirname)
|
||||||
|
d_parent_files = self.find_files(d_parent_folder.id)
|
||||||
|
s_file = find_file(s_parent_files, s_basename)
|
||||||
|
if s_file is not None:
|
||||||
|
d_file = find_file(d_parent_files, d_basename)
|
||||||
|
if d_file is not None:
|
||||||
|
raise IllegalArgumentException(f"File `{d_basename}` already exists.")
|
||||||
|
d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
|
||||||
|
if d_sub_folder is not None:
|
||||||
|
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
|
||||||
|
file_api = FilesApi(self.connection)
|
||||||
|
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
|
||||||
|
file_api.copy(s_file, d_parent_folder.id, normalize("NFC", d_basename))
|
||||||
|
else:
|
||||||
|
s_folder = s_parent_folder.find_sub_folder(s_basename)
|
||||||
|
if s_folder is None:
|
||||||
|
raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
|
||||||
|
if not is_recursive:
|
||||||
|
raise IllegalArgumentException(f"Cannot copy `{s_path}`: Is a folder.")
|
||||||
|
if find_file(d_parent_files, d_basename) is not None:
|
||||||
|
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
|
||||||
|
d_folder = d_parent_folder.find_sub_folder(d_basename)
|
||||||
|
if d_folder is not None:
|
||||||
|
if d_folder.id == s_folder.id:
|
||||||
|
raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
|
||||||
|
raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
|
||||||
|
folder_api = FoldersApi(self.connection)
|
||||||
|
if s_parent_folder.id != d_parent_folder.id or s_basename != d_basename:
|
||||||
|
folder_api.copy(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
|
||||||
|
|
||||||
|
def mv(self, src_path: str, dest_path: str) -> None:
|
||||||
|
s_remote, s_laboratory_name, s_path = self.parse_remote_host_with_path(src_path)
|
||||||
|
d_remote, d_laboratory_name, d_path = self.parse_remote_host_with_path(dest_path)
|
||||||
|
if s_remote != d_remote:
|
||||||
|
raise IllegalArgumentException("Remote host mismatched.")
|
||||||
|
if s_laboratory_name != d_laboratory_name:
|
||||||
|
raise IllegalArgumentException("Laboratory mismatched.")
|
||||||
|
s_path = s_path.rstrip("/")
|
||||||
|
s_dirname = os.path.dirname(s_path)
|
||||||
|
s_basename = os.path.basename(s_path)
|
||||||
|
if d_path.endswith("/"):
|
||||||
|
d_dirname = d_path
|
||||||
|
d_basename = s_basename
|
||||||
|
else:
|
||||||
|
d_dirname = os.path.dirname(d_path)
|
||||||
|
d_basename = os.path.basename(d_path)
|
||||||
|
laboratory = self.find_laboratory(s_laboratory_name)
|
||||||
|
s_parent_folder = self.find_folder(laboratory, s_dirname)
|
||||||
|
s_parent_files = self.find_files(s_parent_folder.id)
|
||||||
|
d_parent_folder = self.find_folder(laboratory, d_dirname)
|
||||||
|
d_parent_files = self.find_files(d_parent_folder.id)
|
||||||
|
s_file = find_file(s_parent_files, s_basename)
|
||||||
|
if s_file is not None:
|
||||||
|
d_file = find_file(d_parent_files, d_basename)
|
||||||
|
if d_file is not None:
|
||||||
|
raise IllegalArgumentException(f"File `{d_basename}` already exists.")
|
||||||
|
d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
|
||||||
|
if d_sub_folder is not None:
|
||||||
|
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
|
||||||
|
file_api = FilesApi(self.connection)
|
||||||
|
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
|
||||||
|
file_api.move(s_file, d_parent_folder.id, normalize("NFC", d_basename))
|
||||||
|
else:
|
||||||
|
s_folder = s_parent_folder.find_sub_folder(s_basename)
|
||||||
|
if s_folder is None:
|
||||||
|
raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
|
||||||
|
if find_file(d_parent_files, d_basename) is not None:
|
||||||
|
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
|
||||||
|
d_folder = d_parent_folder.find_sub_folder(d_basename)
|
||||||
|
if d_folder is not None:
|
||||||
|
if d_folder.id == s_folder.id:
|
||||||
|
raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
|
||||||
|
raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
|
||||||
|
folder_api = FoldersApi(self.connection)
|
||||||
|
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
|
||||||
|
folder_api.move(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
|
||||||
|
|
||||||
|
def chacl(
|
||||||
|
self, remote_path: str, access_level: int, is_recursive: bool = False, password: str | None = None
|
||||||
|
) -> None:
|
||||||
|
remote, laboratory_name, r_path = self.parse_remote_host_with_path(remote_path)
|
||||||
|
r_path = r_path.rstrip("/")
|
||||||
|
laboratory = self.find_laboratory(laboratory_name)
|
||||||
|
folder = self.find_folder(laboratory, r_path)
|
||||||
|
folder_api = FoldersApi(self.connection)
|
||||||
|
folder_api.acl(folder.id, access_level, is_recursive, password)
|
||||||
|
|
||||||
|
def metadata(self, remote_path: str, password: str | None = None) -> dict:
|
||||||
|
folder, laboratory = self.resolve_folder(remote_path, password)
|
||||||
|
folder_api = FoldersApi(self.connection)
|
||||||
|
return folder_api.metadata(folder.id)
|
||||||
|
|
||||||
|
def file_metadata(self, remote_path: str, password: str | None = None) -> dict:
|
||||||
|
folder, laboratory, r_basename = self.resolve_file(remote_path, password)
|
||||||
|
files = self.find_files(folder.id)
|
||||||
|
file = find_file(files, r_basename)
|
||||||
|
if file is None:
|
||||||
|
raise IllegalArgumentException(f"File `{r_basename}` not found.")
|
||||||
|
file_api = FilesApi(self.connection)
|
||||||
|
return file_api.metadata(file)
|
||||||
|
|
||||||
|
def upload(
|
||||||
|
self, local_path: str, remote_path: str, is_recursive: bool = False, is_skip_if_exists: bool = False
|
||||||
|
) -> None:
|
||||||
|
from mdrsclient.transfer import Uploader
|
||||||
|
|
||||||
|
uploader = Uploader(self)
|
||||||
|
uploader.upload(local_path, remote_path, is_recursive, is_skip_if_exists)
|
||||||
|
|
||||||
|
def download(
|
||||||
|
self,
|
||||||
|
remote_path: str,
|
||||||
|
local_path: str,
|
||||||
|
is_recursive: bool = False,
|
||||||
|
is_skip_if_exists: bool = False,
|
||||||
|
password: str | None = None,
|
||||||
|
excludes: list[str] | None = None,
|
||||||
|
) -> None:
|
||||||
|
from mdrsclient.transfer import Downloader
|
||||||
|
|
||||||
|
downloader = Downloader(self)
|
||||||
|
downloader.download(remote_path, local_path, is_recursive, is_skip_if_exists, password, excludes)
|
||||||
|
|
||||||
|
def version(self) -> str:
|
||||||
|
from mdrsclient.__version__ import __version__
|
||||||
|
|
||||||
|
return f"mdrs {__version__}"
|
||||||
|
|
||||||
|
def config_create(self, remote: str, url: str) -> None:
|
||||||
|
remote = self.parse_remote_host(remote)
|
||||||
|
config = self.config_class(remote)
|
||||||
|
if config.url is not None:
|
||||||
|
raise IllegalArgumentException(f"Remote host `{remote}` is already exists.")
|
||||||
|
else:
|
||||||
|
config.url = url
|
||||||
|
|
||||||
|
def config_update(self, remote: str, url: str) -> None:
|
||||||
|
remote = self.parse_remote_host(remote)
|
||||||
|
config = self.config_class(remote)
|
||||||
|
if config.url is None:
|
||||||
|
raise IllegalArgumentException(f"Remote host `{remote}` is not exists.")
|
||||||
|
else:
|
||||||
|
config.url = url
|
||||||
|
|
||||||
|
def config_list(self) -> list:
|
||||||
|
config = self.config_class("")
|
||||||
|
return config.list()
|
||||||
|
|
||||||
|
def config_delete(self, remote: str) -> None:
|
||||||
|
remote = self.parse_remote_host(remote)
|
||||||
|
config = self.config_class(remote)
|
||||||
|
if config.url is None:
|
||||||
|
raise IllegalArgumentException(f"Remote host `{remote}` is not exists.")
|
||||||
|
else:
|
||||||
|
del config.url
|
||||||
@@ -12,6 +12,7 @@ from mdrsclient.commands.mkdir import MkdirCommand
|
|||||||
from mdrsclient.commands.mv import MvCommand
|
from mdrsclient.commands.mv import MvCommand
|
||||||
from mdrsclient.commands.rm import RmCommand
|
from mdrsclient.commands.rm import RmCommand
|
||||||
from mdrsclient.commands.upload import UploadCommand
|
from mdrsclient.commands.upload import UploadCommand
|
||||||
|
from mdrsclient.commands.version import VersionCommand
|
||||||
from mdrsclient.commands.whoami import WhoamiCommand
|
from mdrsclient.commands.whoami import WhoamiCommand
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
@@ -29,5 +30,6 @@ __all__ = [
|
|||||||
"MvCommand",
|
"MvCommand",
|
||||||
"RmCommand",
|
"RmCommand",
|
||||||
"UploadCommand",
|
"UploadCommand",
|
||||||
|
"VersionCommand",
|
||||||
"WhoamiCommand",
|
"WhoamiCommand",
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -1,18 +1,7 @@
|
|||||||
import re
|
|
||||||
from abc import ABC, abstractmethod
|
from abc import ABC, abstractmethod
|
||||||
from typing import Any
|
from typing import Any
|
||||||
from unicodedata import normalize
|
|
||||||
|
|
||||||
from mdrsclient.api import FoldersApi, LaboratoriesApi
|
from mdrsclient.exceptions import UnexpectedException
|
||||||
from mdrsclient.config import ConfigFile
|
|
||||||
from mdrsclient.connection import MDRSConnection
|
|
||||||
from mdrsclient.exceptions import (
|
|
||||||
IllegalArgumentException,
|
|
||||||
MissingConfigurationException,
|
|
||||||
UnauthorizedException,
|
|
||||||
UnexpectedException,
|
|
||||||
)
|
|
||||||
from mdrsclient.models import Folder, Laboratory
|
|
||||||
|
|
||||||
|
|
||||||
class BaseCommand(ABC):
|
class BaseCommand(ABC):
|
||||||
@@ -20,64 +9,3 @@ class BaseCommand(ABC):
|
|||||||
@abstractmethod
|
@abstractmethod
|
||||||
def register(cls, parsers: Any) -> None:
|
def register(cls, parsers: Any) -> None:
|
||||||
raise UnexpectedException("Not implemented.")
|
raise UnexpectedException("Not implemented.")
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def _create_connection(cls, remote: str) -> MDRSConnection:
|
|
||||||
config = ConfigFile(remote)
|
|
||||||
if config.url is None:
|
|
||||||
raise MissingConfigurationException(f"Remote host `{remote}` is not found.")
|
|
||||||
return MDRSConnection(config.remote, config.url)
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def _find_laboratory(cls, connection: MDRSConnection, name: str) -> Laboratory:
|
|
||||||
if connection.laboratories.empty() or connection.token is not None and connection.token.is_expired:
|
|
||||||
laboratory_api = LaboratoriesApi(connection)
|
|
||||||
connection.laboratories = laboratory_api.list()
|
|
||||||
laboratory = connection.laboratories.find_by_name(name)
|
|
||||||
if laboratory is None:
|
|
||||||
raise IllegalArgumentException(f"Laboratory `{name}` not found.")
|
|
||||||
return laboratory
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def _find_folder(
|
|
||||||
cls, connection: MDRSConnection, laboratory: Laboratory, path: str, password: str | None = None
|
|
||||||
) -> Folder:
|
|
||||||
folder_api = FoldersApi(connection)
|
|
||||||
folders = folder_api.list(laboratory.id, normalize("NFC", path))
|
|
||||||
if len(folders) != 1:
|
|
||||||
raise UnexpectedException(f"Folder `{path}` not found.")
|
|
||||||
if folders[0].lock:
|
|
||||||
if password is None:
|
|
||||||
raise UnauthorizedException(f"Folder `{path}` is locked.")
|
|
||||||
folder_api.auth(folders[0].id, password)
|
|
||||||
return folder_api.retrieve(folders[0].id)
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def _parse_remote_host(cls, path: str) -> str:
|
|
||||||
path_array = path.split(":")
|
|
||||||
remote_host = path_array[0]
|
|
||||||
if len(path_array) == 2 and path_array[1] != "" or len(path_array) > 2:
|
|
||||||
raise IllegalArgumentException("Invalid remote host")
|
|
||||||
return remote_host
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def _parse_remote_host_with_path(cls, path: str) -> tuple[str, str, str]:
|
|
||||||
path = re.sub(r"//+|/\./+|/\.$", "/", path)
|
|
||||||
if re.search(r"/\.\./|/\.\.$", path) is not None:
|
|
||||||
raise IllegalArgumentException("Path traversal found.")
|
|
||||||
path_array = path.split(":")
|
|
||||||
if len(path_array) != 2:
|
|
||||||
raise IllegalArgumentException("Invalid remote host.")
|
|
||||||
remote_host = path_array[0]
|
|
||||||
folder_array = path_array[1].split("/")
|
|
||||||
is_absolute_path = folder_array[0] == ""
|
|
||||||
if not is_absolute_path:
|
|
||||||
raise IllegalArgumentException("Must be absolute paths.")
|
|
||||||
del folder_array[0]
|
|
||||||
if len(folder_array) == 0:
|
|
||||||
laboratory = ""
|
|
||||||
folder = ""
|
|
||||||
else:
|
|
||||||
laboratory = folder_array.pop(0)
|
|
||||||
folder = "/" + "/".join(folder_array)
|
|
||||||
return (remote_host, laboratory, folder)
|
|
||||||
|
|||||||
@@ -31,10 +31,8 @@ class ChaclCommand(BaseCommand):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def chacl(cls, remote_path: str, access_level: int, is_recursive: bool, password: str | None) -> None:
|
def chacl(cls, remote_path: str, access_level: int, is_recursive: bool, password: str | None) -> None:
|
||||||
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
|
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
|
||||||
r_path = r_path.rstrip("/")
|
from mdrsclient.client import MdrsClient
|
||||||
connection = cls._create_connection(remote)
|
|
||||||
laboratory = cls._find_laboratory(connection, laboratory_name)
|
client = MdrsClient.from_remote(remote)
|
||||||
folder = cls._find_folder(connection, laboratory, r_path)
|
client.chacl(remote_path, access_level, is_recursive, password)
|
||||||
folder_api = FoldersApi(connection)
|
|
||||||
folder_api.acl(folder.id, access_level, is_recursive, password)
|
|
||||||
|
|||||||
@@ -2,8 +2,6 @@ from argparse import Namespace
|
|||||||
from typing import Any, Callable
|
from typing import Any, Callable
|
||||||
|
|
||||||
from mdrsclient.commands.base import BaseCommand
|
from mdrsclient.commands.base import BaseCommand
|
||||||
from mdrsclient.config import ConfigFile
|
|
||||||
from mdrsclient.exceptions import IllegalArgumentException
|
|
||||||
|
|
||||||
|
|
||||||
class ConfigCommand(BaseCommand):
|
class ConfigCommand(BaseCommand):
|
||||||
@@ -26,10 +24,11 @@ class ConfigCommand(BaseCommand):
|
|||||||
update_parser.set_defaults(func=cls.func_update)
|
update_parser.set_defaults(func=cls.func_update)
|
||||||
# config list
|
# config list
|
||||||
list_parser = config_parsers.add_parser("list", help="list all the remote hosts", aliases=["ls"])
|
list_parser = config_parsers.add_parser("list", help="list all the remote hosts", aliases=["ls"])
|
||||||
list_parser.add_argument("-l", "--long", help="show the api url", action="store_true")
|
|
||||||
list_parser.set_defaults(func=cls.func_list)
|
list_parser.set_defaults(func=cls.func_list)
|
||||||
# config delete
|
# config delete
|
||||||
delete_parser = config_parsers.add_parser("delete", help="delete an existing remote host", aliases=["remove"])
|
delete_parser = config_parsers.add_parser(
|
||||||
|
"delete", help="delete an existing remote host", aliases=["remove", "rm"]
|
||||||
|
)
|
||||||
delete_parser.add_argument("remote", help="label of remote host")
|
delete_parser.add_argument("remote", help="label of remote host")
|
||||||
delete_parser.set_defaults(func=cls.func_delete)
|
delete_parser.set_defaults(func=cls.func_delete)
|
||||||
|
|
||||||
@@ -37,56 +36,29 @@ class ConfigCommand(BaseCommand):
|
|||||||
def func_create(cls, args: Namespace) -> None:
|
def func_create(cls, args: Namespace) -> None:
|
||||||
remote = str(args.remote)
|
remote = str(args.remote)
|
||||||
url = str(args.url)
|
url = str(args.url)
|
||||||
cls.create(remote, url)
|
from mdrsclient.client import MdrsClient
|
||||||
|
|
||||||
|
MdrsClient(None).config_create(remote, url)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def func_update(cls, args: Namespace) -> None:
|
def func_update(cls, args: Namespace) -> None:
|
||||||
remote = str(args.remote)
|
remote = str(args.remote)
|
||||||
url = str(args.url)
|
url = str(args.url)
|
||||||
cls.update(remote, url)
|
from mdrsclient.client import MdrsClient
|
||||||
|
|
||||||
|
MdrsClient(None).config_update(remote, url)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def func_list(cls, args: Namespace) -> None:
|
def func_list(cls, args: Namespace) -> None:
|
||||||
is_long = bool(args.long)
|
from mdrsclient.client import MdrsClient
|
||||||
cls.list(is_long)
|
|
||||||
|
client = MdrsClient(None)
|
||||||
|
for remote, url in client.config_list():
|
||||||
|
print(f"{remote}:\t{url}")
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def func_delete(cls, args: Namespace) -> None:
|
def func_delete(cls, args: Namespace) -> None:
|
||||||
remote = str(args.remote)
|
remote = str(args.remote)
|
||||||
cls.delete(remote)
|
from mdrsclient.client import MdrsClient
|
||||||
|
|
||||||
@classmethod
|
MdrsClient(None).config_delete(remote)
|
||||||
def create(cls, remote: str, url: str) -> None:
|
|
||||||
remote = cls._parse_remote_host(remote)
|
|
||||||
config = ConfigFile(remote)
|
|
||||||
if config.url is not None:
|
|
||||||
raise IllegalArgumentException(f"Remote host `{remote}` is already exists.")
|
|
||||||
else:
|
|
||||||
config.url = url
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def update(cls, remote: str, url: str) -> None:
|
|
||||||
remote = cls._parse_remote_host(remote)
|
|
||||||
config = ConfigFile(remote)
|
|
||||||
if config.url is None:
|
|
||||||
raise IllegalArgumentException(f"Remote host `{remote}` is not exists.")
|
|
||||||
else:
|
|
||||||
config.url = url
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def list(cls, is_long: bool) -> None:
|
|
||||||
config = ConfigFile("")
|
|
||||||
for remote, url in config.list():
|
|
||||||
line = f"{remote}:"
|
|
||||||
if is_long:
|
|
||||||
line += f"\t{url}"
|
|
||||||
print(line)
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def delete(cls, remote: str) -> None:
|
|
||||||
remote = cls._parse_remote_host(remote)
|
|
||||||
config = ConfigFile(remote)
|
|
||||||
if config.url is None:
|
|
||||||
raise IllegalArgumentException(f"Remote host `{remote}` is not exists.")
|
|
||||||
else:
|
|
||||||
del config.url
|
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ from unicodedata import normalize
|
|||||||
from mdrsclient.api import FilesApi, FoldersApi
|
from mdrsclient.api import FilesApi, FoldersApi
|
||||||
from mdrsclient.commands.base import BaseCommand
|
from mdrsclient.commands.base import BaseCommand
|
||||||
from mdrsclient.exceptions import IllegalArgumentException
|
from mdrsclient.exceptions import IllegalArgumentException
|
||||||
|
from mdrsclient.models.file import find_file
|
||||||
|
|
||||||
|
|
||||||
class CpCommand(BaseCommand):
|
class CpCommand(BaseCommand):
|
||||||
@@ -28,51 +29,8 @@ class CpCommand(BaseCommand):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def cp(cls, src_path: str, dest_path: str, is_recursive: bool) -> None:
|
def cp(cls, src_path: str, dest_path: str, is_recursive: bool) -> None:
|
||||||
(s_remote, s_laboratory_name, s_path) = cls._parse_remote_host_with_path(src_path)
|
remote = src_path.split(":", 1)[0] if ":" in src_path else ""
|
||||||
(d_remote, d_laboratory_name, d_path) = cls._parse_remote_host_with_path(dest_path)
|
from mdrsclient.client import MdrsClient
|
||||||
if s_remote != d_remote:
|
|
||||||
raise IllegalArgumentException("Remote host mismatched.")
|
client = MdrsClient.from_remote(remote)
|
||||||
if s_laboratory_name != d_laboratory_name:
|
client.cp(src_path, dest_path, is_recursive)
|
||||||
raise IllegalArgumentException("Laboratory mismatched.")
|
|
||||||
s_path = s_path.rstrip("/")
|
|
||||||
s_dirname = os.path.dirname(s_path)
|
|
||||||
s_basename = os.path.basename(s_path)
|
|
||||||
if d_path.endswith("/"):
|
|
||||||
d_dirname = d_path
|
|
||||||
d_basename = s_basename
|
|
||||||
else:
|
|
||||||
d_dirname = os.path.dirname(d_path)
|
|
||||||
d_basename = os.path.basename(d_path)
|
|
||||||
connection = cls._create_connection(s_remote)
|
|
||||||
laboratory = cls._find_laboratory(connection, s_laboratory_name)
|
|
||||||
s_parent_folder = cls._find_folder(connection, laboratory, s_dirname)
|
|
||||||
d_parent_folder = cls._find_folder(connection, laboratory, d_dirname)
|
|
||||||
s_file = s_parent_folder.find_file(s_basename)
|
|
||||||
if s_file is not None:
|
|
||||||
# source is file
|
|
||||||
d_file = d_parent_folder.find_file(d_basename)
|
|
||||||
if d_file is not None:
|
|
||||||
raise IllegalArgumentException(f"File `{d_basename}` already exists.")
|
|
||||||
d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
|
|
||||||
if d_sub_folder is not None:
|
|
||||||
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
|
|
||||||
file_api = FilesApi(connection)
|
|
||||||
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
|
|
||||||
file_api.copy(s_file, d_parent_folder.id, normalize("NFC", d_basename))
|
|
||||||
else:
|
|
||||||
s_folder = s_parent_folder.find_sub_folder(s_basename)
|
|
||||||
if s_folder is None:
|
|
||||||
raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
|
|
||||||
# source is folder
|
|
||||||
if not is_recursive:
|
|
||||||
raise IllegalArgumentException(f"Cannot copy `{s_path}`: Is a folder.")
|
|
||||||
if d_parent_folder.find_file(d_basename) is not None:
|
|
||||||
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
|
|
||||||
d_folder = d_parent_folder.find_sub_folder(d_basename)
|
|
||||||
if d_folder is not None:
|
|
||||||
if d_folder.id == s_folder.id:
|
|
||||||
raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
|
|
||||||
raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
|
|
||||||
folder_api = FoldersApi(connection)
|
|
||||||
if s_parent_folder.id != d_parent_folder.id or s_basename != d_basename:
|
|
||||||
folder_api.copy(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
|
|
||||||
|
|||||||
@@ -1,28 +1,7 @@
|
|||||||
import os
|
|
||||||
from argparse import Namespace
|
from argparse import Namespace
|
||||||
from concurrent.futures import ThreadPoolExecutor
|
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from pydantic.dataclasses import dataclass
|
|
||||||
|
|
||||||
from mdrsclient.api import FilesApi, FoldersApi
|
|
||||||
from mdrsclient.commands.base import BaseCommand
|
from mdrsclient.commands.base import BaseCommand
|
||||||
from mdrsclient.connection import MDRSConnection
|
|
||||||
from mdrsclient.exceptions import IllegalArgumentException, UnexpectedException
|
|
||||||
from mdrsclient.models import File
|
|
||||||
from mdrsclient.settings import CONCURRENT
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class DownloadFileInfo:
|
|
||||||
file: File
|
|
||||||
path: str
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class DownloadContext:
|
|
||||||
hasError: bool
|
|
||||||
files: list[DownloadFileInfo]
|
|
||||||
|
|
||||||
|
|
||||||
class DownloadCommand(BaseCommand):
|
class DownloadCommand(BaseCommand):
|
||||||
@@ -32,6 +11,15 @@ class DownloadCommand(BaseCommand):
|
|||||||
download_parser.add_argument(
|
download_parser.add_argument(
|
||||||
"-r", "--recursive", help="download folders and their contents recursive", action="store_true"
|
"-r", "--recursive", help="download folders and their contents recursive", action="store_true"
|
||||||
)
|
)
|
||||||
|
download_parser.add_argument(
|
||||||
|
"-s",
|
||||||
|
"--skip-if-exists",
|
||||||
|
help="skip the download if file is already uploaded and file size is the same",
|
||||||
|
action="store_true",
|
||||||
|
)
|
||||||
|
download_parser.add_argument(
|
||||||
|
"-e", "--exclude", help="exclude to download path matched file or folders", action="append"
|
||||||
|
)
|
||||||
download_parser.add_argument("-p", "--password", help="password to use when open locked folder")
|
download_parser.add_argument("-p", "--password", help="password to use when open locked folder")
|
||||||
download_parser.add_argument("remote_path", help="remote file path (remote:/lab/path/file)")
|
download_parser.add_argument("remote_path", help="remote file path (remote:/lab/path/file)")
|
||||||
download_parser.add_argument("local_path", help="local folder path (/foo/bar/)")
|
download_parser.add_argument("local_path", help="local folder path (/foo/bar/)")
|
||||||
@@ -42,69 +30,24 @@ class DownloadCommand(BaseCommand):
|
|||||||
remote_path = str(args.remote_path)
|
remote_path = str(args.remote_path)
|
||||||
local_path = str(args.local_path)
|
local_path = str(args.local_path)
|
||||||
is_recursive = bool(args.recursive)
|
is_recursive = bool(args.recursive)
|
||||||
|
is_skip_if_exists = bool(args.skip_if_exists)
|
||||||
password = str(args.password) if args.password else None
|
password = str(args.password) if args.password else None
|
||||||
cls.download(remote_path, local_path, is_recursive, password)
|
excludes = list(map(lambda x: str(x).rstrip("/").lower(), args.exclude)) if args.exclude is not None else []
|
||||||
|
cls.download(remote_path, local_path, is_recursive, is_skip_if_exists, password, excludes)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def download(cls, remote_path: str, local_path: str, is_recursive: bool, password: str | None) -> None:
|
def download(
|
||||||
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
|
cls,
|
||||||
r_path = r_path.rstrip("/")
|
remote_path: str,
|
||||||
r_dirname = os.path.dirname(r_path)
|
local_path: str,
|
||||||
r_basename = os.path.basename(r_path)
|
is_recursive: bool,
|
||||||
connection = cls._create_connection(remote)
|
is_skip_if_exists: bool,
|
||||||
l_dirname = os.path.realpath(local_path)
|
password: str | None,
|
||||||
if not os.path.isdir(l_dirname):
|
excludes: list[str],
|
||||||
raise IllegalArgumentException(f"Local directory `{local_path}` not found.")
|
|
||||||
laboratory = cls._find_laboratory(connection, laboratory_name)
|
|
||||||
r_parent_folder = cls._find_folder(connection, laboratory, r_dirname, password)
|
|
||||||
file = r_parent_folder.find_file(r_basename)
|
|
||||||
if file is not None:
|
|
||||||
context = DownloadContext(False, [])
|
|
||||||
l_path = os.path.join(l_dirname, r_basename)
|
|
||||||
context.files.append(DownloadFileInfo(file, l_path))
|
|
||||||
cls.__multiple_download(connection, context)
|
|
||||||
else:
|
|
||||||
folder = r_parent_folder.find_sub_folder(r_basename)
|
|
||||||
if folder is None:
|
|
||||||
raise IllegalArgumentException(f"File or folder `{r_path}` not found.")
|
|
||||||
if not is_recursive:
|
|
||||||
raise IllegalArgumentException(f"Cannot download `{r_path}`: Is a folder.")
|
|
||||||
folder_api = FoldersApi(connection)
|
|
||||||
cls.__multiple_download_pickup_recursive_files(connection, folder_api, folder.id, l_dirname)
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def __multiple_download_pickup_recursive_files(
|
|
||||||
cls, connection: MDRSConnection, folder_api: FoldersApi, folder_id: str, basedir: str
|
|
||||||
) -> None:
|
) -> None:
|
||||||
context = DownloadContext(False, [])
|
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
|
||||||
folder = folder_api.retrieve(folder_id)
|
from mdrsclient.client import MdrsClient
|
||||||
dirname = os.path.join(basedir, folder.name)
|
|
||||||
if not os.path.exists(dirname):
|
|
||||||
os.makedirs(dirname)
|
|
||||||
print(dirname)
|
|
||||||
for file in folder.files:
|
|
||||||
path = os.path.join(dirname, file.name)
|
|
||||||
context.files.append(DownloadFileInfo(file, path))
|
|
||||||
cls.__multiple_download(connection, context)
|
|
||||||
if context.hasError:
|
|
||||||
raise UnexpectedException("Some files failed to download.")
|
|
||||||
for sub_folder in folder.sub_folders:
|
|
||||||
cls.__multiple_download_pickup_recursive_files(connection, folder_api, sub_folder.id, dirname)
|
|
||||||
|
|
||||||
@classmethod
|
client = MdrsClient.from_remote(remote)
|
||||||
def __multiple_download(cls, connection: MDRSConnection, context: DownloadContext) -> None:
|
client.download(remote_path, local_path, is_recursive, is_skip_if_exists, password, excludes)
|
||||||
file_api = FilesApi(connection)
|
return
|
||||||
with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
|
|
||||||
results = pool.map(lambda x: cls.__multiple_download_worker(file_api, x), context.files)
|
|
||||||
hasError = next(filter(lambda x: x is False, results), None)
|
|
||||||
if hasError is not None:
|
|
||||||
context.hasError = True
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def __multiple_download_worker(cls, file_api: FilesApi, info: DownloadFileInfo) -> bool:
|
|
||||||
try:
|
|
||||||
file_api.download(info.file, info.path)
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
print(info.path)
|
|
||||||
return True
|
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ from typing import Any
|
|||||||
from mdrsclient.api import FilesApi
|
from mdrsclient.api import FilesApi
|
||||||
from mdrsclient.commands.base import BaseCommand
|
from mdrsclient.commands.base import BaseCommand
|
||||||
from mdrsclient.exceptions import IllegalArgumentException
|
from mdrsclient.exceptions import IllegalArgumentException
|
||||||
|
from mdrsclient.models.file import find_file
|
||||||
|
|
||||||
|
|
||||||
class FileMetadataCommand(BaseCommand):
|
class FileMetadataCommand(BaseCommand):
|
||||||
@@ -24,16 +25,9 @@ class FileMetadataCommand(BaseCommand):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def file_metadata(cls, remote_path: str, password: str | None) -> None:
|
def file_metadata(cls, remote_path: str, password: str | None) -> None:
|
||||||
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
|
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
|
||||||
r_path = r_path.rstrip("/")
|
from mdrsclient.client import MdrsClient
|
||||||
r_dirname = os.path.dirname(r_path)
|
|
||||||
r_basename = os.path.basename(r_path)
|
client = MdrsClient.from_remote(remote)
|
||||||
connection = cls._create_connection(remote)
|
metadata = client.file_metadata(remote_path, password)
|
||||||
laboratory = cls._find_laboratory(connection, laboratory_name)
|
|
||||||
folder = cls._find_folder(connection, laboratory, r_dirname, password)
|
|
||||||
file = folder.find_file(r_basename)
|
|
||||||
if file is None:
|
|
||||||
raise IllegalArgumentException(f"File `{r_basename}` not found.")
|
|
||||||
file_api = FilesApi(connection)
|
|
||||||
metadata = file_api.metadata(file)
|
|
||||||
print(json.dumps(metadata, ensure_ascii=False))
|
print(json.dumps(metadata, ensure_ascii=False))
|
||||||
|
|||||||
@@ -19,11 +19,11 @@ class LabsCommand(BaseCommand):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def labs(cls, remote: str) -> None:
|
def labs(cls, remote: str) -> None:
|
||||||
remote = cls._parse_remote_host(remote)
|
from mdrsclient.client import MdrsClient
|
||||||
connection = cls._create_connection(remote)
|
|
||||||
laboratory_api = LaboratoriesApi(connection)
|
remote_host = MdrsClient.parse_remote_host(remote)
|
||||||
laboratories = laboratory_api.list()
|
client = MdrsClient.from_remote(remote_host)
|
||||||
connection.laboratories = laboratories
|
laboratories = client.get_laboratories()
|
||||||
label = {"id": "ID", "name": "Name", "pi_name": "PI", "full_name": "Laboratory"}
|
label = {"id": "ID", "name": "Name", "pi_name": "PI", "full_name": "Laboratory"}
|
||||||
length: dict[str, int] = {}
|
length: dict[str, int] = {}
|
||||||
for key in label.keys():
|
for key in label.keys():
|
||||||
@@ -34,7 +34,6 @@ class LabsCommand(BaseCommand):
|
|||||||
length["pi_name"] = max(length["pi_name"], len(laboratory.pi_name))
|
length["pi_name"] = max(length["pi_name"], len(laboratory.pi_name))
|
||||||
length["full_name"] = max(length["full_name"], len(laboratory.full_name))
|
length["full_name"] = max(length["full_name"], len(laboratory.full_name))
|
||||||
header = (
|
header = (
|
||||||
# f"{label['id']:{length['id']}}\t{label['name']:{length['name']}}\t"
|
|
||||||
f"{label['name']:{length['name']}}\t"
|
f"{label['name']:{length['name']}}\t"
|
||||||
f"{label['pi_name']:{length['pi_name']}}\t{label['full_name']:{length['full_name']}}"
|
f"{label['pi_name']:{length['pi_name']}}\t{label['full_name']:{length['full_name']}}"
|
||||||
)
|
)
|
||||||
@@ -42,7 +41,6 @@ class LabsCommand(BaseCommand):
|
|||||||
print("-" * len(header.expandtabs()))
|
print("-" * len(header.expandtabs()))
|
||||||
for laboratory in laboratories:
|
for laboratory in laboratories:
|
||||||
print(
|
print(
|
||||||
# f"{laboratory.id:{length['id']}}\t{laboratory.name:{length['name']}}\t"
|
|
||||||
f"{laboratory.name:{length['name']}}\t"
|
f"{laboratory.name:{length['name']}}\t"
|
||||||
f"{laboratory.pi_name:{length['pi_name']}}\t{laboratory.full_name:{length['full_name']}}"
|
f"{laboratory.pi_name:{length['pi_name']}}\t{laboratory.full_name:{length['full_name']}}"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -2,11 +2,7 @@ import getpass
|
|||||||
from argparse import Namespace
|
from argparse import Namespace
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from mdrsclient.api import UsersApi
|
|
||||||
from mdrsclient.commands.base import BaseCommand
|
from mdrsclient.commands.base import BaseCommand
|
||||||
from mdrsclient.config import ConfigFile
|
|
||||||
from mdrsclient.connection import MDRSConnection
|
|
||||||
from mdrsclient.exceptions import MissingConfigurationException
|
|
||||||
|
|
||||||
|
|
||||||
class LoginCommand(BaseCommand):
|
class LoginCommand(BaseCommand):
|
||||||
@@ -21,20 +17,15 @@ class LoginCommand(BaseCommand):
|
|||||||
@classmethod
|
@classmethod
|
||||||
def func(cls, args: Namespace) -> None:
|
def func(cls, args: Namespace) -> None:
|
||||||
remote = str(args.remote)
|
remote = str(args.remote)
|
||||||
username = str(args.username) if args.password else input("Username: ").strip()
|
username = str(args.username) if args.username else input("Username: ").strip()
|
||||||
password = str(args.password) if args.password else getpass.getpass("Password: ").strip()
|
password = str(args.password) if args.password else getpass.getpass("Password: ").strip()
|
||||||
cls.login(remote, username, password)
|
cls.login(remote, username, password)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def login(cls, remote: str, username: str, password: str) -> None:
|
def login(cls, remote: str, username: str, password: str) -> None:
|
||||||
remote = cls._parse_remote_host(remote)
|
from mdrsclient.client import MdrsClient
|
||||||
config = ConfigFile(remote)
|
|
||||||
if config.url is None:
|
remote_host = MdrsClient.parse_remote_host(remote)
|
||||||
raise MissingConfigurationException(f"Remote host `{remote}` is not found.")
|
client = MdrsClient.from_remote(remote_host)
|
||||||
connection = MDRSConnection(config.remote, config.url)
|
client.login(username, password)
|
||||||
user_api = UsersApi(connection)
|
|
||||||
token = user_api.token(username, password)
|
|
||||||
connection.token = token
|
|
||||||
user = user_api.current()
|
|
||||||
connection.user = user
|
|
||||||
print("Login Successful")
|
print("Login Successful")
|
||||||
|
|||||||
@@ -2,9 +2,6 @@ from argparse import Namespace
|
|||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from mdrsclient.commands.base import BaseCommand
|
from mdrsclient.commands.base import BaseCommand
|
||||||
from mdrsclient.config import ConfigFile
|
|
||||||
from mdrsclient.connection import MDRSConnection
|
|
||||||
from mdrsclient.exceptions import MissingConfigurationException
|
|
||||||
|
|
||||||
|
|
||||||
class LogoutCommand(BaseCommand):
|
class LogoutCommand(BaseCommand):
|
||||||
@@ -21,9 +18,8 @@ class LogoutCommand(BaseCommand):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def logout(cls, remote: str) -> None:
|
def logout(cls, remote: str) -> None:
|
||||||
remote = cls._parse_remote_host(remote)
|
from mdrsclient.client import MdrsClient
|
||||||
config = ConfigFile(remote)
|
|
||||||
if config.url is None:
|
remote_host = MdrsClient.parse_remote_host(remote)
|
||||||
raise MissingConfigurationException(f"Remote host `{remote}` is not found.")
|
client = MdrsClient.from_remote(remote_host)
|
||||||
connection = MDRSConnection(config.remote, config.url)
|
client.logout()
|
||||||
connection.logout()
|
|
||||||
|
|||||||
+56
-34
@@ -4,9 +4,9 @@ from typing import Any
|
|||||||
|
|
||||||
from pydantic.dataclasses import dataclass
|
from pydantic.dataclasses import dataclass
|
||||||
|
|
||||||
from mdrsclient.api import FoldersApi
|
from mdrsclient.api import FilesApi, FoldersApi
|
||||||
|
from mdrsclient.client import MdrsClient
|
||||||
from mdrsclient.commands.base import BaseCommand
|
from mdrsclient.commands.base import BaseCommand
|
||||||
from mdrsclient.connection import MDRSConnection
|
|
||||||
from mdrsclient.exceptions import UnauthorizedException
|
from mdrsclient.exceptions import UnauthorizedException
|
||||||
from mdrsclient.models import File, Folder, FolderSimple, Laboratory
|
from mdrsclient.models import File, Folder, FolderSimple, Laboratory
|
||||||
|
|
||||||
@@ -19,11 +19,11 @@ class Config:
|
|||||||
@dataclass(config=Config)
|
@dataclass(config=Config)
|
||||||
class LsCommandContext:
|
class LsCommandContext:
|
||||||
prefix: str
|
prefix: str
|
||||||
connection: MDRSConnection
|
client: MdrsClient
|
||||||
laboratory: Laboratory
|
laboratory: Laboratory
|
||||||
password: str
|
password: str
|
||||||
is_json: bool
|
is_json: bool
|
||||||
is_quick: bool
|
is_quiet: bool
|
||||||
is_recursive: bool
|
is_recursive: bool
|
||||||
|
|
||||||
|
|
||||||
@@ -35,7 +35,7 @@ class LsCommand(BaseCommand):
|
|||||||
ls_parser.add_argument("-J", "--json", help="turn on json output", action="store_true")
|
ls_parser.add_argument("-J", "--json", help="turn on json output", action="store_true")
|
||||||
ls_parser.add_argument(
|
ls_parser.add_argument(
|
||||||
"-q",
|
"-q",
|
||||||
"--quick",
|
"--quiet",
|
||||||
help="don't output header row. this option is forced if the -r option is specified",
|
help="don't output header row. this option is forced if the -r option is specified",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
)
|
)
|
||||||
@@ -49,35 +49,53 @@ class LsCommand(BaseCommand):
|
|||||||
password = str(args.password) if args.password else None
|
password = str(args.password) if args.password else None
|
||||||
is_json = bool(args.json)
|
is_json = bool(args.json)
|
||||||
is_recursive = bool(args.recursive)
|
is_recursive = bool(args.recursive)
|
||||||
is_quick = bool(args.quick) if not is_recursive else True
|
is_quiet = bool(args.quiet) if not is_recursive else True
|
||||||
cls.ls(remote_path, password, is_json, is_recursive, is_quick)
|
cls.ls(remote_path, password, is_json, is_recursive, is_quiet)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def ls(cls, remote_path: str, password: str | None, is_json: bool, is_recursive: bool, is_quick: bool) -> None:
|
def ls(cls, remote_path: str, password: str | None, is_json: bool, is_recursive: bool, is_quiet: bool) -> None:
|
||||||
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
|
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
|
||||||
connection = cls._create_connection(remote)
|
from mdrsclient.client import MdrsClient
|
||||||
laboratory = cls._find_laboratory(connection, laboratory_name)
|
|
||||||
|
client = MdrsClient.from_remote(remote)
|
||||||
|
cls._ls_logic(client, remote_path, password, is_json, is_recursive, is_quiet)
|
||||||
|
return
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def _ls_logic(
|
||||||
|
cls,
|
||||||
|
client: MdrsClient,
|
||||||
|
remote_path: str,
|
||||||
|
password: str | None,
|
||||||
|
is_json: bool,
|
||||||
|
is_recursive: bool,
|
||||||
|
is_quiet: bool,
|
||||||
|
) -> None:
|
||||||
|
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
|
||||||
|
|
||||||
|
folder, laboratory = client.resolve_folder(remote_path, password)
|
||||||
|
laboratory_name = laboratory.name
|
||||||
|
files = client.find_files(folder.id)
|
||||||
context = LsCommandContext(
|
context = LsCommandContext(
|
||||||
f"{remote}:/{laboratory_name}",
|
f"{remote}:/{laboratory_name}",
|
||||||
connection,
|
client,
|
||||||
laboratory,
|
laboratory,
|
||||||
password if password is not None else "",
|
password if password is not None else "",
|
||||||
is_json,
|
is_json,
|
||||||
is_quick,
|
is_quiet,
|
||||||
is_recursive,
|
is_recursive,
|
||||||
)
|
)
|
||||||
folder = cls._find_folder(connection, laboratory, r_path, password)
|
|
||||||
if context.is_json:
|
if context.is_json:
|
||||||
cls._ls_json(context, folder)
|
cls._ls_json(context, folder, files)
|
||||||
else:
|
else:
|
||||||
cls._ls_plain(context, folder)
|
cls._ls_plain(context, folder, files)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _ls_json(cls, context: LsCommandContext, folder: Folder) -> None:
|
def _ls_json(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
|
||||||
print(json.dumps(cls._folder2dict(context, folder), ensure_ascii=False))
|
print(json.dumps(cls._folder2dict(context, folder, files), ensure_ascii=False))
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _ls_plain(cls, context: LsCommandContext, folder: Folder) -> None:
|
def _ls_plain(cls, context: LsCommandContext, folder: Folder, files: list[File]) -> None:
|
||||||
label = {
|
label = {
|
||||||
"type": "Type",
|
"type": "Type",
|
||||||
"acl": "Access",
|
"acl": "Access",
|
||||||
@@ -88,16 +106,16 @@ class LsCommand(BaseCommand):
|
|||||||
}
|
}
|
||||||
length: dict[str, int] = {}
|
length: dict[str, int] = {}
|
||||||
for key in label.keys():
|
for key in label.keys():
|
||||||
length[key] = len(label[key]) if not context.is_quick else 0
|
length[key] = len(label[key]) if not context.is_quiet else 0
|
||||||
for sub_folder in folder.sub_folders:
|
for sub_folder in folder.sub_folders:
|
||||||
sub_laboratory = context.connection.laboratories.find_by_id(sub_folder.laboratory_id)
|
sub_laboratory = context.client.connection.laboratories.find_by_id(sub_folder.laboratory_id)
|
||||||
sub_laboratory_name = sub_laboratory.name if sub_laboratory is not None else "(invalid)"
|
sub_laboratory_name = sub_laboratory.name if sub_laboratory is not None else "(invalid)"
|
||||||
length["acl"] = max(length["acl"], len(sub_folder.access_level_name))
|
length["acl"] = max(length["acl"], len(sub_folder.access_level_name))
|
||||||
length["laboratory"] = max(length["laboratory"], len(sub_laboratory_name))
|
length["laboratory"] = max(length["laboratory"], len(sub_laboratory_name))
|
||||||
length["size"] = max(length["size"], len(str(folder.size)))
|
length["size"] = max(length["size"], len(str(folder.size)))
|
||||||
length["date"] = max(length["date"], len(sub_folder.updated_at_name))
|
length["date"] = max(length["date"], len(sub_folder.updated_at_name))
|
||||||
length["name"] = max(length["name"], len(sub_folder.name))
|
length["name"] = max(length["name"], len(sub_folder.name))
|
||||||
for file in folder.files:
|
for file in files:
|
||||||
length["size"] = max(length["size"], len(str(file.size)))
|
length["size"] = max(length["size"], len(str(file.size)))
|
||||||
length["date"] = max(length["date"], len(file.updated_at_name))
|
length["date"] = max(length["date"], len(file.updated_at_name))
|
||||||
length["name"] = max(length["name"], len(file.name))
|
length["name"] = max(length["name"], len(file.name))
|
||||||
@@ -111,9 +129,9 @@ class LsCommand(BaseCommand):
|
|||||||
|
|
||||||
if context.is_recursive:
|
if context.is_recursive:
|
||||||
print(f"{context.prefix}{folder.path}:")
|
print(f"{context.prefix}{folder.path}:")
|
||||||
print(f"total {sum(f.size for f in folder.files)}")
|
print(f"total {sum(f.size for f in files)}")
|
||||||
|
|
||||||
if not context.is_quick:
|
if not context.is_quiet:
|
||||||
print(header)
|
print(header)
|
||||||
print("-" * len(header.expandtabs()))
|
print("-" * len(header.expandtabs()))
|
||||||
|
|
||||||
@@ -125,7 +143,7 @@ class LsCommand(BaseCommand):
|
|||||||
f"{sub_laboratory_name:{length['laboratory']}}\t{sub_folder.size:{length['size']}}\t"
|
f"{sub_laboratory_name:{length['laboratory']}}\t{sub_folder.size:{length['size']}}\t"
|
||||||
f"{sub_folder.updated_at_name:{length['date']}}\t{sub_folder.name:{length['name']}}"
|
f"{sub_folder.updated_at_name:{length['date']}}\t{sub_folder.name:{length['name']}}"
|
||||||
)
|
)
|
||||||
for file in sorted(folder.files, key=lambda x: x.name):
|
for file in sorted(files, key=lambda x: x.name):
|
||||||
print(
|
print(
|
||||||
f"{'[f]':{length['type']}}\t{folder.access_level_name:{length['acl']}}\t"
|
f"{'[f]':{length['type']}}\t{folder.access_level_name:{length['acl']}}\t"
|
||||||
f"{context.laboratory.name:{length['laboratory']}}\t{file.size:{length['size']}}\t"
|
f"{context.laboratory.name:{length['laboratory']}}\t{file.size:{length['size']}}\t"
|
||||||
@@ -135,17 +153,20 @@ class LsCommand(BaseCommand):
|
|||||||
if context.is_recursive:
|
if context.is_recursive:
|
||||||
print("")
|
print("")
|
||||||
for sub_folder in sorted(folder.sub_folders, key=lambda x: x.name):
|
for sub_folder in sorted(folder.sub_folders, key=lambda x: x.name):
|
||||||
folder_api = FoldersApi(context.connection)
|
folder_api = FoldersApi(context.client.connection)
|
||||||
try:
|
try:
|
||||||
if sub_folder.lock:
|
if sub_folder.lock:
|
||||||
folder_api.auth(sub_folder.id, context.password)
|
folder_api.auth(sub_folder.id, context.password)
|
||||||
folder = folder_api.retrieve(sub_folder.id)
|
folder = folder_api.retrieve(sub_folder.id)
|
||||||
cls._ls_plain(context, folder)
|
files = context.client.find_files(sub_folder.id)
|
||||||
|
cls._ls_plain(context, folder, files)
|
||||||
except UnauthorizedException:
|
except UnauthorizedException:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _folder2dict(cls, context: LsCommandContext, folder: Folder | FolderSimple) -> dict[str, Any]:
|
def _folder2dict(
|
||||||
|
cls, context: LsCommandContext, folder: Folder | FolderSimple, files: list[File]
|
||||||
|
) -> dict[str, Any]:
|
||||||
data: dict[str, Any] = {
|
data: dict[str, Any] = {
|
||||||
"id": folder.id,
|
"id": folder.id,
|
||||||
"pid": folder.pid,
|
"pid": folder.pid,
|
||||||
@@ -159,7 +180,7 @@ class LsCommand(BaseCommand):
|
|||||||
"updated_at": folder.updated_at,
|
"updated_at": folder.updated_at,
|
||||||
}
|
}
|
||||||
if isinstance(folder, Folder):
|
if isinstance(folder, Folder):
|
||||||
folder_api = FoldersApi(context.connection)
|
folder_api = FoldersApi(context.client.connection)
|
||||||
data["metadata"] = folder_api.metadata(folder.id)
|
data["metadata"] = folder_api.metadata(folder.id)
|
||||||
if context.is_recursive:
|
if context.is_recursive:
|
||||||
sub_folders: list[dict[str, Any]] = []
|
sub_folders: list[dict[str, Any]] = []
|
||||||
@@ -168,15 +189,16 @@ class LsCommand(BaseCommand):
|
|||||||
if sub_folder.lock:
|
if sub_folder.lock:
|
||||||
folder_api.auth(sub_folder.id, context.password)
|
folder_api.auth(sub_folder.id, context.password)
|
||||||
folder2 = folder_api.retrieve(sub_folder.id)
|
folder2 = folder_api.retrieve(sub_folder.id)
|
||||||
sub_folders.append(cls._folder2dict(context, folder2))
|
files2 = context.client.find_files(sub_folder.id)
|
||||||
|
sub_folders.append(cls._folder2dict(context, folder2, files2))
|
||||||
except UnauthorizedException:
|
except UnauthorizedException:
|
||||||
pass
|
pass
|
||||||
data["sub_folders"] = sub_folders
|
data["sub_folders"] = sub_folders
|
||||||
else:
|
else:
|
||||||
data["sub_folders"] = list(
|
data["sub_folders"] = list(
|
||||||
map(lambda x: cls._folder2dict(context, x), sorted(folder.sub_folders, key=lambda x: x.name))
|
map(lambda x: cls._folder2dict(context, x, []), sorted(folder.sub_folders, key=lambda x: x.name))
|
||||||
)
|
)
|
||||||
data["files"] = list(map(lambda x: cls._file2dict(context, x), sorted(folder.files, key=lambda x: x.name)))
|
data["files"] = list(map(lambda x: cls._file2dict(context, x), sorted(files, key=lambda x: x.name)))
|
||||||
return data
|
return data
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
@@ -189,7 +211,7 @@ class LsCommand(BaseCommand):
|
|||||||
# "thumbnail": file.thumbnail,
|
# "thumbnail": file.thumbnail,
|
||||||
"description": file.description,
|
"description": file.description,
|
||||||
"metadata": file.metadata,
|
"metadata": file.metadata,
|
||||||
"download_url": f"{context.connection.url}/{file.download_url}",
|
"download_url": f"{context.client.connection.url}/{file.download_url}",
|
||||||
"created_at": file.created_at,
|
"created_at": file.created_at,
|
||||||
"updated_at": file.updated_at,
|
"updated_at": file.updated_at,
|
||||||
}
|
}
|
||||||
@@ -197,5 +219,5 @@ class LsCommand(BaseCommand):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _laboratory_name(cls, context: LsCommandContext, laboratory_id: int) -> str:
|
def _laboratory_name(cls, context: LsCommandContext, laboratory_id: int) -> str:
|
||||||
laboratory = context.connection.laboratories.find_by_id(laboratory_id)
|
laboratory = context.client.connection.laboratories.find_by_id(laboratory_id)
|
||||||
return laboratory.name if laboratory is not None else "(invalid)"
|
return laboratory.name if laboratory is not None else "(invalid)"
|
||||||
|
|||||||
@@ -22,10 +22,9 @@ class MetadataCommand(BaseCommand):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def metadata(cls, remote_path: str, password: str | None) -> None:
|
def metadata(cls, remote_path: str, password: str | None) -> None:
|
||||||
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
|
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
|
||||||
connection = cls._create_connection(remote)
|
from mdrsclient.client import MdrsClient
|
||||||
laboratory = cls._find_laboratory(connection, laboratory_name)
|
|
||||||
folder = cls._find_folder(connection, laboratory, r_path, password)
|
client = MdrsClient.from_remote(remote)
|
||||||
folder_api = FoldersApi(connection)
|
metadata = client.metadata(remote_path, password)
|
||||||
metadata = folder_api.metadata(folder.id)
|
|
||||||
print(json.dumps(metadata, ensure_ascii=False))
|
print(json.dumps(metadata, ensure_ascii=False))
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ from unicodedata import normalize
|
|||||||
from mdrsclient.api import FoldersApi
|
from mdrsclient.api import FoldersApi
|
||||||
from mdrsclient.commands.base import BaseCommand
|
from mdrsclient.commands.base import BaseCommand
|
||||||
from mdrsclient.exceptions import IllegalArgumentException
|
from mdrsclient.exceptions import IllegalArgumentException
|
||||||
|
from mdrsclient.models.file import find_file
|
||||||
|
|
||||||
|
|
||||||
class MkdirCommand(BaseCommand):
|
class MkdirCommand(BaseCommand):
|
||||||
@@ -22,14 +23,8 @@ class MkdirCommand(BaseCommand):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def mkdir(cls, remote_path: str) -> None:
|
def mkdir(cls, remote_path: str) -> None:
|
||||||
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
|
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
|
||||||
r_path = r_path.rstrip("/")
|
from mdrsclient.client import MdrsClient
|
||||||
r_dirname = os.path.dirname(r_path)
|
|
||||||
r_basename = os.path.basename(r_path)
|
client = MdrsClient.from_remote(remote)
|
||||||
connection = cls._create_connection(remote)
|
client.mkdir(remote_path)
|
||||||
laboratory = cls._find_laboratory(connection, laboratory_name)
|
|
||||||
parent_folder = cls._find_folder(connection, laboratory, r_dirname)
|
|
||||||
if parent_folder.find_sub_folder(r_basename) is not None or parent_folder.find_file(r_basename) is not None:
|
|
||||||
raise IllegalArgumentException(f"Cannot create folder `{r_path}`: File exists.")
|
|
||||||
folder_api = FoldersApi(connection)
|
|
||||||
folder_api.create(normalize("NFC", r_basename), parent_folder.id)
|
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ from unicodedata import normalize
|
|||||||
from mdrsclient.api import FilesApi, FoldersApi
|
from mdrsclient.api import FilesApi, FoldersApi
|
||||||
from mdrsclient.commands.base import BaseCommand
|
from mdrsclient.commands.base import BaseCommand
|
||||||
from mdrsclient.exceptions import IllegalArgumentException
|
from mdrsclient.exceptions import IllegalArgumentException
|
||||||
|
from mdrsclient.models.file import find_file
|
||||||
|
|
||||||
|
|
||||||
class MvCommand(BaseCommand):
|
class MvCommand(BaseCommand):
|
||||||
@@ -24,49 +25,8 @@ class MvCommand(BaseCommand):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def mv(cls, src_path: str, dest_path: str) -> None:
|
def mv(cls, src_path: str, dest_path: str) -> None:
|
||||||
(s_remote, s_laboratory_name, s_path) = cls._parse_remote_host_with_path(src_path)
|
remote = src_path.split(":", 1)[0] if ":" in src_path else ""
|
||||||
(d_remote, d_laboratory_name, d_path) = cls._parse_remote_host_with_path(dest_path)
|
from mdrsclient.client import MdrsClient
|
||||||
if s_remote != d_remote:
|
|
||||||
raise IllegalArgumentException("Remote host mismatched.")
|
client = MdrsClient.from_remote(remote)
|
||||||
if s_laboratory_name != d_laboratory_name:
|
client.mv(src_path, dest_path)
|
||||||
raise IllegalArgumentException("Laboratory mismatched.")
|
|
||||||
s_path = s_path.rstrip("/")
|
|
||||||
s_dirname = os.path.dirname(s_path)
|
|
||||||
s_basename = os.path.basename(s_path)
|
|
||||||
if d_path.endswith("/"):
|
|
||||||
d_dirname = d_path
|
|
||||||
d_basename = s_basename
|
|
||||||
else:
|
|
||||||
d_dirname = os.path.dirname(d_path)
|
|
||||||
d_basename = os.path.basename(d_path)
|
|
||||||
connection = cls._create_connection(s_remote)
|
|
||||||
laboratory = cls._find_laboratory(connection, s_laboratory_name)
|
|
||||||
s_parent_folder = cls._find_folder(connection, laboratory, s_dirname)
|
|
||||||
d_parent_folder = cls._find_folder(connection, laboratory, d_dirname)
|
|
||||||
s_file = s_parent_folder.find_file(s_basename)
|
|
||||||
if s_file is not None:
|
|
||||||
# source is file
|
|
||||||
d_file = d_parent_folder.find_file(d_basename)
|
|
||||||
if d_file is not None:
|
|
||||||
raise IllegalArgumentException(f"File `{d_basename}` already exists.")
|
|
||||||
d_sub_folder = d_parent_folder.find_sub_folder(d_basename)
|
|
||||||
if d_sub_folder is not None:
|
|
||||||
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{d_path}`.")
|
|
||||||
file_api = FilesApi(connection)
|
|
||||||
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
|
|
||||||
file_api.move(s_file, d_parent_folder.id, normalize("NFC", d_basename))
|
|
||||||
else:
|
|
||||||
s_folder = s_parent_folder.find_sub_folder(s_basename)
|
|
||||||
if s_folder is None:
|
|
||||||
raise IllegalArgumentException(f"File or folder `{s_basename}` not found.")
|
|
||||||
# source is folder
|
|
||||||
if d_parent_folder.find_file(d_basename) is not None:
|
|
||||||
raise IllegalArgumentException(f"Cannot overwrite non-folder `{d_basename}` with folder `{s_path}`.")
|
|
||||||
d_folder = d_parent_folder.find_sub_folder(d_basename)
|
|
||||||
if d_folder is not None:
|
|
||||||
if d_folder.id == s_folder.id:
|
|
||||||
raise IllegalArgumentException(f"`{s_path}` and `{s_path}` are the same folder.")
|
|
||||||
raise IllegalArgumentException(f"Cannot move `{s_path}` to `{d_path}`: Folder not empty.")
|
|
||||||
folder_api = FoldersApi(connection)
|
|
||||||
if s_parent_folder.id != d_parent_folder.id or d_basename != s_basename:
|
|
||||||
folder_api.move(s_folder, d_parent_folder.id, normalize("NFC", d_basename))
|
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ from typing import Any
|
|||||||
from mdrsclient.api import FilesApi, FoldersApi
|
from mdrsclient.api import FilesApi, FoldersApi
|
||||||
from mdrsclient.commands.base import BaseCommand
|
from mdrsclient.commands.base import BaseCommand
|
||||||
from mdrsclient.exceptions import IllegalArgumentException
|
from mdrsclient.exceptions import IllegalArgumentException
|
||||||
|
from mdrsclient.models.file import find_file
|
||||||
|
|
||||||
|
|
||||||
class RmCommand(BaseCommand):
|
class RmCommand(BaseCommand):
|
||||||
@@ -25,22 +26,8 @@ class RmCommand(BaseCommand):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def rm(cls, remote_path: str, is_recursive: bool) -> None:
|
def rm(cls, remote_path: str, is_recursive: bool) -> None:
|
||||||
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
|
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
|
||||||
r_path = r_path.rstrip("/")
|
from mdrsclient.client import MdrsClient
|
||||||
r_dirname = os.path.dirname(r_path)
|
|
||||||
r_basename = os.path.basename(r_path)
|
client = MdrsClient.from_remote(remote)
|
||||||
connection = cls._create_connection(remote)
|
client.rm(remote_path, is_recursive)
|
||||||
laboratory = cls._find_laboratory(connection, laboratory_name)
|
|
||||||
parent_folder = cls._find_folder(connection, laboratory, r_dirname)
|
|
||||||
file = parent_folder.find_file(r_basename)
|
|
||||||
if file is not None:
|
|
||||||
file_api = FilesApi(connection)
|
|
||||||
file_api.destroy(file)
|
|
||||||
else:
|
|
||||||
folder = parent_folder.find_sub_folder(r_basename)
|
|
||||||
if folder is None:
|
|
||||||
raise IllegalArgumentException(f"Cannot remove `{r_path}`: No such file or folder.")
|
|
||||||
if not is_recursive:
|
|
||||||
raise IllegalArgumentException(f"Cannot remove `{r_path}`: Is a folder.")
|
|
||||||
folder_api = FoldersApi(connection)
|
|
||||||
folder_api.destroy(folder.id, True)
|
|
||||||
|
|||||||
@@ -1,22 +1,7 @@
|
|||||||
import os
|
|
||||||
from argparse import Namespace
|
from argparse import Namespace
|
||||||
from concurrent.futures import ThreadPoolExecutor
|
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from pydantic.dataclasses import dataclass
|
|
||||||
|
|
||||||
from mdrsclient.api import FilesApi, FoldersApi
|
|
||||||
from mdrsclient.commands.base import BaseCommand
|
from mdrsclient.commands.base import BaseCommand
|
||||||
from mdrsclient.connection import MDRSConnection
|
|
||||||
from mdrsclient.exceptions import IllegalArgumentException, MDRSException
|
|
||||||
from mdrsclient.models import Folder
|
|
||||||
from mdrsclient.settings import CONCURRENT
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class UploadFileInfo:
|
|
||||||
folder: Folder
|
|
||||||
path: str
|
|
||||||
|
|
||||||
|
|
||||||
class UploadCommand(BaseCommand):
|
class UploadCommand(BaseCommand):
|
||||||
@@ -46,64 +31,9 @@ class UploadCommand(BaseCommand):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def upload(cls, local_path: str, remote_path: str, is_recursive: bool, is_skip_if_exists: bool) -> None:
|
def upload(cls, local_path: str, remote_path: str, is_recursive: bool, is_skip_if_exists: bool) -> None:
|
||||||
(remote, laboratory_name, r_path) = cls._parse_remote_host_with_path(remote_path)
|
remote = remote_path.split(":", 1)[0] if ":" in remote_path else ""
|
||||||
l_path = os.path.abspath(local_path)
|
from mdrsclient.client import MdrsClient
|
||||||
if not os.path.exists(l_path):
|
|
||||||
raise IllegalArgumentException(f"File or directory `{local_path}` not found.")
|
|
||||||
connection = cls._create_connection(remote)
|
|
||||||
laboratory = cls._find_laboratory(connection, laboratory_name)
|
|
||||||
folder = cls._find_folder(connection, laboratory, r_path)
|
|
||||||
infos: list[UploadFileInfo] = []
|
|
||||||
if os.path.isdir(l_path):
|
|
||||||
if not is_recursive:
|
|
||||||
raise IllegalArgumentException(f"Cannot upload `{local_path}`: Is a directory.")
|
|
||||||
folder_api = FoldersApi(connection)
|
|
||||||
folder_map: dict[str, Folder] = {}
|
|
||||||
folder_map[r_path] = folder
|
|
||||||
l_basename = os.path.basename(l_path)
|
|
||||||
for dirpath, _, filenames in os.walk(l_path, followlinks=True):
|
|
||||||
sub = l_basename if dirpath == l_path else os.path.join(l_basename, os.path.relpath(dirpath, l_path))
|
|
||||||
d_dirname = os.path.join(r_path, sub)
|
|
||||||
d_basename = os.path.basename(d_dirname)
|
|
||||||
# prepare destination parent path
|
|
||||||
d_parent_dirname = os.path.dirname(d_dirname)
|
|
||||||
if folder_map.get(d_parent_dirname) is None:
|
|
||||||
folder_map[d_parent_dirname] = cls._find_folder(connection, laboratory, d_parent_dirname)
|
|
||||||
# prepare destination path
|
|
||||||
if folder_map.get(d_dirname) is None:
|
|
||||||
d_folder = folder_map[d_parent_dirname].find_sub_folder(d_basename)
|
|
||||||
if d_folder is None:
|
|
||||||
d_folder_id = folder_api.create(d_basename, folder_map[d_parent_dirname].id)
|
|
||||||
else:
|
|
||||||
d_folder_id = d_folder.id
|
|
||||||
print(d_dirname)
|
|
||||||
folder_map[d_dirname] = folder_api.retrieve(d_folder_id)
|
|
||||||
if d_folder is None:
|
|
||||||
folder_map[d_parent_dirname].sub_folders.append(folder_map[d_dirname])
|
|
||||||
# register upload file list
|
|
||||||
for filename in filenames:
|
|
||||||
infos.append(UploadFileInfo(folder_map[d_dirname], os.path.join(dirpath, filename)))
|
|
||||||
else:
|
|
||||||
infos.append(UploadFileInfo(folder, l_path))
|
|
||||||
cls.__multiple_upload(connection, infos, is_skip_if_exists)
|
|
||||||
|
|
||||||
@classmethod
|
client = MdrsClient.from_remote(remote)
|
||||||
def __multiple_upload(
|
client.upload(local_path, remote_path, is_recursive, is_skip_if_exists)
|
||||||
cls, connection: MDRSConnection, infos: list[UploadFileInfo], is_skip_if_exists: bool
|
return
|
||||||
) -> None:
|
|
||||||
file_api = FilesApi(connection)
|
|
||||||
with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
|
|
||||||
pool.map(lambda x: cls.__multiple_upload_worker(file_api, x, is_skip_if_exists), infos)
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def __multiple_upload_worker(cls, file_api: FilesApi, info: UploadFileInfo, is_skip_if_exists: bool) -> None:
|
|
||||||
basename = os.path.basename(info.path)
|
|
||||||
file = info.folder.find_file(basename)
|
|
||||||
try:
|
|
||||||
if file is None:
|
|
||||||
file_api.create(info.folder.id, info.path)
|
|
||||||
elif not is_skip_if_exists or file.size != os.path.getsize(info.path):
|
|
||||||
file_api.update(file, info.path)
|
|
||||||
print(os.path.join(info.folder.path, basename))
|
|
||||||
except MDRSException as e:
|
|
||||||
print(f"Error: {e}")
|
|
||||||
|
|||||||
@@ -0,0 +1,24 @@
|
|||||||
|
from argparse import Namespace
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from mdrsclient.__version__ import __version__
|
||||||
|
from mdrsclient.commands.base import BaseCommand
|
||||||
|
|
||||||
|
|
||||||
|
class VersionCommand(BaseCommand):
|
||||||
|
@classmethod
|
||||||
|
def register(cls, parsers: Any) -> None:
|
||||||
|
version_parser = parsers.add_parser("version", help="show the version of this tool")
|
||||||
|
version_parser.set_defaults(func=cls.func)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def func(cls, args: Namespace) -> None:
|
||||||
|
cls.version()
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def version(cls) -> None:
|
||||||
|
from mdrsclient.client import MdrsClient
|
||||||
|
|
||||||
|
# Client initialization is not strictly needed for version, but for consistency:
|
||||||
|
client = MdrsClient(None)
|
||||||
|
print(client.version())
|
||||||
@@ -2,9 +2,6 @@ from argparse import Namespace
|
|||||||
from typing import Any, Final
|
from typing import Any, Final
|
||||||
|
|
||||||
from mdrsclient.commands.base import BaseCommand
|
from mdrsclient.commands.base import BaseCommand
|
||||||
from mdrsclient.config import ConfigFile
|
|
||||||
from mdrsclient.connection import MDRSConnection
|
|
||||||
from mdrsclient.exceptions import MissingConfigurationException
|
|
||||||
|
|
||||||
|
|
||||||
class WhoamiCommand(BaseCommand):
|
class WhoamiCommand(BaseCommand):
|
||||||
@@ -23,12 +20,15 @@ class WhoamiCommand(BaseCommand):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def whoami(cls, remote: str) -> None:
|
def whoami(cls, remote: str) -> None:
|
||||||
remote = cls._parse_remote_host(remote)
|
from mdrsclient.client import MdrsClient
|
||||||
config = ConfigFile(remote)
|
|
||||||
if config.url is None:
|
remote_host = MdrsClient.parse_remote_host(remote)
|
||||||
raise MissingConfigurationException(f"Remote host `{remote}` is not found.")
|
client = MdrsClient.from_remote(remote_host)
|
||||||
connection = MDRSConnection(config.remote, config.url)
|
if client.connection.token is not None and client.connection.token.is_expired:
|
||||||
if connection.token is not None and connection.token.is_expired:
|
client.logout()
|
||||||
connection.logout()
|
try:
|
||||||
username = connection.user.username if connection.user is not None else cls.ANONYMOUS_USERNAME
|
user = client.whoami()
|
||||||
|
username = user.username if user is not None else cls.ANONYMOUS_USERNAME
|
||||||
|
except Exception:
|
||||||
|
username = cls.ANONYMOUS_USERNAME
|
||||||
print(username)
|
print(username)
|
||||||
|
|||||||
+54
-4
@@ -1,15 +1,65 @@
|
|||||||
import configparser
|
import configparser
|
||||||
import os
|
import os
|
||||||
from typing import Final
|
import threading
|
||||||
|
from typing import Final, Protocol, runtime_checkable
|
||||||
|
|
||||||
import validators # type: ignore
|
import validators
|
||||||
|
|
||||||
from mdrsclient.exceptions import IllegalArgumentException
|
from mdrsclient.exceptions import IllegalArgumentException
|
||||||
from mdrsclient.settings import CONFIG_DIRNAME
|
from mdrsclient.settings import CONFIG_DIRNAME
|
||||||
from mdrsclient.utils import FileLock
|
from mdrsclient.utils import FileLock
|
||||||
|
|
||||||
|
|
||||||
class ConfigFile:
|
@runtime_checkable
|
||||||
|
class ConfigInterface(Protocol):
|
||||||
|
remote: str
|
||||||
|
|
||||||
|
def list(self) -> list[tuple[str, str]]: ...
|
||||||
|
@property
|
||||||
|
def url(self) -> str | None: ...
|
||||||
|
@url.setter
|
||||||
|
def url(self, url: str) -> None: ...
|
||||||
|
@url.deleter
|
||||||
|
def url(self) -> None: ...
|
||||||
|
|
||||||
|
|
||||||
|
class InMemoryConfig(ConfigInterface):
|
||||||
|
__configs: dict[str, str] = {}
|
||||||
|
__lock: threading.Lock = threading.Lock()
|
||||||
|
remote: str
|
||||||
|
|
||||||
|
def __init__(self, remote: str) -> None:
|
||||||
|
self.remote = remote
|
||||||
|
|
||||||
|
def list(self) -> list[tuple[str, str]]:
|
||||||
|
with self.__lock:
|
||||||
|
return list(self.__configs.items())
|
||||||
|
|
||||||
|
@property
|
||||||
|
def url(self) -> str | None:
|
||||||
|
with self.__lock:
|
||||||
|
return self.__configs.get(self.remote)
|
||||||
|
|
||||||
|
@url.setter
|
||||||
|
def url(self, url: str) -> None:
|
||||||
|
if not validators.url(url):
|
||||||
|
raise IllegalArgumentException("malformed URI sequence")
|
||||||
|
with self.__lock:
|
||||||
|
self.__configs[self.remote] = url
|
||||||
|
|
||||||
|
@url.deleter
|
||||||
|
def url(self) -> None:
|
||||||
|
with self.__lock:
|
||||||
|
if self.remote in self.__configs:
|
||||||
|
del self.__configs[self.remote]
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def clear(cls) -> None:
|
||||||
|
with cls.__lock:
|
||||||
|
cls.__configs.clear()
|
||||||
|
|
||||||
|
|
||||||
|
class ConfigFile(ConfigInterface):
|
||||||
OPTION_URL: Final[str] = "url"
|
OPTION_URL: Final[str] = "url"
|
||||||
CONFIG_FILENAME: Final[str] = "config.ini"
|
CONFIG_FILENAME: Final[str] = "config.ini"
|
||||||
remote: str
|
remote: str
|
||||||
@@ -41,7 +91,7 @@ class ConfigFile:
|
|||||||
|
|
||||||
@url.setter
|
@url.setter
|
||||||
def url(self, url: str) -> None:
|
def url(self, url: str) -> None:
|
||||||
if not validators.url(url): # type: ignore
|
if not validators.url(url):
|
||||||
raise IllegalArgumentException("malformed URI sequence")
|
raise IllegalArgumentException("malformed URI sequence")
|
||||||
self.__load()
|
self.__load()
|
||||||
if self.__config.has_section(self.remote):
|
if self.__config.has_section(self.remote):
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ from requests_toolbelt.multipart.encoder import MultipartEncoder
|
|||||||
from typing_extensions import Unpack
|
from typing_extensions import Unpack
|
||||||
|
|
||||||
from mdrsclient.__version__ import __version__
|
from mdrsclient.__version__ import __version__
|
||||||
from mdrsclient.cache import CacheFile
|
from mdrsclient.cache import CacheFile, CacheInterface
|
||||||
from mdrsclient.exceptions import MissingConfigurationException
|
from mdrsclient.exceptions import MissingConfigurationException
|
||||||
from mdrsclient.models import Laboratories, Token, User
|
from mdrsclient.models import Laboratories, Token, User
|
||||||
|
|
||||||
@@ -39,14 +39,14 @@ class MDRSConnection:
|
|||||||
url: str
|
url: str
|
||||||
session: Session
|
session: Session
|
||||||
lock: threading.Lock
|
lock: threading.Lock
|
||||||
__cache: CacheFile
|
__cache: CacheInterface
|
||||||
|
|
||||||
def __init__(self, remote: str, url: str) -> None:
|
def __init__(self, remote: str, url: str, cache: CacheInterface | None = None) -> None:
|
||||||
super().__init__()
|
super().__init__()
|
||||||
self.url = url
|
self.url = url
|
||||||
self.session = Session()
|
self.session = Session()
|
||||||
self.lock = threading.Lock()
|
self.lock = threading.Lock()
|
||||||
self.__cache = CacheFile(remote)
|
self.__cache = cache if cache is not None else CacheFile(remote)
|
||||||
self.__prepare_headers()
|
self.__prepare_headers()
|
||||||
|
|
||||||
def get(self, url: str, **kwargs: Unpack[_KwArgsMDRSConnectionGet]) -> Response:
|
def get(self, url: str, **kwargs: Unpack[_KwArgsMDRSConnectionGet]) -> Response:
|
||||||
@@ -92,6 +92,8 @@ class MDRSConnection:
|
|||||||
self.__cache.laboratories = laboratories
|
self.__cache.laboratories = laboratories
|
||||||
|
|
||||||
def __build_url(self, path: str) -> str:
|
def __build_url(self, path: str) -> str:
|
||||||
|
if path.startswith("http://") or path.startswith("https://"):
|
||||||
|
return path
|
||||||
if self.url == "":
|
if self.url == "":
|
||||||
raise MissingConfigurationException("remote host is not configured")
|
raise MissingConfigurationException("remote host is not configured")
|
||||||
return f"{self.url}/{path}"
|
return f"{self.url}/{path}"
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
from mdrsclient.models.doi import Doi
|
||||||
from mdrsclient.models.error import DRFStandardizedErrors
|
from mdrsclient.models.error import DRFStandardizedErrors
|
||||||
from mdrsclient.models.file import File
|
from mdrsclient.models.file import File
|
||||||
from mdrsclient.models.folder import Folder, FolderAccessLevel, FolderSimple
|
from mdrsclient.models.folder import Folder, FolderAccessLevel, FolderSimple
|
||||||
@@ -6,6 +7,7 @@ from mdrsclient.models.user import Token, User
|
|||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"DRFStandardizedErrors",
|
"DRFStandardizedErrors",
|
||||||
|
"Doi",
|
||||||
"File",
|
"File",
|
||||||
"Folder",
|
"Folder",
|
||||||
"FolderAccessLevel",
|
"FolderAccessLevel",
|
||||||
|
|||||||
@@ -0,0 +1,11 @@
|
|||||||
|
from pydantic.dataclasses import dataclass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Doi:
|
||||||
|
"""Model representing a DOI entity (Response from GET v3/doi/{id}/)."""
|
||||||
|
|
||||||
|
# The internal DOI suffix ID returned as a string (e.g. "20260429-001").
|
||||||
|
id: str
|
||||||
|
doi: str
|
||||||
|
folder_id: str
|
||||||
@@ -1,4 +1,5 @@
|
|||||||
from typing import Any
|
from typing import Any
|
||||||
|
from unicodedata import normalize
|
||||||
|
|
||||||
from pydantic.dataclasses import dataclass
|
from pydantic.dataclasses import dataclass
|
||||||
|
|
||||||
@@ -25,3 +26,8 @@ class File:
|
|||||||
@property
|
@property
|
||||||
def updated_at_name(self) -> str:
|
def updated_at_name(self) -> str:
|
||||||
return iso8601_to_user_friendly(self.updated_at)
|
return iso8601_to_user_friendly(self.updated_at)
|
||||||
|
|
||||||
|
|
||||||
|
def find_file(files: list[File], name: str) -> File | None:
|
||||||
|
_name = normalize("NFC", name).lower()
|
||||||
|
return next((x for x in files if x.name.lower() == _name), None)
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ from unicodedata import normalize
|
|||||||
|
|
||||||
from pydantic.dataclasses import dataclass
|
from pydantic.dataclasses import dataclass
|
||||||
|
|
||||||
from mdrsclient.models.file import File
|
|
||||||
from mdrsclient.models.utils import iso8601_to_user_friendly
|
from mdrsclient.models.utils import iso8601_to_user_friendly
|
||||||
|
|
||||||
|
|
||||||
@@ -78,13 +77,8 @@ class FolderSimple:
|
|||||||
class Folder(FolderSimple):
|
class Folder(FolderSimple):
|
||||||
metadata: list[dict[str, Any]]
|
metadata: list[dict[str, Any]]
|
||||||
sub_folders: list[FolderSimple]
|
sub_folders: list[FolderSimple]
|
||||||
files: list[File]
|
|
||||||
path: str
|
path: str
|
||||||
|
|
||||||
def find_sub_folder(self, name: str) -> FolderSimple | None:
|
def find_sub_folder(self, name: str) -> FolderSimple | None:
|
||||||
_name = normalize("NFC", name).lower()
|
_name = normalize("NFC", name).lower()
|
||||||
return next((x for x in self.sub_folders if x.name.lower() == _name), None)
|
return next((x for x in self.sub_folders if x.name.lower() == _name), None)
|
||||||
|
|
||||||
def find_file(self, name: str) -> File | None:
|
|
||||||
_name = normalize("NFC", name).lower()
|
|
||||||
return next((x for x in self.files if x.name.lower() == _name), None)
|
|
||||||
|
|||||||
@@ -0,0 +1,233 @@
|
|||||||
|
import os
|
||||||
|
import re
|
||||||
|
from typing import Any
|
||||||
|
from unicodedata import normalize
|
||||||
|
|
||||||
|
from mdrsclient.api import DoiApi, FilesApi, FoldersApi, LaboratoriesApi, UsersApi
|
||||||
|
from mdrsclient.cache import CacheInterface
|
||||||
|
from mdrsclient.config import ConfigFile, ConfigInterface
|
||||||
|
from mdrsclient.connection import MDRSConnection
|
||||||
|
from mdrsclient.exceptions import (
|
||||||
|
IllegalArgumentException,
|
||||||
|
MissingConfigurationException,
|
||||||
|
UnauthorizedException,
|
||||||
|
UnexpectedException,
|
||||||
|
)
|
||||||
|
from mdrsclient.models import File, Folder, Laboratory, Token, User
|
||||||
|
from mdrsclient.utils import page_num_from_url
|
||||||
|
|
||||||
|
|
||||||
|
class MdrsService:
|
||||||
|
config_class: type[ConfigInterface] = ConfigFile
|
||||||
|
|
||||||
|
def __init__(self, connection: MDRSConnection, config_class: type[ConfigInterface] | None = None):
|
||||||
|
self.connection = connection
|
||||||
|
if config_class is not None:
|
||||||
|
self.config_class = config_class
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def create_connection(
|
||||||
|
cls, remote: str, cache: CacheInterface | None = None, config: ConfigInterface | None = None
|
||||||
|
) -> MDRSConnection:
|
||||||
|
if config is None:
|
||||||
|
config = ConfigFile(remote)
|
||||||
|
elif config.remote != remote:
|
||||||
|
raise IllegalArgumentException("Remote host parameter mismatch.")
|
||||||
|
if config.url is None:
|
||||||
|
raise MissingConfigurationException(f"Remote host `{remote}` is not found.")
|
||||||
|
return MDRSConnection(config.remote, config.url, cache=cache)
|
||||||
|
|
||||||
|
def login(self, username: str, password: str) -> tuple[Token, User]:
|
||||||
|
user_api = UsersApi(self.connection)
|
||||||
|
token = user_api.token(username, password)
|
||||||
|
self.connection.token = token
|
||||||
|
user = user_api.current()
|
||||||
|
self.connection.user = user
|
||||||
|
return token, user
|
||||||
|
|
||||||
|
def logout(self) -> None:
|
||||||
|
self.connection.logout()
|
||||||
|
|
||||||
|
def whoami(self) -> User:
|
||||||
|
user_api = UsersApi(self.connection)
|
||||||
|
return user_api.current()
|
||||||
|
|
||||||
|
def get_laboratories(self) -> list[Laboratory]:
|
||||||
|
laboratory_api = LaboratoriesApi(self.connection)
|
||||||
|
labs = laboratory_api.list()
|
||||||
|
self.connection.laboratories = labs
|
||||||
|
return list(labs)
|
||||||
|
|
||||||
|
def find_laboratory(self, name: str) -> Laboratory:
|
||||||
|
if self.connection.laboratories.empty() or (self.connection.token and self.connection.token.is_expired):
|
||||||
|
self.get_laboratories()
|
||||||
|
laboratory = self.connection.laboratories.find_by_name(name)
|
||||||
|
if laboratory is None:
|
||||||
|
raise IllegalArgumentException(f"Laboratory `{name}` not found.")
|
||||||
|
return laboratory
|
||||||
|
|
||||||
|
def find_folder(self, laboratory: Laboratory, path: str, password: str | None = None) -> Folder:
|
||||||
|
folder_api = FoldersApi(self.connection)
|
||||||
|
folders = folder_api.list(laboratory.id, normalize("NFC", path))
|
||||||
|
if len(folders) != 1:
|
||||||
|
raise UnexpectedException(f"Folder `{path}` not found.")
|
||||||
|
if folders[0].lock:
|
||||||
|
if password is None:
|
||||||
|
raise UnauthorizedException(f"Folder `{path}` is locked.")
|
||||||
|
folder_api.auth(folders[0].id, password)
|
||||||
|
return folder_api.retrieve(folders[0].id)
|
||||||
|
|
||||||
|
def find_files(self, folder_id: str) -> list[File]:
|
||||||
|
files_api = FilesApi(self.connection)
|
||||||
|
page = 1
|
||||||
|
results_file = []
|
||||||
|
while page:
|
||||||
|
result = files_api.list(folder_id, page)
|
||||||
|
results_file.extend(result.results)
|
||||||
|
page = 0
|
||||||
|
if result.next:
|
||||||
|
page = page_num_from_url(result.next)
|
||||||
|
return results_file
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def is_doi(path_component: str) -> bool:
|
||||||
|
return path_component.startswith("10.") and "/" in path_component
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def doi_suffix_id(doi: str) -> str:
|
||||||
|
doi = doi.rstrip("/")
|
||||||
|
slash_pos = doi.find("/")
|
||||||
|
if slash_pos == -1:
|
||||||
|
return doi
|
||||||
|
suffix = doi[slash_pos + 1 :]
|
||||||
|
dot_pos = suffix.rfind(".")
|
||||||
|
return suffix[dot_pos + 1 :] if dot_pos != -1 else suffix
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def split_doi_and_subpath(doi_with_path: str) -> tuple[str, str]:
|
||||||
|
first_slash = doi_with_path.find("/")
|
||||||
|
if first_slash != -1:
|
||||||
|
after_suffix_start = first_slash + 1
|
||||||
|
after_first = doi_with_path[after_suffix_start:]
|
||||||
|
second_slash = after_first.find("/")
|
||||||
|
if second_slash != -1:
|
||||||
|
doi_end = after_suffix_start + second_slash
|
||||||
|
doi = doi_with_path[:doi_end]
|
||||||
|
subpath = doi_with_path[doi_end:]
|
||||||
|
if subpath == "/":
|
||||||
|
return (doi, "")
|
||||||
|
else:
|
||||||
|
return (doi, subpath)
|
||||||
|
else:
|
||||||
|
return (doi_with_path, "")
|
||||||
|
else:
|
||||||
|
return (doi_with_path, "")
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def parse_remote_host(cls, path: str) -> str:
|
||||||
|
path_array = path.split(":")
|
||||||
|
remote_host = path_array[0]
|
||||||
|
if len(path_array) == 2 and path_array[1] != "" or len(path_array) > 2:
|
||||||
|
raise IllegalArgumentException("Invalid remote host")
|
||||||
|
return remote_host
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def parse_remote_host_with_path(cls, path: str) -> tuple[str, str, str]:
|
||||||
|
path = re.sub(r"//+|/\./+|/\.$", "/", path)
|
||||||
|
if re.search(r"/\.\./|/\.\.$", path) is not None:
|
||||||
|
raise IllegalArgumentException("Path traversal found.")
|
||||||
|
path_array = path.split(":")
|
||||||
|
if len(path_array) != 2:
|
||||||
|
raise IllegalArgumentException("Invalid remote host.")
|
||||||
|
remote_host = path_array[0]
|
||||||
|
folder_array = path_array[1].split("/")
|
||||||
|
is_absolute_path = folder_array[0] == ""
|
||||||
|
if not is_absolute_path:
|
||||||
|
raise IllegalArgumentException("Must be absolute paths.")
|
||||||
|
del folder_array[0]
|
||||||
|
if len(folder_array) == 0:
|
||||||
|
laboratory = ""
|
||||||
|
folder = ""
|
||||||
|
else:
|
||||||
|
laboratory = folder_array.pop(0)
|
||||||
|
folder = "/" + "/".join(folder_array)
|
||||||
|
return (remote_host, laboratory, folder)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def parse_doi_remote_host(cls, path: str) -> tuple[str, str, str]:
|
||||||
|
parts = path.split(":", 1)
|
||||||
|
if len(parts) != 2:
|
||||||
|
raise IllegalArgumentException("remote_path must be in the form 'remote:10.xxxx/prefix.ID'")
|
||||||
|
remote, doi_with_path = parts
|
||||||
|
if not cls.is_doi(doi_with_path):
|
||||||
|
raise IllegalArgumentException(f"Path `{doi_with_path}` does not look like a DOI.")
|
||||||
|
doi, subpath = cls.split_doi_and_subpath(doi_with_path)
|
||||||
|
return (remote, doi, subpath)
|
||||||
|
|
||||||
|
def find_folder_by_doi(self, doi: str, password: str | None = None) -> tuple[Folder, Laboratory]:
|
||||||
|
doi_clean = doi.rstrip("/")
|
||||||
|
doi_id = self.doi_suffix_id(doi_clean)
|
||||||
|
doi_api = DoiApi(self.connection)
|
||||||
|
doi_resp = doi_api.retrieve(doi_id)
|
||||||
|
|
||||||
|
returned_doi = doi_resp.doi.rstrip("/")
|
||||||
|
if returned_doi.lower() != doi_clean.lower():
|
||||||
|
raise IllegalArgumentException(
|
||||||
|
f"DOI mismatch: requested `{doi_clean}` but server returned `{returned_doi}`."
|
||||||
|
)
|
||||||
|
|
||||||
|
folder_api = FoldersApi(self.connection)
|
||||||
|
folder = folder_api.retrieve(doi_resp.folder_id)
|
||||||
|
|
||||||
|
if folder.lock:
|
||||||
|
if password is None:
|
||||||
|
raise UnauthorizedException(f"Folder for DOI `{doi_clean}` is locked.")
|
||||||
|
folder_api.auth(doi_resp.folder.id, password)
|
||||||
|
|
||||||
|
lab_api = LaboratoriesApi(self.connection)
|
||||||
|
labs = lab_api.list()
|
||||||
|
lab = labs.find_by_id(folder.laboratory_id)
|
||||||
|
if lab is None:
|
||||||
|
raise UnexpectedException(f"Laboratory with id {folder.laboratory_id} not found.")
|
||||||
|
|
||||||
|
self.connection.laboratories = labs
|
||||||
|
return (folder, lab)
|
||||||
|
|
||||||
|
def resolve_folder(self, remote_path: str, password: str | None = None) -> tuple[Folder, Laboratory]:
|
||||||
|
path_component = remote_path.split(":", 1)[1] if ":" in remote_path else ""
|
||||||
|
if self.is_doi(path_component):
|
||||||
|
remote, doi, subpath = self.parse_doi_remote_host(remote_path)
|
||||||
|
doi_folder, laboratory = self.find_folder_by_doi(doi, password)
|
||||||
|
if not subpath:
|
||||||
|
return (doi_folder, laboratory)
|
||||||
|
else:
|
||||||
|
abs_path = doi_folder.path.rstrip("/") + subpath
|
||||||
|
folder = self.find_folder(laboratory, abs_path, password)
|
||||||
|
return (folder, laboratory)
|
||||||
|
else:
|
||||||
|
remote, laboratory_name, r_path = self.parse_remote_host_with_path(remote_path)
|
||||||
|
laboratory = self.find_laboratory(laboratory_name)
|
||||||
|
folder = self.find_folder(laboratory, r_path, password)
|
||||||
|
return (folder, laboratory)
|
||||||
|
|
||||||
|
def resolve_file(self, remote_path: str, password: str | None = None) -> tuple[Folder, Laboratory, str]:
|
||||||
|
path_component = remote_path.split(":", 1)[1] if ":" in remote_path else ""
|
||||||
|
if self.is_doi(path_component):
|
||||||
|
remote, doi, subpath = self.parse_doi_remote_host(remote_path)
|
||||||
|
doi_folder, laboratory = self.find_folder_by_doi(doi, password)
|
||||||
|
subpath_clean = subpath.rstrip("/")
|
||||||
|
if not subpath_clean:
|
||||||
|
raise IllegalArgumentException("DOI path must point to a file, not a folder.")
|
||||||
|
r_dirname = os.path.dirname(subpath_clean)
|
||||||
|
r_basename = os.path.basename(subpath_clean)
|
||||||
|
abs_path = doi_folder.path.rstrip("/") + r_dirname
|
||||||
|
parent_folder = self.find_folder(laboratory, abs_path, password)
|
||||||
|
return (parent_folder, laboratory, r_basename)
|
||||||
|
else:
|
||||||
|
remote, laboratory_name, r_path = self.parse_remote_host_with_path(remote_path)
|
||||||
|
r_path = r_path.rstrip("/")
|
||||||
|
r_dirname = os.path.dirname(r_path)
|
||||||
|
r_basename = os.path.basename(r_path)
|
||||||
|
laboratory = self.find_laboratory(laboratory_name)
|
||||||
|
parent_folder = self.find_folder(laboratory, r_dirname, password)
|
||||||
|
return (parent_folder, laboratory, r_basename)
|
||||||
@@ -0,0 +1,263 @@
|
|||||||
|
import os
|
||||||
|
from concurrent.futures import ThreadPoolExecutor
|
||||||
|
from typing import Any
|
||||||
|
from unicodedata import normalize
|
||||||
|
|
||||||
|
from pydantic.dataclasses import dataclass
|
||||||
|
|
||||||
|
from mdrsclient.api import FilesApi, FoldersApi
|
||||||
|
from mdrsclient.exceptions import IllegalArgumentException, MDRSException, UnexpectedException
|
||||||
|
from mdrsclient.models import File, Folder, Laboratory
|
||||||
|
from mdrsclient.models.file import find_file
|
||||||
|
from mdrsclient.settings import CONCURRENT
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class UploadFileInfo:
|
||||||
|
folder: Folder
|
||||||
|
files: list[File]
|
||||||
|
path: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class DownloadFileInfo:
|
||||||
|
file: File
|
||||||
|
path: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class DownloadContext:
|
||||||
|
hasError: bool
|
||||||
|
isSkipIfExists: bool
|
||||||
|
files: list[DownloadFileInfo]
|
||||||
|
|
||||||
|
|
||||||
|
class Uploader:
|
||||||
|
def __init__(self, client: Any) -> None:
|
||||||
|
self.client = client
|
||||||
|
|
||||||
|
def upload(
|
||||||
|
self, local_path: str, remote_path: str, is_recursive: bool = False, is_skip_if_exists: bool = False
|
||||||
|
) -> None:
|
||||||
|
remote, laboratory_name, r_path = self.client.parse_remote_host_with_path(remote_path)
|
||||||
|
l_path = os.path.abspath(local_path)
|
||||||
|
if not os.path.exists(l_path):
|
||||||
|
raise IllegalArgumentException(f"File or directory `{local_path}` not found.")
|
||||||
|
|
||||||
|
laboratory = self.client.find_laboratory(laboratory_name)
|
||||||
|
folder = self.client.find_folder(laboratory, r_path)
|
||||||
|
files = self.client.find_files(folder.id)
|
||||||
|
infos: list[UploadFileInfo] = []
|
||||||
|
if os.path.isdir(l_path):
|
||||||
|
if not is_recursive:
|
||||||
|
raise IllegalArgumentException(f"Cannot upload `{local_path}`: Is a directory.")
|
||||||
|
folder_api = FoldersApi(self.client.connection)
|
||||||
|
folder_map: dict[str, Folder] = {}
|
||||||
|
folder_map[r_path] = folder
|
||||||
|
files_map: dict[str, list[File]] = {}
|
||||||
|
files_map[r_path] = files
|
||||||
|
l_basename = os.path.basename(l_path)
|
||||||
|
for dirpath, _, filenames in os.walk(l_path, followlinks=True):
|
||||||
|
sub = l_basename if dirpath == l_path else os.path.join(l_basename, os.path.relpath(dirpath, l_path))
|
||||||
|
d_dirname = os.path.join(r_path, sub)
|
||||||
|
d_basename = os.path.basename(d_dirname)
|
||||||
|
# prepare destination parent path
|
||||||
|
d_parent_dirname = os.path.dirname(d_dirname)
|
||||||
|
if folder_map.get(d_parent_dirname) is None:
|
||||||
|
parent_folder = self.client.find_folder(laboratory, d_parent_dirname)
|
||||||
|
folder_map[d_parent_dirname] = parent_folder
|
||||||
|
parent_files = self.client.find_files(parent_folder.id)
|
||||||
|
files_map[d_parent_dirname] = parent_files
|
||||||
|
# prepare destination path
|
||||||
|
if folder_map.get(d_dirname) is None:
|
||||||
|
d_folder = folder_map[d_parent_dirname].find_sub_folder(d_basename)
|
||||||
|
if d_folder is None:
|
||||||
|
d_folder_id = folder_api.create(normalize("NFC", d_basename), folder_map[d_parent_dirname].id)
|
||||||
|
else:
|
||||||
|
d_folder_id = d_folder.id
|
||||||
|
print(d_dirname)
|
||||||
|
folder_map[d_dirname] = folder_api.retrieve(d_folder_id)
|
||||||
|
files_map[d_dirname] = self.client.find_files(d_folder_id)
|
||||||
|
if d_folder is None:
|
||||||
|
folder_map[d_parent_dirname].sub_folders.append(folder_map[d_dirname])
|
||||||
|
# register upload file list
|
||||||
|
for filename in filenames:
|
||||||
|
infos.append(
|
||||||
|
UploadFileInfo(folder_map[d_dirname], files_map[d_dirname], os.path.join(dirpath, filename))
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
infos.append(UploadFileInfo(folder, files, l_path))
|
||||||
|
self.__multiple_upload(infos, is_skip_if_exists)
|
||||||
|
|
||||||
|
def __multiple_upload(self, infos: list[UploadFileInfo], is_skip_if_exists: bool) -> None:
|
||||||
|
file_api = FilesApi(self.client.connection)
|
||||||
|
with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
|
||||||
|
pool.map(lambda x: self.__multiple_upload_worker(file_api, x, is_skip_if_exists), infos)
|
||||||
|
|
||||||
|
def __multiple_upload_worker(self, file_api: FilesApi, info: UploadFileInfo, is_skip_if_exists: bool) -> None:
|
||||||
|
basename = os.path.basename(info.path)
|
||||||
|
file = find_file(info.files, basename)
|
||||||
|
try:
|
||||||
|
if file is None:
|
||||||
|
file_api.create(info.folder.id, info.path)
|
||||||
|
elif not is_skip_if_exists or file.size != os.path.getsize(info.path):
|
||||||
|
file_api.update(file, info.path)
|
||||||
|
print(os.path.join(info.folder.path, basename))
|
||||||
|
except MDRSException as e:
|
||||||
|
print(f"Error: {e}")
|
||||||
|
|
||||||
|
|
||||||
|
class Downloader:
|
||||||
|
def __init__(self, client: Any) -> None:
|
||||||
|
self.client = client
|
||||||
|
|
||||||
|
def download(
|
||||||
|
self,
|
||||||
|
remote_path: str,
|
||||||
|
local_path: str,
|
||||||
|
is_recursive: bool = False,
|
||||||
|
is_skip_if_exists: bool = False,
|
||||||
|
password: str | None = None,
|
||||||
|
excludes: list[str] | None = None,
|
||||||
|
) -> None:
|
||||||
|
excludes_clean = excludes or []
|
||||||
|
# Detect DOI path: "remote:10.xxxx/prefix.ID[/optional/sub/path]"
|
||||||
|
path_component = remote_path.split(":", 1)[1] if ":" in remote_path else ""
|
||||||
|
if self.client.is_doi(path_component):
|
||||||
|
remote, doi, subpath = self.client.parse_doi_remote_host(remote_path)
|
||||||
|
|
||||||
|
l_dirname = os.path.realpath(local_path)
|
||||||
|
if not os.path.isdir(l_dirname):
|
||||||
|
raise IllegalArgumentException(f"Local directory `{local_path}` not found.")
|
||||||
|
doi_folder, laboratory = self.client.find_folder_by_doi(doi, password)
|
||||||
|
|
||||||
|
subpath_clean = subpath.rstrip("/")
|
||||||
|
if not subpath_clean:
|
||||||
|
folder = doi_folder
|
||||||
|
is_folder = True
|
||||||
|
else:
|
||||||
|
r_dirname = os.path.dirname(subpath_clean)
|
||||||
|
r_basename = os.path.basename(subpath_clean)
|
||||||
|
abs_path = doi_folder.path.rstrip("/") + r_dirname
|
||||||
|
r_parent_folder = self.client.find_folder(laboratory, abs_path, password)
|
||||||
|
r_parent_files = self.client.find_files(r_parent_folder.id)
|
||||||
|
file = find_file(r_parent_files, r_basename)
|
||||||
|
if file is not None:
|
||||||
|
if self.__check_excludes(excludes_clean, laboratory, r_parent_folder, file):
|
||||||
|
return
|
||||||
|
context = DownloadContext(False, is_skip_if_exists, [])
|
||||||
|
l_path = os.path.join(l_dirname, r_basename)
|
||||||
|
context.files.append(DownloadFileInfo(file, l_path))
|
||||||
|
self.__multiple_download(context)
|
||||||
|
return
|
||||||
|
else:
|
||||||
|
folder_simple = r_parent_folder.find_sub_folder(r_basename)
|
||||||
|
if folder_simple is None:
|
||||||
|
raise IllegalArgumentException(f"File or folder `{subpath_clean}` not found.")
|
||||||
|
folder = FoldersApi(self.client.connection).retrieve(folder_simple.id)
|
||||||
|
is_folder = True
|
||||||
|
|
||||||
|
# For a DOI target the whole folder is the download target.
|
||||||
|
if not is_recursive:
|
||||||
|
# Non-recursive: download only the files at the top level of the DOI folder.
|
||||||
|
files = self.client.find_files(folder.id)
|
||||||
|
context = DownloadContext(False, is_skip_if_exists, [])
|
||||||
|
for file in files:
|
||||||
|
if self.__check_excludes(excludes_clean, laboratory, folder, file):
|
||||||
|
continue
|
||||||
|
l_path = os.path.join(l_dirname, file.name)
|
||||||
|
context.files.append(DownloadFileInfo(file, l_path))
|
||||||
|
self.__multiple_download(context)
|
||||||
|
return
|
||||||
|
folder_api = FoldersApi(self.client.connection)
|
||||||
|
self.__multiple_download_pickup_recursive_files(
|
||||||
|
folder_api, laboratory, folder.id, l_dirname, excludes_clean, is_skip_if_exists
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
remote, laboratory_name, r_path = self.client.parse_remote_host_with_path(remote_path)
|
||||||
|
r_path = r_path.rstrip("/")
|
||||||
|
r_dirname = os.path.dirname(r_path)
|
||||||
|
r_basename = os.path.basename(r_path)
|
||||||
|
|
||||||
|
l_dirname = os.path.realpath(local_path)
|
||||||
|
if not os.path.isdir(l_dirname):
|
||||||
|
raise IllegalArgumentException(f"Local directory `{local_path}` not found.")
|
||||||
|
laboratory = self.client.find_laboratory(laboratory_name)
|
||||||
|
r_parent_folder = self.client.find_folder(laboratory, r_dirname, password)
|
||||||
|
r_parent_files = self.client.find_files(r_parent_folder.id)
|
||||||
|
file = find_file(r_parent_files, r_basename)
|
||||||
|
if file is not None:
|
||||||
|
if self.__check_excludes(excludes_clean, laboratory, r_parent_folder, file):
|
||||||
|
return
|
||||||
|
context = DownloadContext(False, is_skip_if_exists, [])
|
||||||
|
l_path = os.path.join(l_dirname, r_basename)
|
||||||
|
context.files.append(DownloadFileInfo(file, l_path))
|
||||||
|
self.__multiple_download(context)
|
||||||
|
else:
|
||||||
|
folder = r_parent_folder.find_sub_folder(r_basename)
|
||||||
|
if folder is None:
|
||||||
|
raise IllegalArgumentException(f"File or folder `{r_path}` not found.")
|
||||||
|
if not is_recursive:
|
||||||
|
raise IllegalArgumentException(f"Cannot download `{r_path}`: Is a folder.")
|
||||||
|
folder_api = FoldersApi(self.client.connection)
|
||||||
|
self.__multiple_download_pickup_recursive_files(
|
||||||
|
folder_api, laboratory, folder.id, l_dirname, excludes_clean, is_skip_if_exists
|
||||||
|
)
|
||||||
|
|
||||||
|
def __multiple_download_pickup_recursive_files(
|
||||||
|
self,
|
||||||
|
folder_api: FoldersApi,
|
||||||
|
laboratory: Laboratory,
|
||||||
|
folder_id: str,
|
||||||
|
basedir: str,
|
||||||
|
excludes: list[str],
|
||||||
|
is_skip_if_exists: bool,
|
||||||
|
) -> None:
|
||||||
|
context = DownloadContext(False, is_skip_if_exists, [])
|
||||||
|
folder = folder_api.retrieve(folder_id)
|
||||||
|
files = self.client.find_files(folder.id)
|
||||||
|
dirname = os.path.join(basedir, folder.name)
|
||||||
|
if self.__check_excludes(excludes, laboratory, folder, None):
|
||||||
|
return
|
||||||
|
if not os.path.exists(dirname):
|
||||||
|
os.makedirs(dirname)
|
||||||
|
print(dirname)
|
||||||
|
for file in files:
|
||||||
|
if self.__check_excludes(excludes, laboratory, folder, file):
|
||||||
|
continue
|
||||||
|
path = os.path.join(dirname, file.name)
|
||||||
|
context.files.append(DownloadFileInfo(file, path))
|
||||||
|
self.__multiple_download(context)
|
||||||
|
if context.hasError:
|
||||||
|
raise UnexpectedException("Some files failed to download.")
|
||||||
|
for sub_folder in folder.sub_folders:
|
||||||
|
self.__multiple_download_pickup_recursive_files(
|
||||||
|
folder_api, laboratory, sub_folder.id, dirname, excludes, is_skip_if_exists
|
||||||
|
)
|
||||||
|
|
||||||
|
def __multiple_download(self, context: DownloadContext) -> None:
|
||||||
|
file_api = FilesApi(self.client.connection)
|
||||||
|
with ThreadPoolExecutor(max_workers=CONCURRENT) as pool:
|
||||||
|
results = pool.map(
|
||||||
|
lambda x: self.__multiple_download_worker(file_api, x, context.isSkipIfExists), context.files
|
||||||
|
)
|
||||||
|
hasError = next(filter(lambda x: x is False, results), None)
|
||||||
|
if hasError is not None:
|
||||||
|
context.hasError = True
|
||||||
|
|
||||||
|
def __multiple_download_worker(self, file_api: FilesApi, info: DownloadFileInfo, is_skip_if_exists: bool) -> bool:
|
||||||
|
if not is_skip_if_exists or not os.path.exists(info.path) or info.file.size != os.path.getsize(info.path):
|
||||||
|
try:
|
||||||
|
file_api.download(info.file, info.path)
|
||||||
|
except Exception:
|
||||||
|
print(f"Failed: {info.path}")
|
||||||
|
if os.path.isfile(info.path):
|
||||||
|
os.remove(info.path)
|
||||||
|
return False
|
||||||
|
print(info.path)
|
||||||
|
return True
|
||||||
|
|
||||||
|
def __check_excludes(self, excludes: list[str], laboratory: Laboratory, folder: Folder, file: File | None) -> bool:
|
||||||
|
path = f"/{laboratory.name}{folder.path}{file.name if file is not None else ''}".rstrip("/").lower()
|
||||||
|
return path in excludes
|
||||||
@@ -1,5 +1,6 @@
|
|||||||
import os
|
import os
|
||||||
from typing import IO, Any
|
from typing import IO, Any
|
||||||
|
from urllib.parse import parse_qs, urlparse
|
||||||
|
|
||||||
if os.name == "nt":
|
if os.name == "nt":
|
||||||
import msvcrt
|
import msvcrt
|
||||||
@@ -21,3 +22,10 @@ class FileLock:
|
|||||||
msvcrt.locking(file.fileno(), msvcrt.LK_UNLCK, 1)
|
msvcrt.locking(file.fileno(), msvcrt.LK_UNLCK, 1)
|
||||||
elif os.name == "posix":
|
elif os.name == "posix":
|
||||||
fcntl.flock(file.fileno(), fcntl.LOCK_UN)
|
fcntl.flock(file.fileno(), fcntl.LOCK_UN)
|
||||||
|
|
||||||
|
|
||||||
|
def page_num_from_url(url: str) -> int | None:
|
||||||
|
parsed_url = urlparse(url)
|
||||||
|
params = parse_qs(parsed_url.query)
|
||||||
|
page = params.get("page", [None])[0]
|
||||||
|
return int(page) if page is not None else None
|
||||||
|
|||||||
+14
-12
@@ -1,18 +1,20 @@
|
|||||||
[tool.poetry]
|
[tool.poetry]
|
||||||
name = "mdrs-client-python"
|
name = "mdrs-client-python"
|
||||||
version = "1.3.6"
|
version = "1.3.18"
|
||||||
description = "The mdrs-client-python is python library and a command-line client for up- and downloading files to and from MDRS based repository."
|
description = "The mdrs-client-python is python library and a command-line client for up- and downloading files to and from MDRS based repository."
|
||||||
authors = ["Yoshihiro OKUMURA <yoshihiro.okumura@riken.jp>"]
|
authors = ["Yoshihiro OKUMURA <yoshihiro.okumura@riken.jp>"]
|
||||||
license = "MIT"
|
license = "MIT"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
classifiers=[
|
classifiers=[
|
||||||
"Development Status :: 3 - Alpha",
|
"Development Status :: 4 - Beta",
|
||||||
"Environment :: Console",
|
"Environment :: Console",
|
||||||
"Intended Audience :: Developers",
|
"Intended Audience :: Developers",
|
||||||
"Intended Audience :: Science/Research",
|
"Intended Audience :: Science/Research",
|
||||||
"Programming Language :: Python :: 3.10",
|
"Programming Language :: Python :: 3.10",
|
||||||
"Programming Language :: Python :: 3.11",
|
"Programming Language :: Python :: 3.11",
|
||||||
"Programming Language :: Python :: 3.12",
|
"Programming Language :: Python :: 3.12",
|
||||||
|
"Programming Language :: Python :: 3.13",
|
||||||
|
"Programming Language :: Python :: 3.14",
|
||||||
"OSI Approved :: MIT License",
|
"OSI Approved :: MIT License",
|
||||||
"Topic :: Utilities",
|
"Topic :: Utilities",
|
||||||
]
|
]
|
||||||
@@ -22,20 +24,20 @@ packages = [
|
|||||||
|
|
||||||
[tool.poetry.dependencies]
|
[tool.poetry.dependencies]
|
||||||
python = "^3.10"
|
python = "^3.10"
|
||||||
requests = "^2.32.3"
|
requests = "^2.34.2"
|
||||||
requests-toolbelt = "^1.0.0"
|
requests-toolbelt = "^1.0.0"
|
||||||
python-dotenv = "^1.0.1"
|
python-dotenv = "^1.1.0"
|
||||||
pydantic = "^2.8.2"
|
pydantic = "^2.13.4"
|
||||||
pydantic-settings = "^2.3.4"
|
pydantic-settings = "^2.14.2"
|
||||||
PyJWT = "^2.8.0"
|
PyJWT = "^2.13.0"
|
||||||
validators = "^0.22.0"
|
validators = "^0.35.0"
|
||||||
|
|
||||||
[tool.poetry.group.dev.dependencies]
|
[tool.poetry.group.dev.dependencies]
|
||||||
black = "^24.2.2"
|
black = "^26.5.1"
|
||||||
flake8 = "^7.1.0"
|
flake8 = "^7.2.0"
|
||||||
Flake8-pyproject = "^1.2.3"
|
Flake8-pyproject = "^1.2.3"
|
||||||
isort = "^5.13.2"
|
isort = "^8.0.1"
|
||||||
pyright = "^1.1.370"
|
pyright = "^1.1.411"
|
||||||
|
|
||||||
[tool.poetry.scripts]
|
[tool.poetry.scripts]
|
||||||
mdrs = 'mdrsclient.__main__:main'
|
mdrs = 'mdrsclient.__main__:main'
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
# Mark tests directory as a Python package
|
||||||
@@ -0,0 +1,418 @@
|
|||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import unittest
|
||||||
|
from io import StringIO
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
from mdrsclient.client import MdrsClient
|
||||||
|
from mdrsclient.commands import (
|
||||||
|
ChaclCommand,
|
||||||
|
ConfigCommand,
|
||||||
|
CpCommand,
|
||||||
|
DownloadCommand,
|
||||||
|
FileMetadataCommand,
|
||||||
|
LabsCommand,
|
||||||
|
LoginCommand,
|
||||||
|
LogoutCommand,
|
||||||
|
LsCommand,
|
||||||
|
MetadataCommand,
|
||||||
|
MkdirCommand,
|
||||||
|
MvCommand,
|
||||||
|
RmCommand,
|
||||||
|
UploadCommand,
|
||||||
|
VersionCommand,
|
||||||
|
WhoamiCommand,
|
||||||
|
)
|
||||||
|
from mdrsclient.models import Folder, Laboratory
|
||||||
|
|
||||||
|
|
||||||
|
class TestCommands(unittest.TestCase):
|
||||||
|
def parse_args(self, cmd_class, args_list):
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
subparsers = parser.add_subparsers(title="subcommands")
|
||||||
|
cmd_class.register(subparsers)
|
||||||
|
return parser.parse_args(args_list)
|
||||||
|
|
||||||
|
def test_version_command(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.return_value = mock_client
|
||||||
|
mock_client.version.return_value = "mdrs 1.3.17"
|
||||||
|
|
||||||
|
args = self.parse_args(VersionCommand, ["version"])
|
||||||
|
with patch("sys.stdout", new=StringIO()) as fake_out:
|
||||||
|
args.func(args)
|
||||||
|
self.assertEqual(fake_out.getvalue().strip(), "mdrs 1.3.17")
|
||||||
|
|
||||||
|
mock_client_class.assert_called_once_with(None)
|
||||||
|
mock_client.version.assert_called_once()
|
||||||
|
|
||||||
|
def test_login_command_with_args(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
mock_client_class.parse_remote_host.return_value = "myremote"
|
||||||
|
|
||||||
|
args = self.parse_args(LoginCommand, ["login", "-u", "myuser", "-p", "mypass", "myremote"])
|
||||||
|
with patch("sys.stdout", new=StringIO()) as fake_out:
|
||||||
|
args.func(args)
|
||||||
|
self.assertIn("Login Successful", fake_out.getvalue())
|
||||||
|
|
||||||
|
mock_client_class.parse_remote_host.assert_called_once_with("myremote")
|
||||||
|
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||||
|
mock_client.login.assert_called_once_with("myuser", "mypass")
|
||||||
|
|
||||||
|
def test_login_command_interactive(self):
|
||||||
|
with (
|
||||||
|
patch("mdrsclient.client.MdrsClient") as mock_client_class,
|
||||||
|
patch("builtins.input", return_value="myuser_int") as mock_input,
|
||||||
|
patch("getpass.getpass", return_value="mypass_int") as mock_getpass,
|
||||||
|
):
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
mock_client_class.parse_remote_host.return_value = "myremote"
|
||||||
|
|
||||||
|
args = self.parse_args(LoginCommand, ["login", "myremote"])
|
||||||
|
with patch("sys.stdout", new=StringIO()) as fake_out:
|
||||||
|
args.func(args)
|
||||||
|
self.assertIn("Login Successful", fake_out.getvalue())
|
||||||
|
|
||||||
|
mock_input.assert_called_once_with("Username: ")
|
||||||
|
mock_getpass.assert_called_once_with("Password: ")
|
||||||
|
mock_client.login.assert_called_once_with("myuser_int", "mypass_int")
|
||||||
|
|
||||||
|
def test_logout_command(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
mock_client_class.parse_remote_host.return_value = "myremote"
|
||||||
|
|
||||||
|
args = self.parse_args(LogoutCommand, ["logout", "myremote"])
|
||||||
|
args.func(args)
|
||||||
|
|
||||||
|
mock_client_class.parse_remote_host.assert_called_once_with("myremote")
|
||||||
|
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||||
|
mock_client.logout.assert_called_once()
|
||||||
|
|
||||||
|
def test_whoami_command_logged_in(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
mock_client_class.parse_remote_host.return_value = "myremote"
|
||||||
|
mock_client.connection.token = None
|
||||||
|
|
||||||
|
mock_user = MagicMock()
|
||||||
|
mock_user.username = "test_user"
|
||||||
|
mock_client.whoami.return_value = mock_user
|
||||||
|
|
||||||
|
args = self.parse_args(WhoamiCommand, ["whoami", "myremote"])
|
||||||
|
with patch("sys.stdout", new=StringIO()) as fake_out:
|
||||||
|
args.func(args)
|
||||||
|
self.assertEqual(fake_out.getvalue().strip(), "test_user")
|
||||||
|
|
||||||
|
mock_client_class.parse_remote_host.assert_called_once_with("myremote")
|
||||||
|
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||||
|
mock_client.whoami.assert_called_once()
|
||||||
|
|
||||||
|
def test_whoami_command_anonymous(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
mock_client_class.parse_remote_host.return_value = "myremote"
|
||||||
|
mock_client.connection.token = None
|
||||||
|
mock_client.whoami.side_effect = Exception("Not logged in")
|
||||||
|
|
||||||
|
args = self.parse_args(WhoamiCommand, ["whoami", "myremote"])
|
||||||
|
with patch("sys.stdout", new=StringIO()) as fake_out:
|
||||||
|
args.func(args)
|
||||||
|
self.assertEqual(fake_out.getvalue().strip(), "(Anonymous)")
|
||||||
|
|
||||||
|
def test_labs_command(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
mock_client_class.parse_remote_host.return_value = "myremote"
|
||||||
|
|
||||||
|
mock_lab = MagicMock()
|
||||||
|
mock_lab.id = 1
|
||||||
|
mock_lab.name = "lab_name"
|
||||||
|
mock_lab.pi_name = "pi_name"
|
||||||
|
mock_lab.full_name = "full_name"
|
||||||
|
mock_client.get_laboratories.return_value = [mock_lab]
|
||||||
|
|
||||||
|
args = self.parse_args(LabsCommand, ["labs", "myremote"])
|
||||||
|
with patch("sys.stdout", new=StringIO()) as fake_out:
|
||||||
|
args.func(args)
|
||||||
|
output = fake_out.getvalue()
|
||||||
|
self.assertIn("Name", output)
|
||||||
|
self.assertIn("lab_name", output)
|
||||||
|
self.assertIn("pi_name", output)
|
||||||
|
|
||||||
|
mock_client_class.parse_remote_host.assert_called_once_with("myremote")
|
||||||
|
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||||
|
mock_client.get_laboratories.assert_called_once()
|
||||||
|
|
||||||
|
def test_ls_command_plain(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock(spec=MdrsClient)
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
|
||||||
|
mock_folder = Folder(
|
||||||
|
id="folder_id",
|
||||||
|
pid=None,
|
||||||
|
name="root",
|
||||||
|
access_level=1,
|
||||||
|
lock=False,
|
||||||
|
size=0,
|
||||||
|
laboratory_id=1,
|
||||||
|
description="",
|
||||||
|
created_at="2026-07-02T00:00:00Z",
|
||||||
|
updated_at="2026-07-02T00:00:00Z",
|
||||||
|
restrict_opened_at=None,
|
||||||
|
metadata=[],
|
||||||
|
sub_folders=[],
|
||||||
|
path="/root",
|
||||||
|
)
|
||||||
|
|
||||||
|
mock_lab = Laboratory(id=1, name="mylab", pi_name="pi_name", full_name="full_name")
|
||||||
|
|
||||||
|
mock_client.resolve_folder.return_value = (mock_folder, mock_lab)
|
||||||
|
|
||||||
|
mock_file = MagicMock()
|
||||||
|
mock_file.name = "file.txt"
|
||||||
|
mock_file.size = 100
|
||||||
|
mock_file.updated_at_name = "2026-07-02"
|
||||||
|
mock_client.find_files.return_value = [mock_file]
|
||||||
|
|
||||||
|
args = self.parse_args(LsCommand, ["ls", "myremote:/mylab/"])
|
||||||
|
with patch("sys.stdout", new=StringIO()) as fake_out:
|
||||||
|
args.func(args)
|
||||||
|
output = fake_out.getvalue()
|
||||||
|
self.assertIn("Type", output)
|
||||||
|
self.assertIn("file.txt", output)
|
||||||
|
|
||||||
|
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||||
|
mock_client.resolve_folder.assert_called_once_with("myremote:/mylab/", None)
|
||||||
|
mock_client.find_files.assert_called_once_with(mock_folder.id)
|
||||||
|
|
||||||
|
def test_ls_command_json(self):
|
||||||
|
with (
|
||||||
|
patch("mdrsclient.client.MdrsClient") as mock_client_class,
|
||||||
|
patch("mdrsclient.commands.ls.FoldersApi") as mock_folders_api_class,
|
||||||
|
):
|
||||||
|
mock_folders_api = MagicMock()
|
||||||
|
mock_folders_api_class.return_value = mock_folders_api
|
||||||
|
mock_folders_api.metadata.return_value = {"folder_meta": "val"}
|
||||||
|
|
||||||
|
mock_client = MagicMock(spec=MdrsClient)
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
mock_client.connection = MagicMock()
|
||||||
|
|
||||||
|
mock_folder = Folder(
|
||||||
|
id="folder_id",
|
||||||
|
pid="parent_id",
|
||||||
|
name="root",
|
||||||
|
access_level=1,
|
||||||
|
lock=False,
|
||||||
|
size=0,
|
||||||
|
laboratory_id=1,
|
||||||
|
description="Root folder",
|
||||||
|
created_at="2026-07-02T00:00:00Z",
|
||||||
|
updated_at="2026-07-02T00:00:00Z",
|
||||||
|
restrict_opened_at=None,
|
||||||
|
metadata=[],
|
||||||
|
sub_folders=[],
|
||||||
|
path="/root",
|
||||||
|
)
|
||||||
|
|
||||||
|
mock_lab = Laboratory(id=1, name="mylab", pi_name="pi_name", full_name="full_name")
|
||||||
|
|
||||||
|
mock_client.resolve_folder.return_value = (mock_folder, mock_lab)
|
||||||
|
mock_client.connection.laboratories.find_by_id.return_value = mock_lab
|
||||||
|
|
||||||
|
mock_file = MagicMock()
|
||||||
|
mock_file.id = "file_id"
|
||||||
|
mock_file.name = "file.txt"
|
||||||
|
mock_file.type = "text"
|
||||||
|
mock_file.size = 100
|
||||||
|
mock_file.description = "A file"
|
||||||
|
mock_file.metadata = {}
|
||||||
|
mock_file.download_url = "download/file"
|
||||||
|
mock_file.created_at = "2026-07-02T00:00:00Z"
|
||||||
|
mock_file.updated_at = "2026-07-02T00:00:00Z"
|
||||||
|
mock_client.find_files.return_value = [mock_file]
|
||||||
|
|
||||||
|
args = self.parse_args(LsCommand, ["ls", "-J", "myremote:/mylab/"])
|
||||||
|
with patch("sys.stdout", new=StringIO()) as fake_out:
|
||||||
|
args.func(args)
|
||||||
|
output = fake_out.getvalue()
|
||||||
|
parsed_json = json.loads(output)
|
||||||
|
self.assertEqual(parsed_json["name"], "root")
|
||||||
|
self.assertEqual(parsed_json["files"][0]["name"], "file.txt")
|
||||||
|
|
||||||
|
def test_mkdir_command(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
|
||||||
|
args = self.parse_args(MkdirCommand, ["mkdir", "myremote:/mylab/newfolder"])
|
||||||
|
args.func(args)
|
||||||
|
|
||||||
|
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||||
|
mock_client.mkdir.assert_called_once_with("myremote:/mylab/newfolder")
|
||||||
|
|
||||||
|
def test_rm_command_file(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
|
||||||
|
args = self.parse_args(RmCommand, ["rm", "myremote:/mylab/file.txt"])
|
||||||
|
args.func(args)
|
||||||
|
|
||||||
|
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||||
|
mock_client.rm.assert_called_once_with("myremote:/mylab/file.txt", False)
|
||||||
|
|
||||||
|
def test_rm_command_recursive(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
|
||||||
|
args = self.parse_args(RmCommand, ["rm", "-r", "myremote:/mylab/folder"])
|
||||||
|
args.func(args)
|
||||||
|
|
||||||
|
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||||
|
mock_client.rm.assert_called_once_with("myremote:/mylab/folder", True)
|
||||||
|
|
||||||
|
def test_cp_command(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
|
||||||
|
args = self.parse_args(CpCommand, ["cp", "myremote:/mylab/src", "myremote:/mylab/dest"])
|
||||||
|
args.func(args)
|
||||||
|
|
||||||
|
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||||
|
mock_client.cp.assert_called_once_with("myremote:/mylab/src", "myremote:/mylab/dest", False)
|
||||||
|
|
||||||
|
def test_mv_command(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
|
||||||
|
args = self.parse_args(MvCommand, ["mv", "myremote:/mylab/src", "myremote:/mylab/dest"])
|
||||||
|
args.func(args)
|
||||||
|
|
||||||
|
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||||
|
mock_client.mv.assert_called_once_with("myremote:/mylab/src", "myremote:/mylab/dest")
|
||||||
|
|
||||||
|
def test_chacl_command(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
|
||||||
|
args = self.parse_args(ChaclCommand, ["chacl", "-r", "-p", "secret", "private", "myremote:/mylab/"])
|
||||||
|
args.func(args)
|
||||||
|
|
||||||
|
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||||
|
mock_client.chacl.assert_called_once_with("myremote:/mylab/", 1, True, "secret")
|
||||||
|
|
||||||
|
def test_metadata_command(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
mock_client.metadata.return_value = {"meta_key": "meta_val"}
|
||||||
|
|
||||||
|
args = self.parse_args(MetadataCommand, ["metadata", "-p", "secret", "myremote:/mylab/"])
|
||||||
|
with patch("sys.stdout", new=StringIO()) as fake_out:
|
||||||
|
args.func(args)
|
||||||
|
parsed_json = json.loads(fake_out.getvalue())
|
||||||
|
self.assertEqual(parsed_json["meta_key"], "meta_val")
|
||||||
|
|
||||||
|
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||||
|
mock_client.metadata.assert_called_once_with("myremote:/mylab/", "secret")
|
||||||
|
|
||||||
|
def test_file_metadata_command(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
mock_client.file_metadata.return_value = {"file_meta": "val"}
|
||||||
|
|
||||||
|
args = self.parse_args(FileMetadataCommand, ["file-metadata", "-p", "secret", "myremote:/mylab/file.txt"])
|
||||||
|
with patch("sys.stdout", new=StringIO()) as fake_out:
|
||||||
|
args.func(args)
|
||||||
|
parsed_json = json.loads(fake_out.getvalue())
|
||||||
|
self.assertEqual(parsed_json["file_meta"], "val")
|
||||||
|
|
||||||
|
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||||
|
mock_client.file_metadata.assert_called_once_with("myremote:/mylab/file.txt", "secret")
|
||||||
|
|
||||||
|
def test_upload_command(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
|
||||||
|
args = self.parse_args(UploadCommand, ["upload", "-r", "-s", "local_file.txt", "myremote:/mylab/"])
|
||||||
|
args.func(args)
|
||||||
|
|
||||||
|
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||||
|
mock_client.upload.assert_called_once_with("local_file.txt", "myremote:/mylab/", True, True)
|
||||||
|
|
||||||
|
def test_download_command(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.from_remote.return_value = mock_client
|
||||||
|
|
||||||
|
args = self.parse_args(
|
||||||
|
DownloadCommand, ["download", "-r", "-s", "-p", "pass", "-e", "ex1", "myremote:/mylab/", "local_dir"]
|
||||||
|
)
|
||||||
|
args.func(args)
|
||||||
|
|
||||||
|
mock_client_class.from_remote.assert_called_once_with("myremote")
|
||||||
|
mock_client.download.assert_called_once_with("myremote:/mylab/", "local_dir", True, True, "pass", ["ex1"])
|
||||||
|
|
||||||
|
def test_config_create(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.return_value = mock_client
|
||||||
|
|
||||||
|
args = self.parse_args(ConfigCommand, ["config", "create", "myremote", "http://example.com"])
|
||||||
|
args.func(args)
|
||||||
|
|
||||||
|
mock_client_class.assert_called_once_with(None)
|
||||||
|
mock_client.config_create.assert_called_once_with("myremote", "http://example.com")
|
||||||
|
|
||||||
|
def test_config_update(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.return_value = mock_client
|
||||||
|
|
||||||
|
args = self.parse_args(ConfigCommand, ["config", "update", "myremote", "http://example.com"])
|
||||||
|
args.func(args)
|
||||||
|
|
||||||
|
mock_client_class.assert_called_once_with(None)
|
||||||
|
mock_client.config_update.assert_called_once_with("myremote", "http://example.com")
|
||||||
|
|
||||||
|
def test_config_delete(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.return_value = mock_client
|
||||||
|
|
||||||
|
args = self.parse_args(ConfigCommand, ["config", "delete", "myremote"])
|
||||||
|
args.func(args)
|
||||||
|
|
||||||
|
mock_client_class.assert_called_once_with(None)
|
||||||
|
mock_client.config_delete.assert_called_once_with("myremote")
|
||||||
|
|
||||||
|
def test_config_list(self):
|
||||||
|
with patch("mdrsclient.client.MdrsClient") as mock_client_class:
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_client_class.return_value = mock_client
|
||||||
|
mock_client.config_list.return_value = [("remote1", "url1"), ("remote2", "url2")]
|
||||||
|
|
||||||
|
args = self.parse_args(ConfigCommand, ["config", "list"])
|
||||||
|
with patch("sys.stdout", new=StringIO()) as fake_out:
|
||||||
|
args.func(args)
|
||||||
|
self.assertEqual(fake_out.getvalue(), "remote1:\turl1\nremote2:\turl2\n")
|
||||||
|
|
||||||
|
mock_client_class.assert_called_once_with(None)
|
||||||
|
mock_client.config_list.assert_called_once()
|
||||||
Reference in New Issue
Block a user