orrisroot
914dd729aa
Upload and download failures were printed and then forgotten: a batch that lost files still exited 0, and a file the client could not write was listed as if it had arrived. Nothing downstream could tell. - Return a verdict from every transfer worker and raise once at the end, so a run that lost a file exits 2. - Raise the error when a downloaded file cannot be written locally, instead of printing it and reporting the path as a success. - Write a download beside its destination and move it into place once complete, and refuse a destination that cannot be written, so a failed transfer no longer leaves a truncated file behind. - Carry on through the remaining sub-folders when a recursive download loses a file or cannot create a folder locally. - Name the file and give the reason in every failure message. - Count a failure the API layer did not raise, such as a file that disappeared between the directory walk and its turn to be sent. - Extract the directory walk from `Uploader.upload` to keep it within the complexity limit. - Cover all of the above in `tests/test_transfer.py`.
mdrs-client-python
The mdrs-client-python is python library and a command-line client for up- and downloading files to and from MDRS based repository.
Installation (Production)
It is recommended to use a virtual environment (venv) to keep your Python environment isolated.
Option 1: Using standard venv + pip
python3 -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install .
Option 2: Using uv
uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
uv pip install .
Development Setup
To set up a local development environment with development tools (testing, formatting, linting):
Option 1: Using standard venv + pip
python3 -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install -e ".[dev]"
Option 2: Using uv (Recommended for developers)
uv sync
CLI Usage
config create
Create remote host configuration
mdrs config create neurodata https://neurodata.riken.jp/api
config update
Update the URL of a registered remote host.
mdrs config update neurodata https://neurodata.riken.jp/api
config list
List registered remote hosts.
mdrs config list
mdrs config ls
config delete
Remove a registered remote host.
mdrs config delete neurodata
mdrs config rm neurodata
login
Login to remote host
mdrs login neurodata:
Username: (enter your login name)
Password: (enter your password)
mdrs login -u USERNAME -p PASSWORD neurodata:
logout
Logout from remote host
mdrs logout neurodata:
whoami
Print current user name
mdrs whoami neurodata:
labs
List all laboratories
mdrs labs neurodata:
ls
List the folder contents. You can also specify a DOI path in the form remote:10.xxxx/yyy.ID[/optional/subpath].
mdrs ls neurodata:/NIU/Repository/
mdrs ls -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/
mdrs ls -r neurodata:/NIU/Repository/Dataset1/
mdrs ls -J -r neurodata:/NIU/Repository/Dataset1/
# DOI access examples:
mdrs ls neurodata:10.60178/cbs.20260429-001
mdrs ls "neurodata:10.60178/cbs.20260429-001/Figure 1"
mkdir
Create a new folder
mdrs mkdir neurodata:/NIU/Repository/TEST
upload
Upload the file or directory
mdrs upload ./sample.dat neurodata:/NIU/Repository/TEST/
mdrs upload -r ./dataset neurodata:/NIU/Repository/TEST/
mdrs upload -r --skip-if-exists ./dataset neurodata:/NIU/Repository/TEST/
download
Download the file or folder. You can also specify a DOI path.
mdrs download neurodata:/NIU/Repository/TEST/sample.dat ./
mdrs download -r neurodata:/NIU/Repository/TEST/dataset/ ./
mdrs download -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/Readme.dat ./
mdrs download -r --exclude /NIU/Repository/TEST/dataset/skip neurodata:/NIU/Repository/TEST/dataset/ ./
mdrs download -r --skip-if-exists neurodata:/NIU/Repository/TEST/dataset/ ./
# DOI access examples:
mdrs download neurodata:10.60178/cbs.20260429-001/README_NeuroData.md ./
mdrs download -r "neurodata:10.60178/cbs.20260429-001/Figure 1" ./
mv
Move or rename the file or folder
mdrs mv neurodata:/NIU/Repository/TEST/sample.dat neurodata:/NIU/Repository/TEST2/sample2.dat
mdrs mv neurodata:/NIU/Repository/TEST/dataset neurodata:/NIU/Repository/TEST2/
cp
Copy the file and folder
mdrs cp neurodata:/NIU/Repository/TEST/sample.dat neurodata:/NIU/Repository/TEST2/sample2.dat
mdrs cp -r neurodata:/NIU/Repository/TEST/dataset neurodata:/NIU/Repository/TEST2/
rm
Remove the file or folder
mdrs rm neurodata:/NIU/Repository/TEST/sample.dat
mdrs rm -r neurodata:/NIU/Repository/TEST/dataset
chacl
Change the folder access level
mdrs chacl private neurodata:/NIU/Repository/Private
mdrs chacl cbs_open -r neurodata:/NIU/Repository/CBS_Open
mdrs chacl pw_open -r -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open
metadata
Get a folder metadata. You can also specify a DOI path.
mdrs metadata neurodata:/NIU/Repository/TEST/
mdrs metadata -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/
# DOI access examples:
mdrs metadata neurodata:10.60178/cbs.20260429-001
file-metadata
Get the file metadata. You can also specify a DOI path.
mdrs file-metadata neurodata:/NIU/Repository/TEST/dataset/sample.dat
mdrs file-metadata -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/Readme.txt
# DOI access examples:
mdrs file-metadata "neurodata:10.60178/cbs.20260429-001/Figure 1/Figure1v3.pdf"
version
Show the tool name and version number
mdrs version
help
Show the help message and exit
mdrs -h
Python API Usage
You can also use this package as a Python library to programmatically interact with MDRS repositories.
from mdrsclient.client import MdrsClient
from mdrsclient.cache import InMemoryCache
from mdrsclient.config import InMemoryConfig
# 1. Setup in-memory configuration and cache to avoid local state files (e.g., config.ini, cache/*.json)
config = InMemoryConfig("neurodata")
config.url = "https://neurodata.riken.jp/api"
cache = InMemoryCache()
# 2. Initialize client with custom configuration and cache
client = MdrsClient.from_remote("neurodata", cache=cache, config=config)
# 3. Login to the remote server
client.login("username", "password")
# 4. Use service methods
labs = client.get_laboratories()
metadata = client.metadata("neurodata:/NIU/Repository/")
# Transfer files programmatically
client.upload("/path/to/local/data", "neurodata:/NIU/Repository/TEST/", is_recursive=True)
client.download("neurodata:/NIU/Repository/TEST/data", "/path/to/local", is_recursive=True)
Testing
You can run the unit test suite using the standard library unittest discover runner:
.venv/bin/python -m unittest discover tests
Changelog
See CHANGELOG.md for the full change history.