15 Commits
Author SHA1 Message Date
Yoshihiro OKUMURA 663956ead7 docs: add a changelog
Release / build-linux-x86_64 (push) Successful in 2m20s
Release / build-linux-aarch64 (push) Successful in 1m57s
The project kept no change history of its own, so the tags were the
only record of what each release contained.

- Summarise 2.0.0, 2.0.1 and 2.0.2 by what changed for the user.
- Link the changelog from the README.
2026-09-04 18:23:33 +09:00
Yoshihiro OKUMURA cb5cff7b64 build: bump to 2.0.2 and update dependencies
- Raise tokio, base64, serde_json and anyhow to the releases they are
  tested against, and refresh the lockfile.
- base64 0.23 needed no change to the code that uses it.
2026-09-04 18:23:33 +09:00
Yoshihiro OKUMURA a16f73543d fix: stream transfers and close gaps against the Python client
A review against the Python client, which shares this client's config
and login cache, found the transfers holding whole files in memory and
several commands answering differently from their Python counterparts.

- Stream uploads and downloads instead of buffering the whole file, so
  memory no longer scales with file size times concurrency. Uploads
  still declare a Content-Length rather than going out chunked.
- Write a download beside its destination and move it into place once
  complete, and refuse a destination that cannot be written, so a
  failed transfer leaves what was already there untouched.
- Report the server's own error detail instead of the bare status.
- List locked sub-folders in `ls --json --recursive` using the given
  password and skip only those that cannot be unlocked, instead of
  failing the whole listing.
- Name each folder's laboratory in JSON output and sort its entries,
  matching the Python client's schema and order.
- Ask the server who is logged in for `whoami`, rather than trusting
  the cached name.
- Collapse repeated separators and refuse `..` in remote paths.
- Fall back to the API when a laboratory is missing from the cache, so
  a newly added one no longer needs a fresh login.
- Verify the login cache digest on read, as the Python client does.
- Accept `-e` as the short form of `--exclude`.
- Satisfy clippy and rustfmt across the crate.
2026-09-04 16:31:14 +09:00
Yoshihiro OKUMURA c24a285cf5 fix(selfupdate): verify the archive before replacing the binary
`mdrs selfupdate` replaced the running binary with whatever the release
endpoint returned, checking only that the transport succeeded. Nothing
proved the archive was the one the release publishes.

- Compare the downloaded archive against the release's `.sha256` asset
  and abort the update on a mismatch.
- Report a release that publishes no checksum as unverified, rather
  than letting its absence pass for a verified download.
- Exclude `.sha256` assets when matching the archive for the build
  target: those assets carry the target name too.
- Write and upload a checksum beside every archive, from the Gitea
  release workflow and the three local build scripts.
2026-09-04 16:31:02 +09:00
Yoshihiro OKUMURA e3026bdfcf feat(auth): send a request again when its token lapsed in the queue
A request can wait to be served for longer than the access token it was
sent with lives, and comes back refused for a token that was valid when
it left. Uploads that take minutes make that wait ordinary.

- send once more when a refusal is answered by a different token, which
  is the only refusal a second attempt can help; a wrong folder
  password, an anonymous request and an ended session all leave the
  token untouched and stay with the caller who knows what they mean
- rebuild the upload and read the file again for its retry, but only
  once the token has moved on: any other refusal would cost a second
  full transfer to be told the same
- say the server may be overloaded when the second attempt is refused
  too, rather than sending the user off to log in over a session that
  is fine
2026-08-14 18:37:40 +09:00
Yoshihiro OKUMURA bcf99dd6d7 fix(auth): keep the refresh token the provider hands back
The reply to a refresh carries a new refresh token and the provider
stops honouring the one that was sent. Only the access half was read,
so the cache kept re-sending a token the server had already retired.

- deserialize the refresh half and write it back to the cache, as an
  Option so a provider that does not rotate leaves the stored one be
- bound the refresh request on its own: the caller holds a lock that
  spans processes while it runs, so a provider that goes quiet would
  stall every other request on the machine
- give config create/update one rule for what a remote URL is, and
  store it without the trailing slash, matching the Python client so
  the two can share config.ini; this drops the validators crate and
  77 transitive dependencies with it
- join the base URL and the API's relative download path with the
  separator neither of them carries, as download.rs already does
2026-08-14 16:36:43 +09:00
Yoshihiro OKUMURA afd08f2499 fix(cache): format sha256 digest manually to resolve trait bound error
Release / build-linux-x86_64 (push) Successful in 2m14s
Release / build-linux-aarch64 (push) Successful in 2m6s
Update digest formatting to manually convert the SHA-256 result bytes
to a hexadecimal string. This resolves a compilation error caused by
upgrading the `sha2` crate to v0.11, where `LowerHex` is no longer
implemented for the return type of `finalize()`.
2026-06-12 10:19:10 +09:00
Yoshihiro OKUMURA 777c5f6533 feat(doi): add DOI-based path access for commands
Support accessing repositories using DOI strings with optional subpaths
across ls, download, metadata, and file-metadata commands.

- Implement GET v3/doi/{id}/ API model and client calls
- Parse and resolve DOI paths into respective folder and files
- Extract common folder and file resolution logic to shared helpers
- Update README with example DOI-based shell commands
2026-06-12 01:28:36 +09:00
Yoshihiro OKUMURA 80b6560030 feat(download): allow anonymous download of public data 2026-06-11 21:04:25 +09:00
Yoshihiro OKUMURAandCopilot beee9b4f41 fix(auth): allow anonymous read-only API access
Match the Python client for read-only commands by falling back to
anonymous API requests when no valid login cache is available.

Keep mutating commands login-only while letting ls, labs, metadata,
and file-metadata resolve laboratories and folders without a cached
token.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-07 15:49:02 +09:00
Yoshihiro OKUMURAandCopilot 7db5c4d53f docs(readme): document selfupdate command
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-07 15:49:02 +09:00
Yoshihiro OKUMURAandCopilot e3cd864a0c chore(release): sync lockfile version
Record the Rust package version bump in Cargo.lock so the
repository stays consistent after updating the crate version
to 2.0.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-07 15:49:02 +09:00
Yoshihiro OKUMURAandCopilot 32149109b4 chore(release): bump version to 2.0.0
Update the Rust package manifest to 2.0.0 so CLI version
reporting and release-related flows pick up the new version
from Cargo metadata.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-07 15:49:02 +09:00
Yoshihiro OKUMURAandCopilot 3f2ca938bd perf(transfer): parallelize folder traversal API calls
Use a shared API request limiter across recursive upload and
download traversal so folder detail fetches, file listings,
folder auth, and transfers can run concurrently under one
budget.

Refactor the traversal loops into task-driven pipelines while
preserving skip-if-exists, excludes, cleanup, and current
output behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-07 15:49:02 +09:00
Yoshihiro OKUMURAandCopilot 4e73766732 perf(cache): reuse auth state in memory
Cache parsed auth state per remote and validate it with on-disk
file metadata so repeated authenticated API calls can skip
redundant open/read/JSON parse work within one process.

Centralize cache load, persist, and removal helpers in the cache
module, reuse them from login, logout, and whoami, and update
the refresh path to persist structured cache data directly.

Add targeted cache tests for memory reuse, invalidation after
external writes, persist updates, and cache removal.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-07 15:49:01 +09:00
41 changed files with 3526 additions and 2132 deletions
+10 -2
View File
@@ -26,12 +26,16 @@ jobs:
TARGET=x86_64-unknown-linux-musl
ARCHIVE="mdrs-${VERSION}-${TARGET}.tar.gz"
tar -czf "${ARCHIVE}" -C target/${TARGET}/release mdrs
# Published alongside the archive so `mdrs selfupdate` can check what it fetched.
sha256sum "${ARCHIVE}" > "${ARCHIVE}.sha256"
echo "ARCHIVE=${ARCHIVE}" >> "$GITHUB_ENV"
- name: Create release and upload asset
uses: akkuman/gitea-release-action@v1
with:
token: ${{ github.token }}
files: ${{ env.ARCHIVE }}
files: |
${{ env.ARCHIVE }}
${{ env.ARCHIVE }}.sha256
build-linux-aarch64:
runs-on: ubuntu-latest
@@ -55,9 +59,13 @@ jobs:
TARGET=aarch64-unknown-linux-musl
ARCHIVE="mdrs-${VERSION}-${TARGET}.tar.gz"
tar -czf "${ARCHIVE}" -C target/${TARGET}/release mdrs
# Published alongside the archive so `mdrs selfupdate` can check what it fetched.
sha256sum "${ARCHIVE}" > "${ARCHIVE}.sha256"
echo "ARCHIVE=${ARCHIVE}" >> "$GITHUB_ENV"
- name: Create release and upload asset
uses: akkuman/gitea-release-action@v1
with:
token: ${{ github.token }}
files: ${{ env.ARCHIVE }}
files: |
${{ env.ARCHIVE }}
${{ env.ARCHIVE }}.sha256
+43
View File
@@ -0,0 +1,43 @@
# Changelog
All notable changes to this project will be documented in this file.
## [2.0.2] - 2026-09-04
### Added
- Accepted `-e` as the short form of `download --exclude`.
### Changed
- Transferred files larger than the memory available, in both directions.
- Left an existing local file intact when a download fails part way through.
- Showed the reason the server gave when it refused a request.
- Updated dependencies.
### Fixed
- Kept a session working across concurrent runs, long server queues, and providers that rotate refresh tokens.
- Listed a tree containing locked folders with `ls --json --recursive` instead of failing on the first one.
- Other minor fixes to `ls --json` output, remote path handling, and the login cache.
### Security
- Verified the downloaded archive against the release checksum before `selfupdate` replaces the binary.
## [2.0.1] - 2026-06-12
### Added
- Accepted a DOI in place of a path, as `remote:10.xxxx/prefix.ID[/sub/path]`.
- Allowed `download` and the read-only commands to run without signing in.
## [2.0.0] - 2026-04-20
First release of the command-line client.
### Added
- Commands `config`, `login`, `logout`, `whoami`, `labs`, `ls`, `mkdir`, `upload`, `download`, `mv`, `cp`, `rm`, `chacl`, `metadata`, `file-metadata`, `version`, and `selfupdate`.
- Release build scripts for Linux, macOS, and Windows.
### Changed
- Sped up transfers by working on folders and files in parallel, bounded by `MDRS_CLIENT_CONCURRENT`.
### Fixed
- Kept the session valid throughout a long-running transfer, and across concurrent runs.
- Handled file and folder names with combining characters consistently.
Generated
+400 -1574
View File
File diff suppressed because it is too large Load Diff
+14 -14
View File
@@ -1,6 +1,6 @@
[package]
name = "mdrs-client-rust"
version = "0.1.1"
version = "2.0.2"
edition = "2024"
license = "MIT"
authors = ["Neuroinformatics Unit, RIKEN CBS"]
@@ -12,25 +12,25 @@ path = "src/main.rs"
[dependencies]
clap = { version = "4.5", features = ["derive"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "multipart", "rustls-tls"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "multipart", "rustls-tls", "stream"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
tokio = { version = "1.37", features = ["full"] }
serde_json = "1.0.151"
tokio = { version = "1.53.1", features = ["full"] }
tokio-util = { version = "0.7", features = ["io"] }
futures = "0.3"
dirs = "5.0"
anyhow = "1.0.102"
configparser = "3.1.0"
validators = "0.25.3"
sha2 = "0.10"
rpassword = "7.0"
base64 = "0.22"
dirs = "6.0.0"
anyhow = "1.0.104"
configparser = "3.2.0"
sha2 = "0.11.0"
rpassword = "7.5.4"
base64 = "0.23.1"
fs2 = "0.4"
ctrlc = "3"
os_info = "3"
os_info = "3.15.0"
dotenvy = "0.15"
unicode-normalization = "0.1"
self-replace = "1"
tar = "0.4"
tar = "0.4.46"
flate2 = "1"
zip = "2"
zip = "8.6.0"
tempfile = "3"
+34 -6
View File
@@ -103,7 +103,7 @@ mdrs labs neurodata:
### ls
List the contents of a remote folder.
List the contents of a remote folder. You can also specify a DOI path in the form `remote:10.xxxx/yyy.ID[/optional/subpath]`.
```shell
mdrs ls neurodata:/NIU/Repository/
@@ -111,6 +111,10 @@ mdrs ls -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/
mdrs ls -r neurodata:/NIU/Repository/Dataset1/
mdrs ls -J -r neurodata:/NIU/Repository/Dataset1/
mdrs ls -q neurodata:/NIU/Repository/
# DOI access examples:
mdrs ls neurodata:10.60178/cbs.20260429-001
mdrs ls "neurodata:10.60178/cbs.20260429-001/Figure 1"
```
### mkdir
@@ -133,14 +137,17 @@ mdrs upload -r --skip-if-exists ./dataset neurodata:/NIU/Repository/TEST/
### download
Download a file or folder from a remote path.
Download a file or folder from a remote path. You can also specify a DOI path.
```shell
mdrs download neurodata:/NIU/Repository/TEST/sample.dat ./
mdrs download -r neurodata:/NIU/Repository/TEST/dataset/ ./
mdrs download -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/Readme.dat ./
mdrs download -r --exclude /NIU/Repository/TEST/dataset/skip neurodata:/NIU/Repository/TEST/dataset/ ./
mdrs download -r -e /NIU/Repository/TEST/dataset/skip neurodata:/NIU/Repository/TEST/dataset/ ./
mdrs download -r --skip-if-exists neurodata:/NIU/Repository/TEST/dataset/ ./
# DOI access examples:
mdrs download -r neurodata:10.60178/cbs.20260429-001 ./
```
### mv
@@ -184,20 +191,26 @@ Available access levels: `private`, `public`, `pw_open`, `cbs_open`, `5kikan_ope
### metadata
Show metadata for a remote folder.
Show metadata for a remote folder. You can also specify a DOI path.
```shell
mdrs metadata neurodata:/NIU/Repository/TEST/
mdrs metadata neurodata:/NIU/Repository/Private/
mdrs metadata -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/
# DOI access examples:
mdrs metadata neurodata:10.60178/cbs.20260429-001
```
### file-metadata
Show metadata for a remote file.
Show metadata for a remote file. You can also specify a DOI path.
```shell
mdrs file-metadata neurodata:/NIU/Repository/TEST/dataset/sample.dat
mdrs file-metadata -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/Readme.txt
# DOI access examples:
mdrs file-metadata "neurodata:10.60178/cbs.20260429-001/Figure 1/Figure1v3.pdf"
```
### version
@@ -208,6 +221,17 @@ Show the tool name and version number.
mdrs version
```
### selfupdate
Update the current `mdrs` binary to the latest published release for
the same build target. The downloaded archive is checked against the `.sha256`
asset the release publishes; a release without one is reported as unverified.
```shell
mdrs selfupdate
mdrs selfupdate -y
```
### help
Show help for a command.
@@ -217,6 +241,10 @@ mdrs --help
mdrs upload --help
```
## Changelog
See [CHANGELOG.md](./CHANGELOG.md) for the full change history.
## License
[MIT](LICENSE) © 2026- Neuroinformatics Unit, RIKEN CBS
+4 -1
View File
@@ -58,7 +58,10 @@ for TARGET in "${TARGETS[@]}"; do
ARCHIVE="mdrs-${VERSION}-${TARGET}.tar.gz"
tar -czf "${ARCHIVE}" -C "target/${TARGET}/release" mdrs
ARCHIVES+=("${ARCHIVE}")
# Uploaded alongside the archive so `mdrs selfupdate` can check what it fetched.
sha256sum "${ARCHIVE}" > "${ARCHIVE}.sha256" 2>/dev/null \
|| shasum -a 256 "${ARCHIVE}" > "${ARCHIVE}.sha256"
ARCHIVES+=("${ARCHIVE}" "${ARCHIVE}.sha256")
echo " Created: ${ARCHIVE}"
done
+4 -1
View File
@@ -46,7 +46,10 @@ for TARGET in "${TARGETS[@]}"; do
ARCHIVE="mdrs-${VERSION}-${TARGET}.tar.gz"
tar -czf "${ARCHIVE}" -C "target/${TARGET}/release" mdrs
ARCHIVES+=("${ARCHIVE}")
# Uploaded alongside the archive so `mdrs selfupdate` can check what it fetched.
sha256sum "${ARCHIVE}" > "${ARCHIVE}.sha256" 2>/dev/null \
|| shasum -a 256 "${ARCHIVE}" > "${ARCHIVE}.sha256"
ARCHIVES+=("${ARCHIVE}" "${ARCHIVE}.sha256")
echo " Created: ${ARCHIVE}"
done
+14 -8
View File
@@ -55,6 +55,10 @@ cargo build --release --target $Target
$Archive = "mdrs-$Version-$Target.zip"
Compress-Archive -Force -Path "target\$Target\release\mdrs.exe" -DestinationPath $Archive
# Written alongside the archive so `mdrs selfupdate` can check what it fetched.
$Checksum = "$Archive.sha256"
$Hash = (Get-FileHash -Algorithm SHA256 -Path $Archive).Hash.ToLower()
"$Hash $Archive" | Set-Content -NoNewline -Encoding ascii $Checksum
Write-Host " Created: $Archive"
# Upload to Gitea if token is provided
@@ -87,14 +91,16 @@ $Release = Invoke-RestMethod -Method Get -Uri "$ServerUrl/api/v1/repos/$Reposito
-Headers @{ Authorization = "Bearer $GiteaToken" }
$ReleaseId = $Release.id
Write-Host "==> Uploading $Archive (release id: $ReleaseId) ..."
$ArchivePath = Join-Path $RepoRoot $Archive
& curl.exe -sf -X POST `
-H "Authorization: Bearer $GiteaToken" `
-F "attachment=@$ArchivePath" `
"$ServerUrl/api/v1/repos/$Repository/releases/$ReleaseId/assets" | Out-Null
if ($LASTEXITCODE -ne 0) { Write-Error "Upload failed (exit code $LASTEXITCODE)"; exit 1 }
Write-Host " Done."
foreach ($Name in @($Archive, $Checksum)) {
Write-Host "==> Uploading $Name (release id: $ReleaseId) ..."
$AssetPath = Join-Path $RepoRoot $Name
& curl.exe -sf -X POST `
-H "Authorization: Bearer $GiteaToken" `
-F "attachment=@$AssetPath" `
"$ServerUrl/api/v1/repos/$Repository/releases/$ReleaseId/assets" | Out-Null
if ($LASTEXITCODE -ne 0) { Write-Error "Upload failed (exit code $LASTEXITCODE)"; exit 1 }
Write-Host " Done."
}
Write-Host ""
Write-Host "==> Upload complete: $ServerUrl/$Repository/releases/tag/$Tag"
+17
View File
@@ -0,0 +1,17 @@
use crate::connection::MDRSConnection;
use crate::error::response_error;
use crate::models::doi::DoiResponse;
impl MDRSConnection {
/// Retrieve the folder associated with a DOI suffix ID (GET v3/doi/{id}/).
///
/// The MDRS DOI format is `10.xxxx/prefix.{id}` where the suffix after the
/// last `.` is the internal system ID passed to this endpoint.
pub async fn retrieve_doi(&self, id: &str) -> Result<DoiResponse, anyhow::Error> {
let resp = self.get(&format!("v3/doi/{}/", id)).await?;
if !resp.status().is_success() {
return Err(response_error("DOI lookup failed", resp).await);
}
Ok(resp.json::<DoiResponse>().await?)
}
}
+415 -18
View File
@@ -1,8 +1,30 @@
use crate::connection::MDRSConnection;
use crate::connection::{ApiRequestLimiter, MDRSConnection};
use crate::error::response_error;
pub use crate::models::file::File;
use anyhow::bail;
use unicode_normalization::UnicodeNormalization;
/// Read and write size for a streamed transfer. Large enough that the syscalls are not
/// what limits the transfer, small enough that ten of them in flight cost little.
const TRANSFER_CHUNK_BYTES: usize = 64 * 1024;
/// Name a scratch file beside the destination for a download in progress.
///
/// Beside it, so the move into place is a rename within one directory and cannot fail
/// half way. Unique, so two transfers heading for the same name cannot clear up after
/// each other.
fn partial_path(dest: &std::path::Path) -> std::path::PathBuf {
use std::sync::atomic::{AtomicU64, Ordering};
static COUNTER: AtomicU64 = AtomicU64::new(0);
let mut name = dest.file_name().unwrap_or_default().to_os_string();
name.push(format!(
".{}-{}.mdrspart",
std::process::id(),
COUNTER.fetch_add(1, Ordering::Relaxed)
));
dest.with_file_name(name)
}
#[derive(serde::Deserialize)]
struct FileListResponse {
pub next: Option<String>,
@@ -21,7 +43,7 @@ impl MDRSConnection {
];
let resp = self.get_with_query("v3/files/", &params).await?;
if !resp.status().is_success() {
anyhow::bail!("List files failed: {}", resp.status());
return Err(response_error("List files failed", resp).await);
}
let list: FileListResponse = resp.json().await?;
let has_next = list.next.is_some();
@@ -34,8 +56,43 @@ impl MDRSConnection {
Ok(all_files)
}
/// Upload a local file into the given remote folder.
pub async fn upload_file(&self, folder_id: &str, file_path: &str) -> Result<(), anyhow::Error> {
/// List all files in a folder while consuming the shared API concurrency budget.
pub async fn list_all_files_limited(
&self,
folder_id: &str,
limiter: &ApiRequestLimiter,
) -> Result<Vec<File>, anyhow::Error> {
let mut all_files = Vec::new();
let mut page: u32 = 1;
loop {
let params = [
("folder_id", folder_id.to_string()),
("page", page.to_string()),
];
let _permit = limiter.acquire().await?;
let resp = self.get_with_query("v3/files/", &params).await?;
if !resp.status().is_success() {
return Err(response_error("List files failed", resp).await);
}
let list: FileListResponse = resp.json().await?;
let has_next = list.next.is_some();
all_files.extend(list.results);
if !has_next {
break;
}
page += 1;
}
Ok(all_files)
}
/// Upload a local file into the given remote folder while consuming the
/// shared API concurrency budget.
pub async fn upload_file_limited(
&self,
folder_id: &str,
file_path: &str,
limiter: &ApiRequestLimiter,
) -> Result<(), anyhow::Error> {
use anyhow::{anyhow, bail};
use reqwest::multipart;
let file_name: String = std::path::Path::new(file_path)
@@ -44,26 +101,366 @@ impl MDRSConnection {
.to_string_lossy()
.nfc()
.collect();
let file_bytes = tokio::fs::read(file_path).await?;
let part = multipart::Part::bytes(file_bytes).file_name(file_name.clone());
let form = multipart::Form::new()
.text("folder_id", folder_id.to_string())
.part("file", part);
let resp = self.post_multipart("v3/files/", form).await?;
// The file is handed to the request as a stream rather than a buffer: a
// repository holds recordings far larger than the machine's memory, and ten of
// them may be in flight at once. The length travels with it so the request keeps
// a Content-Length and does not have to be sent chunked.
let build_form = || async {
let file = tokio::fs::File::open(file_path).await?;
let length = file.metadata().await?.len();
let stream = tokio_util::io::ReaderStream::with_capacity(file, TRANSFER_CHUNK_BYTES);
let part =
multipart::Part::stream_with_length(reqwest::Body::wrap_stream(stream), length)
.file_name(file_name.clone());
Ok::<_, anyhow::Error>(
multipart::Form::new()
.text("folder_id", folder_id.to_string())
.part("file", part),
)
};
let _permit = limiter.acquire().await?;
let sent_with = self.token.clone();
let resp = self
.post_multipart("v3/files/", build_form().await?)
.await?;
if resp.status() != reqwest::StatusCode::UNAUTHORIZED {
if !resp.status().is_success() {
return Err(response_error("Upload failed", resp).await);
}
return Ok(());
}
// The body reached the server, but it may have waited to be served for longer
// than the access token it was sent with lived. Re-reading the file and sending
// it again is only worth it if the token has actually moved on: any other reason
// for the refusal would just cost a second full transfer to be told the same.
let retry_conn = self.connection_with_fresh_token_for_retry().await?;
if retry_conn.token == sent_with {
return Err(response_error("Upload failed", resp).await);
}
let resp = retry_conn
.post_multipart("v3/files/", build_form().await?)
.await?;
if resp.status() == reqwest::StatusCode::UNAUTHORIZED {
// Refused twice, the second time with a token that was current when it left.
bail!(crate::connection::SERVER_BUSY_MESSAGE);
}
if !resp.status().is_success() {
bail!("Upload failed: {}", resp.status());
return Err(response_error("Upload failed", resp).await);
}
Ok(())
}
/// Download a file from `url` and write it to `dest`.
pub async fn download_file(&self, url: &str, dest: &str) -> Result<(), anyhow::Error> {
/// Download a file while consuming the shared API concurrency budget.
pub async fn download_file_limited(
&self,
url: &str,
dest: &str,
limiter: &ApiRequestLimiter,
) -> Result<(), anyhow::Error> {
use futures::StreamExt;
use tokio::io::AsyncWriteExt;
let _permit = limiter.acquire().await?;
let dest_path = std::path::Path::new(dest);
// Checked before anything is fetched. The finished file is moved into place, and
// a rename would replace a destination the mode says is protected, which is not
// what this client or the Python one has ever done.
if dest_path.exists() {
tokio::fs::OpenOptions::new()
.write(true)
.open(dest_path)
.await
.map_err(|e| anyhow::anyhow!("Cannot write `{}`: {}", dest, e))?;
}
let resp = self.get_url(url).await?;
if !resp.status().is_success() {
bail!("Download failed: {}", resp.status());
return Err(response_error("Download failed", resp).await);
}
// Written as it arrives, into a scratch file that is moved into place only once
// the whole body has landed. Holding the body in memory first would cost the size
// of the file, times however many transfers are running; writing straight to the
// destination would destroy whatever is already there the moment the transfer
// starts, and leave a truncated file behind if it does not finish.
let part_path = partial_path(dest_path);
let result = async {
let mut file = tokio::io::BufWriter::with_capacity(
TRANSFER_CHUNK_BYTES,
tokio::fs::File::create(&part_path).await?,
);
let mut stream = resp.bytes_stream();
while let Some(chunk) = stream.next().await {
file.write_all(&chunk?).await?;
}
file.flush().await?;
Ok::<(), anyhow::Error>(())
}
.await;
match result {
Ok(()) => Ok(tokio::fs::rename(&part_path, dest_path).await?),
Err(err) => {
// Only the scratch file goes. Anything already at the destination was not
// written by this transfer and is not this transfer's to remove.
let _ = tokio::fs::remove_file(&part_path).await;
Err(err)
}
}
let bytes = resp.bytes().await?;
tokio::fs::write(dest, &bytes).await?;
Ok(())
}
}
#[cfg(test)]
mod transfer_tests {
use super::*;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
/// Read one HTTP request in full, using its Content-Length to know where it ends.
async fn read_request(stream: &mut tokio::net::TcpStream) -> (String, Vec<u8>) {
let mut buf = Vec::new();
let mut chunk = [0u8; 8192];
let header_end = loop {
let n = stream.read(&mut chunk).await.unwrap();
buf.extend_from_slice(&chunk[..n]);
if let Some(pos) = buf.windows(4).position(|w| w == b"\r\n\r\n") {
break pos + 4;
}
assert!(n > 0, "connection closed before the headers ended");
};
let head = String::from_utf8_lossy(&buf[..header_end]).to_string();
let length: usize = head
.lines()
.find_map(|l| {
l.to_ascii_lowercase()
.strip_prefix("content-length:")
.map(|v| v.trim().to_string())
})
.expect("request must carry a Content-Length")
.parse()
.unwrap();
while buf.len() < header_end + length {
let n = stream.read(&mut chunk).await.unwrap();
assert!(n > 0, "connection closed before the body ended");
buf.extend_from_slice(&chunk[..n]);
}
(head, buf[header_end..].to_vec())
}
/// The file goes out as a stream, and the request still declares its length, so the
/// server is not asked to accept a chunked upload.
#[tokio::test]
async fn an_upload_streams_the_file_with_a_declared_length() {
let payload: Vec<u8> = (0..300_000u32).map(|i| (i % 251) as u8).collect();
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("recording.dat");
std::fs::write(&path, &payload).unwrap();
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let expected = payload.clone();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let (head, body) = read_request(&mut stream).await;
stream
.write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 15\r\nconnection: close\r\n\r\n{\"id\":\"new\"}")
.await
.unwrap();
(head, body, expected)
});
let conn = MDRSConnection::new(&format!("http://{addr}"));
let limiter = ApiRequestLimiter::new(1);
conn.upload_file_limited("folder-1", path.to_str().unwrap(), &limiter)
.await
.unwrap();
let (head, body, expected) = server.await.unwrap();
assert!(
!head
.to_ascii_lowercase()
.contains("transfer-encoding: chunked"),
"the upload must not be chunked:\n{head}"
);
assert!(
body.windows(expected.len()).any(|w| w == expected),
"the file bytes must reach the server"
);
assert!(
body.windows(8).any(|w| w == b"folder-1"),
"the folder id must reach the server"
);
assert!(
String::from_utf8_lossy(&body).contains("recording.dat"),
"the file name must reach the server"
);
}
/// The body is written to disk as it arrives rather than being held whole.
#[tokio::test]
async fn a_download_writes_what_the_server_sent() {
let payload: Vec<u8> = (0..300_000u32).map(|i| (i % 241) as u8).collect();
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let served = payload.clone();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut chunk = [0u8; 4096];
let _ = stream.read(&mut chunk).await.unwrap();
let head = format!(
"HTTP/1.1 200 OK\r\ncontent-length: {}\r\nconnection: close\r\n\r\n",
served.len()
);
stream.write_all(head.as_bytes()).await.unwrap();
stream.write_all(&served).await.unwrap();
});
let dir = tempfile::tempdir().unwrap();
let dest = dir.path().join("out.dat");
let conn = MDRSConnection::new(&format!("http://{addr}"));
let limiter = ApiRequestLimiter::new(1);
conn.download_file_limited(
&format!("http://{addr}/v3/files/1/download/"),
dest.to_str().unwrap(),
&limiter,
)
.await
.unwrap();
server.await.unwrap();
assert_eq!(std::fs::read(&dest).unwrap(), payload);
}
/// A transfer that dies part way must not take the copy already on disk with it.
#[tokio::test]
async fn an_interrupted_download_leaves_the_existing_file_intact() {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut chunk = [0u8; 4096];
let _ = stream.read(&mut chunk).await.unwrap();
// Promises 300000 bytes, sends 1000, then hangs up.
stream
.write_all(
b"HTTP/1.1 200 OK\r\ncontent-length: 300000\r\nconnection: close\r\n\r\n",
)
.await
.unwrap();
stream.write_all(&vec![b'x'; 1000]).await.unwrap();
stream.shutdown().await.unwrap();
});
let dir = tempfile::tempdir().unwrap();
let dest = dir.path().join("existing.dat");
std::fs::write(&dest, b"the copy already here").unwrap();
let conn = MDRSConnection::new(&format!("http://{addr}"));
let limiter = ApiRequestLimiter::new(1);
let outcome = conn
.download_file_limited(
&format!("http://{addr}/v3/files/1/download/"),
dest.to_str().unwrap(),
&limiter,
)
.await;
server.await.unwrap();
assert!(outcome.is_err(), "a truncated body is a failed download");
assert_eq!(std::fs::read(&dest).unwrap(), b"the copy already here");
let leftovers: Vec<_> = std::fs::read_dir(dir.path())
.unwrap()
.map(|e| e.unwrap().file_name().to_string_lossy().to_string())
.filter(|name| name != "existing.dat")
.collect();
assert!(
leftovers.is_empty(),
"no scratch file may be left: {leftovers:?}"
);
}
/// A destination this call could not even open is not a file it may delete.
#[cfg(unix)]
#[tokio::test]
async fn a_download_that_cannot_be_written_leaves_the_existing_file_intact() {
use std::os::unix::fs::PermissionsExt;
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut chunk = [0u8; 4096];
let _ = stream.read(&mut chunk).await.unwrap();
stream
.write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 2\r\nconnection: close\r\n\r\nhi")
.await
.unwrap();
});
// A writable directory holding a file this process may not open for writing.
// That is what separates "could not write it" from "may delete it": unlink needs
// the directory, not the file.
let dir = tempfile::tempdir().unwrap();
let dest = dir.path().join("precious.dat");
std::fs::write(&dest, b"do not touch").unwrap();
let mut perms = std::fs::metadata(&dest).unwrap().permissions();
perms.set_mode(0o444);
std::fs::set_permissions(&dest, perms).unwrap();
if std::fs::OpenOptions::new().write(true).open(&dest).is_ok() {
// Running with rights that ignore the mode, so there is nothing to observe.
return;
}
let conn = MDRSConnection::new(&format!("http://{addr}"));
let limiter = ApiRequestLimiter::new(1);
let outcome = conn
.download_file_limited(
&format!("http://{addr}/v3/files/1/download/"),
dest.to_str().unwrap(),
&limiter,
)
.await;
// Nothing was fetched, so the stub is still waiting to be connected to.
server.abort();
let message = outcome.unwrap_err().to_string();
assert!(
message.starts_with("Cannot write "),
"the refusal must name the problem, got: {message}"
);
assert_eq!(std::fs::read(&dest).unwrap(), b"do not touch");
}
/// A refused download leaves nothing behind that a later run could mistake for the file.
#[tokio::test]
async fn a_refused_download_leaves_no_file() {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut chunk = [0u8; 4096];
let _ = stream.read(&mut chunk).await.unwrap();
let body = r#"{"type":"client_error","errors":[{"code":"permission_denied","detail":"Access is denied.","attr":null}]}"#;
let head = format!(
"HTTP/1.1 403 Forbidden\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{}",
body.len(),
body
);
stream.write_all(head.as_bytes()).await.unwrap();
});
let dir = tempfile::tempdir().unwrap();
let dest = dir.path().join("out.dat");
let conn = MDRSConnection::new(&format!("http://{addr}"));
let limiter = ApiRequestLimiter::new(1);
let err = conn
.download_file_limited(
&format!("http://{addr}/v3/files/1/download/"),
dest.to_str().unwrap(),
&limiter,
)
.await
.unwrap_err();
server.await.unwrap();
assert_eq!(err.to_string(), "Download failed: Access is denied.");
assert!(!dest.exists());
}
}
+93 -5
View File
@@ -1,6 +1,7 @@
use crate::connection::MDRSConnection;
use crate::connection::{ApiRequestLimiter, MDRSConnection};
use crate::error::response_error;
pub use crate::models::folder::{FolderDetail, FolderSimple};
use anyhow::bail;
use anyhow::{anyhow, bail};
impl MDRSConnection {
/// List folders matching the given path under a laboratory (GET v3/folders/?path=...&laboratory_id=...)
@@ -15,7 +16,26 @@ impl MDRSConnection {
];
let resp = self.get_with_query("v3/folders/", &params).await?;
if !resp.status().is_success() {
bail!("List folders failed: {}", resp.status());
return Err(response_error("List folders failed", resp).await);
}
Ok(resp.json::<Vec<FolderSimple>>().await?)
}
/// List folders by path while consuming the shared API concurrency budget.
pub async fn list_folders_by_path_limited(
&self,
lab_id: u32,
path: &str,
limiter: &ApiRequestLimiter,
) -> Result<Vec<FolderSimple>, anyhow::Error> {
let params = [
("laboratory_id", lab_id.to_string()),
("path", path.to_string()),
];
let _permit = limiter.acquire().await?;
let resp = self.get_with_query("v3/folders/", &params).await?;
if !resp.status().is_success() {
return Err(response_error("List folders failed", resp).await);
}
Ok(resp.json::<Vec<FolderSimple>>().await?)
}
@@ -24,7 +44,21 @@ impl MDRSConnection {
pub async fn retrieve_folder(&self, id: &str) -> Result<FolderDetail, anyhow::Error> {
let resp = self.get(&format!("v3/folders/{}/", id)).await?;
if !resp.status().is_success() {
bail!("Retrieve folder failed: {}", resp.status());
return Err(response_error("Retrieve folder failed", resp).await);
}
Ok(resp.json::<FolderDetail>().await?)
}
/// Retrieve folder details while consuming the shared API concurrency budget.
pub async fn retrieve_folder_limited(
&self,
id: &str,
limiter: &ApiRequestLimiter,
) -> Result<FolderDetail, anyhow::Error> {
let _permit = limiter.acquire().await?;
let resp = self.get(&format!("v3/folders/{}/", id)).await?;
if !resp.status().is_success() {
return Err(response_error("Retrieve folder failed", resp).await);
}
Ok(resp.json::<FolderDetail>().await?)
}
@@ -44,6 +78,36 @@ impl MDRSConnection {
self.post_json("v3/folders/", &body).await
}
/// Create a new folder under `parent_id` and return its ID while consuming
/// the shared API concurrency budget.
pub async fn create_folder_id_limited(
&self,
parent_id: &str,
folder_name: &str,
limiter: &ApiRequestLimiter,
) -> Result<String, anyhow::Error> {
let body = serde_json::json!({
"name": folder_name,
"parent_id": parent_id,
"description": "",
"template_id": -1,
});
let _permit = limiter.acquire().await?;
let resp = self.post_json("v3/folders/", &body).await?;
if !resp.status().is_success() {
return Err(response_error(
&format!("Failed to create remote folder `{}`", folder_name),
resp,
)
.await);
}
let json: serde_json::Value = resp.json().await?;
json["id"]
.as_str()
.ok_or_else(|| anyhow!("No id in create_folder response for {}", folder_name))
.map(|s| s.to_string())
}
/// Authenticate against a password-locked folder (POST v3/folders/{id}/auth/).
/// Returns `Err` if the password is incorrect or the request fails.
pub async fn folder_auth(&self, folder_id: &str, password: &str) -> Result<(), anyhow::Error> {
@@ -57,7 +121,31 @@ impl MDRSConnection {
bail!("Password is incorrect.");
}
if !resp.status().is_success() {
bail!("Folder auth failed: {}", resp.status());
return Err(response_error("Folder auth failed", resp).await);
}
Ok(())
}
/// Authenticate against a locked folder while consuming the shared API
/// concurrency budget.
pub async fn folder_auth_limited(
&self,
folder_id: &str,
password: &str,
limiter: &ApiRequestLimiter,
) -> Result<(), anyhow::Error> {
let _permit = limiter.acquire().await?;
let resp = self
.post_json(
&format!("v3/folders/{}/auth/", folder_id),
&serde_json::json!({"password": password}),
)
.await?;
if resp.status() == reqwest::StatusCode::UNAUTHORIZED {
bail!("Password is incorrect.");
}
if !resp.status().is_success() {
return Err(response_error("Folder auth failed", resp).await);
}
Ok(())
}
+2 -1
View File
@@ -1,4 +1,5 @@
use crate::connection::MDRSConnection;
use crate::error::response_error;
use crate::models::laboratory::{Laboratories, Laboratory};
use serde::Deserialize;
@@ -12,7 +13,7 @@ impl MDRSConnection {
pub async fn list_laboratories(&self) -> Result<Laboratories, anyhow::Error> {
let resp = self.get("v3/laboratories/").await?;
if !resp.status().is_success() {
anyhow::bail!("List laboratories failed: {}", resp.status());
return Err(response_error("List laboratories failed", resp).await);
}
// The API may return a paginated object or a direct array
let text = resp.text().await?;
+1
View File
@@ -1,5 +1,6 @@
// API module (add users, files, folders, laboratories, etc. here)
pub mod doi;
pub mod files;
pub mod folders;
pub mod laboratories;
+87 -6
View File
@@ -1,7 +1,14 @@
use crate::connection::MDRSConnection;
use crate::error::response_error;
use crate::models::user::User as ModelUser;
use anyhow::bail;
use serde::Deserialize;
use std::time::Duration;
/// Bound the refresh on its own: the caller holds a lock that spans processes while this
/// runs, so a provider that accepts the connection and then goes quiet would stall every
/// other request on this machine rather than just this one. Uploads are served by a
/// separate instance, so this one is not queued behind them and has no reason to be slow.
const TOKEN_REFRESH_TIMEOUT: Duration = Duration::from_secs(30);
/// Full API response shape from GET v3/users/current/
#[derive(Debug, Deserialize)]
@@ -20,6 +27,16 @@ struct UsersCurrentResponseLaboratory {
#[derive(Deserialize)]
struct TokenRefreshResponse {
access: String,
/// Present when the provider rotates refresh tokens, absent when it does not,
/// so the caller keeps the token it already holds if nothing new arrives.
#[serde(default)]
refresh: Option<String>,
}
/// The token pair a refresh yields.
pub struct RefreshedToken {
pub access: String,
pub refresh: Option<String>,
}
impl MDRSConnection {
@@ -27,7 +44,7 @@ impl MDRSConnection {
pub async fn get_current_user(&self) -> Result<ModelUser, anyhow::Error> {
let resp = self.get("v3/users/current/").await?;
if !resp.status().is_success() {
bail!("Get current user failed: {}", resp.status());
return Err(response_error("Get current user failed", resp).await);
}
let obj = resp.json::<UsersApiCurrentResponse>().await?;
let laboratory_ids = obj.laboratories.into_iter().map(|l| l.id).collect();
@@ -40,19 +57,83 @@ impl MDRSConnection {
}
/// Refresh the access token using the refresh token.
/// POST v3/users/token/refresh/ {refresh: ...} -> {access: new_access}
pub async fn token_refresh(&self, refresh_token: &str) -> Result<String, anyhow::Error> {
/// POST v3/users/token/refresh/ {refresh: ...} -> {access, refresh?}
///
/// A rotating provider answers with a new refresh token and stops honouring the one
/// that was sent, so both halves of the reply have to be kept.
pub async fn token_refresh(
&self,
refresh_token: &str,
) -> Result<RefreshedToken, anyhow::Error> {
let body = serde_json::json!({ "refresh": refresh_token });
let resp = self
.client
.post(self.build_url("v3/users/token/refresh/"))
.json(&body)
.timeout(TOKEN_REFRESH_TIMEOUT)
.send()
.await?;
if !resp.status().is_success() {
bail!("Token refresh failed: {}", resp.status());
return Err(response_error("Token refresh failed", resp).await);
}
let r: TokenRefreshResponse = resp.json().await?;
Ok(r.access)
Ok(RefreshedToken {
access: r.access,
refresh: r.refresh,
})
}
}
#[cfg(test)]
mod tests {
use super::*;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
async fn refresh_against_stub(body: &'static str) -> RefreshedToken {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut buf = [0u8; 4096];
let n = stream.read(&mut buf).await.unwrap();
let req = String::from_utf8_lossy(&buf[..n]);
assert!(req.starts_with("POST /v3/users/token/refresh/ HTTP/1.1"));
assert!(req.contains("\"refresh\":\"old-refresh\""));
let response = format!(
"HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{}",
body.len(),
body
);
stream.write_all(response.as_bytes()).await.unwrap();
});
let conn = MDRSConnection::new(&format!("http://{addr}"));
let refreshed = conn.token_refresh("old-refresh").await.unwrap();
server.await.unwrap();
refreshed
}
/// A rotating provider stops honouring the token that was sent, so the reply's
/// refresh token has to reach the caller rather than being dropped.
#[tokio::test]
async fn token_refresh_returns_the_rotated_refresh_token() {
let refreshed =
refresh_against_stub(r#"{"access":"new-access","refresh":"new-refresh"}"#).await;
assert_eq!(refreshed.access, "new-access");
assert_eq!(refreshed.refresh.as_deref(), Some("new-refresh"));
}
/// A provider that does not rotate answers with the access token alone, and the
/// caller keeps the refresh token it already holds.
#[tokio::test]
async fn token_refresh_reports_no_rotation_when_the_reply_omits_it() {
let refreshed = refresh_against_stub(r#"{"access":"new-access"}"#).await;
assert_eq!(refreshed.access, "new-access");
assert_eq!(refreshed.refresh, None);
}
}
+2 -1
View File
@@ -119,5 +119,6 @@ pub fn compute_digest(
let json_str = python_digest_json(user, access, refresh, labs);
let mut hasher = Sha256::new();
hasher.update(json_str.as_bytes());
format!("{:x}", hasher.finalize())
let result = hasher.finalize();
result.iter().map(|b| format!("{:02x}", b)).collect()
}
+559 -63
View File
@@ -2,13 +2,17 @@ pub mod digest;
pub mod types;
pub use digest::compute_digest;
pub use types::{Cache, CacheLaboratory, CacheLabsWrapper, CacheUser};
pub use types::{Cache, CacheLaboratory, CacheLabsWrapper, CacheToken, CacheUser};
use crate::connection::MDRSConnection;
use anyhow::{anyhow, bail};
use std::collections::HashMap;
use std::fs;
#[cfg(unix)]
use std::os::unix::fs::{MetadataExt, PermissionsExt};
use std::path::{Path, PathBuf};
use std::sync::{Arc, LazyLock, Mutex};
use std::time::UNIX_EPOCH;
// ---------------------------------------------------------------------------
// Per-remote async mutex map (in-process serialization)
@@ -17,6 +21,29 @@ use std::sync::{Arc, LazyLock, Mutex};
static REMOTE_LOCKS: LazyLock<Mutex<HashMap<String, Arc<tokio::sync::Mutex<()>>>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
#[derive(Clone, Debug, Eq, Hash, PartialEq)]
struct CacheStoreKey {
config_dir: PathBuf,
remote: String,
}
#[derive(Clone, Debug, Eq, PartialEq)]
struct CacheFileSnapshot {
len: u64,
modified_nanos: u128,
#[cfg(unix)]
inode: u64,
}
#[derive(Clone)]
struct MemoryCacheEntry {
snapshot: CacheFileSnapshot,
cache: Cache,
}
static MEMORY_CACHE: LazyLock<Mutex<HashMap<CacheStoreKey, MemoryCacheEntry>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
fn get_remote_lock(remote: &str) -> Arc<tokio::sync::Mutex<()>> {
let mut map = REMOTE_LOCKS.lock().unwrap();
map.entry(remote.to_string())
@@ -28,11 +55,220 @@ fn get_remote_lock(remote: &str) -> Arc<tokio::sync::Mutex<()>> {
// Cache file path helpers
// ---------------------------------------------------------------------------
fn cache_file_path(remote: &str) -> std::path::PathBuf {
crate::settings::SETTINGS
.config_dirname
.join("cache")
.join(format!("{}.json", remote))
fn cache_store_key(config_dir: &Path, remote: &str) -> CacheStoreKey {
CacheStoreKey {
config_dir: config_dir.to_path_buf(),
remote: remote.to_string(),
}
}
fn cache_dir_path(config_dir: &Path) -> PathBuf {
config_dir.join("cache")
}
fn cache_file_path_in(config_dir: &Path, remote: &str) -> PathBuf {
cache_dir_path(config_dir).join(format!("{}.json", remote))
}
fn cache_file_path(remote: &str) -> PathBuf {
cache_file_path_in(&crate::settings::SETTINGS.config_dirname, remote)
}
fn cache_snapshot(metadata: &fs::Metadata) -> CacheFileSnapshot {
let modified_nanos = metadata
.modified()
.ok()
.and_then(|time| time.duration_since(UNIX_EPOCH).ok())
.map(|duration| duration.as_nanos())
.unwrap_or_default();
CacheFileSnapshot {
len: metadata.len(),
modified_nanos,
#[cfg(unix)]
inode: metadata.ino(),
}
}
fn read_cache_snapshot(cache_path: &Path) -> Result<CacheFileSnapshot, std::io::Error> {
fs::metadata(cache_path).map(|metadata| cache_snapshot(&metadata))
}
fn cached_entry(config_dir: &Path, remote: &str, snapshot: &CacheFileSnapshot) -> Option<Cache> {
let key = cache_store_key(config_dir, remote);
let map = MEMORY_CACHE.lock().unwrap();
map.get(&key)
.filter(|entry| entry.snapshot == *snapshot)
.map(|entry| entry.cache.clone())
}
fn update_cached_entry(config_dir: &Path, remote: &str, snapshot: CacheFileSnapshot, cache: Cache) {
let key = cache_store_key(config_dir, remote);
let mut map = MEMORY_CACHE.lock().unwrap();
map.insert(key, MemoryCacheEntry { snapshot, cache });
}
fn invalidate_cached_entry(config_dir: &Path, remote: &str) {
let key = cache_store_key(config_dir, remote);
let mut map = MEMORY_CACHE.lock().unwrap();
map.remove(&key);
}
fn ensure_cache_dir(cache_dir: &Path) -> Result<(), anyhow::Error> {
fs::create_dir_all(cache_dir)?;
#[cfg(unix)]
{
let mut perms = fs::metadata(cache_dir)?.permissions();
perms.set_mode(0o700);
fs::set_permissions(cache_dir, perms)?;
}
Ok(())
}
fn write_cache_file(cache_path: &Path, cache: &Cache) -> Result<(), anyhow::Error> {
let tmp_path = cache_path.with_extension("tmp");
fs::write(&tmp_path, serde_json::to_vec_pretty(cache)?)?;
#[cfg(unix)]
{
let mut perms = fs::metadata(&tmp_path)?.permissions();
perms.set_mode(0o600);
fs::set_permissions(&tmp_path, perms)?;
}
fs::rename(&tmp_path, cache_path)?;
Ok(())
}
fn parse_cache(remote: &str, data: &str) -> Result<Cache, anyhow::Error> {
let cache = serde_json::from_str::<Cache>(data).map_err(|e| {
anyhow!(
"Cache for `{}` is invalid or outdated ({}). Run `mdrs login {}` to refresh it.",
remote,
e,
remote
)
})?;
// The digest is what says the file is still the one this client wrote. The Python
// client checks it on every read and the two share the file, so a cache one of them
// would refuse must not be honoured by the other.
let expected = compute_digest(
cache.user.as_ref(),
&cache.token.access,
&cache.token.refresh,
&cache.laboratories,
);
if cache.digest != expected {
bail!(
"Cache for `{}` has been altered or was written by an incompatible version. \
Run `mdrs login {}` to refresh it.",
remote,
remote
);
}
Ok(cache)
}
fn load_cache_from_dir(remote: &str, config_dir: &Path) -> Result<Cache, anyhow::Error> {
let cache_path = cache_file_path_in(config_dir, remote);
let snapshot = match read_cache_snapshot(&cache_path) {
Ok(snapshot) => snapshot,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
invalidate_cached_entry(config_dir, remote);
bail!(
"Not logged in to `{}`. Run `mdrs login {}` first.",
remote,
remote
);
}
Err(e) => return Err(e.into()),
};
if let Some(cache) = cached_entry(config_dir, remote, &snapshot) {
return Ok(cache);
}
let data = match fs::read_to_string(&cache_path) {
Ok(data) => data,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
invalidate_cached_entry(config_dir, remote);
bail!(
"Not logged in to `{}`. Run `mdrs login {}` first.",
remote,
remote
);
}
Err(e) => return Err(e.into()),
};
let cache = parse_cache(remote, &data)?;
update_cached_entry(config_dir, remote, snapshot, cache.clone());
Ok(cache)
}
fn load_cache_if_present_from_dir(
remote: &str,
config_dir: &Path,
) -> Result<Option<Cache>, anyhow::Error> {
let cache_path = cache_file_path_in(config_dir, remote);
let snapshot = match read_cache_snapshot(&cache_path) {
Ok(snapshot) => snapshot,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
invalidate_cached_entry(config_dir, remote);
return Ok(None);
}
Err(e) => return Err(e.into()),
};
if let Some(cache) = cached_entry(config_dir, remote, &snapshot) {
return Ok(Some(cache));
}
let data = match fs::read_to_string(&cache_path) {
Ok(data) => data,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
invalidate_cached_entry(config_dir, remote);
return Ok(None);
}
Err(e) => return Err(e.into()),
};
let cache = match parse_cache(remote, &data) {
Ok(cache) => cache,
Err(e) => {
// Said out loud before the credentials go. The alternative is a command that
// quietly carries on anonymously and fails later for a reason that looks
// unrelated to the cache it just threw away.
eprintln!("{e}");
remove_cache_in_dir(remote, config_dir)?;
return Ok(None);
}
};
update_cached_entry(config_dir, remote, snapshot, cache.clone());
Ok(Some(cache))
}
fn persist_cache_in_dir(
remote: &str,
config_dir: &Path,
cache: &Cache,
) -> Result<(), anyhow::Error> {
let cache_dir = cache_dir_path(config_dir);
ensure_cache_dir(&cache_dir)?;
let cache_path = cache_file_path_in(config_dir, remote);
write_cache_file(&cache_path, cache)?;
let snapshot = read_cache_snapshot(&cache_path)?;
update_cached_entry(config_dir, remote, snapshot, cache.clone());
Ok(())
}
fn remove_cache_in_dir(remote: &str, config_dir: &Path) -> Result<(), anyhow::Error> {
let cache_path = cache_file_path_in(config_dir, remote);
match fs::remove_file(&cache_path) {
Ok(()) => {}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
Err(e) => return Err(e.into()),
}
invalidate_cached_entry(config_dir, remote);
Ok(())
}
// ---------------------------------------------------------------------------
@@ -41,23 +277,17 @@ fn cache_file_path(remote: &str) -> std::path::PathBuf {
/// Load token and laboratories from the login cache file (no token refresh check).
pub fn load_cache(remote: &str) -> Result<Cache, anyhow::Error> {
let cache_path = cache_file_path(remote);
if !cache_path.exists() {
bail!(
"Not logged in to `{}`. Run `mdrs login {}` first.",
remote,
remote
);
}
let data = fs::read_to_string(&cache_path)?;
serde_json::from_str::<Cache>(&data).map_err(|e| {
anyhow!(
"Cache for `{}` is invalid or outdated ({}). Run `mdrs login {}` to refresh it.",
remote,
e,
remote
)
})
load_cache_from_dir(remote, &crate::settings::SETTINGS.config_dirname)
}
/// Persist a cache entry and refresh the in-memory fast path.
pub fn persist_cache(remote: &str, cache: &Cache) -> Result<(), anyhow::Error> {
persist_cache_in_dir(remote, &crate::settings::SETTINGS.config_dirname, cache)
}
/// Remove a cache entry from disk and memory.
pub fn remove_cache(remote: &str) -> Result<(), anyhow::Error> {
remove_cache_in_dir(remote, &crate::settings::SETTINGS.config_dirname)
}
// ---------------------------------------------------------------------------
@@ -76,11 +306,15 @@ pub async fn load_cache_with_token_refresh(remote: &str) -> Result<Cache, anyhow
let lock = get_remote_lock(remote);
let _guard = lock.lock().await;
ensure_cache_dir(&cache_dir_path(&crate::settings::SETTINGS.config_dirname))?;
let lock_path = cache_file_path(remote).with_extension("lock");
use fs2::FileExt;
let lock_file = fs::OpenOptions::new()
.write(true)
.create(true)
// Nothing is ever written into it: the file exists only to be flocked, and
// truncating it would touch a file other processes are holding open.
.truncate(false)
.open(&lock_path)?;
lock_file.lock_exclusive()?;
@@ -98,8 +332,7 @@ pub async fn load_cache_with_token_refresh(remote: &str) -> Result<Cache, anyhow
}
if crate::token::is_refresh_required(&cache.token.access, &cache.token.refresh) {
let new_access = refresh_and_persist(remote, &cache).await?;
cache.token.access = new_access;
cache = refresh_and_persist(remote, &cache).await?;
}
Ok(cache)
@@ -110,51 +343,95 @@ pub async fn load_cache_with_token_refresh(remote: &str) -> Result<Cache, anyhow
result
}
async fn load_cache_with_token_refresh_optional_from_dir(
remote: &str,
config_dir: &Path,
) -> Result<Option<Cache>, anyhow::Error> {
let lock = get_remote_lock(remote);
let _guard = lock.lock().await;
ensure_cache_dir(&cache_dir_path(config_dir))?;
let lock_path = cache_file_path_in(config_dir, remote).with_extension("lock");
use fs2::FileExt;
let lock_file = fs::OpenOptions::new()
.write(true)
.create(true)
// Nothing is ever written into it: the file exists only to be flocked, and
// truncating it would touch a file other processes are holding open.
.truncate(false)
.open(&lock_path)?;
lock_file.lock_exclusive()?;
let result: Result<Option<Cache>, anyhow::Error> = async {
let Some(mut cache) = load_cache_if_present_from_dir(remote, config_dir)? else {
return Ok(None);
};
if crate::token::is_expired(&cache.token.refresh) {
remove_cache_in_dir(remote, config_dir)?;
return Ok(None);
}
if crate::token::is_refresh_required(&cache.token.access, &cache.token.refresh) {
cache = refresh_and_persist_in_dir(remote, config_dir, &cache).await?;
}
Ok(Some(cache))
}
.await;
lock_file.unlock()?;
result
}
/// Load cache when present and refresh its token if needed.
///
/// Unlike `load_cache_with_token_refresh`, this returns `Ok(None)` when the user
/// is effectively anonymous: no cache file exists, the cache is invalid, or the
/// refresh token has already expired. This mirrors the Python client behavior
/// used by read-only commands.
pub async fn load_cache_with_token_refresh_optional(
remote: &str,
) -> Result<Option<Cache>, anyhow::Error> {
load_cache_with_token_refresh_optional_from_dir(
remote,
&crate::settings::SETTINGS.config_dirname,
)
.await
}
/// Call the token-refresh endpoint and write the new access token back to the
/// cache file. The caller must already hold the per-remote async mutex.
async fn refresh_and_persist(remote: &str, cache: &Cache) -> Result<String, anyhow::Error> {
async fn refresh_and_persist(remote: &str, cache: &Cache) -> Result<Cache, anyhow::Error> {
refresh_and_persist_in_dir(remote, &crate::settings::SETTINGS.config_dirname, cache).await
}
async fn refresh_and_persist_in_dir(
remote: &str,
config_dir: &Path,
cache: &Cache,
) -> Result<Cache, anyhow::Error> {
let url = crate::commands::config::get_remote_url(remote)?
.ok_or_else(|| anyhow!("Remote `{}` is not configured.", remote))?;
let conn = MDRSConnection::new(&url);
let new_access = conn.token_refresh(&cache.token.refresh).await?;
let refreshed = conn.token_refresh(&cache.token.refresh).await?;
let new_digest = compute_digest(
cache.user.as_ref(),
&new_access,
&cache.token.refresh,
&cache.laboratories,
let mut updated_cache = cache.clone();
updated_cache.token.access = refreshed.access;
if let Some(refresh) = refreshed.refresh {
updated_cache.token.refresh = refresh;
}
updated_cache.digest = compute_digest(
updated_cache.user.as_ref(),
&updated_cache.token.access,
&updated_cache.token.refresh,
&updated_cache.laboratories,
);
let cache_path = cache_file_path(remote);
let raw = fs::read_to_string(&cache_path)?;
let mut obj: serde_json::Value = serde_json::from_str(&raw)?;
persist_cache_in_dir(remote, config_dir, &updated_cache)?;
obj["token"]["access"] = serde_json::Value::String(new_access.clone());
obj["digest"] = serde_json::Value::String(new_digest);
// Write atomically: write to .tmp then rename.
let tmp_path = cache_path.with_extension("tmp");
{
use std::io::Write;
let mut tmp_file = fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.open(&tmp_path)?;
tmp_file.write_all(serde_json::to_string(&obj)?.as_bytes())?;
tmp_file.flush()?;
}
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
let mut perms = fs::metadata(&tmp_path)?.permissions();
perms.set_mode(0o600);
fs::set_permissions(&tmp_path, perms)?;
}
fs::rename(&tmp_path, &cache_path)?;
Ok(new_access)
Ok(updated_cache)
}
// ---------------------------------------------------------------------------
@@ -166,9 +443,228 @@ pub fn create_authenticated_conn(
remote: &str,
cache: &Cache,
) -> Result<MDRSConnection, anyhow::Error> {
Ok(create_remote_conn(remote)?.with_token(cache.token.access.clone()))
}
/// Create an unauthenticated `MDRSConnection` for the given remote label.
pub fn create_remote_conn(remote: &str) -> Result<MDRSConnection, anyhow::Error> {
let url = crate::commands::config::get_remote_url(remote)?
.ok_or_else(|| anyhow!("Remote `{}` is not configured.", remote))?;
Ok(MDRSConnection::new(&url)
.with_remote(remote)
.with_token(cache.token.access.clone()))
Ok(MDRSConnection::new(&url).with_remote(remote))
}
/// Create a connection for read-only commands, attaching a bearer token only
/// when a valid login cache is available.
pub async fn create_readonly_conn(
remote: &str,
) -> Result<(MDRSConnection, Option<Cache>), anyhow::Error> {
let conn = create_remote_conn(remote)?;
match load_cache_with_token_refresh_optional(remote).await? {
Some(cache) => Ok((conn.with_token(cache.token.access.clone()), Some(cache))),
None => Ok((conn, None)),
}
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::tempdir;
fn sample_cache(username: &str) -> Cache {
let mut cache = Cache {
user: Some(CacheUser {
id: 1,
username: username.to_string(),
laboratory_ids: vec![10, 20],
is_reviewer: false,
}),
token: types::CacheToken {
access: format!("access-{username}"),
refresh: format!("refresh-{username}"),
},
laboratories: CacheLabsWrapper {
items: vec![CacheLaboratory {
id: 10,
name: "lab".to_string(),
pi_name: String::new(),
full_name: "Laboratory".to_string(),
}],
},
digest: String::new(),
};
cache.digest = compute_digest(
cache.user.as_ref(),
&cache.token.access,
&cache.token.refresh,
&cache.laboratories,
);
cache
}
fn remote_name(prefix: &str, config_dir: &Path) -> String {
format!(
"{prefix}-{}",
config_dir
.file_name()
.unwrap_or_default()
.to_string_lossy()
.replace('.', "_")
)
}
#[cfg(unix)]
#[test]
fn load_cache_uses_memory_fast_path_when_snapshot_matches() {
let dir = tempdir().unwrap();
let remote = remote_name("fast-path", dir.path());
let cache = sample_cache("alice");
persist_cache_in_dir(&remote, dir.path(), &cache).unwrap();
let first = load_cache_from_dir(&remote, dir.path()).unwrap();
assert_eq!(first.user.unwrap().username, "alice");
let cache_path = cache_file_path_in(dir.path(), &remote);
let mut perms = fs::metadata(&cache_path).unwrap().permissions();
perms.set_mode(0o000);
fs::set_permissions(&cache_path, perms).unwrap();
let second = load_cache_from_dir(&remote, dir.path()).unwrap();
assert_eq!(second.user.unwrap().username, "alice");
}
#[test]
fn load_cache_reloads_when_external_writer_changes_file() {
let dir = tempdir().unwrap();
let remote = remote_name("reload", dir.path());
let original = sample_cache("alice");
let updated = sample_cache("bob");
persist_cache_in_dir(&remote, dir.path(), &original).unwrap();
let first = load_cache_from_dir(&remote, dir.path()).unwrap();
assert_eq!(first.user.unwrap().username, "alice");
let cache_dir = cache_dir_path(dir.path());
ensure_cache_dir(&cache_dir).unwrap();
let cache_path = cache_file_path_in(dir.path(), &remote);
write_cache_file(&cache_path, &updated).unwrap();
let second = load_cache_from_dir(&remote, dir.path()).unwrap();
assert_eq!(second.user.unwrap().username, "bob");
}
#[cfg(unix)]
#[test]
fn persist_cache_refreshes_memory_entry() {
let dir = tempdir().unwrap();
let remote = remote_name("persist", dir.path());
let original = sample_cache("alice");
let updated = sample_cache("bob");
persist_cache_in_dir(&remote, dir.path(), &original).unwrap();
let _ = load_cache_from_dir(&remote, dir.path()).unwrap();
persist_cache_in_dir(&remote, dir.path(), &updated).unwrap();
let cache_path = cache_file_path_in(dir.path(), &remote);
let mut perms = fs::metadata(&cache_path).unwrap().permissions();
perms.set_mode(0o000);
fs::set_permissions(&cache_path, perms).unwrap();
let loaded = load_cache_from_dir(&remote, dir.path()).unwrap();
assert_eq!(loaded.user.unwrap().username, "bob");
}
#[test]
fn remove_cache_invalidates_memory_entry() {
let dir = tempdir().unwrap();
let remote = remote_name("remove", dir.path());
let cache = sample_cache("alice");
persist_cache_in_dir(&remote, dir.path(), &cache).unwrap();
let _ = load_cache_from_dir(&remote, dir.path()).unwrap();
remove_cache_in_dir(&remote, dir.path()).unwrap();
let err = load_cache_from_dir(&remote, dir.path()).unwrap_err();
assert!(
err.to_string()
.contains(&format!("Not logged in to `{remote}`"))
);
}
fn make_jwt_with_exp(exp: i64) -> String {
use base64::{Engine, engine::general_purpose::URL_SAFE_NO_PAD};
let header = URL_SAFE_NO_PAD.encode(r#"{"alg":"none","typ":"JWT"}"#);
let payload = URL_SAFE_NO_PAD.encode(format!(r#"{{"exp":{exp}}}"#));
format!("{header}.{payload}.")
}
#[test]
fn load_cache_if_present_returns_none_when_cache_missing() {
let dir = tempdir().unwrap();
let remote = remote_name("missing", dir.path());
let loaded = load_cache_if_present_from_dir(&remote, dir.path()).unwrap();
assert!(loaded.is_none());
}
/// A cache whose contents no longer match its digest is not a session to act on.
#[test]
fn load_cache_refuses_a_tampered_cache() {
let dir = tempdir().unwrap();
let remote = remote_name("tampered", dir.path());
let mut cache = sample_cache("alice");
persist_cache_in_dir(&remote, dir.path(), &cache).unwrap();
// Someone edits the stored token but leaves the digest as it was.
cache.token.access = "someone-elses-access".to_string();
let cache_path = cache_file_path_in(dir.path(), &remote);
fs::write(&cache_path, serde_json::to_vec_pretty(&cache).unwrap()).unwrap();
invalidate_cached_entry(dir.path(), &remote);
let err = load_cache_from_dir(&remote, dir.path()).unwrap_err();
assert!(err.to_string().contains("has been altered"));
}
#[test]
fn load_cache_if_present_clears_invalid_cache() {
let dir = tempdir().unwrap();
let remote = remote_name("invalid", dir.path());
let cache_dir = cache_dir_path(dir.path());
ensure_cache_dir(&cache_dir).unwrap();
let cache_path = cache_file_path_in(dir.path(), &remote);
fs::write(&cache_path, b"{invalid json").unwrap();
let loaded = load_cache_if_present_from_dir(&remote, dir.path()).unwrap();
assert!(loaded.is_none());
assert!(!cache_path.exists());
}
#[tokio::test]
async fn optional_cache_load_treats_expired_session_as_anonymous() {
let dir = tempdir().unwrap();
let remote = remote_name("expired", dir.path());
let mut cache = sample_cache("alice");
cache.token.access = make_jwt_with_exp(0);
cache.token.refresh = make_jwt_with_exp(0);
// Re-stamped, so this exercises the expired session rather than the digest check.
cache.digest = compute_digest(
cache.user.as_ref(),
&cache.token.access,
&cache.token.refresh,
&cache.laboratories,
);
persist_cache_in_dir(&remote, dir.path(), &cache).unwrap();
let loaded = load_cache_with_token_refresh_optional_from_dir(&remote, dir.path())
.await
.unwrap();
assert!(loaded.is_none());
assert!(!cache_file_path_in(dir.path(), &remote).exists());
}
}
+8 -6
View File
@@ -1,14 +1,14 @@
use serde::Deserialize;
use serde::{Deserialize, Serialize};
/// Access and refresh token pair stored in the login cache file.
#[derive(Deserialize, Clone)]
#[derive(Deserialize, Serialize, Clone, Debug, PartialEq, Eq)]
pub struct CacheToken {
pub access: String,
pub refresh: String,
}
/// Minimal user fields stored in the cache, matching Python's `User` dataclass.
#[derive(Deserialize, Clone)]
#[derive(Deserialize, Serialize, Clone, Debug, PartialEq, Eq)]
pub struct CacheUser {
pub id: u32,
pub username: String,
@@ -17,7 +17,7 @@ pub struct CacheUser {
}
/// All four laboratory fields needed for digest computation.
#[derive(Deserialize, Clone)]
#[derive(Deserialize, Serialize, Clone, Debug, PartialEq, Eq)]
pub struct CacheLaboratory {
pub id: u32,
pub name: String,
@@ -28,15 +28,17 @@ pub struct CacheLaboratory {
}
/// Wrapper matching Python's `Laboratories` serialization: `{"items": [...]}`.
#[derive(Deserialize, Clone, Default)]
#[derive(Deserialize, Serialize, Clone, Debug, Default, PartialEq, Eq)]
pub struct CacheLabsWrapper {
pub items: Vec<CacheLaboratory>,
}
/// Full login cache, corresponding to the `<remote>.json` file written by `login`.
#[derive(Deserialize, Clone)]
#[derive(Deserialize, Serialize, Clone, Debug, PartialEq, Eq)]
pub struct Cache {
pub user: Option<CacheUser>,
pub token: CacheToken,
pub laboratories: CacheLabsWrapper,
#[serde(default)]
pub digest: String,
}
+1 -1
View File
@@ -41,7 +41,7 @@ pub enum Commands {
skip_if_exists: bool,
#[arg(short = 'p', long)]
password: Option<String>,
#[arg(long)]
#[arg(short = 'e', long)]
exclude: Vec<String>,
remote_path: String,
local_path: String,
+4 -3
View File
@@ -1,5 +1,6 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::commands::shared::{find_folder, find_lab_in_cache, parse_remote_path};
use crate::commands::shared::{find_folder, find_laboratory, parse_remote_path};
use crate::error::response_error;
use anyhow::bail;
pub async fn chacl(
@@ -23,7 +24,7 @@ pub async fn chacl(
let (remote, labname, folder_path) = parse_remote_path(remote_path)?;
let cache = load_cache_with_token_refresh(&remote).await?;
let conn = create_authenticated_conn(&remote, &cache)?;
let lab = find_lab_in_cache(&cache, &labname)?;
let lab = find_laboratory(&conn, Some(&cache), &labname).await?;
let folder = find_folder(&conn, lab.id, &folder_path, None).await?;
let mut data = serde_json::Map::new();
@@ -45,7 +46,7 @@ pub async fn chacl(
.await?;
if !resp.status().is_success() {
bail!("ACL change failed: {}", resp.status());
return Err(response_error("ACL change failed", resp).await);
}
Ok(())
}
+68 -18
View File
@@ -13,18 +13,18 @@ fn sanitize_config_file(path: &PathBuf) -> Result<(), anyhow::Error> {
}
let text = fs::read_to_string(path)?;
let lines: Vec<&str> = text.lines().collect();
if let Some(first) = lines.first() {
if first.trim() == path.to_string_lossy() {
// remove the first line and write atomically
let new_text = if lines.len() > 1 {
lines[1..].join("\n")
} else {
String::new()
};
let tmp = path.with_extension("tmp");
fs::write(&tmp, new_text.as_bytes())?;
fs::rename(&tmp, path)?;
}
if let Some(first) = lines.first()
&& first.trim() == path.to_string_lossy()
{
// remove the first line and write atomically
let new_text = if lines.len() > 1 {
lines[1..].join("\n")
} else {
String::new()
};
let tmp = path.with_extension("tmp");
fs::write(&tmp, new_text.as_bytes())?;
fs::rename(&tmp, path)?;
}
Ok(())
}
@@ -49,9 +49,10 @@ pub fn get_remote_url(remote: &str) -> Result<Option<String>, anyhow::Error> {
}
pub fn config_create(remote: &str, url: &str) -> Result<(), anyhow::Error> {
if !validate_url(url) {
let Some(url) = normalize_url(url) else {
bail!("Malformed URL");
}
};
let url = url.as_str();
let path = config_path();
sanitize_config_file(&path)?;
let path_str = path.to_string_lossy().to_string();
@@ -78,9 +79,10 @@ pub fn config_create(remote: &str, url: &str) -> Result<(), anyhow::Error> {
}
pub fn config_update(remote: &str, url: &str) -> Result<(), anyhow::Error> {
if !validate_url(url) {
let Some(url) = normalize_url(url) else {
bail!("Malformed URL");
}
};
let url = url.as_str();
let path = config_path();
sanitize_config_file(&path)?;
let path_str = path.to_string_lossy().to_string();
@@ -145,6 +147,54 @@ pub fn config_delete(remote: &str) -> Result<(), anyhow::Error> {
Ok(())
}
fn validate_url(url: &str) -> bool {
validators::url::Url::parse(url).is_ok()
/// Check a remote URL and put it in the one form every client agrees on.
///
/// A bare hostname is accepted, so a development server on `localhost` is as acceptable
/// as a deployment behind a domain name. The trailing slash goes because the URL is
/// joined with a path that brings its own.
fn normalize_url(url: &str) -> Option<String> {
let parsed = reqwest::Url::parse(url).ok()?;
if !matches!(parsed.scheme(), "http" | "https") {
return None;
}
parsed.host_str()?;
Some(url.trim_end_matches('/').to_string())
}
#[cfg(test)]
mod url_tests {
use super::normalize_url;
/// Both clients share config.ini, so they have to agree on what a remote URL is.
#[test]
fn trailing_slash_is_dropped() {
assert_eq!(
normalize_url("http://127.0.0.1:8000/api/").as_deref(),
Some("http://127.0.0.1:8000/api")
);
assert_eq!(
normalize_url("https://neurodata.riken.jp/api/").as_deref(),
Some("https://neurodata.riken.jp/api")
);
}
#[test]
fn bare_hostname_is_accepted() {
assert_eq!(
normalize_url("http://localhost:8000/api").as_deref(),
Some("http://localhost:8000/api")
);
}
#[test]
fn only_http_schemes_are_accepted() {
for url in [
"ftp://x.example.com/",
"file:///etc/passwd",
"not-a-url",
"http://",
] {
assert_eq!(normalize_url(url), None, "{url} should be rejected");
}
}
}
+5 -5
View File
@@ -1,8 +1,8 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::commands::shared::{
find_file_by_name, find_folder, find_lab_in_cache, find_subfolder_by_name, nfc,
parse_remote_path,
find_file_by_name, find_folder, find_laboratory, find_subfolder_by_name, nfc, parse_remote_path,
};
use crate::error::response_error;
use anyhow::bail;
pub async fn cp(src_path: &str, dest_path: &str, recursive: bool) -> Result<(), anyhow::Error> {
@@ -19,7 +19,7 @@ pub async fn cp(src_path: &str, dest_path: &str, recursive: bool) -> Result<(),
let cache = load_cache_with_token_refresh(&s_remote).await?;
let conn = create_authenticated_conn(&s_remote, &cache)?;
let lab = find_lab_in_cache(&cache, &s_lab)?;
let lab = find_laboratory(&conn, Some(&cache), &s_lab).await?;
let lab_id = lab.id;
// Split source path into parent directory and target name
@@ -62,7 +62,7 @@ pub async fn cp(src_path: &str, dest_path: &str, recursive: bool) -> Result<(),
.post_json(&format!("v3/files/{}/copy/", src_file_id), &body)
.await?;
if !resp.status().is_success() {
bail!("Copy failed: {}", resp.status());
return Err(response_error("Copy failed", resp).await);
}
return Ok(());
}
@@ -102,7 +102,7 @@ pub async fn cp(src_path: &str, dest_path: &str, recursive: bool) -> Result<(),
.post_json(&format!("v3/folders/{}/copy/", src_folder_id), &body)
.await?;
if !resp.status().is_success() {
bail!("Copy failed: {}", resp.status());
return Err(response_error("Copy failed", resp).await);
}
Ok(())
}
+454 -84
View File
@@ -1,12 +1,13 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::cache::create_readonly_conn;
use crate::commands::shared::{
find_file_by_name, find_folder, find_lab_in_cache, find_subfolder_by_name, parse_remote_path,
find_file_by_name, find_folder_by_doi, find_folder_limited, find_laboratory,
find_subfolder_by_name, is_doi, parse_doi_remote_path, parse_remote_path,
};
use crate::connection::MDRSConnection;
use crate::connection::{ApiRequestLimiter, MDRSConnection};
use anyhow::{anyhow, bail};
use futures::stream::{FuturesUnordered, StreamExt};
use std::path::PathBuf;
use std::sync::Arc;
use tokio::task::JoinSet;
pub async fn download(
remote_path: &str,
@@ -16,10 +17,7 @@ pub async fn download(
password: Option<&str>,
excludes: Vec<String>,
) -> Result<(), anyhow::Error> {
let (remote, labname, r_path) = parse_remote_path(remote_path)?;
let cache = load_cache_with_token_refresh(&remote).await?;
let conn = Arc::new(create_authenticated_conn(&remote, &cache)?);
let lab = find_lab_in_cache(&cache, &labname)?;
let limiter = ApiRequestLimiter::new(crate::settings::SETTINGS.concurrent);
// Validate that local_path is an existing directory (matching Python's behaviour).
let local_real = std::fs::canonicalize(local_path)
@@ -28,6 +26,179 @@ pub async fn download(
bail!("Local directory `{}` not found.", local_path);
}
// Detect DOI path: "remote:10.xxxx/prefix.ID[/optional/sub/path]"
if is_doi(remote_path.split_once(':').map(|x| x.1).unwrap_or("")) {
let (remote, doi, subpath) = parse_doi_remote_path(remote_path)?;
let (raw_conn, _cache) = create_readonly_conn(&remote).await?;
let (doi_folder, lab) = find_folder_by_doi(&raw_conn, &doi, password).await?;
let abs_path = format!("{}{}", doi_folder.path.trim_end_matches('/'), subpath);
let abs_path_clean = abs_path.trim_end_matches('/');
// Check if the target is a folder.
// If it is, we download the directory. If it is not, we download the file or subfolder.
let (folder, is_folder) = match find_folder_limited(
&raw_conn,
&limiter,
lab.id,
abs_path_clean,
password,
)
.await
{
Ok(f) => (f, true),
Err(_) => {
let (parent_path, _) = match abs_path_clean.rfind('/') {
Some(0) => ("/".to_string(), abs_path_clean[1..].to_string()),
Some(pos) => (
abs_path_clean[..pos].to_string(),
abs_path_clean[pos + 1..].to_string(),
),
None => ("/".to_string(), abs_path_clean.to_string()),
};
let parent =
find_folder_limited(&raw_conn, &limiter, lab.id, &parent_path, password)
.await?;
(parent, false)
}
};
let conn = Arc::new(raw_conn);
let excludes = Arc::new(excludes);
if is_folder {
let path = folder.path.clone();
let lab_name = Arc::new(lab.name);
let top_local = local_real.join(&folder.name);
let password_owned = password.map(str::to_string);
let mut folder_tasks: JoinSet<Result<DownloadFolderTaskResult, anyhow::Error>> =
JoinSet::new();
let mut download_tasks: JoinSet<Result<(), anyhow::Error>> = JoinSet::new();
let mut errors = Vec::new();
if !recursive {
// Single-folder, non-recursive: download the files inside the DOI folder.
let files = conn.list_all_files_limited(&folder.id, &limiter).await?;
for file in &files {
if is_excluded(&excludes, lab_name.as_str(), &path, Some(&file.name)) {
continue;
}
let dest = local_real.join(&file.name);
if skip_if_exists
&& dest.exists()
&& let Ok(meta) = std::fs::metadata(&dest)
&& meta.len() == file.size
{
println!("{}", dest.display());
continue;
}
let url = make_absolute_url(&conn, &file.download_url);
conn.download_file_limited(&url, &dest.to_string_lossy(), &limiter)
.await?;
println!("{}", dest.display());
}
return Ok(());
}
let session = DownloadSession {
conn: conn.clone(),
limiter,
lab_name,
excludes,
password: password_owned,
skip_if_exists,
};
spawn_download_folder_task(
&mut folder_tasks,
session.clone(),
folder.id.clone(),
top_local,
);
drive_download_tasks(&mut folder_tasks, &mut download_tasks, &mut errors, session)
.await;
if !errors.is_empty() {
bail!(errors.join("\n"));
}
return Ok(());
}
// Case: target is a file or subfolder inside the resolved folder.
let basename = match abs_path_clean.rfind('/') {
Some(pos) => abs_path_clean[pos + 1..].to_string(),
None => abs_path_clean.to_string(),
};
let files = conn.list_all_files_limited(&folder.id, &limiter).await?;
// Case 1: basename matches a file in the folder.
if let Some(file) = find_file_by_name(&files, &basename) {
if is_excluded(&excludes, &lab.name, &folder.path, Some(&file.name)) {
return Ok(());
}
let dest = local_real.join(&file.name);
if skip_if_exists
&& dest.exists()
&& let Ok(meta) = std::fs::metadata(&dest)
&& meta.len() == file.size
{
println!("{}", dest.display());
return Ok(());
}
let url = make_absolute_url(&conn, &file.download_url);
conn.download_file_limited(&url, &dest.to_string_lossy(), &limiter)
.await?;
println!("{}", dest.display());
return Ok(());
}
// Case 2: basename matches a sub-folder.
let subfolder = find_subfolder_by_name(&folder.sub_folders, &basename);
if let Some(sub) = subfolder {
if !recursive {
bail!("Cannot download `{}`: Is a folder.", abs_path_clean);
}
let top_local = local_real.join(&sub.name);
let mut folder_tasks: JoinSet<Result<DownloadFolderTaskResult, anyhow::Error>> =
JoinSet::new();
let mut download_tasks: JoinSet<Result<(), anyhow::Error>> = JoinSet::new();
let mut errors = Vec::new();
let lab_name = Arc::new(lab.name.clone());
let password_owned = password.map(str::to_string);
let session = DownloadSession {
conn: conn.clone(),
limiter,
lab_name,
excludes,
password: password_owned,
skip_if_exists,
};
spawn_download_folder_task(
&mut folder_tasks,
session.clone(),
sub.id.clone(),
top_local,
);
drive_download_tasks(&mut folder_tasks, &mut download_tasks, &mut errors, session)
.await;
if !errors.is_empty() {
bail!(errors.join("\n"));
}
return Ok(());
}
bail!("File or folder `{}` not found.", abs_path_clean);
}
// Normal path: "remote:/labname/path/..."
let (remote, labname, r_path) = parse_remote_path(remote_path)?;
let (raw_conn, cache) = create_readonly_conn(&remote).await?;
let conn = Arc::new(raw_conn);
let lab = find_laboratory(&conn, cache.as_ref(), &labname).await?;
// Split r_path into the parent directory path and the target basename.
// Trailing slashes are already stripped by parse_remote_path, so this is safe.
let r_path_clean = r_path.trim_end_matches('/');
@@ -40,8 +211,11 @@ pub async fn download(
None => ("/".to_string(), r_path_clean.to_string()),
};
let parent_folder = find_folder(&conn, lab.id, &parent_path, password).await?;
let files = conn.list_all_files(&parent_folder.id).await?;
let parent_folder =
find_folder_limited(&conn, &limiter, lab.id, &parent_path, password).await?;
let files = conn
.list_all_files_limited(&parent_folder.id, &limiter)
.await?;
// Case 1: basename matches a file in the parent folder.
if let Some(file) = find_file_by_name(&files, &basename) {
@@ -50,18 +224,17 @@ pub async fn download(
}
// Python always places the downloaded file inside the local directory.
let dest = local_real.join(&file.name);
if skip_if_exists {
if dest.exists() {
if let Ok(meta) = std::fs::metadata(&dest) {
if meta.len() == file.size {
println!("{}", dest.display());
return Ok(());
}
}
}
if skip_if_exists
&& dest.exists()
&& let Ok(meta) = std::fs::metadata(&dest)
&& meta.len() == file.size
{
println!("{}", dest.display());
return Ok(());
}
let url = make_absolute_url(&conn, &file.download_url);
conn.download_file(&url, &dest.to_string_lossy()).await?;
conn.download_file_limited(&url, &dest.to_string_lossy(), &limiter)
.await?;
println!("{}", dest.display());
return Ok(());
}
@@ -76,73 +249,33 @@ pub async fn download(
// We create that subdirectory first, then recurse into it.
let top_local = local_real.join(&sub.name);
// Iterative DFS: each entry is (remote_folder_id, local_dir)
let mut stack: Vec<(String, PathBuf)> = vec![(sub.id.clone(), top_local)];
let mut folder_tasks: JoinSet<Result<DownloadFolderTaskResult, anyhow::Error>> =
JoinSet::new();
let mut download_tasks: JoinSet<Result<(), anyhow::Error>> = JoinSet::new();
let mut errors = Vec::new();
let excludes = Arc::new(excludes);
let lab_name = Arc::new(lab.name.clone());
let password = password.map(str::to_string);
while let Some((folder_id, local_dir)) = stack.pop() {
let folder = conn.retrieve_folder(&folder_id).await?;
let session = DownloadSession {
conn: conn.clone(),
limiter,
lab_name,
excludes,
password,
skip_if_exists,
};
spawn_download_folder_task(
&mut folder_tasks,
session.clone(),
sub.id.clone(),
top_local,
);
if is_excluded(&excludes, &lab.name, &folder.path, None) {
continue;
}
drive_download_tasks(&mut folder_tasks, &mut download_tasks, &mut errors, session).await;
tokio::fs::create_dir_all(&local_dir).await?;
println!("{}", local_dir.display());
let dir_files = conn.list_all_files(&folder_id).await?;
// Download files in this folder (up to 10 concurrent).
let mut futs: FuturesUnordered<tokio::task::JoinHandle<()>> = FuturesUnordered::new();
for f in &dir_files {
if is_excluded(&excludes, &lab.name, &folder.path, Some(&f.name)) {
continue;
}
let dest_path = local_dir.join(&f.name);
if skip_if_exists {
if dest_path.exists() {
if let Ok(meta) = std::fs::metadata(&dest_path) {
if meta.len() == f.size {
println!("{}", dest_path.display());
continue;
}
}
}
}
let url = make_absolute_url(&conn, &f.download_url);
let conn = conn.clone();
futs.push(tokio::spawn(async move {
let dest_str = dest_path.to_string_lossy().to_string();
match conn.download_file(&url, &dest_str).await {
Ok(_) => println!("{}", dest_path.display()),
Err(_) => {
eprintln!("Failed: {}", dest_path.display());
if dest_path.is_file() {
let _ = std::fs::remove_file(&dest_path);
}
}
}
}));
if futs.len() >= crate::settings::SETTINGS.concurrent {
let _ = futs.next().await;
}
}
while futs.next().await.is_some() {}
// Push sub-folders onto the stack for recursive processing.
for sf in &folder.sub_folders {
if sf.lock {
match password {
Some(pw) => {
if conn.folder_auth(&sf.id, pw).await.is_err() {
continue;
}
}
None => continue,
}
}
let sub_local = local_dir.join(&sf.name);
stack.push((sf.id.clone(), sub_local));
}
if !errors.is_empty() {
bail!(errors.join("\n"));
}
return Ok(());
}
@@ -181,3 +314,240 @@ fn make_absolute_url(conn: &MDRSConnection, url: &str) -> String {
)
}
}
struct DownloadFolderTaskResult {
child_folders: Vec<(String, PathBuf)>,
download_jobs: Vec<DownloadJob>,
}
struct DownloadJob {
url: String,
dest_path: PathBuf,
}
/// What every folder of one recursive download shares: the connection and its request
/// budget, the laboratory the paths are named against, what to leave out, the password
/// for locked folders, and whether files already on disk may be left alone.
#[derive(Clone)]
struct DownloadSession {
conn: Arc<MDRSConnection>,
limiter: ApiRequestLimiter,
lab_name: Arc<String>,
excludes: Arc<Vec<String>>,
password: Option<String>,
skip_if_exists: bool,
}
fn spawn_download_folder_task(
folder_tasks: &mut JoinSet<Result<DownloadFolderTaskResult, anyhow::Error>>,
session: DownloadSession,
folder_id: String,
local_dir: PathBuf,
) {
folder_tasks.spawn(async move { process_download_folder(session, folder_id, local_dir).await });
}
fn spawn_download_task(
download_tasks: &mut JoinSet<Result<(), anyhow::Error>>,
conn: Arc<MDRSConnection>,
limiter: ApiRequestLimiter,
job: DownloadJob,
) {
download_tasks.spawn(async move {
let dest_str = job.dest_path.to_string_lossy().to_string();
match conn
.download_file_limited(&job.url, &dest_str, &limiter)
.await
{
Ok(()) => {
println!("{}", job.dest_path.display());
Ok(())
}
Err(err) => {
// Nothing to clear up: a failed transfer writes only to its own scratch
// file, and removes that itself.
Err(anyhow!(
"Failed to download {}: {}",
job.dest_path.display(),
err
))
}
}
});
}
async fn process_download_folder(
session: DownloadSession,
folder_id: String,
local_dir: PathBuf,
) -> Result<DownloadFolderTaskResult, anyhow::Error> {
let DownloadSession {
conn,
limiter,
lab_name,
excludes,
password,
skip_if_exists,
} = session;
let folder = conn.retrieve_folder_limited(&folder_id, &limiter).await?;
if is_excluded(excludes.as_slice(), lab_name.as_str(), &folder.path, None) {
return Ok(DownloadFolderTaskResult {
child_folders: Vec::new(),
download_jobs: Vec::new(),
});
}
tokio::fs::create_dir_all(&local_dir).await?;
println!("{}", local_dir.display());
let dir_files = conn.list_all_files_limited(&folder_id, &limiter).await?;
let mut download_jobs = Vec::new();
for file in &dir_files {
if is_excluded(
excludes.as_slice(),
lab_name.as_str(),
&folder.path,
Some(&file.name),
) {
continue;
}
let dest_path = local_dir.join(&file.name);
if skip_if_exists
&& dest_path.exists()
&& let Ok(meta) = std::fs::metadata(&dest_path)
&& meta.len() == file.size
{
println!("{}", dest_path.display());
continue;
}
download_jobs.push(DownloadJob {
url: make_absolute_url(&conn, &file.download_url),
dest_path,
});
}
let mut child_folder_tasks: JoinSet<Result<Option<(String, PathBuf)>, anyhow::Error>> =
JoinSet::new();
for sub_folder in folder.sub_folders {
let conn = conn.clone();
let limiter = limiter.clone();
let password = password.clone();
let sub_local = local_dir.join(&sub_folder.name);
child_folder_tasks.spawn(async move {
if sub_folder.lock {
match password.as_deref() {
Some(pw) => {
if conn
.folder_auth_limited(&sub_folder.id, pw, &limiter)
.await
.is_err()
{
return Ok(None);
}
}
None => return Ok(None),
}
}
Ok(Some((sub_folder.id, sub_local)))
});
}
let mut child_folders = Vec::new();
while let Some(result) = child_folder_tasks.join_next().await {
if let Some(child) = flatten_join_result(result)? {
child_folders.push(child);
}
}
Ok(DownloadFolderTaskResult {
child_folders,
download_jobs,
})
}
async fn drive_download_tasks(
folder_tasks: &mut JoinSet<Result<DownloadFolderTaskResult, anyhow::Error>>,
download_tasks: &mut JoinSet<Result<(), anyhow::Error>>,
errors: &mut Vec<String>,
session: DownloadSession,
) {
loop {
match (folder_tasks.is_empty(), download_tasks.is_empty()) {
(true, true) => break,
(false, true) => {
if let Some(result) = folder_tasks.join_next().await {
handle_download_folder_result(
result,
folder_tasks,
download_tasks,
errors,
&session,
);
}
}
(true, false) => {
if let Some(result) = download_tasks.join_next().await
&& let Err(err) = flatten_join_result(result)
{
errors.push(err.to_string());
}
}
(false, false) => {
tokio::select! {
result = folder_tasks.join_next() => {
if let Some(result) = result {
handle_download_folder_result(
result,
folder_tasks,
download_tasks,
errors,
&session,
);
}
}
result = download_tasks.join_next() => {
if let Some(result) = result
&& let Err(err) = flatten_join_result(result) {
errors.push(err.to_string());
}
}
}
}
}
}
}
fn handle_download_folder_result(
result: Result<Result<DownloadFolderTaskResult, anyhow::Error>, tokio::task::JoinError>,
folder_tasks: &mut JoinSet<Result<DownloadFolderTaskResult, anyhow::Error>>,
download_tasks: &mut JoinSet<Result<(), anyhow::Error>>,
errors: &mut Vec<String>,
session: &DownloadSession,
) {
match flatten_join_result(result) {
Ok(task_result) => {
for (folder_id, local_dir) in task_result.child_folders {
spawn_download_folder_task(folder_tasks, session.clone(), folder_id, local_dir);
}
for job in task_result.download_jobs {
spawn_download_task(
download_tasks,
session.conn.clone(),
session.limiter.clone(),
job,
);
}
}
Err(err) => errors.push(err.to_string()),
}
}
fn flatten_join_result<T>(
result: Result<Result<T, anyhow::Error>, tokio::task::JoinError>,
) -> Result<T, anyhow::Error> {
match result {
Ok(inner) => inner,
Err(err) => Err(anyhow!("Task join failed: {}", err)),
}
}
+13 -20
View File
@@ -1,33 +1,26 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::commands::shared::{
find_file_by_name, find_folder, find_lab_in_cache, parse_remote_path,
};
use crate::cache::create_readonly_conn;
use crate::commands::shared::{find_file_by_name, resolve_remote_file};
use crate::error::response_error;
use anyhow::anyhow;
pub async fn file_metadata(remote_path: &str, password: Option<&str>) -> Result<(), anyhow::Error> {
let (remote, labname, r_path) = parse_remote_path(remote_path)?;
let remote = remote_path
.split(':')
.next()
.ok_or_else(|| anyhow!("Invalid remote path"))?;
let (conn, cache) = create_readonly_conn(remote).await?;
let (parent_folder, basename) =
resolve_remote_file(&conn, cache.as_ref(), remote_path, password).await?;
let cache = load_cache_with_token_refresh(&remote).await?;
let conn = create_authenticated_conn(&remote, &cache)?;
let lab = find_lab_in_cache(&cache, &labname)?;
let lab_id = lab.id;
// Split the file path into parent directory and filename
let path = r_path.trim_end_matches('/');
let (dirname, basename) = if let Some(pos) = path.rfind('/') {
let d = if pos == 0 { "/" } else { &path[..pos] };
(d.to_string(), path[pos + 1..].to_string())
} else {
("/".to_string(), path.to_string())
};
let parent_folder = find_folder(&conn, lab_id, &dirname, password).await?;
let files = conn.list_all_files(&parent_folder.id).await?;
let file = find_file_by_name(&files, &basename)
.ok_or_else(|| anyhow!("File `{}` not found.", basename))?;
let resp = conn.get(&format!("v3/files/{}/metadata/", file.id)).await?;
if !resp.status().is_success() {
return Err(response_error("Failed to get file metadata", resp).await);
}
let json: serde_json::Value = resp.json().await?;
println!("{}", serde_json::to_string(&json)?);
Ok(())
+2 -3
View File
@@ -1,8 +1,7 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::cache::create_readonly_conn;
pub async fn labs(remote: &str) -> Result<(), anyhow::Error> {
let cache = load_cache_with_token_refresh(remote).await?;
let conn = create_authenticated_conn(remote, &cache)?;
let (conn, _) = create_readonly_conn(remote).await?;
let labs = conn.list_laboratories().await?;
let header = ("Name", "PI", "Laboratory");
+17 -64
View File
@@ -1,14 +1,10 @@
use crate::cache::{CacheLabsWrapper, CacheUser, compute_digest};
use crate::cache::{Cache, CacheLabsWrapper, CacheToken, CacheUser, compute_digest, persist_cache};
use crate::connection::MDRSConnection;
use crate::models::laboratory::Laboratories;
use crate::models::user::User;
use anyhow::{anyhow, bail};
use reqwest::Client;
use serde::Deserialize;
use serde_json::{Value, json};
use std::fs;
#[cfg(unix)]
use std::os::unix::fs::PermissionsExt;
/// Prompt for credentials if not supplied and then perform login.
/// This is the entry point called from `main`.
@@ -72,66 +68,23 @@ pub async fn login(username: &str, password: &str, remote: &str) -> Result<(), a
let cache_labs: CacheLabsWrapper = (&labs).into();
// compute Python-compatible digest
let digest = compute_digest(
cache_user_opt.as_ref(),
&token.access,
&token.refresh,
&cache_labs,
);
// build the cache JSON — field order matches Python's dataclass layout:
// user (id, username, laboratory_ids, is_reviewer)
// token (access, refresh)
// laboratories (items)
// digest
let user_val: Value = match &cache_user_opt {
Some(u) => json!({
"id": u.id,
"username": u.username,
"laboratory_ids": u.laboratory_ids,
"is_reviewer": u.is_reviewer
}),
None => Value::Null,
let cache = Cache {
user: cache_user_opt,
token: CacheToken {
access: token.access,
refresh: token.refresh,
},
laboratories: cache_labs,
digest: String::new(),
};
let labs_items: Vec<Value> = cache_labs
.items
.iter()
.map(|l| {
json!({
"id": l.id,
"name": l.name,
"pi_name": l.pi_name,
"full_name": l.full_name
})
})
.collect();
let obj = json!({
"user": user_val,
"token": {"access": token.access, "refresh": token.refresh},
"laboratories": {"items": labs_items},
"digest": digest
});
// write cache file: {config_dirname}/cache/<remote>.json
let cache_dir = crate::settings::SETTINGS.config_dirname.join("cache");
fs::create_dir_all(&cache_dir)?;
#[cfg(unix)]
{
let mut perms = fs::metadata(&cache_dir)?.permissions();
perms.set_mode(0o700);
fs::set_permissions(&cache_dir, perms)?;
}
let cache_file = cache_dir.join(format!("{}.json", remote));
let tmp = cache_file.with_extension("tmp");
fs::write(&tmp, serde_json::to_vec_pretty(&obj)?)?;
#[cfg(unix)]
{
let mut perms = fs::metadata(&tmp)?.permissions();
perms.set_mode(0o600);
fs::set_permissions(&tmp, perms)?;
}
fs::rename(&tmp, &cache_file)?;
let mut cache = cache;
cache.digest = compute_digest(
cache.user.as_ref(),
&cache.token.access,
&cache.token.refresh,
&cache.laboratories,
);
persist_cache(remote, &cache)?;
println!("Login Successful");
Ok(())
+1 -8
View File
@@ -1,10 +1,3 @@
pub fn logout(remote: &str) -> Result<(), anyhow::Error> {
let cache_path = crate::settings::SETTINGS
.config_dirname
.join("cache")
.join(format!("{}.json", remote));
if cache_path.exists() {
std::fs::remove_file(&cache_path)?;
}
Ok(())
crate::cache::remove_cache(remote)
}
+92 -25
View File
@@ -1,12 +1,19 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::commands::shared::{find_folder, find_lab_in_cache, fmt_datetime, parse_remote_path};
use crate::cache::{Cache, create_readonly_conn};
use crate::commands::shared::{fmt_datetime, resolve_remote_folder};
use crate::connection::MDRSConnection;
use crate::models::file::File;
use crate::models::folder::{FolderDetail, FolderSimple};
use serde_json::{Value, json};
use std::collections::HashMap;
use std::future::Future;
use std::pin::Pin;
/// Laboratory names by id, for the JSON output's per-folder `laboratory` field.
type LaboratoryNames = HashMap<u32, String>;
/// Shown for a folder whose laboratory the client cannot name, matching the Python client.
const UNKNOWN_LABORATORY: &str = "(invalid)";
pub async fn ls(
remote_path: &str,
password: Option<&str>,
@@ -14,22 +21,25 @@ pub async fn ls(
is_recursive: bool,
is_quiet: bool,
) -> Result<(), anyhow::Error> {
let (remote, labname, path) = parse_remote_path(remote_path)?;
let cache = load_cache_with_token_refresh(&remote).await?;
let conn = create_authenticated_conn(&remote, &cache)?;
let lab = find_lab_in_cache(&cache, &labname)?;
let folder = find_folder(&conn, lab.id, &path, password).await?;
let remote = remote_path
.split(':')
.next()
.ok_or_else(|| anyhow::anyhow!("Invalid remote path"))?;
let (conn, cache) = create_readonly_conn(remote).await?;
let (folder, lab) =
resolve_remote_folder(&conn, cache.as_ref(), None, remote_path, password).await?;
let labname = lab.name;
if is_json {
let names = laboratory_names(&conn, cache.as_ref()).await;
let output = if is_recursive {
build_folder_json_recursive(&conn, folder, &labname).await?
build_folder_json_recursive(&conn, folder, &names, password).await?
} else {
build_folder_json_flat(&conn, &folder, &labname).await?
build_folder_json_flat(&conn, &folder, &names).await?
};
println!("{}", serde_json::to_string(&output)?);
} else if is_recursive {
let prefix = format!("{}:/{}", remote, labname);
let prefix = format!("{}:/{}", conn.remote.as_deref().unwrap_or(""), labname);
ls_plain_recursive(&conn, folder, &labname, &prefix, password).await?;
} else {
let files = conn.list_all_files(&folder.id).await?;
@@ -212,7 +222,13 @@ fn file_to_json(f: &File, base_url: &str) -> Value {
let download_url = if f.download_url.starts_with("http") {
f.download_url.clone()
} else {
format!("{}{}", base_url.trim_end_matches('/'), f.download_url)
// The API answers with a relative path and no leading separator, so supply one
// rather than running the two together.
format!(
"{}/{}",
base_url.trim_end_matches('/'),
f.download_url.trim_start_matches('/')
)
};
json!({
"id": f.id,
@@ -227,7 +243,7 @@ fn file_to_json(f: &File, base_url: &str) -> Value {
})
}
fn folder_simple_to_json(sf: &FolderSimple) -> Value {
fn folder_simple_to_json(sf: &FolderSimple, names: &LaboratoryNames) -> Value {
json!({
"id": sf.id,
"pid": sf.pid,
@@ -235,26 +251,58 @@ fn folder_simple_to_json(sf: &FolderSimple) -> Value {
"access_level": sf.access_level_name(),
"lock": sf.lock,
"size": sf.size,
"laboratory_id": sf.laboratory_id,
"laboratory": laboratory_name(names, sf.laboratory_id),
"description": sf.description,
"created_at": sf.created_at,
"updated_at": sf.updated_at,
})
}
/// Name the laboratory a folder belongs to, so a listing does not hand out an id the
/// reader has no way to resolve.
fn laboratory_name(names: &LaboratoryNames, id: u32) -> String {
names
.get(&id)
.cloned()
.unwrap_or_else(|| UNKNOWN_LABORATORY.to_string())
}
/// Collect laboratory names, preferring what the login cache already holds so a listing
/// does not pay for a request it does not need.
async fn laboratory_names(conn: &MDRSConnection, cache: Option<&Cache>) -> LaboratoryNames {
if let Some(cache) = cache.filter(|c| !c.laboratories.items.is_empty()) {
return cache
.laboratories
.items
.iter()
.map(|lab| (lab.id, lab.name.clone()))
.collect();
}
match conn.list_laboratories().await {
Ok(labs) => labs
.items
.into_iter()
.map(|lab| (lab.id, lab.name))
.collect(),
Err(_) => LaboratoryNames::new(),
}
}
/// Build JSON for the top-level folder without recursing into sub-folders.
async fn build_folder_json_flat(
conn: &MDRSConnection,
folder: &FolderDetail,
labname: &str,
names: &LaboratoryNames,
) -> Result<Value, anyhow::Error> {
let metadata = get_folder_metadata(conn, &folder.id).await?;
let files = conn.list_all_files(&folder.id).await?;
let mut files = conn.list_all_files(&folder.id).await?;
files.sort_by(|a, b| a.name.cmp(&b.name));
let files_json: Vec<Value> = files.iter().map(|f| file_to_json(f, &conn.url)).collect();
let sub_folders_json: Vec<Value> = folder
.sub_folders
let mut sub_folders = folder.sub_folders.clone();
sub_folders.sort_by(|a, b| a.name.cmp(&b.name));
let sub_folders_json: Vec<Value> = sub_folders
.iter()
.map(folder_simple_to_json)
.map(|sf| folder_simple_to_json(sf, names))
.collect();
Ok(json!({
@@ -264,7 +312,7 @@ async fn build_folder_json_flat(
"size": folder.size,
"access_level": folder.access_level_name(),
"lock": folder.lock,
"laboratory": labname,
"laboratory": laboratory_name(names, folder.laboratory_id),
"description": folder.description,
"created_at": folder.created_at,
"updated_at": folder.updated_at,
@@ -278,17 +326,36 @@ async fn build_folder_json_flat(
fn build_folder_json_recursive<'a>(
conn: &'a MDRSConnection,
folder: FolderDetail,
labname: &'a str,
names: &'a LaboratoryNames,
password: Option<&'a str>,
) -> Pin<Box<dyn Future<Output = Result<Value, anyhow::Error>> + 'a>> {
Box::pin(async move {
let metadata = get_folder_metadata(conn, &folder.id).await?;
let files = conn.list_all_files(&folder.id).await?;
let mut files = conn.list_all_files(&folder.id).await?;
files.sort_by(|a, b| a.name.cmp(&b.name));
let files_json: Vec<Value> = files.iter().map(|f| file_to_json(f, &conn.url)).collect();
let mut sub_folders = folder.sub_folders.clone();
sub_folders.sort_by(|a, b| a.name.cmp(&b.name));
let mut sub_folders_json = Vec::new();
for sf in &folder.sub_folders {
for sf in &sub_folders {
// A folder the caller cannot unlock is left out rather than ending the whole
// listing: a tree usually holds a locked folder or two, and the rest of it is
// still what the user asked to see. Anything else is reported, because a
// listing that quietly drops a branch cannot be told from one where the
// branch is empty, and this output is read by programs.
if sf.lock {
match password {
None => continue,
Some(pw) => {
if conn.folder_auth(&sf.id, pw).await.is_err() {
continue;
}
}
}
}
let sf_detail = conn.retrieve_folder(&sf.id).await?;
let sf_json = build_folder_json_recursive(conn, sf_detail, labname).await?;
let sf_json = build_folder_json_recursive(conn, sf_detail, names, password).await?;
sub_folders_json.push(sf_json);
}
@@ -299,7 +366,7 @@ fn build_folder_json_recursive<'a>(
"size": folder.size,
"access_level": folder.access_level_name(),
"lock": folder.lock,
"laboratory": labname,
"laboratory": laboratory_name(names, folder.laboratory_id),
"description": folder.description,
"created_at": folder.created_at,
"updated_at": folder.updated_at,
+13 -7
View File
@@ -1,16 +1,22 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::commands::shared::{find_folder, find_lab_in_cache, parse_remote_path};
use crate::cache::create_readonly_conn;
use crate::commands::shared::resolve_remote_folder;
use crate::error::response_error;
pub async fn metadata(remote_path: &str, password: Option<&str>) -> Result<(), anyhow::Error> {
let (remote, labname, folder_path) = parse_remote_path(remote_path)?;
let cache = load_cache_with_token_refresh(&remote).await?;
let conn = create_authenticated_conn(&remote, &cache)?;
let lab = find_lab_in_cache(&cache, &labname)?;
let folder = find_folder(&conn, lab.id, &folder_path, password).await?;
let remote = remote_path
.split(':')
.next()
.ok_or_else(|| anyhow::anyhow!("Invalid remote path"))?;
let (conn, cache) = create_readonly_conn(remote).await?;
let (folder, _) =
resolve_remote_folder(&conn, cache.as_ref(), None, remote_path, password).await?;
let resp = conn
.get(&format!("v3/folders/{}/metadata/", folder.id))
.await?;
if !resp.status().is_success() {
return Err(response_error("Failed to get folder metadata", resp).await);
}
let json: serde_json::Value = resp.json().await?;
println!("{}", serde_json::to_string(&json)?);
Ok(())
+4 -4
View File
@@ -1,8 +1,8 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::commands::shared::{
find_file_by_name, find_folder, find_lab_in_cache, find_subfolder_by_name, nfc,
parse_remote_path,
find_file_by_name, find_folder, find_laboratory, find_subfolder_by_name, nfc, parse_remote_path,
};
use crate::error::response_error;
use anyhow::{anyhow, bail};
pub async fn mkdir(remote_path: &str) -> Result<(), anyhow::Error> {
@@ -25,7 +25,7 @@ pub async fn mkdir(remote_path: &str) -> Result<(), anyhow::Error> {
let cache = load_cache_with_token_refresh(&remote).await?;
let conn = create_authenticated_conn(&remote, &cache)?;
let lab = find_lab_in_cache(&cache, &labname)?;
let lab = find_laboratory(&conn, Some(&cache), &labname).await?;
let parent_folder = find_folder(&conn, lab.id, parent_path, None).await?;
// Check for name conflict in sub-folders or files
@@ -40,7 +40,7 @@ pub async fn mkdir(remote_path: &str) -> Result<(), anyhow::Error> {
.create_folder(&parent_folder.id, &nfc(new_folder_name))
.await?;
if !resp.status().is_success() {
bail!("Failed to create folder: {}", resp.status());
return Err(response_error("Failed to create folder", resp).await);
}
Ok(())
}
+5 -5
View File
@@ -1,8 +1,8 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::commands::shared::{
find_file_by_name, find_folder, find_lab_in_cache, find_subfolder_by_name, nfc,
parse_remote_path,
find_file_by_name, find_folder, find_laboratory, find_subfolder_by_name, nfc, parse_remote_path,
};
use crate::error::response_error;
use anyhow::bail;
pub async fn mv(src_path: &str, dest_path: &str) -> Result<(), anyhow::Error> {
@@ -19,7 +19,7 @@ pub async fn mv(src_path: &str, dest_path: &str) -> Result<(), anyhow::Error> {
let cache = load_cache_with_token_refresh(&s_remote).await?;
let conn = create_authenticated_conn(&s_remote, &cache)?;
let lab = find_lab_in_cache(&cache, &s_lab)?;
let lab = find_laboratory(&conn, Some(&cache), &s_lab).await?;
let lab_id = lab.id;
// Split source path into parent directory and target name
@@ -62,7 +62,7 @@ pub async fn mv(src_path: &str, dest_path: &str) -> Result<(), anyhow::Error> {
.post_json(&format!("v3/files/{}/move/", src_file_id), &body)
.await?;
if !resp.status().is_success() {
bail!("Move failed: {}", resp.status());
return Err(response_error("Move failed", resp).await);
}
return Ok(());
}
@@ -99,7 +99,7 @@ pub async fn mv(src_path: &str, dest_path: &str) -> Result<(), anyhow::Error> {
.post_json(&format!("v3/folders/{}/move/", src_folder_id), &body)
.await?;
if !resp.status().is_success() {
bail!("Move failed: {}", resp.status());
return Err(response_error("Move failed", resp).await);
}
Ok(())
}
+5 -4
View File
@@ -1,7 +1,8 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::commands::shared::{
find_file_by_name, find_folder, find_lab_in_cache, find_subfolder_by_name, parse_remote_path,
find_file_by_name, find_folder, find_laboratory, find_subfolder_by_name, parse_remote_path,
};
use crate::error::response_error;
use anyhow::{anyhow, bail};
pub async fn rm(remote_path: &str, recursive: bool) -> Result<(), anyhow::Error> {
@@ -22,7 +23,7 @@ pub async fn rm(remote_path: &str, recursive: bool) -> Result<(), anyhow::Error>
let cache = load_cache_with_token_refresh(&remote).await?;
let conn = create_authenticated_conn(&remote, &cache)?;
let lab = find_lab_in_cache(&cache, &labname)?;
let lab = find_laboratory(&conn, Some(&cache), &labname).await?;
let parent_folder = find_folder(&conn, lab.id, parent_path, None).await?;
// Check if target is a file
@@ -30,7 +31,7 @@ pub async fn rm(remote_path: &str, recursive: bool) -> Result<(), anyhow::Error>
if let Some(file) = find_file_by_name(&files, target_name) {
let resp = conn.delete(&format!("v3/files/{}/", file.id)).await?;
if !resp.status().is_success() {
bail!("Failed to delete file: {}", resp.status());
return Err(response_error("Failed to delete file", resp).await);
}
return Ok(());
}
@@ -47,7 +48,7 @@ pub async fn rm(remote_path: &str, recursive: bool) -> Result<(), anyhow::Error>
)
.await?;
if !resp.status().is_success() {
bail!("Failed to delete folder: {}", resp.status());
return Err(response_error("Failed to delete folder", resp).await);
}
return Ok(());
}
+101 -1
View File
@@ -48,6 +48,25 @@ fn is_newer(current: &str, latest: &str) -> bool {
false
}
/// Read the digest out of a `sha256sum` line: the hex digest, then the file it covers.
fn parse_sha256_line(text: &str) -> Option<String> {
let digest = text.split_whitespace().next()?.to_ascii_lowercase();
let is_hex = digest.len() == 64 && digest.chars().all(|c| c.is_ascii_hexdigit());
is_hex.then_some(digest)
}
/// Hex SHA-256 of a slice, in the form `sha256sum` prints.
fn sha256_hex(bytes: &[u8]) -> String {
use sha2::{Digest, Sha256};
let mut hasher = Sha256::new();
hasher.update(bytes);
hasher
.finalize()
.iter()
.map(|b| format!("{:02x}", b))
.collect()
}
/// Extract the binary named `bin_name` from a `.tar.gz` archive at `archive_path`
/// and write it to `dest_path`.
fn extract_from_tar_gz(
@@ -138,10 +157,11 @@ pub async fn selfupdate(yes: bool) -> anyhow::Result<()> {
println!("New version available: {latest_version}");
// Find the asset matching the current build target.
// The checksum asset carries the target in its name too, so say which one is wanted.
let asset = release
.assets
.iter()
.find(|a| a.name.contains(BUILD_TARGET))
.find(|a| a.name.contains(BUILD_TARGET) && !a.name.ends_with(".sha256"))
.ok_or_else(|| {
let names: Vec<&str> = release.assets.iter().map(|a| a.name.as_str()).collect();
anyhow!(
@@ -186,6 +206,54 @@ pub async fn selfupdate(yes: bool) -> anyhow::Result<()> {
}
let bytes = download_resp.bytes().await?;
// This binary is about to replace the one the user is running, so what arrived has to
// be what the release says it is. Transport already proves who served it; the digest
// proves the archive was not swapped or truncated on the way.
match release
.assets
.iter()
.find(|a| a.name == format!("{}.sha256", asset.name))
{
Some(checksum_asset) => {
let mut checksum_req = client
.get(&checksum_asset.browser_download_url)
.header(USER_AGENT, format!("mdrs/{current_version}"));
if let Ok(token) = env::var("GITEA_TOKEN") {
checksum_req = checksum_req.header(AUTHORIZATION, format!("Bearer {token}"));
}
let checksum_resp = checksum_req.send().await?;
if !checksum_resp.status().is_success() {
bail!(
"Failed to download checksum: HTTP {}",
checksum_resp.status()
);
}
let expected = parse_sha256_line(&checksum_resp.text().await?)
.ok_or_else(|| anyhow!("Checksum asset '{}' is malformed", checksum_asset.name))?;
let actual = sha256_hex(&bytes);
if actual != expected {
bail!(
"Checksum mismatch for '{}': expected {}, got {}. The download was not what \
the release publishes, so this binary has not been replaced.",
asset.name,
expected,
actual
);
}
println!("Checksum verified.");
}
None => {
// Releases built before the workflow published checksums have none to check
// against. Say so rather than letting the absence pass for a verified download.
eprintln!(
"Warning: release {latest_version} publishes no checksum for '{}', \
so the download could not be verified.",
asset.name
);
}
}
std::fs::write(&archive_path, &bytes)?;
// Extract the binary from the archive.
@@ -216,3 +284,35 @@ pub async fn selfupdate(yes: bool) -> anyhow::Result<()> {
println!("Successfully updated to version {latest_version}.");
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_sha256sum_line_yields_its_digest() {
let digest = "a".repeat(64);
assert_eq!(
parse_sha256_line(&format!(
"{digest} mdrs-2.0.1-x86_64-unknown-linux-musl.tar.gz\n"
)),
Some(digest)
);
}
#[test]
fn a_line_that_is_not_a_digest_is_refused() {
for text in ["", "not-a-digest file", "abc123 file"] {
assert_eq!(parse_sha256_line(text), None, "{text:?} should be refused");
}
}
#[test]
fn the_digest_matches_what_sha256sum_prints() {
// Verified against `printf mdrs | sha256sum`.
assert_eq!(
sha256_hex(b"mdrs"),
"bd9845248df877e411ec8bef10b823bd4c0900a530157eedfdfbfb63ccbde2fa"
);
}
}
+506 -6
View File
@@ -1,15 +1,167 @@
use crate::cache::{Cache, CacheLaboratory};
use crate::connection::ApiRequestLimiter;
use crate::connection::MDRSConnection;
use crate::models::file::File;
use crate::models::folder::{FolderDetail, FolderSimple};
use crate::models::laboratory::Laboratory;
use anyhow::{anyhow, bail};
use unicode_normalization::UnicodeNormalization;
// ---------------------------------------------------------------------------
// DOI helpers
// ---------------------------------------------------------------------------
/// Return true if the path component (after the remote: prefix) looks like a
/// DOI string, i.e. starts with "10." and contains a "/".
pub fn is_doi(path: &str) -> bool {
path.starts_with("10.") && path.contains('/')
}
/// Extract the DOI suffix ID from a full DOI string.
///
/// MDRS uses the segment after the last `.` in the DOI as its internal
/// identifier, e.g. `10.xxxx/prefix.20230511-001` → `20230511-001`.
/// If there is no `.` after the `/`, the entire suffix after `/` is used.
pub fn doi_suffix_id(doi: &str) -> &str {
// Find the slash that separates DOI prefix from suffix.
if let Some(slash_pos) = doi.find('/') {
let suffix = &doi[slash_pos + 1..];
// Use the part after the last `.` within the suffix.
if let Some(dot_pos) = suffix.rfind('.') {
&suffix[dot_pos + 1..]
} else {
suffix
}
} else {
doi
}
}
/// Split a DOI-with-optional-path string into `(doi, subpath)`.
///
/// A DOI has the form `10.REGISTRANT/SUFFIX` where SUFFIX contains no `/`.
/// Anything after the first `/` following SUFFIX is treated as a subfolder path.
///
/// Examples:
/// - `"10.1234/prefix.ID"` → `("10.1234/prefix.ID", "")`
/// - `"10.1234/prefix.ID/"` → `("10.1234/prefix.ID", "")`
/// - `"10.1234/prefix.ID/sub"` → `("10.1234/prefix.ID", "/sub")`
/// - `"10.1234/prefix.ID/sub/deep"` → `("10.1234/prefix.ID", "/sub/deep")`
pub fn split_doi_and_subpath(doi_with_path: &str) -> (&str, &str) {
// Find the first `/` that separates registrant from suffix.
if let Some(first_slash) = doi_with_path.find('/') {
let after_suffix_start = first_slash + 1;
let after_first = &doi_with_path[after_suffix_start..];
// Find the next `/` inside the suffix portion — this starts the subpath.
if let Some(second_slash) = after_first.find('/') {
let doi_end = after_suffix_start + second_slash;
let doi = &doi_with_path[..doi_end];
let subpath = &doi_with_path[doi_end..]; // begins with "/"
// Treat a bare trailing slash as no subpath (root of DOI folder).
if subpath == "/" {
(doi, "")
} else {
(doi, subpath)
}
} else {
// No second slash — the whole string is the DOI, no subpath.
(doi_with_path, "")
}
} else {
(doi_with_path, "")
}
}
/// Parse `"remote:10.xxxx/prefix.ID[/optional/sub/path]"` into
/// `(remote, doi, subpath)`.
///
/// `subpath` is empty when the remote path points directly at the DOI root
/// folder. Otherwise it is an absolute path string like `"/subfolder/deep"`.
pub fn parse_doi_remote_path(remote_path: &str) -> Result<(String, String, String), anyhow::Error> {
let parts: Vec<&str> = remote_path.splitn(2, ':').collect();
if parts.len() != 2 {
bail!("remote_path must be in the form 'remote:10.xxxx/prefix.ID'");
}
let remote = parts[0].to_string();
let doi_with_path = parts[1];
if !is_doi(doi_with_path) {
bail!(
"Path `{}` does not look like a DOI (must start with '10.' and contain '/').",
doi_with_path
);
}
let (doi, subpath) = split_doi_and_subpath(doi_with_path);
Ok((remote, doi.to_string(), subpath.to_string()))
}
/// Resolve a DOI string to a (FolderDetail, Laboratory) pair.
///
/// Calls GET v3/doi/{id}/ to look up the folder ID, then fetches the full
/// folder detail (which carries `laboratory_id`) and resolves the laboratory.
pub async fn find_folder_by_doi(
conn: &MDRSConnection,
doi: &str,
password: Option<&str>,
) -> Result<(FolderDetail, Laboratory), anyhow::Error> {
// Strip any trailing slash from the DOI before extracting the suffix ID.
let doi_clean = doi.trim_end_matches('/');
let id = doi_suffix_id(doi_clean);
let doi_resp = conn.retrieve_doi(id).await?;
// Verify that the returned DOI matches the one supplied by the caller
// (case-insensitive, trimming trailing slashes).
let returned = doi_resp.doi.trim_end_matches('/');
if !returned.eq_ignore_ascii_case(doi_clean) {
bail!(
"DOI mismatch: requested `{}` but server returned `{}`.",
doi_clean,
returned
);
}
let folder_id = &doi_resp.folder.id;
// Fetch the full folder detail; laboratory_id is available here.
let folder = conn
.retrieve_folder(folder_id)
.await
.map_err(|e| anyhow!("Failed to retrieve folder for DOI `{}`: {}", doi_clean, e))?;
// Handle password-locked folder.
if folder.lock {
match password {
None => {
bail!(
"Folder for DOI `{}` is locked. Use -p/--password to provide a password.",
doi_clean
);
}
Some(pw) => conn.folder_auth(folder_id, pw).await?,
}
}
// Resolve laboratory using the laboratory_id from the folder detail.
let lab_id = folder.laboratory_id;
let lab = conn
.list_laboratories()
.await?
.items
.into_iter()
.find(|l| l.id == lab_id)
.ok_or_else(|| anyhow!("Laboratory with id {} not found.", lab_id))?;
Ok((folder, lab))
}
// ---------------------------------------------------------------------------
// Path helpers
// ---------------------------------------------------------------------------
/// Parse "remote:/labname/path/" into (remote, labname, folder_path).
///
/// The path is put in one form before anything is done with it: repeated and trailing
/// separators go, `.` segments go, and a `..` is refused rather than sent on. The Python
/// client does the same, and both talk to the same server.
pub fn parse_remote_path(remote_path: &str) -> Result<(String, String, String), anyhow::Error> {
let parts: Vec<&str> = remote_path.splitn(2, ':').collect();
if parts.len() != 2 {
@@ -20,12 +172,22 @@ pub fn parse_remote_path(remote_path: &str) -> Result<(String, String, String),
if !rest.starts_with('/') {
bail!("Path must be absolute (start with '/')");
}
let folder_parts: Vec<&str> = rest.trim_start_matches('/').splitn(2, '/').collect();
let labname = folder_parts[0].to_string();
let path = if folder_parts.len() > 1 && !folder_parts[1].is_empty() {
format!("/{}", folder_parts[1].trim_end_matches('/'))
} else {
let mut segments = Vec::new();
for segment in rest.split('/') {
match segment {
"" | "." => continue,
".." => bail!("Path traversal found."),
other => segments.push(other),
}
}
if segments.is_empty() {
bail!("Laboratory name is missing from `{}`.", remote_path);
}
let labname = segments.remove(0).to_string();
let path = if segments.is_empty() {
"/".to_string()
} else {
format!("/{}", segments.join("/"))
};
Ok((remote, labname, path))
}
@@ -47,6 +209,32 @@ pub fn find_lab_in_cache<'a>(
.ok_or_else(|| anyhow!("Laboratory `{}` not found.", labname))
}
/// Resolve a laboratory by name using cached laboratories when available, and
/// falling back to the API when the user is anonymous.
pub async fn find_laboratory(
conn: &MDRSConnection,
cache: Option<&Cache>,
labname: &str,
) -> Result<Laboratory, anyhow::Error> {
if let Some(cache) = cache
&& let Ok(lab) = find_lab_in_cache(cache, labname)
{
return Ok(Laboratory {
id: lab.id,
name: lab.name.clone(),
pi_name: lab.pi_name.clone(),
full_name: lab.full_name.clone(),
});
}
conn.list_laboratories()
.await?
.items
.into_iter()
.find(|lab| lab.name == labname)
.ok_or_else(|| anyhow!("Laboratory `{}` not found.", labname))
}
// ---------------------------------------------------------------------------
// Unicode helpers
// ---------------------------------------------------------------------------
@@ -95,6 +283,49 @@ pub async fn find_folder(
Ok(folder)
}
/// Resolve a folder by path while consuming the shared API concurrency budget.
pub async fn find_folder_limited(
conn: &MDRSConnection,
limiter: &ApiRequestLimiter,
lab_id: u32,
path: &str,
password: Option<&str>,
) -> Result<FolderDetail, anyhow::Error> {
let normalized_path = nfc(path);
let folders = conn
.list_folders_by_path_limited(lab_id, &normalized_path, limiter)
.await?;
if folders.is_empty() {
bail!("Folder `{}` not found.", path);
}
if folders.len() != 1 {
bail!(
"Ambiguous path `{}`: {} folders matched.",
path,
folders.len()
);
}
let folder_simple = &folders[0];
if folder_simple.lock {
match password {
None => {
bail!(
"Folder `{}` is locked. Use -p/--password to provide a password.",
path
);
}
Some(pw) => {
conn.folder_auth_limited(&folder_simple.id, pw, limiter)
.await?
}
}
}
let folder = conn
.retrieve_folder_limited(&folder_simple.id, limiter)
.await?;
Ok(folder)
}
/// Find a file by name (NFC-normalized, case-insensitive) in a file list.
pub fn find_file_by_name<'a>(files: &'a [File], name: &str) -> Option<&'a File> {
let name_lower = nfc(name).to_lowercase();
@@ -121,7 +352,7 @@ pub fn find_subfolder_by_name<'a>(
/// Format an ISO 8601 timestamp as "YYYY/MM/DD HH:MM:SS".
pub fn fmt_datetime(iso: &str) -> String {
let s = iso.trim();
let s = if let Some(pos) = s[10..].find(|c: char| c == '+' || c == '-') {
let s = if let Some(pos) = s[10..].find(['+', '-']) {
&s[..10 + pos]
} else {
s.trim_end_matches('Z')
@@ -134,3 +365,272 @@ pub fn fmt_datetime(iso: &str) -> String {
iso.to_string()
}
}
/// Resolve any remote path (normal or DOI-based) into a FolderDetail and Laboratory.
/// Takes an optional API limiter for download command compatibility.
pub async fn resolve_remote_folder(
conn: &MDRSConnection,
cache: Option<&Cache>,
limiter: Option<&ApiRequestLimiter>,
remote_path: &str,
password: Option<&str>,
) -> Result<(FolderDetail, Laboratory), anyhow::Error> {
let path_component = remote_path.split_once(':').map(|x| x.1).unwrap_or("");
if is_doi(path_component) {
let (_, doi, subpath) = parse_doi_remote_path(remote_path)?;
let (doi_folder, lab) = find_folder_by_doi(conn, &doi, password).await?;
if subpath.is_empty() {
Ok((doi_folder, lab))
} else {
let abs_path = format!("{}{}", doi_folder.path.trim_end_matches('/'), subpath);
let folder = if let Some(l) = limiter {
find_folder_limited(conn, l, lab.id, &abs_path, password).await?
} else {
find_folder(conn, lab.id, &abs_path, password).await?
};
Ok((folder, lab))
}
} else {
let (_, labname, folder_path) = parse_remote_path(remote_path)?;
let lab = find_laboratory(conn, cache, &labname).await?;
let folder = if let Some(l) = limiter {
find_folder_limited(conn, l, lab.id, &folder_path, password).await?
} else {
find_folder(conn, lab.id, &folder_path, password).await?
};
Ok((folder, lab))
}
}
/// Resolves a remote path pointing to a file into the parent FolderDetail and the file's basename.
pub async fn resolve_remote_file(
conn: &MDRSConnection,
cache: Option<&Cache>,
remote_path: &str,
password: Option<&str>,
) -> Result<(FolderDetail, String), anyhow::Error> {
let path_component = remote_path.split_once(':').map(|x| x.1).unwrap_or("");
if is_doi(path_component) {
let (_, doi, subpath) = parse_doi_remote_path(remote_path)?;
let (doi_folder, lab) = find_folder_by_doi(conn, &doi, password).await?;
let subpath_clean = subpath.trim_end_matches('/');
if subpath_clean.is_empty() {
bail!("DOI path must point to a file, not a folder.");
}
let (sub_dir, basename) = if let Some(pos) = subpath_clean.rfind('/') {
let d = if pos == 0 { "/" } else { &subpath_clean[..pos] };
(d.to_string(), subpath_clean[pos + 1..].to_string())
} else {
("/".to_string(), subpath_clean.to_string())
};
let abs_path = format!(
"{}{}",
doi_folder.path.trim_end_matches('/'),
if sub_dir.starts_with('/') {
sub_dir
} else {
format!("/{}", sub_dir)
}
);
let parent = find_folder(conn, lab.id, &abs_path, password).await?;
Ok((parent, basename))
} else {
let (_, labname, r_path) = parse_remote_path(remote_path)?;
let lab = find_laboratory(conn, cache, &labname).await?;
let path = r_path.trim_end_matches('/');
let (dirname, basename) = if let Some(pos) = path.rfind('/') {
let d = if pos == 0 { "/" } else { &path[..pos] };
(d.to_string(), path[pos + 1..].to_string())
} else {
("/".to_string(), path.to_string())
};
let parent = find_folder(conn, lab.id, &dirname, password).await?;
Ok((parent, basename))
}
}
#[cfg(test)]
mod tests {
use super::*;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
#[tokio::test]
async fn find_laboratory_falls_back_to_api_without_authorization() {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut buf = [0u8; 4096];
let n = stream.read(&mut buf).await.unwrap();
let req = String::from_utf8_lossy(&buf[..n]);
assert!(req.starts_with("GET /v3/laboratories/ HTTP/1.1"));
assert!(!req.contains("\r\nAuthorization: Bearer "));
let body =
r#"[{"id":1,"name":"public-lab","pi_name":"PI","full_name":"Public Laboratory"}]"#;
let response = format!(
"HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{}",
body.len(),
body
);
stream.write_all(response.as_bytes()).await.unwrap();
});
let conn = MDRSConnection::new(&format!("http://{addr}"));
let lab = find_laboratory(&conn, None, "public-lab").await.unwrap();
assert_eq!(lab.id, 1);
assert_eq!(lab.name, "public-lab");
server.await.unwrap();
}
// ------------------------------------------------------------------
// Remote path parsing
// ------------------------------------------------------------------
#[test]
fn parse_remote_path_splits_remote_lab_and_path() {
let (remote, lab, path) = parse_remote_path("neurodata:/mylab/a/b/").unwrap();
assert_eq!(
(remote.as_str(), lab.as_str(), path.as_str()),
("neurodata", "mylab", "/a/b")
);
}
#[test]
fn parse_remote_path_gives_the_lab_root_its_own_form() {
let (_, lab, path) = parse_remote_path("neurodata:/mylab").unwrap();
assert_eq!((lab.as_str(), path.as_str()), ("mylab", "/"));
}
/// Both clients talk to the same server, so a doubled separator or a `.` segment has
/// to reach it as the same path from either one.
#[test]
fn parse_remote_path_collapses_redundant_segments() {
let (_, lab, path) = parse_remote_path("neurodata://mylab//a/./b//").unwrap();
assert_eq!((lab.as_str(), path.as_str()), ("mylab", "/a/b"));
}
#[test]
fn parse_remote_path_refuses_a_parent_segment() {
for path in [
"neurodata:/mylab/../other",
"neurodata:/mylab/a/..",
"neurodata:/../mylab",
] {
let err = parse_remote_path(path).unwrap_err();
assert!(
err.to_string().contains("Path traversal found."),
"{path} should be refused"
);
}
}
#[test]
fn parse_remote_path_refuses_a_path_without_a_laboratory() {
assert!(parse_remote_path("neurodata:/").is_err());
}
// ------------------------------------------------------------------
// DOI helper unit tests
// ------------------------------------------------------------------
#[test]
fn is_doi_returns_true_for_valid_doi() {
assert!(is_doi("10.12345/prefix.20230511-001"));
assert!(is_doi("10.1234/abc"));
}
#[test]
fn is_doi_returns_false_for_normal_paths() {
assert!(!is_doi("/labname/path/to/folder"));
assert!(!is_doi("labname/path"));
assert!(!is_doi("10.1234")); // no slash
}
#[test]
fn doi_suffix_id_extracts_last_dot_segment() {
assert_eq!(
doi_suffix_id("10.12345/prefix.20230511-001"),
"20230511-001"
);
}
#[test]
fn doi_suffix_id_no_dot_in_suffix_returns_whole_suffix() {
assert_eq!(doi_suffix_id("10.1234/nodot"), "nodot");
}
#[test]
fn doi_suffix_id_no_slash_returns_whole_input() {
assert_eq!(doi_suffix_id("10.1234"), "10.1234");
}
#[test]
fn parse_doi_remote_path_valid_no_subpath() {
let (remote, doi, subpath) =
parse_doi_remote_path("neurodata:10.12345/prefix.20230511-001").unwrap();
assert_eq!(remote, "neurodata");
assert_eq!(doi, "10.12345/prefix.20230511-001");
assert_eq!(subpath, "");
}
#[test]
fn parse_doi_remote_path_valid_trailing_slash() {
let (remote, doi, subpath) =
parse_doi_remote_path("neurodata:10.12345/prefix.20230511-001/").unwrap();
assert_eq!(remote, "neurodata");
assert_eq!(doi, "10.12345/prefix.20230511-001");
assert_eq!(subpath, ""); // trailing slash treated as no subpath
}
#[test]
fn parse_doi_remote_path_valid_with_subpath() {
let (remote, doi, subpath) =
parse_doi_remote_path("neurodata:10.12345/prefix.20230511-001/sub/folder").unwrap();
assert_eq!(remote, "neurodata");
assert_eq!(doi, "10.12345/prefix.20230511-001");
assert_eq!(subpath, "/sub/folder");
}
#[test]
fn parse_doi_remote_path_rejects_normal_path() {
let err = parse_doi_remote_path("neurodata:/lab/path").unwrap_err();
assert!(err.to_string().contains("does not look like a DOI"));
}
#[test]
fn split_doi_and_subpath_no_subpath() {
assert_eq!(
split_doi_and_subpath("10.1234/prefix.ID"),
("10.1234/prefix.ID", "")
);
}
#[test]
fn split_doi_and_subpath_trailing_slash_only() {
assert_eq!(
split_doi_and_subpath("10.1234/prefix.ID/"),
("10.1234/prefix.ID", "")
);
}
#[test]
fn split_doi_and_subpath_single_level() {
assert_eq!(
split_doi_and_subpath("10.1234/prefix.ID/sub"),
("10.1234/prefix.ID", "/sub")
);
}
#[test]
fn split_doi_and_subpath_multi_level() {
assert_eq!(
split_doi_and_subpath("10.1234/prefix.ID/sub/deep/path"),
("10.1234/prefix.ID", "/sub/deep/path")
);
}
}
+265 -83
View File
@@ -1,13 +1,14 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::commands::shared::{
find_file_by_name, find_folder, find_lab_in_cache, nfc, parse_remote_path,
find_file_by_name, find_folder_limited, find_laboratory, nfc, parse_remote_path,
};
use crate::connection::{ApiRequestLimiter, MDRSConnection};
use crate::models::folder::FolderSimple;
use anyhow::{anyhow, bail};
use futures::stream::{FuturesUnordered, StreamExt};
use std::path::PathBuf;
use std::sync::Arc;
use tokio::fs;
use tokio::task::JoinSet;
pub async fn upload(
local_path: &str,
@@ -18,8 +19,9 @@ pub async fn upload(
let (remote, labname, r_path) = parse_remote_path(remote_path)?;
let cache = load_cache_with_token_refresh(&remote).await?;
let conn = Arc::new(create_authenticated_conn(&remote, &cache)?);
let lab = find_lab_in_cache(&cache, &labname)?;
let dest_folder = find_folder(&conn, lab.id, &r_path, None).await?;
let limiter = ApiRequestLimiter::new(crate::settings::SETTINGS.concurrent);
let lab = find_laboratory(&conn, Some(&cache), &labname).await?;
let dest_folder = find_folder_limited(&conn, &limiter, lab.id, &r_path, None).await?;
// Normalize local_path: resolve to an absolute canonical path so that
// trailing slashes and "./" prefixes are handled consistently (matching
@@ -30,16 +32,17 @@ pub async fn upload(
if local.is_file() {
let filename = local.file_name().unwrap().to_string_lossy().to_string();
let remote_files = conn.list_all_files(&dest_folder.id).await?;
if skip_if_exists {
if let Some(rf) = find_file_by_name(&remote_files, &filename) {
if rf.size == std::fs::metadata(local)?.len() {
println!("{}{}", dest_folder.path, filename);
return Ok(());
}
}
let remote_files = conn
.list_all_files_limited(&dest_folder.id, &limiter)
.await?;
if skip_if_exists
&& let Some(rf) = find_file_by_name(&remote_files, &filename)
&& rf.size == std::fs::metadata(local)?.len()
{
println!("{}{}", dest_folder.path, filename);
return Ok(());
}
conn.upload_file(&dest_folder.id, &local.to_string_lossy())
conn.upload_file_limited(&dest_folder.id, &local.to_string_lossy(), &limiter)
.await?;
println!("{}{}", dest_folder.path, filename);
} else if local.is_dir() {
@@ -52,75 +55,43 @@ pub async fn upload(
let local_basename = local.file_name().unwrap().to_string_lossy().to_string();
let top_remote_id = find_or_create_folder(
&conn,
&limiter,
&dest_folder.id,
&dest_folder.sub_folders,
&local_basename,
)
.await?;
let top_folder = conn.retrieve_folder(&top_remote_id).await?;
println!("{}", top_folder.path.trim_end_matches('/'));
println!(
"{}",
format!("{}{}", dest_folder.path, local_basename).trim_end_matches('/')
);
// Iterative depth-first walk: each entry is (local_dir, remote_folder_id)
let mut stack: Vec<(PathBuf, String)> = vec![(local.to_path_buf(), top_remote_id)];
let mut folder_tasks: JoinSet<Result<UploadFolderTaskResult, anyhow::Error>> =
JoinSet::new();
let mut upload_tasks: JoinSet<Result<(), anyhow::Error>> = JoinSet::new();
let mut errors = Vec::new();
while let Some((local_dir, remote_id)) = stack.pop() {
let folder_detail = conn.retrieve_folder(&remote_id).await?;
let remote_files = conn.list_all_files(&remote_id).await?;
spawn_upload_folder_task(
&mut folder_tasks,
conn.clone(),
limiter.clone(),
local.to_path_buf(),
top_remote_id,
skip_if_exists,
);
let mut entries = fs::read_dir(&local_dir).await?;
let mut subdirs: Vec<PathBuf> = Vec::new();
let mut files: Vec<PathBuf> = Vec::new();
while let Some(entry) = entries.next_entry().await? {
let p = entry.path();
if p.is_dir() {
subdirs.push(p);
} else {
files.push(p);
}
}
drive_upload_tasks(
&mut folder_tasks,
&mut upload_tasks,
&mut errors,
conn.clone(),
limiter,
skip_if_exists,
)
.await;
// Ensure each local sub-directory exists on the remote side
for subdir in subdirs {
let dirname = subdir.file_name().unwrap().to_string_lossy().to_string();
let sub_remote_id =
find_or_create_folder(&conn, &remote_id, &folder_detail.sub_folders, &dirname)
.await?;
let sub_folder = conn.retrieve_folder(&sub_remote_id).await?;
println!("{}", sub_folder.path.trim_end_matches('/'));
stack.push((subdir, sub_remote_id));
}
// Upload files in this directory (up to 10 concurrent)
let mut futs: FuturesUnordered<tokio::task::JoinHandle<()>> = FuturesUnordered::new();
for file_path in files {
let filename = file_path.file_name().unwrap().to_string_lossy().to_string();
let file_path_str = file_path.to_string_lossy().to_string();
if skip_if_exists {
if let Some(rf) = find_file_by_name(&remote_files, &filename) {
if let Ok(meta) = std::fs::metadata(&file_path) {
if rf.size == meta.len() {
let remote_path_prefix = folder_detail.path.clone();
println!("{}{}", remote_path_prefix, filename);
continue;
}
}
}
}
let conn = conn.clone();
let folder_id = remote_id.clone();
let remote_path_prefix = folder_detail.path.clone();
let fname = filename.clone();
futs.push(tokio::spawn(async move {
match conn.upload_file(&folder_id, &file_path_str).await {
Ok(_) => println!("{}{}", remote_path_prefix, fname),
Err(e) => eprintln!("Error: {}", e),
}
}));
if futs.len() >= crate::settings::SETTINGS.concurrent {
let _ = futs.next().await;
}
}
while futs.next().await.is_some() {}
if !errors.is_empty() {
bail!(errors.join("\n"));
}
} else {
bail!("File or directory `{}` not found.", local_path);
@@ -131,7 +102,8 @@ pub async fn upload(
/// Find an existing sub-folder by name or create it, returning its ID.
async fn find_or_create_folder(
conn: &crate::connection::MDRSConnection,
conn: &MDRSConnection,
limiter: &ApiRequestLimiter,
parent_id: &str,
existing: &[FolderSimple],
name: &str,
@@ -142,13 +114,223 @@ async fn find_or_create_folder(
{
return Ok(sf.id.clone());
}
let resp = conn.create_folder(parent_id, &nfc(name)).await?;
if !resp.status().is_success() {
bail!("Failed to create remote folder: {}", name);
}
let json: serde_json::Value = resp.json().await?;
json["id"]
.as_str()
.ok_or_else(|| anyhow!("No id in create_folder response for {}", name))
.map(|s| s.to_string())
conn.create_folder_id_limited(parent_id, &nfc(name), limiter)
.await
}
struct UploadFolderTaskResult {
child_folders: Vec<(PathBuf, String)>,
upload_jobs: Vec<UploadJob>,
}
struct UploadJob {
folder_id: String,
file_path: String,
remote_path: String,
}
fn spawn_upload_folder_task(
folder_tasks: &mut JoinSet<Result<UploadFolderTaskResult, anyhow::Error>>,
conn: Arc<MDRSConnection>,
limiter: ApiRequestLimiter,
local_dir: PathBuf,
remote_id: String,
skip_if_exists: bool,
) {
folder_tasks.spawn(async move {
process_upload_folder(conn, limiter, local_dir, remote_id, skip_if_exists).await
});
}
fn spawn_upload_task(
upload_tasks: &mut JoinSet<Result<(), anyhow::Error>>,
conn: Arc<MDRSConnection>,
limiter: ApiRequestLimiter,
job: UploadJob,
) {
upload_tasks.spawn(async move {
conn.upload_file_limited(&job.folder_id, &job.file_path, &limiter)
.await?;
println!("{}", job.remote_path);
Ok(())
});
}
async fn process_upload_folder(
conn: Arc<MDRSConnection>,
limiter: ApiRequestLimiter,
local_dir: PathBuf,
remote_id: String,
skip_if_exists: bool,
) -> Result<UploadFolderTaskResult, anyhow::Error> {
let (folder_detail, remote_files) = tokio::try_join!(
conn.retrieve_folder_limited(&remote_id, &limiter),
conn.list_all_files_limited(&remote_id, &limiter),
)?;
let mut entries = fs::read_dir(&local_dir).await?;
let mut subdirs: Vec<PathBuf> = Vec::new();
let mut files: Vec<PathBuf> = Vec::new();
while let Some(entry) = entries.next_entry().await? {
let path = entry.path();
if path.is_dir() {
subdirs.push(path);
} else {
files.push(path);
}
}
let mut subdir_tasks: JoinSet<Result<(PathBuf, String, String), anyhow::Error>> =
JoinSet::new();
let existing_subfolders = Arc::new(folder_detail.sub_folders.clone());
let folder_path_prefix = folder_detail.path.clone();
for subdir in subdirs {
let conn = conn.clone();
let limiter = limiter.clone();
let remote_id = remote_id.clone();
let existing_subfolders = existing_subfolders.clone();
let folder_path_prefix = folder_path_prefix.clone();
subdir_tasks.spawn(async move {
let dirname = subdir.file_name().unwrap().to_string_lossy().to_string();
let sub_remote_id = find_or_create_folder(
&conn,
&limiter,
&remote_id,
existing_subfolders.as_slice(),
&dirname,
)
.await?;
Ok((
subdir,
sub_remote_id,
format!("{}{}", folder_path_prefix, dirname),
))
});
}
let mut child_folders = Vec::new();
while let Some(result) = subdir_tasks.join_next().await {
let (subdir, sub_remote_id, remote_path) = flatten_join_result(result)?;
println!("{}", remote_path.trim_end_matches('/'));
child_folders.push((subdir, sub_remote_id));
}
let mut upload_jobs = Vec::new();
for file_path in files {
let filename = file_path.file_name().unwrap().to_string_lossy().to_string();
if skip_if_exists
&& let Some(rf) = find_file_by_name(&remote_files, &filename)
&& let Ok(meta) = std::fs::metadata(&file_path)
&& rf.size == meta.len()
{
println!("{}{}", folder_detail.path, filename);
continue;
}
upload_jobs.push(UploadJob {
folder_id: remote_id.clone(),
file_path: file_path.to_string_lossy().to_string(),
remote_path: format!("{}{}", folder_detail.path, filename),
});
}
Ok(UploadFolderTaskResult {
child_folders,
upload_jobs,
})
}
async fn drive_upload_tasks(
folder_tasks: &mut JoinSet<Result<UploadFolderTaskResult, anyhow::Error>>,
upload_tasks: &mut JoinSet<Result<(), anyhow::Error>>,
errors: &mut Vec<String>,
conn: Arc<MDRSConnection>,
limiter: ApiRequestLimiter,
skip_if_exists: bool,
) {
loop {
match (folder_tasks.is_empty(), upload_tasks.is_empty()) {
(true, true) => break,
(false, true) => {
if let Some(result) = folder_tasks.join_next().await {
handle_upload_folder_result(
result,
folder_tasks,
upload_tasks,
errors,
conn.clone(),
limiter.clone(),
skip_if_exists,
);
}
}
(true, false) => {
if let Some(result) = upload_tasks.join_next().await
&& let Err(err) = flatten_join_result(result)
{
errors.push(err.to_string());
}
}
(false, false) => {
tokio::select! {
result = folder_tasks.join_next() => {
if let Some(result) = result {
handle_upload_folder_result(
result,
folder_tasks,
upload_tasks,
errors,
conn.clone(),
limiter.clone(),
skip_if_exists,
);
}
}
result = upload_tasks.join_next() => {
if let Some(result) = result
&& let Err(err) = flatten_join_result(result) {
errors.push(err.to_string());
}
}
}
}
}
}
}
fn handle_upload_folder_result(
result: Result<Result<UploadFolderTaskResult, anyhow::Error>, tokio::task::JoinError>,
folder_tasks: &mut JoinSet<Result<UploadFolderTaskResult, anyhow::Error>>,
upload_tasks: &mut JoinSet<Result<(), anyhow::Error>>,
errors: &mut Vec<String>,
conn: Arc<MDRSConnection>,
limiter: ApiRequestLimiter,
skip_if_exists: bool,
) {
match flatten_join_result(result) {
Ok(task_result) => {
for (local_dir, remote_id) in task_result.child_folders {
spawn_upload_folder_task(
folder_tasks,
conn.clone(),
limiter.clone(),
local_dir,
remote_id,
skip_if_exists,
);
}
for job in task_result.upload_jobs {
spawn_upload_task(upload_tasks, conn.clone(), limiter.clone(), job);
}
}
Err(err) => errors.push(err.to_string()),
}
}
fn flatten_join_result<T>(
result: Result<Result<T, anyhow::Error>, tokio::task::JoinError>,
) -> Result<T, anyhow::Error> {
match result {
Ok(inner) => inner,
Err(err) => Err(anyhow!("Task join failed: {}", err)),
}
}
+9 -26
View File
@@ -1,31 +1,14 @@
use serde::Deserialize;
use std::fs;
#[derive(Deserialize)]
struct CacheUser {
username: String,
}
#[derive(Deserialize)]
struct WhoamiCache {
user: Option<CacheUser>,
}
use crate::cache::create_readonly_conn;
/// Ask the server who it thinks is calling.
///
/// The cached name is only what was true at login: an account disabled or removed since
/// then would still answer from the cache, which is the one thing this command exists to
/// tell the user. A session the server no longer honours reads as anonymous.
pub async fn whoami(remote: &str) -> Result<(), anyhow::Error> {
let cache_path = crate::settings::SETTINGS
.config_dirname
.join("cache")
.join(format!("{}.json", remote));
if !cache_path.exists() {
println!("(Anonymous)");
return Ok(());
}
let data = fs::read_to_string(&cache_path)?;
match serde_json::from_str::<WhoamiCache>(&data) {
Ok(cache) => match cache.user {
Some(user) => println!("{}", user.username),
None => println!("(Anonymous)"),
},
let (conn, _cache) = create_readonly_conn(remote).await?;
match conn.get_current_user().await {
Ok(user) => println!("{}", user.username),
Err(_) => println!("(Anonymous)"),
}
Ok(())
+148 -49
View File
@@ -1,6 +1,13 @@
use anyhow::bail;
use reqwest::header::{ACCEPT, AUTHORIZATION, HeaderMap, HeaderValue, USER_AGENT};
use reqwest::{Client, Response};
use serde::Serialize;
use std::sync::Arc;
use tokio::sync::{OwnedSemaphorePermit, Semaphore};
/// Shown when a request was refused twice for a token that was current when it left.
pub const SERVER_BUSY_MESSAGE: &str = "The server took too long to start handling the request and may be overloaded. \
Try again, or reduce the number of parallel transfers.";
fn build_user_agent() -> String {
let info = os_info::get();
@@ -36,6 +43,27 @@ pub struct MDRSConnection {
pub token: Option<String>,
}
#[derive(Clone)]
pub struct ApiRequestLimiter {
semaphore: Arc<Semaphore>,
}
impl ApiRequestLimiter {
pub fn new(limit: usize) -> Self {
Self {
semaphore: Arc::new(Semaphore::new(limit.max(1))),
}
}
pub async fn acquire(&self) -> Result<OwnedSemaphorePermit, anyhow::Error> {
self.semaphore
.clone()
.acquire_owned()
.await
.map_err(|_| anyhow::anyhow!("API request limiter was closed."))
}
}
impl MDRSConnection {
pub fn new(url: &str) -> Self {
MDRSConnection {
@@ -67,6 +95,12 @@ impl MDRSConnection {
}
}
/// Same as `connection_with_fresh_token`, for callers that manage their own retry
/// because the request body cannot simply be sent twice.
pub async fn connection_with_fresh_token_for_retry(&self) -> Result<Self, anyhow::Error> {
self.connection_with_fresh_token().await
}
async fn connection_with_fresh_token(&self) -> Result<Self, anyhow::Error> {
match (&self.remote, &self.token) {
(Some(remote), Some(_)) => {
@@ -99,56 +133,87 @@ impl MDRSConnection {
headers
}
pub async fn get(&self, path: &str) -> Result<Response, anyhow::Error> {
/// Send a request, and send it once more if the server found the access token
/// expired. A request can wait in the server's queue for longer than the token it
/// was sent with lives, so a refusal here need not mean the session is over.
///
/// The retry goes back through `connection_with_fresh_token`, which re-reads the
/// cache under the lock: if another process refreshed while this request waited,
/// its token is reused rather than a second one being minted.
async fn send_with_retry<F>(&self, build: F) -> Result<Response, anyhow::Error>
where
F: Fn(&MDRSConnection) -> reqwest::RequestBuilder,
{
let conn = self.connection_with_fresh_token().await?;
Ok(conn
.client
.get(conn.build_url(path))
.headers(conn.prepare_headers())
.send()
.await?)
let response = build(&conn).send().await?;
if response.status() != reqwest::StatusCode::UNAUTHORIZED {
return Ok(response);
}
// Only a stale access token is worth a second attempt, and the sign of one is a
// different token coming back. A wrong folder password, an anonymous request, or
// a session that has really ended all leave it untouched, and those refusals
// belong to the caller to report.
let retry_conn = self.connection_with_fresh_token().await?;
if retry_conn.token == conn.token {
return Ok(response);
}
let response = build(&retry_conn).send().await?;
if response.status() == reqwest::StatusCode::UNAUTHORIZED {
// Refused twice, the second time with a token that was current when it left.
// The session is fine; the server is not starting requests before their
// credentials lapse, which is worth saying plainly rather than sending the
// user off to log in again.
bail!(SERVER_BUSY_MESSAGE);
}
Ok(response)
}
pub async fn get(&self, path: &str) -> Result<Response, anyhow::Error> {
self.send_with_retry(|conn| {
conn.client
.get(conn.build_url(path))
.headers(conn.prepare_headers())
})
.await
}
pub async fn get_with_query<Q>(&self, path: &str, query: &Q) -> Result<Response, anyhow::Error>
where
Q: Serialize + ?Sized,
{
let conn = self.connection_with_fresh_token().await?;
Ok(conn
.client
.get(conn.build_url(path))
.headers(conn.prepare_headers())
.query(query)
.send()
.await?)
self.send_with_retry(|conn| {
conn.client
.get(conn.build_url(path))
.headers(conn.prepare_headers())
.query(query)
})
.await
}
pub async fn get_url(&self, url: &str) -> Result<Response, anyhow::Error> {
let conn = self.connection_with_fresh_token().await?;
Ok(conn
.client
.get(if url.starts_with("http") {
url.to_string()
} else {
conn.build_url(url)
})
.headers(conn.prepare_headers())
.send()
.await?)
self.send_with_retry(|conn| {
conn.client
.get(if url.starts_with("http") {
url.to_string()
} else {
conn.build_url(url)
})
.headers(conn.prepare_headers())
})
.await
}
pub async fn post_json<B>(&self, path: &str, body: &B) -> Result<Response, anyhow::Error>
where
B: Serialize + ?Sized,
{
let conn = self.connection_with_fresh_token().await?;
Ok(conn
.client
.post(conn.build_url(path))
.headers(conn.prepare_headers())
.json(body)
.send()
.await?)
self.send_with_retry(|conn| {
conn.client
.post(conn.build_url(path))
.headers(conn.prepare_headers())
.json(body)
})
.await
}
pub async fn post_multipart(
@@ -167,13 +232,12 @@ impl MDRSConnection {
}
pub async fn delete(&self, path: &str) -> Result<Response, anyhow::Error> {
let conn = self.connection_with_fresh_token().await?;
Ok(conn
.client
.delete(conn.build_url(path))
.headers(conn.prepare_headers())
.send()
.await?)
self.send_with_retry(|conn| {
conn.client
.delete(conn.build_url(path))
.headers(conn.prepare_headers())
})
.await
}
pub async fn delete_with_query<Q>(
@@ -184,13 +248,48 @@ impl MDRSConnection {
where
Q: Serialize + ?Sized,
{
let conn = self.connection_with_fresh_token().await?;
Ok(conn
.client
.delete(conn.build_url(path))
.headers(conn.prepare_headers())
.query(query)
.send()
.await?)
self.send_with_retry(|conn| {
conn.client
.delete(conn.build_url(path))
.headers(conn.prepare_headers())
.query(query)
})
.await
}
}
#[cfg(test)]
mod retry_tests {
use super::*;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
/// A refusal that no new token could answer belongs to the caller: an anonymous
/// request, a wrong folder password and an ended session all look like this, and
/// sending the same thing twice would only repeat the refusal.
#[tokio::test]
async fn a_refusal_is_handed_back_when_the_token_cannot_change() {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut buf = [0u8; 4096];
let _ = stream.read(&mut buf).await.unwrap();
stream
.write_all(
b"HTTP/1.1 401 Unauthorized\r\ncontent-length: 0\r\nconnection: close\r\n\r\n",
)
.await
.unwrap();
});
// No remote and no token, so nothing can be refreshed and the request must not
// be sent a second time. The stub answers once and would hang on a retry.
let conn = MDRSConnection::new(&format!("http://{addr}"));
let response = conn.get("v3/anything/").await.unwrap();
server.await.unwrap();
assert_eq!(response.status(), reqwest::StatusCode::UNAUTHORIZED);
}
}
+80
View File
@@ -1,3 +1,33 @@
use anyhow::anyhow;
use serde::Deserialize;
/// The error envelope the API answers with (drf-standardized-errors).
#[derive(Deserialize)]
struct ApiErrors {
errors: Vec<ApiError>,
}
#[derive(Deserialize)]
struct ApiError {
detail: String,
}
/// Turn a refused response into an error carrying what the server actually said.
///
/// A status code on its own leaves the user guessing. The reason - a permission, a
/// quota, a name already taken - is in the body the API sends along with it, and that
/// is the part worth showing.
pub async fn response_error(context: &str, resp: reqwest::Response) -> anyhow::Error {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
match serde_json::from_str::<ApiErrors>(&body) {
Ok(parsed) if !parsed.errors.is_empty() => {
anyhow!("{}: {}", context, parsed.errors[0].detail)
}
_ => anyhow!("{}: {}", context, status),
}
}
/// Print the error message and exit with code 2.
/// JSON deserialization errors produce a friendlier message matching Python's
/// JSONDecodeError handling.
@@ -23,3 +53,53 @@ fn is_json_error(e: &(dyn std::error::Error + 'static)) -> bool {
}
false
}
#[cfg(test)]
mod response_error_tests {
use super::*;
async fn refused_with(body: &'static str, content_type: &'static str) -> String {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut buf = [0u8; 4096];
let _ = stream.read(&mut buf).await.unwrap();
let response = format!(
"HTTP/1.1 403 Forbidden\r\ncontent-type: {}\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{}",
content_type,
body.len(),
body
);
stream.write_all(response.as_bytes()).await.unwrap();
});
let resp = reqwest::get(format!("http://{addr}/")).await.unwrap();
let message = response_error("Upload failed", resp).await.to_string();
server.await.unwrap();
message
}
/// The reason the server gave is what the user needs, not the number that carried it.
#[tokio::test]
async fn the_servers_own_message_is_reported() {
let message = refused_with(
r#"{"type":"client_error","errors":[{"code":"permission_denied","detail":"You do not have permission.","attr":null}]}"#,
"application/json",
)
.await;
assert_eq!(message, "Upload failed: You do not have permission.");
}
/// A gateway that answers in HTML has nothing to quote, so the status still stands in.
#[tokio::test]
async fn the_status_stands_in_when_the_body_says_nothing() {
let message = refused_with("<html>nope</html>", "text/html").await;
assert_eq!(message, "Upload failed: 403 Forbidden");
}
}
+1 -1
View File
@@ -201,7 +201,7 @@ fn run(cli: Cli) {
}
Commands::SelfUpdate { yes } => {
if let Err(e) = build_rt().block_on(commands::selfupdate::selfupdate(yes)) {
handle_error(e.into());
handle_error(e);
}
}
}
+18
View File
@@ -0,0 +1,18 @@
use serde::{Deserialize, Serialize};
/// Nested folder information returned inside a DOI response.
/// The DOI endpoint only returns the folder `id`; `laboratory_id` must be
/// obtained by subsequently calling the folder retrieve endpoint.
#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct DoiFolderRef {
pub id: String,
}
/// Response from GET v3/doi/{id}/
#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct DoiResponse {
/// The internal DOI suffix ID (e.g. "20260429-001") returned as a string.
pub id: String,
pub doi: String,
pub folder: DoiFolderRef,
}
+1
View File
@@ -1,5 +1,6 @@
// Model definitions (User, Laboratory, File, Folder, etc.)
pub mod doi;
pub mod file;
pub mod folder;
pub mod laboratory;
+6 -9
View File
@@ -4,7 +4,7 @@ pub struct Settings {
/// Base directory for config and cache files.
/// Controlled by `MDRS_CLIENT_CONFIG_DIRNAME` env var (default: `~/.mdrs-client`).
pub config_dirname: std::path::PathBuf,
/// Maximum number of concurrent upload/download workers.
/// Maximum number of concurrent MDRS API requests used by upload/download.
/// Controlled by `MDRS_CLIENT_CONCURRENT` env var (default: 10).
pub concurrent: usize,
}
@@ -13,14 +13,11 @@ impl Settings {
fn load() -> Self {
let config_dirname = std::env::var("MDRS_CLIENT_CONFIG_DIRNAME")
.ok()
.map(|s| {
if s.starts_with("~/") {
dirs::home_dir()
.unwrap_or_else(|| std::path::PathBuf::from("."))
.join(&s[2..])
} else {
std::path::PathBuf::from(&s)
}
.map(|s| match s.strip_prefix("~/") {
Some(rest) => dirs::home_dir()
.unwrap_or_else(|| std::path::PathBuf::from("."))
.join(rest),
None => std::path::PathBuf::from(&s),
})
.unwrap_or_else(|| {
dirs::home_dir()