6 Commits
Author SHA1 Message Date
Yoshihiro OKUMURA 663956ead7 docs: add a changelog
Release / build-linux-x86_64 (push) Successful in 2m20s
Release / build-linux-aarch64 (push) Successful in 1m57s
The project kept no change history of its own, so the tags were the
only record of what each release contained.

- Summarise 2.0.0, 2.0.1 and 2.0.2 by what changed for the user.
- Link the changelog from the README.
2026-09-04 18:23:33 +09:00
Yoshihiro OKUMURA cb5cff7b64 build: bump to 2.0.2 and update dependencies
- Raise tokio, base64, serde_json and anyhow to the releases they are
  tested against, and refresh the lockfile.
- base64 0.23 needed no change to the code that uses it.
2026-09-04 18:23:33 +09:00
Yoshihiro OKUMURA a16f73543d fix: stream transfers and close gaps against the Python client
A review against the Python client, which shares this client's config
and login cache, found the transfers holding whole files in memory and
several commands answering differently from their Python counterparts.

- Stream uploads and downloads instead of buffering the whole file, so
  memory no longer scales with file size times concurrency. Uploads
  still declare a Content-Length rather than going out chunked.
- Write a download beside its destination and move it into place once
  complete, and refuse a destination that cannot be written, so a
  failed transfer leaves what was already there untouched.
- Report the server's own error detail instead of the bare status.
- List locked sub-folders in `ls --json --recursive` using the given
  password and skip only those that cannot be unlocked, instead of
  failing the whole listing.
- Name each folder's laboratory in JSON output and sort its entries,
  matching the Python client's schema and order.
- Ask the server who is logged in for `whoami`, rather than trusting
  the cached name.
- Collapse repeated separators and refuse `..` in remote paths.
- Fall back to the API when a laboratory is missing from the cache, so
  a newly added one no longer needs a fresh login.
- Verify the login cache digest on read, as the Python client does.
- Accept `-e` as the short form of `--exclude`.
- Satisfy clippy and rustfmt across the crate.
2026-09-04 16:31:14 +09:00
Yoshihiro OKUMURA c24a285cf5 fix(selfupdate): verify the archive before replacing the binary
`mdrs selfupdate` replaced the running binary with whatever the release
endpoint returned, checking only that the transport succeeded. Nothing
proved the archive was the one the release publishes.

- Compare the downloaded archive against the release's `.sha256` asset
  and abort the update on a mismatch.
- Report a release that publishes no checksum as unverified, rather
  than letting its absence pass for a verified download.
- Exclude `.sha256` assets when matching the archive for the build
  target: those assets carry the target name too.
- Write and upload a checksum beside every archive, from the Gitea
  release workflow and the three local build scripts.
2026-09-04 16:31:02 +09:00
Yoshihiro OKUMURA e3026bdfcf feat(auth): send a request again when its token lapsed in the queue
A request can wait to be served for longer than the access token it was
sent with lives, and comes back refused for a token that was valid when
it left. Uploads that take minutes make that wait ordinary.

- send once more when a refusal is answered by a different token, which
  is the only refusal a second attempt can help; a wrong folder
  password, an anonymous request and an ended session all leave the
  token untouched and stay with the caller who knows what they mean
- rebuild the upload and read the file again for its retry, but only
  once the token has moved on: any other refusal would cost a second
  full transfer to be told the same
- say the server may be overloaded when the second attempt is refused
  too, rather than sending the user off to log in over a session that
  is fine
2026-08-14 18:37:40 +09:00
Yoshihiro OKUMURA bcf99dd6d7 fix(auth): keep the refresh token the provider hands back
The reply to a refresh carries a new refresh token and the provider
stops honouring the one that was sent. Only the access half was read,
so the cache kept re-sending a token the server had already retired.

- deserialize the refresh half and write it back to the cache, as an
  Option so a provider that does not rotate leaves the stored one be
- bound the refresh request on its own: the caller holds a lock that
  spans processes while it runs, so a provider that goes quiet would
  stall every other request on the machine
- give config create/update one rule for what a remote URL is, and
  store it without the trailing slash, matching the Python client so
  the two can share config.ini; this drops the validators crate and
  77 transitive dependencies with it
- join the base URL and the API's relative download path with the
  separator neither of them carries, as download.rs already does
2026-08-14 16:36:43 +09:00
33 changed files with 1618 additions and 1674 deletions
+10 -2
View File
@@ -26,12 +26,16 @@ jobs:
TARGET=x86_64-unknown-linux-musl
ARCHIVE="mdrs-${VERSION}-${TARGET}.tar.gz"
tar -czf "${ARCHIVE}" -C target/${TARGET}/release mdrs
# Published alongside the archive so `mdrs selfupdate` can check what it fetched.
sha256sum "${ARCHIVE}" > "${ARCHIVE}.sha256"
echo "ARCHIVE=${ARCHIVE}" >> "$GITHUB_ENV"
- name: Create release and upload asset
uses: akkuman/gitea-release-action@v1
with:
token: ${{ github.token }}
files: ${{ env.ARCHIVE }}
files: |
${{ env.ARCHIVE }}
${{ env.ARCHIVE }}.sha256
build-linux-aarch64:
runs-on: ubuntu-latest
@@ -55,9 +59,13 @@ jobs:
TARGET=aarch64-unknown-linux-musl
ARCHIVE="mdrs-${VERSION}-${TARGET}.tar.gz"
tar -czf "${ARCHIVE}" -C target/${TARGET}/release mdrs
# Published alongside the archive so `mdrs selfupdate` can check what it fetched.
sha256sum "${ARCHIVE}" > "${ARCHIVE}.sha256"
echo "ARCHIVE=${ARCHIVE}" >> "$GITHUB_ENV"
- name: Create release and upload asset
uses: akkuman/gitea-release-action@v1
with:
token: ${{ github.token }}
files: ${{ env.ARCHIVE }}
files: |
${{ env.ARCHIVE }}
${{ env.ARCHIVE }}.sha256
+43
View File
@@ -0,0 +1,43 @@
# Changelog
All notable changes to this project will be documented in this file.
## [2.0.2] - 2026-09-04
### Added
- Accepted `-e` as the short form of `download --exclude`.
### Changed
- Transferred files larger than the memory available, in both directions.
- Left an existing local file intact when a download fails part way through.
- Showed the reason the server gave when it refused a request.
- Updated dependencies.
### Fixed
- Kept a session working across concurrent runs, long server queues, and providers that rotate refresh tokens.
- Listed a tree containing locked folders with `ls --json --recursive` instead of failing on the first one.
- Other minor fixes to `ls --json` output, remote path handling, and the login cache.
### Security
- Verified the downloaded archive against the release checksum before `selfupdate` replaces the binary.
## [2.0.1] - 2026-06-12
### Added
- Accepted a DOI in place of a path, as `remote:10.xxxx/prefix.ID[/sub/path]`.
- Allowed `download` and the read-only commands to run without signing in.
## [2.0.0] - 2026-04-20
First release of the command-line client.
### Added
- Commands `config`, `login`, `logout`, `whoami`, `labs`, `ls`, `mkdir`, `upload`, `download`, `mv`, `cp`, `rm`, `chacl`, `metadata`, `file-metadata`, `version`, and `selfupdate`.
- Release build scripts for Linux, macOS, and Windows.
### Changed
- Sped up transfers by working on folders and files in parallel, bounded by `MDRS_CLIENT_CONCURRENT`.
### Fixed
- Kept the session valid throughout a long-running transfer, and across concurrent runs.
- Handled file and folder names with combining characters consistently.
Generated
+272 -1285
View File
File diff suppressed because it is too large Load Diff
+7 -7
View File
@@ -1,6 +1,6 @@
[package]
name = "mdrs-client-rust"
version = "2.0.1"
version = "2.0.2"
edition = "2024"
license = "MIT"
authors = ["Neuroinformatics Unit, RIKEN CBS"]
@@ -12,18 +12,18 @@ path = "src/main.rs"
[dependencies]
clap = { version = "4.5", features = ["derive"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "multipart", "rustls-tls"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "multipart", "rustls-tls", "stream"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0.150"
tokio = { version = "1.52.3", features = ["full"] }
serde_json = "1.0.151"
tokio = { version = "1.53.1", features = ["full"] }
tokio-util = { version = "0.7", features = ["io"] }
futures = "0.3"
dirs = "6.0.0"
anyhow = "1.0.102"
anyhow = "1.0.104"
configparser = "3.2.0"
validators = "0.25.3"
sha2 = "0.11.0"
rpassword = "7.5.4"
base64 = "0.22"
base64 = "0.23.1"
fs2 = "0.4"
ctrlc = "3"
os_info = "3.15.0"
+7 -2
View File
@@ -143,7 +143,7 @@ Download a file or folder from a remote path. You can also specify a DOI path.
mdrs download neurodata:/NIU/Repository/TEST/sample.dat ./
mdrs download -r neurodata:/NIU/Repository/TEST/dataset/ ./
mdrs download -p SHARING_PASSWORD neurodata:/NIU/Repository/PW_Open/Readme.dat ./
mdrs download -r --exclude /NIU/Repository/TEST/dataset/skip neurodata:/NIU/Repository/TEST/dataset/ ./
mdrs download -r -e /NIU/Repository/TEST/dataset/skip neurodata:/NIU/Repository/TEST/dataset/ ./
mdrs download -r --skip-if-exists neurodata:/NIU/Repository/TEST/dataset/ ./
# DOI access examples:
@@ -224,7 +224,8 @@ mdrs version
### selfupdate
Update the current `mdrs` binary to the latest published release for
the same build target.
the same build target. The downloaded archive is checked against the `.sha256`
asset the release publishes; a release without one is reported as unverified.
```shell
mdrs selfupdate
@@ -240,6 +241,10 @@ mdrs --help
mdrs upload --help
```
## Changelog
See [CHANGELOG.md](./CHANGELOG.md) for the full change history.
## License
[MIT](LICENSE) © 2026- Neuroinformatics Unit, RIKEN CBS
+4 -1
View File
@@ -58,7 +58,10 @@ for TARGET in "${TARGETS[@]}"; do
ARCHIVE="mdrs-${VERSION}-${TARGET}.tar.gz"
tar -czf "${ARCHIVE}" -C "target/${TARGET}/release" mdrs
ARCHIVES+=("${ARCHIVE}")
# Uploaded alongside the archive so `mdrs selfupdate` can check what it fetched.
sha256sum "${ARCHIVE}" > "${ARCHIVE}.sha256" 2>/dev/null \
|| shasum -a 256 "${ARCHIVE}" > "${ARCHIVE}.sha256"
ARCHIVES+=("${ARCHIVE}" "${ARCHIVE}.sha256")
echo " Created: ${ARCHIVE}"
done
+4 -1
View File
@@ -46,7 +46,10 @@ for TARGET in "${TARGETS[@]}"; do
ARCHIVE="mdrs-${VERSION}-${TARGET}.tar.gz"
tar -czf "${ARCHIVE}" -C "target/${TARGET}/release" mdrs
ARCHIVES+=("${ARCHIVE}")
# Uploaded alongside the archive so `mdrs selfupdate` can check what it fetched.
sha256sum "${ARCHIVE}" > "${ARCHIVE}.sha256" 2>/dev/null \
|| shasum -a 256 "${ARCHIVE}" > "${ARCHIVE}.sha256"
ARCHIVES+=("${ARCHIVE}" "${ARCHIVE}.sha256")
echo " Created: ${ARCHIVE}"
done
+14 -8
View File
@@ -55,6 +55,10 @@ cargo build --release --target $Target
$Archive = "mdrs-$Version-$Target.zip"
Compress-Archive -Force -Path "target\$Target\release\mdrs.exe" -DestinationPath $Archive
# Written alongside the archive so `mdrs selfupdate` can check what it fetched.
$Checksum = "$Archive.sha256"
$Hash = (Get-FileHash -Algorithm SHA256 -Path $Archive).Hash.ToLower()
"$Hash $Archive" | Set-Content -NoNewline -Encoding ascii $Checksum
Write-Host " Created: $Archive"
# Upload to Gitea if token is provided
@@ -87,14 +91,16 @@ $Release = Invoke-RestMethod -Method Get -Uri "$ServerUrl/api/v1/repos/$Reposito
-Headers @{ Authorization = "Bearer $GiteaToken" }
$ReleaseId = $Release.id
Write-Host "==> Uploading $Archive (release id: $ReleaseId) ..."
$ArchivePath = Join-Path $RepoRoot $Archive
& curl.exe -sf -X POST `
-H "Authorization: Bearer $GiteaToken" `
-F "attachment=@$ArchivePath" `
"$ServerUrl/api/v1/repos/$Repository/releases/$ReleaseId/assets" | Out-Null
if ($LASTEXITCODE -ne 0) { Write-Error "Upload failed (exit code $LASTEXITCODE)"; exit 1 }
Write-Host " Done."
foreach ($Name in @($Archive, $Checksum)) {
Write-Host "==> Uploading $Name (release id: $ReleaseId) ..."
$AssetPath = Join-Path $RepoRoot $Name
& curl.exe -sf -X POST `
-H "Authorization: Bearer $GiteaToken" `
-F "attachment=@$AssetPath" `
"$ServerUrl/api/v1/repos/$Repository/releases/$ReleaseId/assets" | Out-Null
if ($LASTEXITCODE -ne 0) { Write-Error "Upload failed (exit code $LASTEXITCODE)"; exit 1 }
Write-Host " Done."
}
Write-Host ""
Write-Host "==> Upload complete: $ServerUrl/$Repository/releases/tag/$Tag"
+2 -2
View File
@@ -1,6 +1,6 @@
use crate::connection::MDRSConnection;
use crate::error::response_error;
use crate::models::doi::DoiResponse;
use anyhow::bail;
impl MDRSConnection {
/// Retrieve the folder associated with a DOI suffix ID (GET v3/doi/{id}/).
@@ -10,7 +10,7 @@ impl MDRSConnection {
pub async fn retrieve_doi(&self, id: &str) -> Result<DoiResponse, anyhow::Error> {
let resp = self.get(&format!("v3/doi/{}/", id)).await?;
if !resp.status().is_success() {
bail!("DOI lookup failed: {}", resp.status());
return Err(response_error("DOI lookup failed", resp).await);
}
Ok(resp.json::<DoiResponse>().await?)
}
+369 -15
View File
@@ -1,8 +1,30 @@
use crate::connection::{ApiRequestLimiter, MDRSConnection};
use crate::error::response_error;
pub use crate::models::file::File;
use anyhow::bail;
use unicode_normalization::UnicodeNormalization;
/// Read and write size for a streamed transfer. Large enough that the syscalls are not
/// what limits the transfer, small enough that ten of them in flight cost little.
const TRANSFER_CHUNK_BYTES: usize = 64 * 1024;
/// Name a scratch file beside the destination for a download in progress.
///
/// Beside it, so the move into place is a rename within one directory and cannot fail
/// half way. Unique, so two transfers heading for the same name cannot clear up after
/// each other.
fn partial_path(dest: &std::path::Path) -> std::path::PathBuf {
use std::sync::atomic::{AtomicU64, Ordering};
static COUNTER: AtomicU64 = AtomicU64::new(0);
let mut name = dest.file_name().unwrap_or_default().to_os_string();
name.push(format!(
".{}-{}.mdrspart",
std::process::id(),
COUNTER.fetch_add(1, Ordering::Relaxed)
));
dest.with_file_name(name)
}
#[derive(serde::Deserialize)]
struct FileListResponse {
pub next: Option<String>,
@@ -21,7 +43,7 @@ impl MDRSConnection {
];
let resp = self.get_with_query("v3/files/", &params).await?;
if !resp.status().is_success() {
anyhow::bail!("List files failed: {}", resp.status());
return Err(response_error("List files failed", resp).await);
}
let list: FileListResponse = resp.json().await?;
let has_next = list.next.is_some();
@@ -50,7 +72,7 @@ impl MDRSConnection {
let _permit = limiter.acquire().await?;
let resp = self.get_with_query("v3/files/", &params).await?;
if !resp.status().is_success() {
anyhow::bail!("List files failed: {}", resp.status());
return Err(response_error("List files failed", resp).await);
}
let list: FileListResponse = resp.json().await?;
let has_next = list.next.is_some();
@@ -79,15 +101,51 @@ impl MDRSConnection {
.to_string_lossy()
.nfc()
.collect();
let file_bytes = tokio::fs::read(file_path).await?;
let part = multipart::Part::bytes(file_bytes).file_name(file_name.clone());
let form = multipart::Form::new()
.text("folder_id", folder_id.to_string())
.part("file", part);
// The file is handed to the request as a stream rather than a buffer: a
// repository holds recordings far larger than the machine's memory, and ten of
// them may be in flight at once. The length travels with it so the request keeps
// a Content-Length and does not have to be sent chunked.
let build_form = || async {
let file = tokio::fs::File::open(file_path).await?;
let length = file.metadata().await?.len();
let stream = tokio_util::io::ReaderStream::with_capacity(file, TRANSFER_CHUNK_BYTES);
let part =
multipart::Part::stream_with_length(reqwest::Body::wrap_stream(stream), length)
.file_name(file_name.clone());
Ok::<_, anyhow::Error>(
multipart::Form::new()
.text("folder_id", folder_id.to_string())
.part("file", part),
)
};
let _permit = limiter.acquire().await?;
let resp = self.post_multipart("v3/files/", form).await?;
let sent_with = self.token.clone();
let resp = self
.post_multipart("v3/files/", build_form().await?)
.await?;
if resp.status() != reqwest::StatusCode::UNAUTHORIZED {
if !resp.status().is_success() {
return Err(response_error("Upload failed", resp).await);
}
return Ok(());
}
// The body reached the server, but it may have waited to be served for longer
// than the access token it was sent with lived. Re-reading the file and sending
// it again is only worth it if the token has actually moved on: any other reason
// for the refusal would just cost a second full transfer to be told the same.
let retry_conn = self.connection_with_fresh_token_for_retry().await?;
if retry_conn.token == sent_with {
return Err(response_error("Upload failed", resp).await);
}
let resp = retry_conn
.post_multipart("v3/files/", build_form().await?)
.await?;
if resp.status() == reqwest::StatusCode::UNAUTHORIZED {
// Refused twice, the second time with a token that was current when it left.
bail!(crate::connection::SERVER_BUSY_MESSAGE);
}
if !resp.status().is_success() {
bail!("Upload failed: {}", resp.status());
return Err(response_error("Upload failed", resp).await);
}
Ok(())
}
@@ -99,14 +157,310 @@ impl MDRSConnection {
dest: &str,
limiter: &ApiRequestLimiter,
) -> Result<(), anyhow::Error> {
use futures::StreamExt;
use tokio::io::AsyncWriteExt;
let _permit = limiter.acquire().await?;
let dest_path = std::path::Path::new(dest);
// Checked before anything is fetched. The finished file is moved into place, and
// a rename would replace a destination the mode says is protected, which is not
// what this client or the Python one has ever done.
if dest_path.exists() {
tokio::fs::OpenOptions::new()
.write(true)
.open(dest_path)
.await
.map_err(|e| anyhow::anyhow!("Cannot write `{}`: {}", dest, e))?;
}
let resp = self.get_url(url).await?;
if !resp.status().is_success() {
bail!("Download failed: {}", resp.status());
return Err(response_error("Download failed", resp).await);
}
// Written as it arrives, into a scratch file that is moved into place only once
// the whole body has landed. Holding the body in memory first would cost the size
// of the file, times however many transfers are running; writing straight to the
// destination would destroy whatever is already there the moment the transfer
// starts, and leave a truncated file behind if it does not finish.
let part_path = partial_path(dest_path);
let result = async {
let mut file = tokio::io::BufWriter::with_capacity(
TRANSFER_CHUNK_BYTES,
tokio::fs::File::create(&part_path).await?,
);
let mut stream = resp.bytes_stream();
while let Some(chunk) = stream.next().await {
file.write_all(&chunk?).await?;
}
file.flush().await?;
Ok::<(), anyhow::Error>(())
}
.await;
match result {
Ok(()) => Ok(tokio::fs::rename(&part_path, dest_path).await?),
Err(err) => {
// Only the scratch file goes. Anything already at the destination was not
// written by this transfer and is not this transfer's to remove.
let _ = tokio::fs::remove_file(&part_path).await;
Err(err)
}
}
let bytes = resp.bytes().await?;
drop(_permit);
tokio::fs::write(dest, &bytes).await?;
Ok(())
}
}
#[cfg(test)]
mod transfer_tests {
use super::*;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
/// Read one HTTP request in full, using its Content-Length to know where it ends.
async fn read_request(stream: &mut tokio::net::TcpStream) -> (String, Vec<u8>) {
let mut buf = Vec::new();
let mut chunk = [0u8; 8192];
let header_end = loop {
let n = stream.read(&mut chunk).await.unwrap();
buf.extend_from_slice(&chunk[..n]);
if let Some(pos) = buf.windows(4).position(|w| w == b"\r\n\r\n") {
break pos + 4;
}
assert!(n > 0, "connection closed before the headers ended");
};
let head = String::from_utf8_lossy(&buf[..header_end]).to_string();
let length: usize = head
.lines()
.find_map(|l| {
l.to_ascii_lowercase()
.strip_prefix("content-length:")
.map(|v| v.trim().to_string())
})
.expect("request must carry a Content-Length")
.parse()
.unwrap();
while buf.len() < header_end + length {
let n = stream.read(&mut chunk).await.unwrap();
assert!(n > 0, "connection closed before the body ended");
buf.extend_from_slice(&chunk[..n]);
}
(head, buf[header_end..].to_vec())
}
/// The file goes out as a stream, and the request still declares its length, so the
/// server is not asked to accept a chunked upload.
#[tokio::test]
async fn an_upload_streams_the_file_with_a_declared_length() {
let payload: Vec<u8> = (0..300_000u32).map(|i| (i % 251) as u8).collect();
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("recording.dat");
std::fs::write(&path, &payload).unwrap();
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let expected = payload.clone();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let (head, body) = read_request(&mut stream).await;
stream
.write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 15\r\nconnection: close\r\n\r\n{\"id\":\"new\"}")
.await
.unwrap();
(head, body, expected)
});
let conn = MDRSConnection::new(&format!("http://{addr}"));
let limiter = ApiRequestLimiter::new(1);
conn.upload_file_limited("folder-1", path.to_str().unwrap(), &limiter)
.await
.unwrap();
let (head, body, expected) = server.await.unwrap();
assert!(
!head
.to_ascii_lowercase()
.contains("transfer-encoding: chunked"),
"the upload must not be chunked:\n{head}"
);
assert!(
body.windows(expected.len()).any(|w| w == expected),
"the file bytes must reach the server"
);
assert!(
body.windows(8).any(|w| w == b"folder-1"),
"the folder id must reach the server"
);
assert!(
String::from_utf8_lossy(&body).contains("recording.dat"),
"the file name must reach the server"
);
}
/// The body is written to disk as it arrives rather than being held whole.
#[tokio::test]
async fn a_download_writes_what_the_server_sent() {
let payload: Vec<u8> = (0..300_000u32).map(|i| (i % 241) as u8).collect();
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let served = payload.clone();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut chunk = [0u8; 4096];
let _ = stream.read(&mut chunk).await.unwrap();
let head = format!(
"HTTP/1.1 200 OK\r\ncontent-length: {}\r\nconnection: close\r\n\r\n",
served.len()
);
stream.write_all(head.as_bytes()).await.unwrap();
stream.write_all(&served).await.unwrap();
});
let dir = tempfile::tempdir().unwrap();
let dest = dir.path().join("out.dat");
let conn = MDRSConnection::new(&format!("http://{addr}"));
let limiter = ApiRequestLimiter::new(1);
conn.download_file_limited(
&format!("http://{addr}/v3/files/1/download/"),
dest.to_str().unwrap(),
&limiter,
)
.await
.unwrap();
server.await.unwrap();
assert_eq!(std::fs::read(&dest).unwrap(), payload);
}
/// A transfer that dies part way must not take the copy already on disk with it.
#[tokio::test]
async fn an_interrupted_download_leaves_the_existing_file_intact() {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut chunk = [0u8; 4096];
let _ = stream.read(&mut chunk).await.unwrap();
// Promises 300000 bytes, sends 1000, then hangs up.
stream
.write_all(
b"HTTP/1.1 200 OK\r\ncontent-length: 300000\r\nconnection: close\r\n\r\n",
)
.await
.unwrap();
stream.write_all(&vec![b'x'; 1000]).await.unwrap();
stream.shutdown().await.unwrap();
});
let dir = tempfile::tempdir().unwrap();
let dest = dir.path().join("existing.dat");
std::fs::write(&dest, b"the copy already here").unwrap();
let conn = MDRSConnection::new(&format!("http://{addr}"));
let limiter = ApiRequestLimiter::new(1);
let outcome = conn
.download_file_limited(
&format!("http://{addr}/v3/files/1/download/"),
dest.to_str().unwrap(),
&limiter,
)
.await;
server.await.unwrap();
assert!(outcome.is_err(), "a truncated body is a failed download");
assert_eq!(std::fs::read(&dest).unwrap(), b"the copy already here");
let leftovers: Vec<_> = std::fs::read_dir(dir.path())
.unwrap()
.map(|e| e.unwrap().file_name().to_string_lossy().to_string())
.filter(|name| name != "existing.dat")
.collect();
assert!(
leftovers.is_empty(),
"no scratch file may be left: {leftovers:?}"
);
}
/// A destination this call could not even open is not a file it may delete.
#[cfg(unix)]
#[tokio::test]
async fn a_download_that_cannot_be_written_leaves_the_existing_file_intact() {
use std::os::unix::fs::PermissionsExt;
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut chunk = [0u8; 4096];
let _ = stream.read(&mut chunk).await.unwrap();
stream
.write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 2\r\nconnection: close\r\n\r\nhi")
.await
.unwrap();
});
// A writable directory holding a file this process may not open for writing.
// That is what separates "could not write it" from "may delete it": unlink needs
// the directory, not the file.
let dir = tempfile::tempdir().unwrap();
let dest = dir.path().join("precious.dat");
std::fs::write(&dest, b"do not touch").unwrap();
let mut perms = std::fs::metadata(&dest).unwrap().permissions();
perms.set_mode(0o444);
std::fs::set_permissions(&dest, perms).unwrap();
if std::fs::OpenOptions::new().write(true).open(&dest).is_ok() {
// Running with rights that ignore the mode, so there is nothing to observe.
return;
}
let conn = MDRSConnection::new(&format!("http://{addr}"));
let limiter = ApiRequestLimiter::new(1);
let outcome = conn
.download_file_limited(
&format!("http://{addr}/v3/files/1/download/"),
dest.to_str().unwrap(),
&limiter,
)
.await;
// Nothing was fetched, so the stub is still waiting to be connected to.
server.abort();
let message = outcome.unwrap_err().to_string();
assert!(
message.starts_with("Cannot write "),
"the refusal must name the problem, got: {message}"
);
assert_eq!(std::fs::read(&dest).unwrap(), b"do not touch");
}
/// A refused download leaves nothing behind that a later run could mistake for the file.
#[tokio::test]
async fn a_refused_download_leaves_no_file() {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut chunk = [0u8; 4096];
let _ = stream.read(&mut chunk).await.unwrap();
let body = r#"{"type":"client_error","errors":[{"code":"permission_denied","detail":"Access is denied.","attr":null}]}"#;
let head = format!(
"HTTP/1.1 403 Forbidden\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{}",
body.len(),
body
);
stream.write_all(head.as_bytes()).await.unwrap();
});
let dir = tempfile::tempdir().unwrap();
let dest = dir.path().join("out.dat");
let conn = MDRSConnection::new(&format!("http://{addr}"));
let limiter = ApiRequestLimiter::new(1);
let err = conn
.download_file_limited(
&format!("http://{addr}/v3/files/1/download/"),
dest.to_str().unwrap(),
&limiter,
)
.await
.unwrap_err();
server.await.unwrap();
assert_eq!(err.to_string(), "Download failed: Access is denied.");
assert!(!dest.exists());
}
}
+12 -7
View File
@@ -1,4 +1,5 @@
use crate::connection::{ApiRequestLimiter, MDRSConnection};
use crate::error::response_error;
pub use crate::models::folder::{FolderDetail, FolderSimple};
use anyhow::{anyhow, bail};
@@ -15,7 +16,7 @@ impl MDRSConnection {
];
let resp = self.get_with_query("v3/folders/", &params).await?;
if !resp.status().is_success() {
bail!("List folders failed: {}", resp.status());
return Err(response_error("List folders failed", resp).await);
}
Ok(resp.json::<Vec<FolderSimple>>().await?)
}
@@ -34,7 +35,7 @@ impl MDRSConnection {
let _permit = limiter.acquire().await?;
let resp = self.get_with_query("v3/folders/", &params).await?;
if !resp.status().is_success() {
bail!("List folders failed: {}", resp.status());
return Err(response_error("List folders failed", resp).await);
}
Ok(resp.json::<Vec<FolderSimple>>().await?)
}
@@ -43,7 +44,7 @@ impl MDRSConnection {
pub async fn retrieve_folder(&self, id: &str) -> Result<FolderDetail, anyhow::Error> {
let resp = self.get(&format!("v3/folders/{}/", id)).await?;
if !resp.status().is_success() {
bail!("Retrieve folder failed: {}", resp.status());
return Err(response_error("Retrieve folder failed", resp).await);
}
Ok(resp.json::<FolderDetail>().await?)
}
@@ -57,7 +58,7 @@ impl MDRSConnection {
let _permit = limiter.acquire().await?;
let resp = self.get(&format!("v3/folders/{}/", id)).await?;
if !resp.status().is_success() {
bail!("Retrieve folder failed: {}", resp.status());
return Err(response_error("Retrieve folder failed", resp).await);
}
Ok(resp.json::<FolderDetail>().await?)
}
@@ -94,7 +95,11 @@ impl MDRSConnection {
let _permit = limiter.acquire().await?;
let resp = self.post_json("v3/folders/", &body).await?;
if !resp.status().is_success() {
bail!("Failed to create remote folder: {}", folder_name);
return Err(response_error(
&format!("Failed to create remote folder `{}`", folder_name),
resp,
)
.await);
}
let json: serde_json::Value = resp.json().await?;
json["id"]
@@ -116,7 +121,7 @@ impl MDRSConnection {
bail!("Password is incorrect.");
}
if !resp.status().is_success() {
bail!("Folder auth failed: {}", resp.status());
return Err(response_error("Folder auth failed", resp).await);
}
Ok(())
}
@@ -140,7 +145,7 @@ impl MDRSConnection {
bail!("Password is incorrect.");
}
if !resp.status().is_success() {
bail!("Folder auth failed: {}", resp.status());
return Err(response_error("Folder auth failed", resp).await);
}
Ok(())
}
+2 -1
View File
@@ -1,4 +1,5 @@
use crate::connection::MDRSConnection;
use crate::error::response_error;
use crate::models::laboratory::{Laboratories, Laboratory};
use serde::Deserialize;
@@ -12,7 +13,7 @@ impl MDRSConnection {
pub async fn list_laboratories(&self) -> Result<Laboratories, anyhow::Error> {
let resp = self.get("v3/laboratories/").await?;
if !resp.status().is_success() {
anyhow::bail!("List laboratories failed: {}", resp.status());
return Err(response_error("List laboratories failed", resp).await);
}
// The API may return a paginated object or a direct array
let text = resp.text().await?;
+87 -6
View File
@@ -1,7 +1,14 @@
use crate::connection::MDRSConnection;
use crate::error::response_error;
use crate::models::user::User as ModelUser;
use anyhow::bail;
use serde::Deserialize;
use std::time::Duration;
/// Bound the refresh on its own: the caller holds a lock that spans processes while this
/// runs, so a provider that accepts the connection and then goes quiet would stall every
/// other request on this machine rather than just this one. Uploads are served by a
/// separate instance, so this one is not queued behind them and has no reason to be slow.
const TOKEN_REFRESH_TIMEOUT: Duration = Duration::from_secs(30);
/// Full API response shape from GET v3/users/current/
#[derive(Debug, Deserialize)]
@@ -20,6 +27,16 @@ struct UsersCurrentResponseLaboratory {
#[derive(Deserialize)]
struct TokenRefreshResponse {
access: String,
/// Present when the provider rotates refresh tokens, absent when it does not,
/// so the caller keeps the token it already holds if nothing new arrives.
#[serde(default)]
refresh: Option<String>,
}
/// The token pair a refresh yields.
pub struct RefreshedToken {
pub access: String,
pub refresh: Option<String>,
}
impl MDRSConnection {
@@ -27,7 +44,7 @@ impl MDRSConnection {
pub async fn get_current_user(&self) -> Result<ModelUser, anyhow::Error> {
let resp = self.get("v3/users/current/").await?;
if !resp.status().is_success() {
bail!("Get current user failed: {}", resp.status());
return Err(response_error("Get current user failed", resp).await);
}
let obj = resp.json::<UsersApiCurrentResponse>().await?;
let laboratory_ids = obj.laboratories.into_iter().map(|l| l.id).collect();
@@ -40,19 +57,83 @@ impl MDRSConnection {
}
/// Refresh the access token using the refresh token.
/// POST v3/users/token/refresh/ {refresh: ...} -> {access: new_access}
pub async fn token_refresh(&self, refresh_token: &str) -> Result<String, anyhow::Error> {
/// POST v3/users/token/refresh/ {refresh: ...} -> {access, refresh?}
///
/// A rotating provider answers with a new refresh token and stops honouring the one
/// that was sent, so both halves of the reply have to be kept.
pub async fn token_refresh(
&self,
refresh_token: &str,
) -> Result<RefreshedToken, anyhow::Error> {
let body = serde_json::json!({ "refresh": refresh_token });
let resp = self
.client
.post(self.build_url("v3/users/token/refresh/"))
.json(&body)
.timeout(TOKEN_REFRESH_TIMEOUT)
.send()
.await?;
if !resp.status().is_success() {
bail!("Token refresh failed: {}", resp.status());
return Err(response_error("Token refresh failed", resp).await);
}
let r: TokenRefreshResponse = resp.json().await?;
Ok(r.access)
Ok(RefreshedToken {
access: r.access,
refresh: r.refresh,
})
}
}
#[cfg(test)]
mod tests {
use super::*;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
async fn refresh_against_stub(body: &'static str) -> RefreshedToken {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut buf = [0u8; 4096];
let n = stream.read(&mut buf).await.unwrap();
let req = String::from_utf8_lossy(&buf[..n]);
assert!(req.starts_with("POST /v3/users/token/refresh/ HTTP/1.1"));
assert!(req.contains("\"refresh\":\"old-refresh\""));
let response = format!(
"HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{}",
body.len(),
body
);
stream.write_all(response.as_bytes()).await.unwrap();
});
let conn = MDRSConnection::new(&format!("http://{addr}"));
let refreshed = conn.token_refresh("old-refresh").await.unwrap();
server.await.unwrap();
refreshed
}
/// A rotating provider stops honouring the token that was sent, so the reply's
/// refresh token has to reach the caller rather than being dropped.
#[tokio::test]
async fn token_refresh_returns_the_rotated_refresh_token() {
let refreshed =
refresh_against_stub(r#"{"access":"new-access","refresh":"new-refresh"}"#).await;
assert_eq!(refreshed.access, "new-access");
assert_eq!(refreshed.refresh.as_deref(), Some("new-refresh"));
}
/// A provider that does not rotate answers with the access token alone, and the
/// caller keeps the refresh token it already holds.
#[tokio::test]
async fn token_refresh_reports_no_rotation_when_the_reply_omits_it() {
let refreshed = refresh_against_stub(r#"{"access":"new-access"}"#).await;
assert_eq!(refreshed.access, "new-access");
assert_eq!(refreshed.refresh, None);
}
}
+72 -8
View File
@@ -139,14 +139,32 @@ fn write_cache_file(cache_path: &Path, cache: &Cache) -> Result<(), anyhow::Erro
}
fn parse_cache(remote: &str, data: &str) -> Result<Cache, anyhow::Error> {
serde_json::from_str::<Cache>(data).map_err(|e| {
let cache = serde_json::from_str::<Cache>(data).map_err(|e| {
anyhow!(
"Cache for `{}` is invalid or outdated ({}). Run `mdrs login {}` to refresh it.",
remote,
e,
remote
)
})
})?;
// The digest is what says the file is still the one this client wrote. The Python
// client checks it on every read and the two share the file, so a cache one of them
// would refuse must not be honoured by the other.
let expected = compute_digest(
cache.user.as_ref(),
&cache.token.access,
&cache.token.refresh,
&cache.laboratories,
);
if cache.digest != expected {
bail!(
"Cache for `{}` has been altered or was written by an incompatible version. \
Run `mdrs login {}` to refresh it.",
remote,
remote
);
}
Ok(cache)
}
fn load_cache_from_dir(remote: &str, config_dir: &Path) -> Result<Cache, anyhow::Error> {
@@ -213,7 +231,11 @@ fn load_cache_if_present_from_dir(
};
let cache = match parse_cache(remote, &data) {
Ok(cache) => cache,
Err(_) => {
Err(e) => {
// Said out loud before the credentials go. The alternative is a command that
// quietly carries on anonymously and fails later for a reason that looks
// unrelated to the cache it just threw away.
eprintln!("{e}");
remove_cache_in_dir(remote, config_dir)?;
return Ok(None);
}
@@ -290,6 +312,9 @@ pub async fn load_cache_with_token_refresh(remote: &str) -> Result<Cache, anyhow
let lock_file = fs::OpenOptions::new()
.write(true)
.create(true)
// Nothing is ever written into it: the file exists only to be flocked, and
// truncating it would touch a file other processes are holding open.
.truncate(false)
.open(&lock_path)?;
lock_file.lock_exclusive()?;
@@ -331,6 +356,9 @@ async fn load_cache_with_token_refresh_optional_from_dir(
let lock_file = fs::OpenOptions::new()
.write(true)
.create(true)
// Nothing is ever written into it: the file exists only to be flocked, and
// truncating it would touch a file other processes are holding open.
.truncate(false)
.open(&lock_path)?;
lock_file.lock_exclusive()?;
@@ -387,10 +415,13 @@ async fn refresh_and_persist_in_dir(
.ok_or_else(|| anyhow!("Remote `{}` is not configured.", remote))?;
let conn = MDRSConnection::new(&url);
let new_access = conn.token_refresh(&cache.token.refresh).await?;
let refreshed = conn.token_refresh(&cache.token.refresh).await?;
let mut updated_cache = cache.clone();
updated_cache.token.access = new_access;
updated_cache.token.access = refreshed.access;
if let Some(refresh) = refreshed.refresh {
updated_cache.token.refresh = refresh;
}
updated_cache.digest = compute_digest(
updated_cache.user.as_ref(),
&updated_cache.token.access,
@@ -440,7 +471,7 @@ mod tests {
use tempfile::tempdir;
fn sample_cache(username: &str) -> Cache {
Cache {
let mut cache = Cache {
user: Some(CacheUser {
id: 1,
username: username.to_string(),
@@ -459,8 +490,15 @@ mod tests {
full_name: "Laboratory".to_string(),
}],
},
digest: format!("digest-{username}"),
}
digest: String::new(),
};
cache.digest = compute_digest(
cache.user.as_ref(),
&cache.token.access,
&cache.token.refresh,
&cache.laboratories,
);
cache
}
fn remote_name(prefix: &str, config_dir: &Path) -> String {
@@ -572,6 +610,25 @@ mod tests {
assert!(loaded.is_none());
}
/// A cache whose contents no longer match its digest is not a session to act on.
#[test]
fn load_cache_refuses_a_tampered_cache() {
let dir = tempdir().unwrap();
let remote = remote_name("tampered", dir.path());
let mut cache = sample_cache("alice");
persist_cache_in_dir(&remote, dir.path(), &cache).unwrap();
// Someone edits the stored token but leaves the digest as it was.
cache.token.access = "someone-elses-access".to_string();
let cache_path = cache_file_path_in(dir.path(), &remote);
fs::write(&cache_path, serde_json::to_vec_pretty(&cache).unwrap()).unwrap();
invalidate_cached_entry(dir.path(), &remote);
let err = load_cache_from_dir(&remote, dir.path()).unwrap_err();
assert!(err.to_string().contains("has been altered"));
}
#[test]
fn load_cache_if_present_clears_invalid_cache() {
let dir = tempdir().unwrap();
@@ -594,6 +651,13 @@ mod tests {
let mut cache = sample_cache("alice");
cache.token.access = make_jwt_with_exp(0);
cache.token.refresh = make_jwt_with_exp(0);
// Re-stamped, so this exercises the expired session rather than the digest check.
cache.digest = compute_digest(
cache.user.as_ref(),
&cache.token.access,
&cache.token.refresh,
&cache.laboratories,
);
persist_cache_in_dir(&remote, dir.path(), &cache).unwrap();
let loaded = load_cache_with_token_refresh_optional_from_dir(&remote, dir.path())
+1 -1
View File
@@ -41,7 +41,7 @@ pub enum Commands {
skip_if_exists: bool,
#[arg(short = 'p', long)]
password: Option<String>,
#[arg(long)]
#[arg(short = 'e', long)]
exclude: Vec<String>,
remote_path: String,
local_path: String,
+4 -3
View File
@@ -1,5 +1,6 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::commands::shared::{find_folder, find_lab_in_cache, parse_remote_path};
use crate::commands::shared::{find_folder, find_laboratory, parse_remote_path};
use crate::error::response_error;
use anyhow::bail;
pub async fn chacl(
@@ -23,7 +24,7 @@ pub async fn chacl(
let (remote, labname, folder_path) = parse_remote_path(remote_path)?;
let cache = load_cache_with_token_refresh(&remote).await?;
let conn = create_authenticated_conn(&remote, &cache)?;
let lab = find_lab_in_cache(&cache, &labname)?;
let lab = find_laboratory(&conn, Some(&cache), &labname).await?;
let folder = find_folder(&conn, lab.id, &folder_path, None).await?;
let mut data = serde_json::Map::new();
@@ -45,7 +46,7 @@ pub async fn chacl(
.await?;
if !resp.status().is_success() {
bail!("ACL change failed: {}", resp.status());
return Err(response_error("ACL change failed", resp).await);
}
Ok(())
}
+68 -18
View File
@@ -13,18 +13,18 @@ fn sanitize_config_file(path: &PathBuf) -> Result<(), anyhow::Error> {
}
let text = fs::read_to_string(path)?;
let lines: Vec<&str> = text.lines().collect();
if let Some(first) = lines.first() {
if first.trim() == path.to_string_lossy() {
// remove the first line and write atomically
let new_text = if lines.len() > 1 {
lines[1..].join("\n")
} else {
String::new()
};
let tmp = path.with_extension("tmp");
fs::write(&tmp, new_text.as_bytes())?;
fs::rename(&tmp, path)?;
}
if let Some(first) = lines.first()
&& first.trim() == path.to_string_lossy()
{
// remove the first line and write atomically
let new_text = if lines.len() > 1 {
lines[1..].join("\n")
} else {
String::new()
};
let tmp = path.with_extension("tmp");
fs::write(&tmp, new_text.as_bytes())?;
fs::rename(&tmp, path)?;
}
Ok(())
}
@@ -49,9 +49,10 @@ pub fn get_remote_url(remote: &str) -> Result<Option<String>, anyhow::Error> {
}
pub fn config_create(remote: &str, url: &str) -> Result<(), anyhow::Error> {
if !validate_url(url) {
let Some(url) = normalize_url(url) else {
bail!("Malformed URL");
}
};
let url = url.as_str();
let path = config_path();
sanitize_config_file(&path)?;
let path_str = path.to_string_lossy().to_string();
@@ -78,9 +79,10 @@ pub fn config_create(remote: &str, url: &str) -> Result<(), anyhow::Error> {
}
pub fn config_update(remote: &str, url: &str) -> Result<(), anyhow::Error> {
if !validate_url(url) {
let Some(url) = normalize_url(url) else {
bail!("Malformed URL");
}
};
let url = url.as_str();
let path = config_path();
sanitize_config_file(&path)?;
let path_str = path.to_string_lossy().to_string();
@@ -145,6 +147,54 @@ pub fn config_delete(remote: &str) -> Result<(), anyhow::Error> {
Ok(())
}
fn validate_url(url: &str) -> bool {
validators::url::Url::parse(url).is_ok()
/// Check a remote URL and put it in the one form every client agrees on.
///
/// A bare hostname is accepted, so a development server on `localhost` is as acceptable
/// as a deployment behind a domain name. The trailing slash goes because the URL is
/// joined with a path that brings its own.
fn normalize_url(url: &str) -> Option<String> {
let parsed = reqwest::Url::parse(url).ok()?;
if !matches!(parsed.scheme(), "http" | "https") {
return None;
}
parsed.host_str()?;
Some(url.trim_end_matches('/').to_string())
}
#[cfg(test)]
mod url_tests {
use super::normalize_url;
/// Both clients share config.ini, so they have to agree on what a remote URL is.
#[test]
fn trailing_slash_is_dropped() {
assert_eq!(
normalize_url("http://127.0.0.1:8000/api/").as_deref(),
Some("http://127.0.0.1:8000/api")
);
assert_eq!(
normalize_url("https://neurodata.riken.jp/api/").as_deref(),
Some("https://neurodata.riken.jp/api")
);
}
#[test]
fn bare_hostname_is_accepted() {
assert_eq!(
normalize_url("http://localhost:8000/api").as_deref(),
Some("http://localhost:8000/api")
);
}
#[test]
fn only_http_schemes_are_accepted() {
for url in [
"ftp://x.example.com/",
"file:///etc/passwd",
"not-a-url",
"http://",
] {
assert_eq!(normalize_url(url), None, "{url} should be rejected");
}
}
}
+5 -5
View File
@@ -1,8 +1,8 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::commands::shared::{
find_file_by_name, find_folder, find_lab_in_cache, find_subfolder_by_name, nfc,
parse_remote_path,
find_file_by_name, find_folder, find_laboratory, find_subfolder_by_name, nfc, parse_remote_path,
};
use crate::error::response_error;
use anyhow::bail;
pub async fn cp(src_path: &str, dest_path: &str, recursive: bool) -> Result<(), anyhow::Error> {
@@ -19,7 +19,7 @@ pub async fn cp(src_path: &str, dest_path: &str, recursive: bool) -> Result<(),
let cache = load_cache_with_token_refresh(&s_remote).await?;
let conn = create_authenticated_conn(&s_remote, &cache)?;
let lab = find_lab_in_cache(&cache, &s_lab)?;
let lab = find_laboratory(&conn, Some(&cache), &s_lab).await?;
let lab_id = lab.id;
// Split source path into parent directory and target name
@@ -62,7 +62,7 @@ pub async fn cp(src_path: &str, dest_path: &str, recursive: bool) -> Result<(),
.post_json(&format!("v3/files/{}/copy/", src_file_id), &body)
.await?;
if !resp.status().is_success() {
bail!("Copy failed: {}", resp.status());
return Err(response_error("Copy failed", resp).await);
}
return Ok(());
}
@@ -102,7 +102,7 @@ pub async fn cp(src_path: &str, dest_path: &str, recursive: bool) -> Result<(),
.post_json(&format!("v3/folders/{}/copy/", src_folder_id), &body)
.await?;
if !resp.status().is_success() {
bail!("Copy failed: {}", resp.status());
return Err(response_error("Copy failed", resp).await);
}
Ok(())
}
+107 -163
View File
@@ -27,7 +27,7 @@ pub async fn download(
}
// Detect DOI path: "remote:10.xxxx/prefix.ID[/optional/sub/path]"
if is_doi(remote_path.splitn(2, ':').nth(1).unwrap_or("")) {
if is_doi(remote_path.split_once(':').map(|x| x.1).unwrap_or("")) {
let (remote, doi, subpath) = parse_doi_remote_path(remote_path)?;
let (raw_conn, _cache) = create_readonly_conn(&remote).await?;
let (doi_folder, lab) = find_folder_by_doi(&raw_conn, &doi, password).await?;
@@ -84,15 +84,13 @@ pub async fn download(
continue;
}
let dest = local_real.join(&file.name);
if skip_if_exists {
if dest.exists() {
if let Ok(meta) = std::fs::metadata(&dest) {
if meta.len() == file.size {
println!("{}", dest.display());
continue;
}
}
}
if skip_if_exists
&& dest.exists()
&& let Ok(meta) = std::fs::metadata(&dest)
&& meta.len() == file.size
{
println!("{}", dest.display());
continue;
}
let url = make_absolute_url(&conn, &file.download_url);
conn.download_file_limited(&url, &dest.to_string_lossy(), &limiter)
@@ -102,30 +100,23 @@ pub async fn download(
return Ok(());
}
spawn_download_folder_task(
&mut folder_tasks,
conn.clone(),
limiter.clone(),
lab_name.clone(),
excludes.clone(),
folder.id.clone(),
top_local,
password_owned.clone(),
skip_if_exists,
);
drive_download_tasks(
&mut folder_tasks,
&mut download_tasks,
&mut errors,
conn.clone(),
let session = DownloadSession {
conn: conn.clone(),
limiter,
lab_name,
excludes,
password_owned,
password: password_owned,
skip_if_exists,
)
.await;
};
spawn_download_folder_task(
&mut folder_tasks,
session.clone(),
folder.id.clone(),
top_local,
);
drive_download_tasks(&mut folder_tasks, &mut download_tasks, &mut errors, session)
.await;
if !errors.is_empty() {
bail!(errors.join("\n"));
@@ -146,15 +137,13 @@ pub async fn download(
return Ok(());
}
let dest = local_real.join(&file.name);
if skip_if_exists {
if dest.exists() {
if let Ok(meta) = std::fs::metadata(&dest) {
if meta.len() == file.size {
println!("{}", dest.display());
return Ok(());
}
}
}
if skip_if_exists
&& dest.exists()
&& let Ok(meta) = std::fs::metadata(&dest)
&& meta.len() == file.size
{
println!("{}", dest.display());
return Ok(());
}
let url = make_absolute_url(&conn, &file.download_url);
conn.download_file_limited(&url, &dest.to_string_lossy(), &limiter)
@@ -177,30 +166,23 @@ pub async fn download(
let lab_name = Arc::new(lab.name.clone());
let password_owned = password.map(str::to_string);
spawn_download_folder_task(
&mut folder_tasks,
conn.clone(),
limiter.clone(),
lab_name.clone(),
excludes.clone(),
sub.id.clone(),
top_local,
password_owned.clone(),
skip_if_exists,
);
drive_download_tasks(
&mut folder_tasks,
&mut download_tasks,
&mut errors,
conn.clone(),
let session = DownloadSession {
conn: conn.clone(),
limiter,
lab_name,
excludes,
password_owned,
password: password_owned,
skip_if_exists,
)
.await;
};
spawn_download_folder_task(
&mut folder_tasks,
session.clone(),
sub.id.clone(),
top_local,
);
drive_download_tasks(&mut folder_tasks, &mut download_tasks, &mut errors, session)
.await;
if !errors.is_empty() {
bail!(errors.join("\n"));
@@ -242,15 +224,13 @@ pub async fn download(
}
// Python always places the downloaded file inside the local directory.
let dest = local_real.join(&file.name);
if skip_if_exists {
if dest.exists() {
if let Ok(meta) = std::fs::metadata(&dest) {
if meta.len() == file.size {
println!("{}", dest.display());
return Ok(());
}
}
}
if skip_if_exists
&& dest.exists()
&& let Ok(meta) = std::fs::metadata(&dest)
&& meta.len() == file.size
{
println!("{}", dest.display());
return Ok(());
}
let url = make_absolute_url(&conn, &file.download_url);
conn.download_file_limited(&url, &dest.to_string_lossy(), &limiter)
@@ -277,30 +257,22 @@ pub async fn download(
let lab_name = Arc::new(lab.name.clone());
let password = password.map(str::to_string);
spawn_download_folder_task(
&mut folder_tasks,
conn.clone(),
limiter.clone(),
lab_name.clone(),
excludes.clone(),
sub.id.clone(),
top_local,
password.clone(),
skip_if_exists,
);
drive_download_tasks(
&mut folder_tasks,
&mut download_tasks,
&mut errors,
conn.clone(),
let session = DownloadSession {
conn: conn.clone(),
limiter,
lab_name,
excludes,
password,
skip_if_exists,
)
.await;
};
spawn_download_folder_task(
&mut folder_tasks,
session.clone(),
sub.id.clone(),
top_local,
);
drive_download_tasks(&mut folder_tasks, &mut download_tasks, &mut errors, session).await;
if !errors.is_empty() {
bail!(errors.join("\n"));
@@ -353,30 +325,26 @@ struct DownloadJob {
dest_path: PathBuf,
}
fn spawn_download_folder_task(
folder_tasks: &mut JoinSet<Result<DownloadFolderTaskResult, anyhow::Error>>,
/// What every folder of one recursive download shares: the connection and its request
/// budget, the laboratory the paths are named against, what to leave out, the password
/// for locked folders, and whether files already on disk may be left alone.
#[derive(Clone)]
struct DownloadSession {
conn: Arc<MDRSConnection>,
limiter: ApiRequestLimiter,
lab_name: Arc<String>,
excludes: Arc<Vec<String>>,
folder_id: String,
local_dir: PathBuf,
password: Option<String>,
skip_if_exists: bool,
}
fn spawn_download_folder_task(
folder_tasks: &mut JoinSet<Result<DownloadFolderTaskResult, anyhow::Error>>,
session: DownloadSession,
folder_id: String,
local_dir: PathBuf,
) {
folder_tasks.spawn(async move {
process_download_folder(
conn,
limiter,
lab_name,
excludes,
folder_id,
local_dir,
password,
skip_if_exists,
)
.await
});
folder_tasks.spawn(async move { process_download_folder(session, folder_id, local_dir).await });
}
fn spawn_download_task(
@@ -396,9 +364,8 @@ fn spawn_download_task(
Ok(())
}
Err(err) => {
if job.dest_path.is_file() {
let _ = std::fs::remove_file(&job.dest_path);
}
// Nothing to clear up: a failed transfer writes only to its own scratch
// file, and removes that itself.
Err(anyhow!(
"Failed to download {}: {}",
job.dest_path.display(),
@@ -410,15 +377,18 @@ fn spawn_download_task(
}
async fn process_download_folder(
conn: Arc<MDRSConnection>,
limiter: ApiRequestLimiter,
lab_name: Arc<String>,
excludes: Arc<Vec<String>>,
session: DownloadSession,
folder_id: String,
local_dir: PathBuf,
password: Option<String>,
skip_if_exists: bool,
) -> Result<DownloadFolderTaskResult, anyhow::Error> {
let DownloadSession {
conn,
limiter,
lab_name,
excludes,
password,
skip_if_exists,
} = session;
let folder = conn.retrieve_folder_limited(&folder_id, &limiter).await?;
if is_excluded(excludes.as_slice(), lab_name.as_str(), &folder.path, None) {
@@ -443,13 +413,13 @@ async fn process_download_folder(
continue;
}
let dest_path = local_dir.join(&file.name);
if skip_if_exists && dest_path.exists() {
if let Ok(meta) = std::fs::metadata(&dest_path) {
if meta.len() == file.size {
println!("{}", dest_path.display());
continue;
}
}
if skip_if_exists
&& dest_path.exists()
&& let Ok(meta) = std::fs::metadata(&dest_path)
&& meta.len() == file.size
{
println!("{}", dest_path.display());
continue;
}
download_jobs.push(DownloadJob {
url: make_absolute_url(&conn, &file.download_url),
@@ -500,12 +470,7 @@ async fn drive_download_tasks(
folder_tasks: &mut JoinSet<Result<DownloadFolderTaskResult, anyhow::Error>>,
download_tasks: &mut JoinSet<Result<(), anyhow::Error>>,
errors: &mut Vec<String>,
conn: Arc<MDRSConnection>,
limiter: ApiRequestLimiter,
lab_name: Arc<String>,
excludes: Arc<Vec<String>>,
password: Option<String>,
skip_if_exists: bool,
session: DownloadSession,
) {
loop {
match (folder_tasks.is_empty(), download_tasks.is_empty()) {
@@ -517,20 +482,15 @@ async fn drive_download_tasks(
folder_tasks,
download_tasks,
errors,
conn.clone(),
limiter.clone(),
lab_name.clone(),
excludes.clone(),
password.clone(),
skip_if_exists,
&session,
);
}
}
(true, false) => {
if let Some(result) = download_tasks.join_next().await {
if let Err(err) = flatten_join_result(result) {
errors.push(err.to_string());
}
if let Some(result) = download_tasks.join_next().await
&& let Err(err) = flatten_join_result(result)
{
errors.push(err.to_string());
}
}
(false, false) => {
@@ -542,21 +502,15 @@ async fn drive_download_tasks(
folder_tasks,
download_tasks,
errors,
conn.clone(),
limiter.clone(),
lab_name.clone(),
excludes.clone(),
password.clone(),
skip_if_exists,
&session,
);
}
}
result = download_tasks.join_next() => {
if let Some(result) = result {
if let Err(err) = flatten_join_result(result) {
if let Some(result) = result
&& let Err(err) = flatten_join_result(result) {
errors.push(err.to_string());
}
}
}
}
}
@@ -569,30 +523,20 @@ fn handle_download_folder_result(
folder_tasks: &mut JoinSet<Result<DownloadFolderTaskResult, anyhow::Error>>,
download_tasks: &mut JoinSet<Result<(), anyhow::Error>>,
errors: &mut Vec<String>,
conn: Arc<MDRSConnection>,
limiter: ApiRequestLimiter,
lab_name: Arc<String>,
excludes: Arc<Vec<String>>,
password: Option<String>,
skip_if_exists: bool,
session: &DownloadSession,
) {
match flatten_join_result(result) {
Ok(task_result) => {
for (folder_id, local_dir) in task_result.child_folders {
spawn_download_folder_task(
folder_tasks,
conn.clone(),
limiter.clone(),
lab_name.clone(),
excludes.clone(),
folder_id,
local_dir,
password.clone(),
skip_if_exists,
);
spawn_download_folder_task(folder_tasks, session.clone(), folder_id, local_dir);
}
for job in task_result.download_jobs {
spawn_download_task(download_tasks, conn.clone(), limiter.clone(), job);
spawn_download_task(
download_tasks,
session.conn.clone(),
session.limiter.clone(),
job,
);
}
}
Err(err) => errors.push(err.to_string()),
+5 -1
View File
@@ -1,10 +1,11 @@
use crate::cache::create_readonly_conn;
use crate::commands::shared::{find_file_by_name, resolve_remote_file};
use crate::error::response_error;
use anyhow::anyhow;
pub async fn file_metadata(remote_path: &str, password: Option<&str>) -> Result<(), anyhow::Error> {
let remote = remote_path
.splitn(2, ':')
.split(':')
.next()
.ok_or_else(|| anyhow!("Invalid remote path"))?;
let (conn, cache) = create_readonly_conn(remote).await?;
@@ -17,6 +18,9 @@ pub async fn file_metadata(remote_path: &str, password: Option<&str>) -> Result<
.ok_or_else(|| anyhow!("File `{}` not found.", basename))?;
let resp = conn.get(&format!("v3/files/{}/metadata/", file.id)).await?;
if !resp.status().is_success() {
return Err(response_error("Failed to get file metadata", resp).await);
}
let json: serde_json::Value = resp.json().await?;
println!("{}", serde_json::to_string(&json)?);
Ok(())
+83 -18
View File
@@ -1,12 +1,19 @@
use crate::cache::create_readonly_conn;
use crate::cache::{Cache, create_readonly_conn};
use crate::commands::shared::{fmt_datetime, resolve_remote_folder};
use crate::connection::MDRSConnection;
use crate::models::file::File;
use crate::models::folder::{FolderDetail, FolderSimple};
use serde_json::{Value, json};
use std::collections::HashMap;
use std::future::Future;
use std::pin::Pin;
/// Laboratory names by id, for the JSON output's per-folder `laboratory` field.
type LaboratoryNames = HashMap<u32, String>;
/// Shown for a folder whose laboratory the client cannot name, matching the Python client.
const UNKNOWN_LABORATORY: &str = "(invalid)";
pub async fn ls(
remote_path: &str,
password: Option<&str>,
@@ -15,7 +22,7 @@ pub async fn ls(
is_quiet: bool,
) -> Result<(), anyhow::Error> {
let remote = remote_path
.splitn(2, ':')
.split(':')
.next()
.ok_or_else(|| anyhow::anyhow!("Invalid remote path"))?;
let (conn, cache) = create_readonly_conn(remote).await?;
@@ -24,10 +31,11 @@ pub async fn ls(
let labname = lab.name;
if is_json {
let names = laboratory_names(&conn, cache.as_ref()).await;
let output = if is_recursive {
build_folder_json_recursive(&conn, folder, &labname).await?
build_folder_json_recursive(&conn, folder, &names, password).await?
} else {
build_folder_json_flat(&conn, &folder, &labname).await?
build_folder_json_flat(&conn, &folder, &names).await?
};
println!("{}", serde_json::to_string(&output)?);
} else if is_recursive {
@@ -214,7 +222,13 @@ fn file_to_json(f: &File, base_url: &str) -> Value {
let download_url = if f.download_url.starts_with("http") {
f.download_url.clone()
} else {
format!("{}{}", base_url.trim_end_matches('/'), f.download_url)
// The API answers with a relative path and no leading separator, so supply one
// rather than running the two together.
format!(
"{}/{}",
base_url.trim_end_matches('/'),
f.download_url.trim_start_matches('/')
)
};
json!({
"id": f.id,
@@ -229,7 +243,7 @@ fn file_to_json(f: &File, base_url: &str) -> Value {
})
}
fn folder_simple_to_json(sf: &FolderSimple) -> Value {
fn folder_simple_to_json(sf: &FolderSimple, names: &LaboratoryNames) -> Value {
json!({
"id": sf.id,
"pid": sf.pid,
@@ -237,26 +251,58 @@ fn folder_simple_to_json(sf: &FolderSimple) -> Value {
"access_level": sf.access_level_name(),
"lock": sf.lock,
"size": sf.size,
"laboratory_id": sf.laboratory_id,
"laboratory": laboratory_name(names, sf.laboratory_id),
"description": sf.description,
"created_at": sf.created_at,
"updated_at": sf.updated_at,
})
}
/// Name the laboratory a folder belongs to, so a listing does not hand out an id the
/// reader has no way to resolve.
fn laboratory_name(names: &LaboratoryNames, id: u32) -> String {
names
.get(&id)
.cloned()
.unwrap_or_else(|| UNKNOWN_LABORATORY.to_string())
}
/// Collect laboratory names, preferring what the login cache already holds so a listing
/// does not pay for a request it does not need.
async fn laboratory_names(conn: &MDRSConnection, cache: Option<&Cache>) -> LaboratoryNames {
if let Some(cache) = cache.filter(|c| !c.laboratories.items.is_empty()) {
return cache
.laboratories
.items
.iter()
.map(|lab| (lab.id, lab.name.clone()))
.collect();
}
match conn.list_laboratories().await {
Ok(labs) => labs
.items
.into_iter()
.map(|lab| (lab.id, lab.name))
.collect(),
Err(_) => LaboratoryNames::new(),
}
}
/// Build JSON for the top-level folder without recursing into sub-folders.
async fn build_folder_json_flat(
conn: &MDRSConnection,
folder: &FolderDetail,
labname: &str,
names: &LaboratoryNames,
) -> Result<Value, anyhow::Error> {
let metadata = get_folder_metadata(conn, &folder.id).await?;
let files = conn.list_all_files(&folder.id).await?;
let mut files = conn.list_all_files(&folder.id).await?;
files.sort_by(|a, b| a.name.cmp(&b.name));
let files_json: Vec<Value> = files.iter().map(|f| file_to_json(f, &conn.url)).collect();
let sub_folders_json: Vec<Value> = folder
.sub_folders
let mut sub_folders = folder.sub_folders.clone();
sub_folders.sort_by(|a, b| a.name.cmp(&b.name));
let sub_folders_json: Vec<Value> = sub_folders
.iter()
.map(folder_simple_to_json)
.map(|sf| folder_simple_to_json(sf, names))
.collect();
Ok(json!({
@@ -266,7 +312,7 @@ async fn build_folder_json_flat(
"size": folder.size,
"access_level": folder.access_level_name(),
"lock": folder.lock,
"laboratory": labname,
"laboratory": laboratory_name(names, folder.laboratory_id),
"description": folder.description,
"created_at": folder.created_at,
"updated_at": folder.updated_at,
@@ -280,17 +326,36 @@ async fn build_folder_json_flat(
fn build_folder_json_recursive<'a>(
conn: &'a MDRSConnection,
folder: FolderDetail,
labname: &'a str,
names: &'a LaboratoryNames,
password: Option<&'a str>,
) -> Pin<Box<dyn Future<Output = Result<Value, anyhow::Error>> + 'a>> {
Box::pin(async move {
let metadata = get_folder_metadata(conn, &folder.id).await?;
let files = conn.list_all_files(&folder.id).await?;
let mut files = conn.list_all_files(&folder.id).await?;
files.sort_by(|a, b| a.name.cmp(&b.name));
let files_json: Vec<Value> = files.iter().map(|f| file_to_json(f, &conn.url)).collect();
let mut sub_folders = folder.sub_folders.clone();
sub_folders.sort_by(|a, b| a.name.cmp(&b.name));
let mut sub_folders_json = Vec::new();
for sf in &folder.sub_folders {
for sf in &sub_folders {
// A folder the caller cannot unlock is left out rather than ending the whole
// listing: a tree usually holds a locked folder or two, and the rest of it is
// still what the user asked to see. Anything else is reported, because a
// listing that quietly drops a branch cannot be told from one where the
// branch is empty, and this output is read by programs.
if sf.lock {
match password {
None => continue,
Some(pw) => {
if conn.folder_auth(&sf.id, pw).await.is_err() {
continue;
}
}
}
}
let sf_detail = conn.retrieve_folder(&sf.id).await?;
let sf_json = build_folder_json_recursive(conn, sf_detail, labname).await?;
let sf_json = build_folder_json_recursive(conn, sf_detail, names, password).await?;
sub_folders_json.push(sf_json);
}
@@ -301,7 +366,7 @@ fn build_folder_json_recursive<'a>(
"size": folder.size,
"access_level": folder.access_level_name(),
"lock": folder.lock,
"laboratory": labname,
"laboratory": laboratory_name(names, folder.laboratory_id),
"description": folder.description,
"created_at": folder.created_at,
"updated_at": folder.updated_at,
+5 -1
View File
@@ -1,9 +1,10 @@
use crate::cache::create_readonly_conn;
use crate::commands::shared::resolve_remote_folder;
use crate::error::response_error;
pub async fn metadata(remote_path: &str, password: Option<&str>) -> Result<(), anyhow::Error> {
let remote = remote_path
.splitn(2, ':')
.split(':')
.next()
.ok_or_else(|| anyhow::anyhow!("Invalid remote path"))?;
let (conn, cache) = create_readonly_conn(remote).await?;
@@ -13,6 +14,9 @@ pub async fn metadata(remote_path: &str, password: Option<&str>) -> Result<(), a
let resp = conn
.get(&format!("v3/folders/{}/metadata/", folder.id))
.await?;
if !resp.status().is_success() {
return Err(response_error("Failed to get folder metadata", resp).await);
}
let json: serde_json::Value = resp.json().await?;
println!("{}", serde_json::to_string(&json)?);
Ok(())
+4 -4
View File
@@ -1,8 +1,8 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::commands::shared::{
find_file_by_name, find_folder, find_lab_in_cache, find_subfolder_by_name, nfc,
parse_remote_path,
find_file_by_name, find_folder, find_laboratory, find_subfolder_by_name, nfc, parse_remote_path,
};
use crate::error::response_error;
use anyhow::{anyhow, bail};
pub async fn mkdir(remote_path: &str) -> Result<(), anyhow::Error> {
@@ -25,7 +25,7 @@ pub async fn mkdir(remote_path: &str) -> Result<(), anyhow::Error> {
let cache = load_cache_with_token_refresh(&remote).await?;
let conn = create_authenticated_conn(&remote, &cache)?;
let lab = find_lab_in_cache(&cache, &labname)?;
let lab = find_laboratory(&conn, Some(&cache), &labname).await?;
let parent_folder = find_folder(&conn, lab.id, parent_path, None).await?;
// Check for name conflict in sub-folders or files
@@ -40,7 +40,7 @@ pub async fn mkdir(remote_path: &str) -> Result<(), anyhow::Error> {
.create_folder(&parent_folder.id, &nfc(new_folder_name))
.await?;
if !resp.status().is_success() {
bail!("Failed to create folder: {}", resp.status());
return Err(response_error("Failed to create folder", resp).await);
}
Ok(())
}
+5 -5
View File
@@ -1,8 +1,8 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::commands::shared::{
find_file_by_name, find_folder, find_lab_in_cache, find_subfolder_by_name, nfc,
parse_remote_path,
find_file_by_name, find_folder, find_laboratory, find_subfolder_by_name, nfc, parse_remote_path,
};
use crate::error::response_error;
use anyhow::bail;
pub async fn mv(src_path: &str, dest_path: &str) -> Result<(), anyhow::Error> {
@@ -19,7 +19,7 @@ pub async fn mv(src_path: &str, dest_path: &str) -> Result<(), anyhow::Error> {
let cache = load_cache_with_token_refresh(&s_remote).await?;
let conn = create_authenticated_conn(&s_remote, &cache)?;
let lab = find_lab_in_cache(&cache, &s_lab)?;
let lab = find_laboratory(&conn, Some(&cache), &s_lab).await?;
let lab_id = lab.id;
// Split source path into parent directory and target name
@@ -62,7 +62,7 @@ pub async fn mv(src_path: &str, dest_path: &str) -> Result<(), anyhow::Error> {
.post_json(&format!("v3/files/{}/move/", src_file_id), &body)
.await?;
if !resp.status().is_success() {
bail!("Move failed: {}", resp.status());
return Err(response_error("Move failed", resp).await);
}
return Ok(());
}
@@ -99,7 +99,7 @@ pub async fn mv(src_path: &str, dest_path: &str) -> Result<(), anyhow::Error> {
.post_json(&format!("v3/folders/{}/move/", src_folder_id), &body)
.await?;
if !resp.status().is_success() {
bail!("Move failed: {}", resp.status());
return Err(response_error("Move failed", resp).await);
}
Ok(())
}
+5 -4
View File
@@ -1,7 +1,8 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::commands::shared::{
find_file_by_name, find_folder, find_lab_in_cache, find_subfolder_by_name, parse_remote_path,
find_file_by_name, find_folder, find_laboratory, find_subfolder_by_name, parse_remote_path,
};
use crate::error::response_error;
use anyhow::{anyhow, bail};
pub async fn rm(remote_path: &str, recursive: bool) -> Result<(), anyhow::Error> {
@@ -22,7 +23,7 @@ pub async fn rm(remote_path: &str, recursive: bool) -> Result<(), anyhow::Error>
let cache = load_cache_with_token_refresh(&remote).await?;
let conn = create_authenticated_conn(&remote, &cache)?;
let lab = find_lab_in_cache(&cache, &labname)?;
let lab = find_laboratory(&conn, Some(&cache), &labname).await?;
let parent_folder = find_folder(&conn, lab.id, parent_path, None).await?;
// Check if target is a file
@@ -30,7 +31,7 @@ pub async fn rm(remote_path: &str, recursive: bool) -> Result<(), anyhow::Error>
if let Some(file) = find_file_by_name(&files, target_name) {
let resp = conn.delete(&format!("v3/files/{}/", file.id)).await?;
if !resp.status().is_success() {
bail!("Failed to delete file: {}", resp.status());
return Err(response_error("Failed to delete file", resp).await);
}
return Ok(());
}
@@ -47,7 +48,7 @@ pub async fn rm(remote_path: &str, recursive: bool) -> Result<(), anyhow::Error>
)
.await?;
if !resp.status().is_success() {
bail!("Failed to delete folder: {}", resp.status());
return Err(response_error("Failed to delete folder", resp).await);
}
return Ok(());
}
+101 -1
View File
@@ -48,6 +48,25 @@ fn is_newer(current: &str, latest: &str) -> bool {
false
}
/// Read the digest out of a `sha256sum` line: the hex digest, then the file it covers.
fn parse_sha256_line(text: &str) -> Option<String> {
let digest = text.split_whitespace().next()?.to_ascii_lowercase();
let is_hex = digest.len() == 64 && digest.chars().all(|c| c.is_ascii_hexdigit());
is_hex.then_some(digest)
}
/// Hex SHA-256 of a slice, in the form `sha256sum` prints.
fn sha256_hex(bytes: &[u8]) -> String {
use sha2::{Digest, Sha256};
let mut hasher = Sha256::new();
hasher.update(bytes);
hasher
.finalize()
.iter()
.map(|b| format!("{:02x}", b))
.collect()
}
/// Extract the binary named `bin_name` from a `.tar.gz` archive at `archive_path`
/// and write it to `dest_path`.
fn extract_from_tar_gz(
@@ -138,10 +157,11 @@ pub async fn selfupdate(yes: bool) -> anyhow::Result<()> {
println!("New version available: {latest_version}");
// Find the asset matching the current build target.
// The checksum asset carries the target in its name too, so say which one is wanted.
let asset = release
.assets
.iter()
.find(|a| a.name.contains(BUILD_TARGET))
.find(|a| a.name.contains(BUILD_TARGET) && !a.name.ends_with(".sha256"))
.ok_or_else(|| {
let names: Vec<&str> = release.assets.iter().map(|a| a.name.as_str()).collect();
anyhow!(
@@ -186,6 +206,54 @@ pub async fn selfupdate(yes: bool) -> anyhow::Result<()> {
}
let bytes = download_resp.bytes().await?;
// This binary is about to replace the one the user is running, so what arrived has to
// be what the release says it is. Transport already proves who served it; the digest
// proves the archive was not swapped or truncated on the way.
match release
.assets
.iter()
.find(|a| a.name == format!("{}.sha256", asset.name))
{
Some(checksum_asset) => {
let mut checksum_req = client
.get(&checksum_asset.browser_download_url)
.header(USER_AGENT, format!("mdrs/{current_version}"));
if let Ok(token) = env::var("GITEA_TOKEN") {
checksum_req = checksum_req.header(AUTHORIZATION, format!("Bearer {token}"));
}
let checksum_resp = checksum_req.send().await?;
if !checksum_resp.status().is_success() {
bail!(
"Failed to download checksum: HTTP {}",
checksum_resp.status()
);
}
let expected = parse_sha256_line(&checksum_resp.text().await?)
.ok_or_else(|| anyhow!("Checksum asset '{}' is malformed", checksum_asset.name))?;
let actual = sha256_hex(&bytes);
if actual != expected {
bail!(
"Checksum mismatch for '{}': expected {}, got {}. The download was not what \
the release publishes, so this binary has not been replaced.",
asset.name,
expected,
actual
);
}
println!("Checksum verified.");
}
None => {
// Releases built before the workflow published checksums have none to check
// against. Say so rather than letting the absence pass for a verified download.
eprintln!(
"Warning: release {latest_version} publishes no checksum for '{}', \
so the download could not be verified.",
asset.name
);
}
}
std::fs::write(&archive_path, &bytes)?;
// Extract the binary from the archive.
@@ -216,3 +284,35 @@ pub async fn selfupdate(yes: bool) -> anyhow::Result<()> {
println!("Successfully updated to version {latest_version}.");
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_sha256sum_line_yields_its_digest() {
let digest = "a".repeat(64);
assert_eq!(
parse_sha256_line(&format!(
"{digest} mdrs-2.0.1-x86_64-unknown-linux-musl.tar.gz\n"
)),
Some(digest)
);
}
#[test]
fn a_line_that_is_not_a_digest_is_refused() {
for text in ["", "not-a-digest file", "abc123 file"] {
assert_eq!(parse_sha256_line(text), None, "{text:?} should be refused");
}
}
#[test]
fn the_digest_matches_what_sha256sum_prints() {
// Verified against `printf mdrs | sha256sum`.
assert_eq!(
sha256_hex(b"mdrs"),
"bd9845248df877e411ec8bef10b823bd4c0900a530157eedfdfbfb63ccbde2fa"
);
}
}
+78 -17
View File
@@ -158,6 +158,10 @@ pub async fn find_folder_by_doi(
// ---------------------------------------------------------------------------
/// Parse "remote:/labname/path/" into (remote, labname, folder_path).
///
/// The path is put in one form before anything is done with it: repeated and trailing
/// separators go, `.` segments go, and a `..` is refused rather than sent on. The Python
/// client does the same, and both talk to the same server.
pub fn parse_remote_path(remote_path: &str) -> Result<(String, String, String), anyhow::Error> {
let parts: Vec<&str> = remote_path.splitn(2, ':').collect();
if parts.len() != 2 {
@@ -168,12 +172,22 @@ pub fn parse_remote_path(remote_path: &str) -> Result<(String, String, String),
if !rest.starts_with('/') {
bail!("Path must be absolute (start with '/')");
}
let folder_parts: Vec<&str> = rest.trim_start_matches('/').splitn(2, '/').collect();
let labname = folder_parts[0].to_string();
let path = if folder_parts.len() > 1 && !folder_parts[1].is_empty() {
format!("/{}", folder_parts[1].trim_end_matches('/'))
} else {
let mut segments = Vec::new();
for segment in rest.split('/') {
match segment {
"" | "." => continue,
".." => bail!("Path traversal found."),
other => segments.push(other),
}
}
if segments.is_empty() {
bail!("Laboratory name is missing from `{}`.", remote_path);
}
let labname = segments.remove(0).to_string();
let path = if segments.is_empty() {
"/".to_string()
} else {
format!("/{}", segments.join("/"))
};
Ok((remote, labname, path))
}
@@ -202,15 +216,15 @@ pub async fn find_laboratory(
cache: Option<&Cache>,
labname: &str,
) -> Result<Laboratory, anyhow::Error> {
if let Some(cache) = cache {
if let Ok(lab) = find_lab_in_cache(cache, labname) {
return Ok(Laboratory {
id: lab.id,
name: lab.name.clone(),
pi_name: lab.pi_name.clone(),
full_name: lab.full_name.clone(),
});
}
if let Some(cache) = cache
&& let Ok(lab) = find_lab_in_cache(cache, labname)
{
return Ok(Laboratory {
id: lab.id,
name: lab.name.clone(),
pi_name: lab.pi_name.clone(),
full_name: lab.full_name.clone(),
});
}
conn.list_laboratories()
@@ -338,7 +352,7 @@ pub fn find_subfolder_by_name<'a>(
/// Format an ISO 8601 timestamp as "YYYY/MM/DD HH:MM:SS".
pub fn fmt_datetime(iso: &str) -> String {
let s = iso.trim();
let s = if let Some(pos) = s[10..].find(|c: char| c == '+' || c == '-') {
let s = if let Some(pos) = s[10..].find(['+', '-']) {
&s[..10 + pos]
} else {
s.trim_end_matches('Z')
@@ -361,7 +375,7 @@ pub async fn resolve_remote_folder(
remote_path: &str,
password: Option<&str>,
) -> Result<(FolderDetail, Laboratory), anyhow::Error> {
let path_component = remote_path.splitn(2, ':').nth(1).unwrap_or("");
let path_component = remote_path.split_once(':').map(|x| x.1).unwrap_or("");
if is_doi(path_component) {
let (_, doi, subpath) = parse_doi_remote_path(remote_path)?;
let (doi_folder, lab) = find_folder_by_doi(conn, &doi, password).await?;
@@ -395,7 +409,7 @@ pub async fn resolve_remote_file(
remote_path: &str,
password: Option<&str>,
) -> Result<(FolderDetail, String), anyhow::Error> {
let path_component = remote_path.splitn(2, ':').nth(1).unwrap_or("");
let path_component = remote_path.split_once(':').map(|x| x.1).unwrap_or("");
if is_doi(path_component) {
let (_, doi, subpath) = parse_doi_remote_path(remote_path)?;
let (doi_folder, lab) = find_folder_by_doi(conn, &doi, password).await?;
@@ -473,6 +487,53 @@ mod tests {
server.await.unwrap();
}
// ------------------------------------------------------------------
// Remote path parsing
// ------------------------------------------------------------------
#[test]
fn parse_remote_path_splits_remote_lab_and_path() {
let (remote, lab, path) = parse_remote_path("neurodata:/mylab/a/b/").unwrap();
assert_eq!(
(remote.as_str(), lab.as_str(), path.as_str()),
("neurodata", "mylab", "/a/b")
);
}
#[test]
fn parse_remote_path_gives_the_lab_root_its_own_form() {
let (_, lab, path) = parse_remote_path("neurodata:/mylab").unwrap();
assert_eq!((lab.as_str(), path.as_str()), ("mylab", "/"));
}
/// Both clients talk to the same server, so a doubled separator or a `.` segment has
/// to reach it as the same path from either one.
#[test]
fn parse_remote_path_collapses_redundant_segments() {
let (_, lab, path) = parse_remote_path("neurodata://mylab//a/./b//").unwrap();
assert_eq!((lab.as_str(), path.as_str()), ("mylab", "/a/b"));
}
#[test]
fn parse_remote_path_refuses_a_parent_segment() {
for path in [
"neurodata:/mylab/../other",
"neurodata:/mylab/a/..",
"neurodata:/../mylab",
] {
let err = parse_remote_path(path).unwrap_err();
assert!(
err.to_string().contains("Path traversal found."),
"{path} should be refused"
);
}
}
#[test]
fn parse_remote_path_refuses_a_path_without_a_laboratory() {
assert!(parse_remote_path("neurodata:/").is_err());
}
// ------------------------------------------------------------------
// DOI helper unit tests
// ------------------------------------------------------------------
+21 -25
View File
@@ -1,6 +1,6 @@
use crate::cache::{create_authenticated_conn, load_cache_with_token_refresh};
use crate::commands::shared::{
find_file_by_name, find_folder_limited, find_lab_in_cache, nfc, parse_remote_path,
find_file_by_name, find_folder_limited, find_laboratory, nfc, parse_remote_path,
};
use crate::connection::{ApiRequestLimiter, MDRSConnection};
use crate::models::folder::FolderSimple;
@@ -20,7 +20,7 @@ pub async fn upload(
let cache = load_cache_with_token_refresh(&remote).await?;
let conn = Arc::new(create_authenticated_conn(&remote, &cache)?);
let limiter = ApiRequestLimiter::new(crate::settings::SETTINGS.concurrent);
let lab = find_lab_in_cache(&cache, &labname)?;
let lab = find_laboratory(&conn, Some(&cache), &labname).await?;
let dest_folder = find_folder_limited(&conn, &limiter, lab.id, &r_path, None).await?;
// Normalize local_path: resolve to an absolute canonical path so that
@@ -35,13 +35,12 @@ pub async fn upload(
let remote_files = conn
.list_all_files_limited(&dest_folder.id, &limiter)
.await?;
if skip_if_exists {
if let Some(rf) = find_file_by_name(&remote_files, &filename) {
if rf.size == std::fs::metadata(local)?.len() {
println!("{}{}", dest_folder.path, filename);
return Ok(());
}
}
if skip_if_exists
&& let Some(rf) = find_file_by_name(&remote_files, &filename)
&& rf.size == std::fs::metadata(local)?.len()
{
println!("{}{}", dest_folder.path, filename);
return Ok(());
}
conn.upload_file_limited(&dest_folder.id, &local.to_string_lossy(), &limiter)
.await?;
@@ -219,15 +218,13 @@ async fn process_upload_folder(
let mut upload_jobs = Vec::new();
for file_path in files {
let filename = file_path.file_name().unwrap().to_string_lossy().to_string();
if skip_if_exists {
if let Some(rf) = find_file_by_name(&remote_files, &filename) {
if let Ok(meta) = std::fs::metadata(&file_path) {
if rf.size == meta.len() {
println!("{}{}", folder_detail.path, filename);
continue;
}
}
}
if skip_if_exists
&& let Some(rf) = find_file_by_name(&remote_files, &filename)
&& let Ok(meta) = std::fs::metadata(&file_path)
&& rf.size == meta.len()
{
println!("{}{}", folder_detail.path, filename);
continue;
}
upload_jobs.push(UploadJob {
folder_id: remote_id.clone(),
@@ -267,10 +264,10 @@ async fn drive_upload_tasks(
}
}
(true, false) => {
if let Some(result) = upload_tasks.join_next().await {
if let Err(err) = flatten_join_result(result) {
errors.push(err.to_string());
}
if let Some(result) = upload_tasks.join_next().await
&& let Err(err) = flatten_join_result(result)
{
errors.push(err.to_string());
}
}
(false, false) => {
@@ -289,11 +286,10 @@ async fn drive_upload_tasks(
}
}
result = upload_tasks.join_next() => {
if let Some(result) = result {
if let Err(err) = flatten_join_result(result) {
if let Some(result) = result
&& let Err(err) = flatten_join_result(result) {
errors.push(err.to_string());
}
}
}
}
}
+10 -5
View File
@@ -1,9 +1,14 @@
use crate::cache::create_readonly_conn;
/// Ask the server who it thinks is calling.
///
/// The cached name is only what was true at login: an account disabled or removed since
/// then would still answer from the cache, which is the one thing this command exists to
/// tell the user. A session the server no longer honours reads as anonymous.
pub async fn whoami(remote: &str) -> Result<(), anyhow::Error> {
match crate::cache::load_cache(remote) {
Ok(cache) => match cache.user {
Some(user) => println!("{}", user.username),
None => println!("(Anonymous)"),
},
let (conn, _cache) = create_readonly_conn(remote).await?;
match conn.get_current_user().await {
Ok(user) => println!("{}", user.username),
Err(_) => println!("(Anonymous)"),
}
Ok(())
+125 -49
View File
@@ -1,9 +1,14 @@
use anyhow::bail;
use reqwest::header::{ACCEPT, AUTHORIZATION, HeaderMap, HeaderValue, USER_AGENT};
use reqwest::{Client, Response};
use serde::Serialize;
use std::sync::Arc;
use tokio::sync::{OwnedSemaphorePermit, Semaphore};
/// Shown when a request was refused twice for a token that was current when it left.
pub const SERVER_BUSY_MESSAGE: &str = "The server took too long to start handling the request and may be overloaded. \
Try again, or reduce the number of parallel transfers.";
fn build_user_agent() -> String {
let info = os_info::get();
let mut parts = vec![info.os_type().to_string()];
@@ -90,6 +95,12 @@ impl MDRSConnection {
}
}
/// Same as `connection_with_fresh_token`, for callers that manage their own retry
/// because the request body cannot simply be sent twice.
pub async fn connection_with_fresh_token_for_retry(&self) -> Result<Self, anyhow::Error> {
self.connection_with_fresh_token().await
}
async fn connection_with_fresh_token(&self) -> Result<Self, anyhow::Error> {
match (&self.remote, &self.token) {
(Some(remote), Some(_)) => {
@@ -122,56 +133,87 @@ impl MDRSConnection {
headers
}
pub async fn get(&self, path: &str) -> Result<Response, anyhow::Error> {
/// Send a request, and send it once more if the server found the access token
/// expired. A request can wait in the server's queue for longer than the token it
/// was sent with lives, so a refusal here need not mean the session is over.
///
/// The retry goes back through `connection_with_fresh_token`, which re-reads the
/// cache under the lock: if another process refreshed while this request waited,
/// its token is reused rather than a second one being minted.
async fn send_with_retry<F>(&self, build: F) -> Result<Response, anyhow::Error>
where
F: Fn(&MDRSConnection) -> reqwest::RequestBuilder,
{
let conn = self.connection_with_fresh_token().await?;
Ok(conn
.client
.get(conn.build_url(path))
.headers(conn.prepare_headers())
.send()
.await?)
let response = build(&conn).send().await?;
if response.status() != reqwest::StatusCode::UNAUTHORIZED {
return Ok(response);
}
// Only a stale access token is worth a second attempt, and the sign of one is a
// different token coming back. A wrong folder password, an anonymous request, or
// a session that has really ended all leave it untouched, and those refusals
// belong to the caller to report.
let retry_conn = self.connection_with_fresh_token().await?;
if retry_conn.token == conn.token {
return Ok(response);
}
let response = build(&retry_conn).send().await?;
if response.status() == reqwest::StatusCode::UNAUTHORIZED {
// Refused twice, the second time with a token that was current when it left.
// The session is fine; the server is not starting requests before their
// credentials lapse, which is worth saying plainly rather than sending the
// user off to log in again.
bail!(SERVER_BUSY_MESSAGE);
}
Ok(response)
}
pub async fn get(&self, path: &str) -> Result<Response, anyhow::Error> {
self.send_with_retry(|conn| {
conn.client
.get(conn.build_url(path))
.headers(conn.prepare_headers())
})
.await
}
pub async fn get_with_query<Q>(&self, path: &str, query: &Q) -> Result<Response, anyhow::Error>
where
Q: Serialize + ?Sized,
{
let conn = self.connection_with_fresh_token().await?;
Ok(conn
.client
.get(conn.build_url(path))
.headers(conn.prepare_headers())
.query(query)
.send()
.await?)
self.send_with_retry(|conn| {
conn.client
.get(conn.build_url(path))
.headers(conn.prepare_headers())
.query(query)
})
.await
}
pub async fn get_url(&self, url: &str) -> Result<Response, anyhow::Error> {
let conn = self.connection_with_fresh_token().await?;
Ok(conn
.client
.get(if url.starts_with("http") {
url.to_string()
} else {
conn.build_url(url)
})
.headers(conn.prepare_headers())
.send()
.await?)
self.send_with_retry(|conn| {
conn.client
.get(if url.starts_with("http") {
url.to_string()
} else {
conn.build_url(url)
})
.headers(conn.prepare_headers())
})
.await
}
pub async fn post_json<B>(&self, path: &str, body: &B) -> Result<Response, anyhow::Error>
where
B: Serialize + ?Sized,
{
let conn = self.connection_with_fresh_token().await?;
Ok(conn
.client
.post(conn.build_url(path))
.headers(conn.prepare_headers())
.json(body)
.send()
.await?)
self.send_with_retry(|conn| {
conn.client
.post(conn.build_url(path))
.headers(conn.prepare_headers())
.json(body)
})
.await
}
pub async fn post_multipart(
@@ -190,13 +232,12 @@ impl MDRSConnection {
}
pub async fn delete(&self, path: &str) -> Result<Response, anyhow::Error> {
let conn = self.connection_with_fresh_token().await?;
Ok(conn
.client
.delete(conn.build_url(path))
.headers(conn.prepare_headers())
.send()
.await?)
self.send_with_retry(|conn| {
conn.client
.delete(conn.build_url(path))
.headers(conn.prepare_headers())
})
.await
}
pub async fn delete_with_query<Q>(
@@ -207,13 +248,48 @@ impl MDRSConnection {
where
Q: Serialize + ?Sized,
{
let conn = self.connection_with_fresh_token().await?;
Ok(conn
.client
.delete(conn.build_url(path))
.headers(conn.prepare_headers())
.query(query)
.send()
.await?)
self.send_with_retry(|conn| {
conn.client
.delete(conn.build_url(path))
.headers(conn.prepare_headers())
.query(query)
})
.await
}
}
#[cfg(test)]
mod retry_tests {
use super::*;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
/// A refusal that no new token could answer belongs to the caller: an anonymous
/// request, a wrong folder password and an ended session all look like this, and
/// sending the same thing twice would only repeat the refusal.
#[tokio::test]
async fn a_refusal_is_handed_back_when_the_token_cannot_change() {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut buf = [0u8; 4096];
let _ = stream.read(&mut buf).await.unwrap();
stream
.write_all(
b"HTTP/1.1 401 Unauthorized\r\ncontent-length: 0\r\nconnection: close\r\n\r\n",
)
.await
.unwrap();
});
// No remote and no token, so nothing can be refreshed and the request must not
// be sent a second time. The stub answers once and would hang on a retry.
let conn = MDRSConnection::new(&format!("http://{addr}"));
let response = conn.get("v3/anything/").await.unwrap();
server.await.unwrap();
assert_eq!(response.status(), reqwest::StatusCode::UNAUTHORIZED);
}
}
+80
View File
@@ -1,3 +1,33 @@
use anyhow::anyhow;
use serde::Deserialize;
/// The error envelope the API answers with (drf-standardized-errors).
#[derive(Deserialize)]
struct ApiErrors {
errors: Vec<ApiError>,
}
#[derive(Deserialize)]
struct ApiError {
detail: String,
}
/// Turn a refused response into an error carrying what the server actually said.
///
/// A status code on its own leaves the user guessing. The reason - a permission, a
/// quota, a name already taken - is in the body the API sends along with it, and that
/// is the part worth showing.
pub async fn response_error(context: &str, resp: reqwest::Response) -> anyhow::Error {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
match serde_json::from_str::<ApiErrors>(&body) {
Ok(parsed) if !parsed.errors.is_empty() => {
anyhow!("{}: {}", context, parsed.errors[0].detail)
}
_ => anyhow!("{}: {}", context, status),
}
}
/// Print the error message and exit with code 2.
/// JSON deserialization errors produce a friendlier message matching Python's
/// JSONDecodeError handling.
@@ -23,3 +53,53 @@ fn is_json_error(e: &(dyn std::error::Error + 'static)) -> bool {
}
false
}
#[cfg(test)]
mod response_error_tests {
use super::*;
async fn refused_with(body: &'static str, content_type: &'static str) -> String {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut buf = [0u8; 4096];
let _ = stream.read(&mut buf).await.unwrap();
let response = format!(
"HTTP/1.1 403 Forbidden\r\ncontent-type: {}\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{}",
content_type,
body.len(),
body
);
stream.write_all(response.as_bytes()).await.unwrap();
});
let resp = reqwest::get(format!("http://{addr}/")).await.unwrap();
let message = response_error("Upload failed", resp).await.to_string();
server.await.unwrap();
message
}
/// The reason the server gave is what the user needs, not the number that carried it.
#[tokio::test]
async fn the_servers_own_message_is_reported() {
let message = refused_with(
r#"{"type":"client_error","errors":[{"code":"permission_denied","detail":"You do not have permission.","attr":null}]}"#,
"application/json",
)
.await;
assert_eq!(message, "Upload failed: You do not have permission.");
}
/// A gateway that answers in HTML has nothing to quote, so the status still stands in.
#[tokio::test]
async fn the_status_stands_in_when_the_body_says_nothing() {
let message = refused_with("<html>nope</html>", "text/html").await;
assert_eq!(message, "Upload failed: 403 Forbidden");
}
}
+1 -1
View File
@@ -201,7 +201,7 @@ fn run(cli: Cli) {
}
Commands::SelfUpdate { yes } => {
if let Err(e) = build_rt().block_on(commands::selfupdate::selfupdate(yes)) {
handle_error(e.into());
handle_error(e);
}
}
}
+5 -8
View File
@@ -13,14 +13,11 @@ impl Settings {
fn load() -> Self {
let config_dirname = std::env::var("MDRS_CLIENT_CONFIG_DIRNAME")
.ok()
.map(|s| {
if s.starts_with("~/") {
dirs::home_dir()
.unwrap_or_else(|| std::path::PathBuf::from("."))
.join(&s[2..])
} else {
std::path::PathBuf::from(&s)
}
.map(|s| match s.strip_prefix("~/") {
Some(rest) => dirs::home_dir()
.unwrap_or_else(|| std::path::PathBuf::from("."))
.join(rest),
None => std::path::PathBuf::from(&s),
})
.unwrap_or_else(|| {
dirs::home_dir()