Fundamentally that wouldn’t work because stalwart always buffers requests and responses into RAM. This also means that someone with Permission::UnlimitedUploads can OOM Stalwart, which is neat.
I propose that a newtype be created that’s an enum of either Vec<u8> or some kind of AsyncRead which will be used in BlobStore.put_blob and BlobStore.get_blob. http_proto::request::fetch_body would have to be changed to have this newtype too. Said newtype could also have a method to coerce into a Vec<u8> for “normal” sized requests.
The back-ends for BlobStore::Fs, BlobStore::S3, and BlobStore::Azure all seem to work pretty well with ranges too, one could create a readable stream that only fetches within the specified range.
Though there’s some caveats with uploading new blobs
This won’t be useful in deployments that re-use the data store as the blob store. (How would this be handled? Error?)
The stream will have to be reset-able since…
BlobHash’s need to be computed before the blobs are committed
It’s a requirement for azure
Since the streams need to be reset-able when creating blobs, maybe requests could be buffered to temporary files instead of RAM if the in-memory size limit is exceeded. From my exploration so far, streams that would be created by fetching blobs don’t seem to need to be reset-able.
Been cooking for a bit, will post my results soon.
Note to self, larger-than-ram file uploads using JAMP can’t be done with serde, as serde always buffers the whole string (which is the JSON parser needs to un-base-64-ify), and I don’t feel like re-writing the JMAP request pareser right now. Claude suggested struson as a solution but the crate considers itself unstable.
Either way, it seems like my desire to share files that are 10s of GBs using stalwart will be limited to WebDAV-only for now.
Edit: Nope, that apparently only applies to the a POST request on /jmap. /jmap/upload allows the bytes as-is. ngl pretty excited about this.
Hello @mdecimus! (apologies if this ping is inappropriate)
It was a lot of work, but I managed to get a version of stalwart that ostensibly supports large files to compile! This is branched from 0.16.17
High-level changes (in no particular order):
Some new structs have been created
utils::jumbo_bytes::JumboBytesMut: Basically a Cursor<Vec<u8>> when the buffer size is less than a (currently hard-coded) size, or a tokio::fs::File otherwise, pointing to a temporary file. Currently hardcoded to reside in std::env::tmpdir().
The idea is that the spillover threshold and the tempdir would be user-configurable, perhaps defaulting to 32 bytes larger than the largest size limit. (This is to allow people to downgrade, more on that later)
The spillover threshold should still be greater than or equal to most size limits (mail, sieve scripts, calendar events, etc.) since I didn’t touch their parsers, and dumping to disk only to buffer it again would be inefficient.
I considered having the spillover go to the configured blob store, but creating a seek+read+write IO stream would have been a lot more work, especially considering Azure. Oh god, Azure.
utils::jumbo_bytes::JumboBytes: Clonable, read-only counterpart. This only exists because Azure wants a seekable and clonable read stream
utils::jumbo_bytes::JumboBytesMutSync: Same as JumboBytesMut, but is a std::fs::File is the spillover threshold was reached. This was created for the backup/restore system, as that used a thread without a tokio runtime to do the file writing.
store::async_lz4::{AsyncFrameDecoder, AsyncFrameEncoder}: Async wrappers for lz4_flex::frame::{FrameEncoder, FrameDecoder}. This was needed to support compressed blobs larger than the spillover threshold. It’s a little hacky, a Mutex had to be used in order to compress/decompress in a non-blocking fashion. Note that this introduces a file suffix LZ4_STREAM_MARKER. More on that later.
groupware::file::File: has been altered to use a u64 for file sizes. (de)serialization is backward/forward compatible as long as the file size is below u32::MAX
http_proto::HttpResponseBodyError: currently an alias for std::io::Error. Possible as no instances of hyper::Error were actually constructed during a response stream. This was needed to support BlobReadStream
store::stream::BlobReadStream: a read-only non-seekable stream, represents the result of a (potentially partial) blob read.
Some function signatures and types were changed
http_proto::HttpResponse was reworked to use a http_body_util::combinators::UnsyncBoxBody<Bytes, HttpResponseBodyError> instead of a http_body_util::combinators::BoxBody. These are the same except UnsyncBoxBody is less restrictive, which was needed to support store::stream::BlobReadStream as a response body, because the readable s3 stream doesn’t implement Sync. The error type was changed because store::stream::BlobReadStream may return std::io::Error while streaming, and as previously noted, hyper::Error wasn’t actually constructed anywhere.
store::BlobStore::get_blob was changed to Fn(&self, &[u8], Range<u64>) -> trc::Result<Option<BlobReadStream, u64)>>. Where the returned u64 represents the total size of the stream
store::BlobStore::put_blob was changed to take a JumboBytesMut instead of a &[u8] for the data param.
This function always cloned the data anyway, so having it require owned data at least makes it more honest.
This function also uses the new LZ4_STREAM_MARKER if the JumboBytesMut’s spillover threshold was reached and compression is enabled.
Various size-related counters (quota, etc.) were updated to use a u64 instead of usize where needed.
common::Manager::Core was reworked to use JumboBytesMutSync for it’s backup/restore functions.
http_proto::request::fetch_body now returns a JumboBytesMut
…and all the side-effects that entails.
WebDAV PUT/POST and PATCH requests now use groupware.max_file_sizeinstead ofgroupware.max_request_size
Some functions were added
store::BlobStore::get_blob_length gets the uncompressed blob length without actually need to read the full blob to do so.
store::BlobStore::get_blob_vec has store::BlobStore::get_blob’s old behaviour.
BlobHash::generate_from_stream: does what it says, requires a seekable read stream (like JumboBytesMut)
http_proto::HttpResponse::with_io_body takes a Box<dyn AsyncRead + Send + 'static> and optional length. Used for BlobReadStream, could be used for others in the future.
Current downsides
store::BlobStore::get_blob now does 3 requests when using s3.
HEAD the blob to get the size
GET the last byte of the blob in order to get the compression type
GET the (potentially partial) blob with the last byte omitted, decompress as-needed.
JumboBytesMut spillover threshold and tempfile dir is currently hard-coded. Though this won’t be the case for long.
Server::blob_get still buffers the entire underlying blob since that what the JSON serializer needs. We’ll have to place size limitations on reading blobs in this case. Haven’t 100% figured out what to enfoce yet, would appreciate feedback. ./jmap/download still streams fine, though.
I really didn’t do anything to improve the “use the data store as the blob store” situation beyond store::BlobStore::get_blob_length. In fact, partial requests still get the whole blob internally.
Backwards/Forwards compatibility.
As long as users don’t save files larger than 4GiB, or save data larger than JumboByteMut’s spillover threshold while blob compression is enabled, or save files larger than the system memory accessible to stalwart, users should be able to downgrade freely. I did my best to make sure that any points of no return would be fully opt-in. (As soon as those 2 new configs are added)
Please let me know what you think. Yeah, this touched more things than I’d like, but I’m still eager for any feedback to better help stalwart support large files. Happy to discuss further or submit a PR anytime!
I’ve kept the thing squashed to a single commit to make rebasing easier. Here it is!
I plan to keep up with upstream every time a new release is created.
I wasn’t able to create a new config option as that requires changing the schema which I don’t have access to, so the tempfile dir is set with an env var, and it’s currently hard-coded to spill over after 1MiB.
Ha, apologies (and thank you for visiting my thread of ramblings), to really condense over a week’s worth of effort:
u64’s for file storage sizes and quotas
don’t buffer whole blobs in RAM when possible, either just use AsyncRead/AsyncWrite streams or create a temporary file if needed.
I’m actively storing files that are larger than total system RAM right now.
The giant list I’ve made a couple posts ago just went into the specifics of what had to be done to achieve this. (Though my approach to Server::blob_get is different now than it was before) I’m personally using the fs blob store uncompressed, but I made sure to make it work with all other blob stores with compression enabled.
