refactor(mimefactory): separate rendering of message payload and sendable message - #8345
Conversation
45102bc to
1956c42
Compare
5f3c57c to
f9c33de
Compare
f9c33de to
e314fc0
Compare
3d4610c to
b1b0e3f
Compare
ce43887 to
6abd90c
Compare
6abd90c to
e350dd6
Compare
a18ab11 to
2e20218
Compare
a39da96 to
1c19dd8
Compare
598ac45 to
72d547a
Compare
Could you check if removing compression support from Asymmetric encryption is constant-time and should not depend on the message size, and symmetric AES encryption is supposed to be cheap and not taking seconds for 10 MB message, so I suspect the slowest part is OpenPGP compression. If it is that slow, it's one more reason to get rid of it. For encrypted attachments we don't need to worry about compatibility, so can even send them as binary MIME without base64-encoding, this will also reduce memory usage for the receivers and may help iOS. Large attachments are likely don't compress well as videos and images are already compressed, webxdcs are zip archives, and zlib compression cannot compress them further, so even just dropping compression already might be an improvement. The only reason for OpenPGP compression currently is compensating base64-encoding of binary attachments. Attachments are base64-encoded, then compressed, then signed and encrypted, then the whole encrypted messages is ASCII-armored. Compression negates the first base64-encoding, but we don't have to do it in the first place, mailparse at least theoretically supports binary attachments. |
|
Removing compression only helped a little bit, reducing the time from 3.5s to 2.6s. Additionally Ordering Rust to optimize for speed rather than binary size gets it down to 1.6s (I did not check how this affects the APK size). |
de45e20 to
95aba6d
Compare
af647e7 to
5eb92c2
Compare
5eb92c2 to
eef6ea4
Compare
hpk42
left a comment
There was a problem hiding this comment.
To me this looks good modulo a couple of medium/minor nits. The unencrypted MDN is probably most relevant, and then maybe the missing block_in_place. I am also in favor of merging this PR soon, and run main on some dev devices.
| // Never add outer multipart/mixed wrapper to MDN | ||
| // as multipart/report Content-Type is used to recognize MDNs | ||
| // by Delta Chat receiver and Chatmail servers | ||
| // allowing them to be unencrypted and not contain Autocrypt header | ||
| // without resetting Autocrypt encryption or triggering Chatmail filter | ||
| // that normally only allows encrypted mails. | ||
|
|
||
| // Hidden headers are dropped. | ||
| message |
There was a problem hiding this comment.
Pretty sure we want to get rid of unencrypted MDNs completely. Not implemented yet but it's better to just bail out here and drop the complex comment.
| // MDNs are only sent encrypted. Unencrypted MDNs are being | |
| // phased out but not fully prevented yet, so just refuse to render them here. | |
| bail!("unencrypted MDNs are not sent"); | |
| } |
There was a problem hiding this comment.
I think these changes should be outside of this PR, i'm trying to keep this a refactoring so it breaks as little things as possible.
| msg.chat_id | ||
| .set_selfavatar_timestamp(context, now) | ||
| .await | ||
| .context("Failed to set selfavatar timestamp")?; |
There was a problem hiding this comment.
why is this changed from just logging an error in the background to aborting create_send_msg_jobs?
There was a problem hiding this comment.
I assume that the reason is that if one SQL statement errors, then there is usually not much point in continuing to try using the database (except there is a syntax error in the SQL statement, but then every test touching this code would fail)
| } | ||
| unprotected_headers.push(("To", hidden_recipients().into())); | ||
| } else if header_name == "chat-broadcast-secret" { | ||
| if is_encrypted { |
There was a problem hiding this comment.
not sure about just dropping this guard. the secret should never go out in unencrypted messages. Maybe some ensure_and_debug_assert!(...no-Chat-Broadcast-Secret..) in the unencrypted path makes sense?
There was a problem hiding this comment.
There is no is_encrypted in this function, the whole function should be removed together with hidden headers in some follow-up.
| let date = chrono::DateTime::<chrono::Utc>::from_timestamp(self.timestamp, 0) | ||
| .unwrap() | ||
| .to_rfc2822(); | ||
| headers.push(("Date", mail_builder::headers::raw::Raw::new(date).into())); |
There was a problem hiding this comment.
Date now precedes To/Subject? nobody should care, just unusual for old-school readers like me :)
There was a problem hiding this comment.
Yes, Date cannot go below To without parsing headers because it is rendered late.
| .iter() | ||
| .map(|(name, addr)| { | ||
| Address::new_address( | ||
| if name.is_empty() { | ||
| None | ||
| } else { | ||
| Some(name.to_string()) | ||
| }, | ||
| addr.clone(), | ||
| ) | ||
| }) | ||
| .collect() |
There was a problem hiding this comment.
sidenote: i really dislike such waste of vertical lines, as it impedes readability. It's not new code but while i am at it, i think something like self.to.iter().map(to_address).collect() would be preferrable (with an approprirate to_address helper).
There was a problem hiding this comment.
Makes sense. This becomes easier because there already is fn new_address_with_name, although can't be used as .map(new_address_with_name) takes (&str, String) rather than (&String, &String)
| msg.chat_id | ||
| .set_selfavatar_timestamp(context, now) | ||
| .await | ||
| .context("Failed to set selfavatar timestamp")?; |
There was a problem hiding this comment.
I assume that the reason is that if one SQL statement errors, then there is usually not much point in continuing to try using the database (except there is a syntax error in the SQL statement, but then every test touching this code would fail)
| .iter() | ||
| .map(|(name, addr)| { | ||
| Address::new_address( | ||
| if name.is_empty() { | ||
| None | ||
| } else { | ||
| Some(name.to_string()) | ||
| }, | ||
| addr.clone(), | ||
| ) | ||
| }) | ||
| .collect() |
There was a problem hiding this comment.
Makes sense. This becomes easier because there already is fn new_address_with_name, although can't be used as .map(new_address_with_name) takes (&str, String) rather than (&String, &String)
cd0eb02 to
6515b30
Compare
This adds a test for the unencrypted headers, because #8345 changes how these are rendered, and so far we didn't have any tests for them.
…able message This change separates rendering into two separate steps: 1. Rendering of the message payload without the From, Date and Autocrypt headers. 2. Adding the From, Date and Autocrypt headers and possibly encrypting the message. The goal is to have serializable result of the first step that can be persisted in the database and sent later with any email address. This way it will be possible to send queued messages over any relay. This will make it possible not to remove all messages from the queue when the sending relay is changed. Currently changing `configured_addr` deletes everything from `smtp` table. This change is however only a refactoring and does not implement any features.
This adds a test for the unencrypted headers, because #8345 changes how these are rendered, and so far we didn't have any tests for them.
3b0555d to
3891ceb
Compare
This change separates rendering into two separate steps:
The goal is to have serializable result of the first step
that can be persisted in the database and sent later with any email address.
This way it will be possible to send queued messages over any relay.
This will make it possible not to remove all messages from the queue
when the sending relay is changed.
Currently changing
configured_addrdeletes everything fromsmtptable.This change is however only a refactoring and does not implement any features.
This is a refactoring PR in preparation for automatic relay failover.
As a side effect it also makes possible to change the Date of the message for #8112 if we decide on this approach (unlikely).
Serializable mail is currently called
mimefactory::QueuedMail. Everything except the public keys is trivially serializable, public keys should likely be serialized as recipient fingerprints rather than as OpenPGP certificates.I also noticed that we likely can remove the concept of "hidden headers" which are headers that are sent on the mulipart/mixed level of unencrypted messages. They are used to send
Chat-Editheaders and avatars in unencrypted messages. Sending avatars in unencrypted messages is not useful because they are not displayed anyway. And we can decide to make it impossible to edit and delete unencrypted messages. I have not changed anything in this PR, however, hidden headers work as before.