Ignore invalid SMTP EHLO response lines

Fixes #6062

EHLO keyword contains invalid character

EHLO keyword contains invalid character
### Checklist

- [X] I have used the search function to see if someone else has already submitted the same bug report.
- [X] I will describe the problem with as much detail as possible.

### App version

d13352a4cd7f5ea540eba738155ff377ee48da62 - 2205202769770bc37feb98414bb8a1ae084b9c98

### Where did you get the app from?

Other

### Android version

11 (LineageOS for microG)

### Device model

Fairphone 3

### Steps to reproduce

1. Set up an email account
2. Go to outgoing server settings
3. Fill in everything and click next

SMTP server: `<REDACTED>`
Security: SSL/TLS
Port: 465
Require sign-in: Yes
Username: `<REDACTED>`
Authentication: Normal password
Password: `<REDACTED>`
Client certificate: No client certificate

Note: This happens on my own self-hosted mailserver, not on a public provider. I obviously don't want to make authentication info for it public, but if it helps to debug please send me an email and I'll get you some login credentials.

### Expected behavior

Succesfully set up SMTP, like it did on K-9 Mail v6.000

### Actual behavior

An error message shows:

# Setup could not finish
Cannot connect to server.
(EHLO keyword contains invalid character)
[ Continue ] [ Edit Details ]

![image](https://user-images.githubusercontent.com/1885159/167604225-40801927-8e97-44b4-b177-0732d62d22df.png)

### Logs

Using git bisect, I was able to pinpoint it to the following commit:

```
d13352a4cd7f5ea540eba738155ff377ee48da62 is the first bad commit
commit d13352a4cd7f5ea540eba738155ff377ee48da62
Author: cketti <cketti@gmail.com>
Date:   Wed Mar 23 04:07:37 2022 +0100

    Use new `SmtpResponseParser` in `SmtpTransport`

 .../smtp/EnhancedNegativeSmtpReplyException.java   |  12 +-
 .../fsck/k9/mail/transport/smtp/SmtpResponse.kt    |   3 +
 .../fsck/k9/mail/transport/smtp/SmtpTransport.java | 268 +++++++--------------
 .../k9/mail/transport/smtp/StatusCodeClass.java    |  12 -
 .../k9/mail/transport/smtp/StatusCodeDetail.java   |  80 ------
 .../k9/mail/transport/smtp/StatusCodeSubject.java  |  31 ---
 .../k9/mail/transport/smtp/SmtpTransportTest.java  |   8 +-
 7 files changed, 100 insertions(+), 314 deletions(-)
 delete mode 100644 mail/protocols/smtp/src/main/java/com/fsck/k9/mail/transport/smtp/StatusCodeDetail.java
 delete mode 100644 mail/protocols/smtp/src/main/java/com/fsck/k9/mail/transport/smtp/StatusCodeSubject.java
```

Please note: In the attached log file, I replaced the mailserver identification with `mail.example.com` and the Hello's hostname and IP with `<REDACTED>`.

[k9-log-redacted.txt](https://github.com/k9mail/k-9/files/8660865/k9-log-redacted.txt)

The server's response to the `EHLO` command starts with:

```text
250-mail.example.com Hello <REDACTED> [<REDACTED>]
250-SIZE 52428800
250-8BITMIME
250-PIPELINING
250-PIPE_CONNECT
```

`PIPE_CONNECT` contains an underscore, which is not allowed. See https://datatracker.ietf.org/doc/html/rfc5321#section-4.1.1.1

It looks like this is a known issue that hasn't received much attention: https://www.mail-archive.com/exim-users@exim.org/msg56497.html

I'd prefer for this to be fixed in exim rather than working around the issue in K-9 Mail. When I rewrote the SMTP response parser, I made it throw an error rather than ignore protocol violations to uncover cases like this – so that the code can be fixed.

Unfortunately, it will be a while before I have time to work on trying to get the issue fixed in exim. Given that this involves changing the keyword `PIPE_CONNECT` to something else, maybe you could get the ball rolling by starting a discussion with the exim developers. If not a lot of people are using the extension, a name change could be relatively simple. If the extension has a significant number of users, a migration strategy might be required.
Thanks for the detailed response. I agree that standard-compliance is important and that this is technically not a K-9 Mail issue but an Exim one. I do however worry that K-9 enforcing this compliance the next release will break the use case of sending emails for many users.

**How common is this misconfiguration?**

In my case, I run Exim4 as configured by MyVestaCP and looking through my Exim4 config, the related config options named in https://www.exim.org/exim-html-current/doc/html/spec_html/ch-main_configuration.html (`pipelining_connect_advertise_hosts` and `hosts_pipe_connect`) are not configured anywhere.

Grabbing [the exim4-config (4.94.2-7) package from Debian bullseye](https://packages.debian.org/bullseye/exim4-config) and grepping through it, I don't see any related options defined anywhere either, so I would assume this is the default configuration for at very least Debian Stable as well.

**Are there any other issues on this configuration?**

Just to try, I have edited the `checkAndNormalizeEhloKeyword` function in K-9 Mail to allowlist the `PIPE_CONNECT` keyword as follows:
```
    private fun checkAndNormalizeEhloKeyword(text: String): String {
        val keyword = text.uppercase()
        if (!keyword[0].isCapitalAlphaDigit() || keyword.any { !it.isCapitalAlphaDigit() && it != DASH }) {
            if (keyword != "PIPE_CONNECT") {
                parserError("EHLO keyword contains invalid character")
            }
        }

        return keyword
    }
```

Doing this allows K-9 Mail to connect, so it seems to me like `PIPE_CONNECT` is the only compliance issue Exim currently has (as far as K-9 can detect).

**Difficulty for me reporting this issue upstream**

Frankly, I don't know that much about email so even though this is apparently enabled on my server I have no idea if I am actively using this or not. As the server is in use by more people than just me, I also can't simply turn it off and see if anything breaks. Given my limited knowledge, I doubt I can have any productive discussion with the Exim developers about this, as I wouldn't be able to answer many "basic" questions.

Currently, my understanding of this subject pretty much boils down to:
1. Exim4's default configuration sends a non-compliant keyword
2. K-9 Mail was likely not using this extensions, as previous versions simply ignored invalid options and thus K-9 probably never understood this extension in the first place

As such, all I could really do on the Exim side is create an issue on https://bugs.exim.org/buglist.cgi?quicksearch=PIPE_CONNECT stating the above.

Given this, I believe it would be more sensible for someone with more knowledge about email to report this issue.

**My suggestion**

Given the scale of this misconfiguration and the otherwise seeming compliance, would it be reasonable for K-9 Mail to ignore only `PIPE_CONNECT` until this gets resolved upstream? That way other issues should still become visible without breaking K-9 Mail for users.
You can fix your server easily, I had the same issue on my self hosted mail server, if you are running postfix, open /etc/postfix/main.cf and look for "broken_sasl_auth_clients = yes" then change it to  "broken_sasl_auth_clients = no" restart the postfix service and dovecot and your issue will be gone.

check  https://www.postfix.org/postconf.5.html for more information.
In my case I am using Exim4. Are you sure the Postfix issue you're running into is the exact same issue? If you check logs, do you also see `250-PIPE_CONNECT`?
I got the same  (Cannot connect to server.
(EHLO keyword contains invalid character) sorry  I did not check the logcat.
With that option enabled Postfix seems to add a syntactically invalid keyword (starting with `AUTH=`):

https://github.com/vdukhovni/postfix/blob/abf156923c91c8c4f2bbd1a591114eb45bb77e81/postfix/src/smtpd/smtpd.c#L1952-L1953

./gradlew :mail:protocols:smtp:test --tests "com.fsck.k9.mail.transport.smtp.SmtpResponseParserTest"

diff --git a/mail/protocols/smtp/src/main/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParser.kt b/mail/protocols/smtp/src/main/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParser.kt
index e67febebd16..a7ceefaa101 100644
--- a/mail/protocols/smtp/src/main/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParser.kt
+++ b/mail/protocols/smtp/src/main/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParser.kt
@@ -115,20 +115,25 @@ internal class SmtpResponseParser(

     private fun parseEhloLine(ehloLine: String, keywords: MutableMap<String, List<String>>) {
         val parts = ehloLine.split(" ")
-        val keyword = checkAndNormalizeEhloKeyword(parts[0])
-        val parameters = checkEhloParameters(parts)

-        if (keywords.containsKey(keyword)) {
-            parserError("Same EHLO keyword present in more than one response line")
-        }
+        try {
+            val keyword = checkAndNormalizeEhloKeyword(parts[0])
+            val parameters = checkEhloParameters(parts)
+
+            if (keywords.containsKey(keyword)) {
+                parserError("Same EHLO keyword present in more than one response line", logging = false)
+            }

-        keywords[keyword] = parameters
+            keywords[keyword] = parameters
+        } catch (e: SmtpResponseParserException) {
+            logger.log(e, "Ignoring EHLO keyword line: $ehloLine")
+        }
     }

     private fun checkAndNormalizeEhloKeyword(text: String): String {
         val keyword = text.uppercase()
         if (!keyword[0].isCapitalAlphaDigit() || keyword.any { !it.isCapitalAlphaDigit() && it != DASH }) {
-            parserError("EHLO keyword contains invalid character")
+            parserError("EHLO keyword contains invalid character", logging = false)
         }

         return keyword
@@ -138,9 +143,9 @@ internal class SmtpResponseParser(
         for (i in 1..parts.lastIndex) {
             val parameter = parts[i]
             if (parameter.isEmpty()) {
-                parserError("EHLO parameter must not be empty")
+                parserError("EHLO parameter must not be empty", logging = false)
             } else if (parameter.any { it.code !in 33..126 }) {
-                parserError("EHLO parameter contains invalid character")
+                parserError("EHLO parameter contains invalid character", logging = false)
             }
         }

diff --git a/mail/protocols/smtp/src/test/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParserTest.kt b/mail/protocols/smtp/src/test/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParserTest.kt
index e3f349201c6..9b813c0ae5c 100644
--- a/mail/protocols/smtp/src/test/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParserTest.kt
+++ b/mail/protocols/smtp/src/test/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParserTest.kt
@@ -145,26 +145,36 @@ class SmtpResponseParserTest {
     }

     @Test
-    fun `read EHLO response with invalid keyword`() {
+    fun `read EHLO response with invalid keywords`() {
         val input = """
             250-smtp.domain.example
-            250 KEY:WORD
+            250-SIZE 52428800
+            250-8BITMIME
+            250-PIPELINING
+            250-PIPE_CONNECT
+            250-AUTH=PLAIN
+            250 HELP
             """.toPeekableInputStream()
         val parser = SmtpResponseParser(logger, input)

-        assertFailsWithMessage("EHLO keyword contains invalid character") {
-            parser.readHelloResponse()
-        }
+        val response = parser.readHelloResponse()

-        assertThat(logger.logEntries).containsExactly(
-            LogEntry(
-                throwable = null,
-                message = """
-                    SMTP response data on parser error:
-                    250-smtp.domain.example
-                    250 KEY:WORD
-                """.trimIndent()
+        assertType<SmtpHelloResponse.Hello>(response) { hello ->
+            assertThat(hello.keywords.keys).containsExactly(
+                "SIZE",
+                "8BITMIME",
+                "PIPELINING",
+                "HELP"
             )
+        }
+
+        assertThat(logger.logEntries.map { it.message }).containsExactly(
+            "Ignoring EHLO keyword line: PIPE_CONNECT",
+            "Ignoring EHLO keyword line: AUTH=PLAIN"
+        )
+        assertThat(logger.logEntries.map { it.throwable?.message }).containsExactly(
+            "EHLO keyword contains invalid character",
+            "EHLO keyword contains invalid character"
         )
     }

@@ -176,44 +186,37 @@ class SmtpResponseParserTest {
             """.toPeekableInputStream()
         val parser = SmtpResponseParser(logger, input)

-        assertFailsWithMessage("EHLO parameter must not be empty") {
-            parser.readHelloResponse()
+        val response = parser.readHelloResponse()
+
+        assertType<SmtpHelloResponse.Hello>(response) { hello ->
+            assertThat(hello.keywords.keys).isEmpty()
         }

-        assertThat(logger.logEntries).containsExactly(
-            LogEntry(
-                throwable = null,
-                message = """
-                    SMTP response data on parser error:
-                    250-smtp.domain.example
-                    250 KEYWORD${" "}
-                """.trimIndent()
-            )
-        )
+        assertThat(logger.logEntries).hasSize(1)
+        assertThat(logger.logEntries.first().throwable).hasMessageThat().isEqualTo("EHLO parameter must not be empty")
+        assertThat(logger.logEntries.first().message).isEqualTo("Ignoring EHLO keyword line: KEYWORD ")
     }

     @Test
     fun `read EHLO response with invalid parameter`() {
         val input = """
             250-smtp.domain.example
+            250-8BITMIME
             250 KEYWORD para${"\t"}meter
             """.toPeekableInputStream()
         val parser = SmtpResponseParser(logger, input)

-        assertFailsWithMessage("EHLO parameter contains invalid character") {
-            parser.readHelloResponse()
+        val response = parser.readHelloResponse()
+
+        assertType<SmtpHelloResponse.Hello>(response) { hello ->
+            assertThat(hello.keywords.keys).containsExactly("8BITMIME")
         }

-        assertThat(logger.logEntries).containsExactly(
-            LogEntry(
-                throwable = null,
-                message = """
-                    SMTP response data on parser error:
-                    250-smtp.domain.example
-                    250 KEYWORD para${"\t"}meter
-                """.trimIndent()
-            )
-        )
+        assertThat(logger.logEntries).hasSize(1)
+        assertThat(logger.logEntries.first().throwable)
+            .hasMessageThat().isEqualTo("EHLO parameter contains invalid character")
+        assertThat(logger.logEntries.first().message)
+            .isEqualTo("Ignoring EHLO keyword line: KEYWORD para${"\t"}meter")
     }

     @Test


Move `DefaultTrustedSocketFactory`

Closes #6012

Move `DefaultTrustedSocketFactory`

Move `DefaultTrustedSocketFactory`
`DefaultTrustedSocketFactory` depends on the Android SDK and therefore should not live inside the `:mail:common` Gradle project.

Related class: `KeyChainKeyManager`

./gradlew :mail:protocols:webdav:testDebugUnitTest --tests "com.fsck.k9.mail.store.webdav.WebDavStoreTest"

diff --git a/app/core/src/main/java/com/fsck/k9/KoinModule.kt b/app/core/src/main/java/com/fsck/k9/KoinModule.kt
index 05c30287cc1..85cacac5b97 100644
--- a/app/core/src/main/java/com/fsck/k9/KoinModule.kt
+++ b/app/core/src/main/java/com/fsck/k9/KoinModule.kt
@@ -2,7 +2,7 @@ package com.fsck.k9

 import android.content.Context
 import com.fsck.k9.helper.Contacts
-import com.fsck.k9.mail.ssl.DefaultTrustedSocketFactory
+import com.fsck.k9.helper.DefaultTrustedSocketFactory
 import com.fsck.k9.mail.ssl.LocalKeyStore
 import com.fsck.k9.mail.ssl.TrustManagerFactory
 import com.fsck.k9.mail.ssl.TrustedSocketFactory
diff --git a/mail/common/src/main/java/com/fsck/k9/mail/ssl/DefaultTrustedSocketFactory.java b/app/core/src/main/java/com/fsck/k9/helper/DefaultTrustedSocketFactory.java
similarity index 97%
rename from mail/common/src/main/java/com/fsck/k9/mail/ssl/DefaultTrustedSocketFactory.java
rename to app/core/src/main/java/com/fsck/k9/helper/DefaultTrustedSocketFactory.java
index 4c9857e024a..ce046daa9e1 100644
--- a/mail/common/src/main/java/com/fsck/k9/mail/ssl/DefaultTrustedSocketFactory.java
+++ b/app/core/src/main/java/com/fsck/k9/helper/DefaultTrustedSocketFactory.java
@@ -1,4 +1,4 @@
-package com.fsck.k9.mail.ssl;
+package com.fsck.k9.helper;

 import java.io.IOException;
@@ -14,6 +14,8 @@
 import android.text.TextUtils;

 import com.fsck.k9.mail.MessagingException;
+import com.fsck.k9.mail.ssl.TrustManagerFactory;
+import com.fsck.k9.mail.ssl.TrustedSocketFactory;
 import javax.net.ssl.KeyManager;
 import javax.net.ssl.SSLContext;
 import javax.net.ssl.SSLSocket;
diff --git a/mail/common/src/main/java/com/fsck/k9/mail/ssl/KeyChainKeyManager.java b/app/core/src/main/java/com/fsck/k9/helper/KeyChainKeyManager.java
similarity index 99%
rename from mail/common/src/main/java/com/fsck/k9/mail/ssl/KeyChainKeyManager.java
rename to app/core/src/main/java/com/fsck/k9/helper/KeyChainKeyManager.java
index bfd90c196c9..77e9fc4f834 100644
--- a/mail/common/src/main/java/com/fsck/k9/mail/ssl/KeyChainKeyManager.java
+++ b/app/core/src/main/java/com/fsck/k9/helper/KeyChainKeyManager.java
@@ -1,5 +1,5 @@

-package com.fsck.k9.mail.ssl;
+package com.fsck.k9.helper;

 import java.net.Socket;
 import java.security.Principal;
diff --git a/app/k9mail/src/main/java/com/fsck/k9/backends/KoinModule.kt b/app/k9mail/src/main/java/com/fsck/k9/backends/KoinModule.kt
index 8dd9438a3cc..5585a2bdcc1 100644
--- a/app/k9mail/src/main/java/com/fsck/k9/backends/KoinModule.kt
+++ b/app/k9mail/src/main/java/com/fsck/k9/backends/KoinModule.kt
@@ -5,7 +5,9 @@ import app.k9mail.dev.developmentModuleAdditions
 import com.fsck.k9.backend.BackendManager
 import com.fsck.k9.backend.imap.BackendIdleRefreshManager
 import com.fsck.k9.backend.imap.SystemAlarmManager
+import com.fsck.k9.helper.DefaultTrustedSocketFactory
 import com.fsck.k9.mail.store.imap.IdleRefreshManager
+import com.fsck.k9.mail.store.webdav.SniHostSetter
 import org.koin.dsl.module

 val backendsModule = module {
@@ -30,7 +32,19 @@ val backendsModule = module {
     single<SystemAlarmManager> { AndroidAlarmManager(context = get(), alarmManager = get()) }
     single<IdleRefreshManager> { BackendIdleRefreshManager(alarmManager = get()) }
     single { Pop3BackendFactory(get(), get()) }
-    single { WebDavBackendFactory(get(), get(), get()) }
+    single {
+        WebDavBackendFactory(
+            backendStorageFactory = get(),
+            trustManagerFactory = get(),
+            sniHostSetter = get(),
+            folderRepository = get()
+        )
+    }
+    single {
+        SniHostSetter { factory, socket, hostname ->
+            DefaultTrustedSocketFactory.setSniHost(factory, socket, hostname)
+        }
+    }

     developmentModuleAdditions()
 }
diff --git a/app/k9mail/src/main/java/com/fsck/k9/backends/WebDavBackendFactory.kt b/app/k9mail/src/main/java/com/fsck/k9/backends/WebDavBackendFactory.kt
index 0098b96e468..0e4e588e335 100644
--- a/app/k9mail/src/main/java/com/fsck/k9/backends/WebDavBackendFactory.kt
+++ b/app/k9mail/src/main/java/com/fsck/k9/backends/WebDavBackendFactory.kt
@@ -6,6 +6,7 @@ import com.fsck.k9.backend.api.Backend
 import com.fsck.k9.backend.webdav.WebDavBackend
 import com.fsck.k9.mail.ssl.TrustManagerFactory
 import com.fsck.k9.mail.store.webdav.DraftsFolderProvider
+import com.fsck.k9.mail.store.webdav.SniHostSetter
 import com.fsck.k9.mail.store.webdav.WebDavStore
 import com.fsck.k9.mail.transport.WebDavTransport
 import com.fsck.k9.mailstore.FolderRepository
@@ -14,6 +15,7 @@ import com.fsck.k9.mailstore.K9BackendStorageFactory
 class WebDavBackendFactory(
     private val backendStorageFactory: K9BackendStorageFactory,
     private val trustManagerFactory: TrustManagerFactory,
+    private val sniHostSetter: SniHostSetter,
     private val folderRepository: FolderRepository
 ) : BackendFactory {
     override fun createBackend(account: Account): Backend {
@@ -21,8 +23,8 @@ class WebDavBackendFactory(
         val backendStorage = backendStorageFactory.createBackendStorage(account)
         val serverSettings = account.incomingServerSettings
         val draftsFolderProvider = createDraftsFolderProvider(account)
-        val webDavStore = WebDavStore(trustManagerFactory, serverSettings, draftsFolderProvider)
-        val webDavTransport = WebDavTransport(trustManagerFactory, serverSettings, draftsFolderProvider)
+        val webDavStore = WebDavStore(trustManagerFactory, sniHostSetter, serverSettings, draftsFolderProvider)
+        val webDavTransport = WebDavTransport(trustManagerFactory, sniHostSetter, serverSettings, draftsFolderProvider)
         return WebDavBackend(accountName, backendStorage, webDavStore, webDavTransport)
     }

diff --git a/mail/protocols/webdav/src/main/java/com/fsck/k9/mail/store/webdav/SniHostSetter.java b/mail/protocols/webdav/src/main/java/com/fsck/k9/mail/store/webdav/SniHostSetter.java
new file mode 100644
index 00000000000..df323946ac2
--- /dev/null
+++ b/mail/protocols/webdav/src/main/java/com/fsck/k9/mail/store/webdav/SniHostSetter.java
@@ -0,0 +1,10 @@
+package com.fsck.k9.mail.store.webdav;
+
+
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.SSLSocketFactory;
+
+
+public interface SniHostSetter {
+    void setSniHost(SSLSocketFactory factory, SSLSocket socket, String hostname);
+}
diff --git a/mail/protocols/webdav/src/main/java/com/fsck/k9/mail/store/webdav/WebDavSocketFactory.java b/mail/protocols/webdav/src/main/java/com/fsck/k9/mail/store/webdav/WebDavSocketFactory.java
index 20ee771fd50..36d7abd8ffa 100644
--- a/mail/protocols/webdav/src/main/java/com/fsck/k9/mail/store/webdav/WebDavSocketFactory.java
+++ b/mail/protocols/webdav/src/main/java/com/fsck/k9/mail/store/webdav/WebDavSocketFactory.java
@@ -7,7 +7,6 @@
 import java.security.KeyManagementException;
 import java.security.NoSuchAlgorithmException;

-import com.fsck.k9.mail.ssl.DefaultTrustedSocketFactory;
 import com.fsck.k9.mail.ssl.TrustManagerFactory;
 import javax.net.ssl.SSLContext;
 import javax.net.ssl.SSLSocket;
@@ -22,10 +21,13 @@
  * Using two socket factories looks suspicious.
  */
 public class WebDavSocketFactory implements LayeredSocketFactory {
+    private final SniHostSetter sniHostSetter;
     private SSLSocketFactory mSocketFactory;
     private org.apache.http.conn.ssl.SSLSocketFactory mSchemeSocketFactory;

-    public WebDavSocketFactory(TrustManagerFactory trustManagerFactory, String host, int port) throws NoSuchAlgorithmException, KeyManagementException {
+    public WebDavSocketFactory(TrustManagerFactory trustManagerFactory, SniHostSetter sniHostSetter,
+            String host, int port) throws NoSuchAlgorithmException, KeyManagementException {
+        this.sniHostSetter = sniHostSetter;
         SSLContext sslContext = SSLContext.getInstance("TLS");
         sslContext.init(null, new TrustManager[] {
                 trustManagerFactory.getTrustManagerForDomain(host, port)
@@ -62,7 +64,7 @@ public Socket createSocket(
                 port,
                 autoClose
         );
-        DefaultTrustedSocketFactory.setSniHost(mSocketFactory, sslSocket, host);
+        sniHostSetter.setSniHost(mSocketFactory, sslSocket, host);
         //hostnameVerifier.verify(host, sslSocket);
         // verifyHostName() didn't blowup - good!
         return sslSocket;
diff --git a/mail/protocols/webdav/src/main/java/com/fsck/k9/mail/store/webdav/WebDavStore.java b/mail/protocols/webdav/src/main/java/com/fsck/k9/mail/store/webdav/WebDavStore.java
index bad815cca93..c11282ac4ca 100644
--- a/mail/protocols/webdav/src/main/java/com/fsck/k9/mail/store/webdav/WebDavStore.java
+++ b/mail/protocols/webdav/src/main/java/com/fsck/k9/mail/store/webdav/WebDavStore.java
@@ -73,6 +73,7 @@ public class WebDavStore {
     private String mailboxPath;

     private final TrustManagerFactory trustManagerFactory;
+    private final SniHostSetter sniHostSetter;
     private final WebDavHttpClient.WebDavHttpClientFactory httpClientFactory;
     private WebDavHttpClient httpClient = null;
     private HttpContext httpContext = null;
@@ -84,16 +85,19 @@ public class WebDavStore {
     private WebDavFolder sendFolder = null;
     private Map<String, WebDavFolder> folderList = new HashMap<>();

-    public WebDavStore(TrustManagerFactory trustManagerFactory, ServerSettings serverSettings,
-            DraftsFolderProvider draftsFolderProvider) {
-        this(trustManagerFactory, serverSettings, draftsFolderProvider, new WebDavHttpClient.WebDavHttpClientFactory());
+    public WebDavStore(TrustManagerFactory trustManagerFactory, SniHostSetter sniHostSetter,
+            ServerSettings serverSettings, DraftsFolderProvider draftsFolderProvider) {
+        this(trustManagerFactory, sniHostSetter, serverSettings, draftsFolderProvider,
+                new WebDavHttpClient.WebDavHttpClientFactory());
     }

-    public WebDavStore(TrustManagerFactory trustManagerFactory, ServerSettings serverSettings,
-            DraftsFolderProvider draftsFolderProvider, WebDavHttpClientFactory clientFactory) {
+    public WebDavStore(TrustManagerFactory trustManagerFactory, SniHostSetter sniHostSetter,
+            ServerSettings serverSettings, DraftsFolderProvider draftsFolderProvider,
+            WebDavHttpClientFactory clientFactory) {
         this.draftsFolderProvider = draftsFolderProvider;
         httpClientFactory = clientFactory;
         this.trustManagerFactory = trustManagerFactory;
+        this.sniHostSetter = sniHostSetter;

         hostname = serverSettings.host;
         port = serverSettings.port;
@@ -765,7 +769,9 @@ public WebDavHttpClient getHttpClient() throws MessagingException {

             SchemeRegistry reg = httpClient.getConnectionManager().getSchemeRegistry();
             try {
-                Scheme s = new Scheme("https", new WebDavSocketFactory(trustManagerFactory, hostname, 443), 443);
+                WebDavSocketFactory socketFactory =
+                        new WebDavSocketFactory(trustManagerFactory, sniHostSetter, hostname, 443);
+                Scheme s = new Scheme("https", socketFactory, 443);
                 reg.register(s);
             } catch (NoSuchAlgorithmException nsa) {
                 Timber.e(nsa, "NoSuchAlgorithmException in getHttpClient");
diff --git a/mail/protocols/webdav/src/main/java/com/fsck/k9/mail/transport/WebDavTransport.java b/mail/protocols/webdav/src/main/java/com/fsck/k9/mail/transport/WebDavTransport.java
index a9e8cac0ac8..d133b0e740e 100644
--- a/mail/protocols/webdav/src/main/java/com/fsck/k9/mail/transport/WebDavTransport.java
+++ b/mail/protocols/webdav/src/main/java/com/fsck/k9/mail/transport/WebDavTransport.java
@@ -10,15 +10,16 @@
 import com.fsck.k9.mail.Transport;
 import com.fsck.k9.mail.ssl.TrustManagerFactory;
 import com.fsck.k9.mail.store.webdav.DraftsFolderProvider;
+import com.fsck.k9.mail.store.webdav.SniHostSetter;
 import com.fsck.k9.mail.store.webdav.WebDavStore;
 import timber.log.Timber;

 public class WebDavTransport extends Transport {
     private WebDavStore store;

-    public WebDavTransport(TrustManagerFactory trustManagerFactory, ServerSettings serverSettings,
-            DraftsFolderProvider draftsFolderProvider) {
-        store = new WebDavStore(trustManagerFactory, serverSettings, draftsFolderProvider);
+    public WebDavTransport(TrustManagerFactory trustManagerFactory, SniHostSetter sniHostSetter,
+            ServerSettings serverSettings, DraftsFolderProvider draftsFolderProvider) {
+        store = new WebDavStore(trustManagerFactory, sniHostSetter, serverSettings, draftsFolderProvider);

         if (K9MailLib.isDebug())
             Timber.d(">>> New WebDavTransport creation complete");

diff --git a/mail/protocols/webdav/src/test/java/com/fsck/k9/mail/store/webdav/WebDavStoreTest.java b/mail/protocols/webdav/src/test/java/com/fsck/k9/mail/store/webdav/WebDavStoreTest.java
index 958ee1f1599..8cb66466e2e 100644
--- a/mail/protocols/webdav/src/test/java/com/fsck/k9/mail/store/webdav/WebDavStoreTest.java
+++ b/mail/protocols/webdav/src/test/java/com/fsck/k9/mail/store/webdav/WebDavStoreTest.java
@@ -70,6 +70,8 @@ public class WebDavStoreTest {
     @Mock
     private TrustManagerFactory trustManagerFactory;
     @Mock
+    private SniHostSetter sniHostSetter;
+    @Mock
     private DraftsFolderProvider draftsFolderProvider;

     private ArgumentCaptor<HttpGeneric> requestCaptor;
@@ -352,12 +354,14 @@ private ServerSettings createServerSettings(ConnectionSecurity connectionSecurit
     }

     private WebDavStore createWebDavStore() {
-        return new WebDavStore(trustManagerFactory, serverSettings, draftsFolderProvider, mockHttpClientFactory);
+        return new WebDavStore(trustManagerFactory, sniHostSetter, serverSettings, draftsFolderProvider,
+                mockHttpClientFactory);
     }

     private WebDavStore createWebDavStore(ConnectionSecurity connectionSecurity) {
         ServerSettings serverSettings = createServerSettings(connectionSecurity);
-        return new WebDavStore(trustManagerFactory, serverSettings, draftsFolderProvider, mockHttpClientFactory);
+        return new WebDavStore(trustManagerFactory, sniHostSetter, serverSettings, draftsFolderProvider,
+                mockHttpClientFactory);
     }

     private void configureHttpResponses(HttpResponse... responses) throws IOException {


Move `MailTo` and `ListHeaders` classes

Closes #6014

Move `MailTo` and `ListHeaders`

Move `MailTo` and `ListHeaders`
These classes depend on the Android SDK and therefore should not live inside the `:mail:common` Gradle project.

Alternatively, we could change the implementation to not use Android's `Uri` class.

./gradlew :app:core:testDebugUnitTest --tests "com.fsck.k9.helper.MailToTest" --tests "com.fsck.k9.helper.ListHeadersTest" --tests "com.fsck.k9.helper.ReplyToParserTest"

diff --git a/mail/common/src/main/java/com/fsck/k9/mail/internet/ListHeaders.java b/app/core/src/main/java/com/fsck/k9/helper/ListHeaders.java
similarity index 96%
rename from mail/common/src/main/java/com/fsck/k9/mail/internet/ListHeaders.java
rename to app/core/src/main/java/com/fsck/k9/helper/ListHeaders.java
index 2bed0bafcc4..52f16846078 100644
--- a/mail/common/src/main/java/com/fsck/k9/mail/internet/ListHeaders.java
+++ b/app/core/src/main/java/com/fsck/k9/helper/ListHeaders.java
@@ -1,4 +1,4 @@
-package com.fsck.k9.mail.internet;
+package com.fsck.k9.helper;

 import android.net.Uri;
@@ -8,7 +8,6 @@
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;

-import com.fsck.k9.helper.MailTo;
 import com.fsck.k9.mail.Address;
 import com.fsck.k9.mail.Message;

diff --git a/mail/common/src/main/java/com/fsck/k9/helper/MailTo.java b/app/core/src/main/java/com/fsck/k9/helper/MailTo.java
similarity index 100%
rename from mail/common/src/main/java/com/fsck/k9/helper/MailTo.java
rename to app/core/src/main/java/com/fsck/k9/helper/MailTo.java
diff --git a/app/core/src/main/java/com/fsck/k9/helper/ReplyToParser.java b/app/core/src/main/java/com/fsck/k9/helper/ReplyToParser.java
index 7cbb15e714f..2d53a81392d 100644
--- a/app/core/src/main/java/com/fsck/k9/helper/ReplyToParser.java
+++ b/app/core/src/main/java/com/fsck/k9/helper/ReplyToParser.java
@@ -12,7 +12,6 @@
 import com.fsck.k9.mail.Address;
 import com.fsck.k9.mail.Message;
 import com.fsck.k9.mail.Message.RecipientType;
-import com.fsck.k9.mail.internet.ListHeaders;

 public class ReplyToParser {

diff --git a/mail/common/src/test/java/com/fsck/k9/mail/internet/ListHeadersTest.java b/app/core/src/test/java/com/fsck/k9/helper/ListHeadersTest.java
similarity index 98%
rename from mail/common/src/test/java/com/fsck/k9/mail/internet/ListHeadersTest.java
rename to app/core/src/test/java/com/fsck/k9/helper/ListHeadersTest.java
index ba6bfbee31b..b3d77e2ca94 100644
--- a/mail/common/src/test/java/com/fsck/k9/mail/internet/ListHeadersTest.java
+++ b/app/core/src/test/java/com/fsck/k9/helper/ListHeadersTest.java
@@ -1,9 +1,10 @@
-package com.fsck.k9.mail.internet;
+package com.fsck.k9.helper;

 import com.fsck.k9.mail.Address;
 import com.fsck.k9.mail.K9LibRobolectricTestRunner;
 import com.fsck.k9.mail.Message;
+import com.fsck.k9.mail.internet.MimeMessage;
 import org.junit.Test;
 import org.junit.runner.RunWith;

diff --git a/mail/common/src/test/java/com/fsck/k9/helper/MailToTest.java b/app/core/src/test/java/com/fsck/k9/helper/MailToTest.java
similarity index 100%
rename from mail/common/src/test/java/com/fsck/k9/helper/MailToTest.java
rename to app/core/src/test/java/com/fsck/k9/helper/MailToTest.java
diff --git a/app/core/src/test/java/com/fsck/k9/helper/ReplyToParserTest.java b/app/core/src/test/java/com/fsck/k9/helper/ReplyToParserTest.java
index 501b2731b96..6093ed15fdc 100644
--- a/app/core/src/test/java/com/fsck/k9/helper/ReplyToParserTest.java
+++ b/app/core/src/test/java/com/fsck/k9/helper/ReplyToParserTest.java
@@ -10,7 +10,6 @@
 import com.fsck.k9.mail.Address;
 import com.fsck.k9.mail.Message;
 import com.fsck.k9.mail.Message.RecipientType;
-import com.fsck.k9.mail.internet.ListHeaders;
 import org.junit.Before;
 import org.junit.Test;


Fix bug in `FlowedMessageUtils.deflow()`

Fixes #6029

Broken quoting of plain text mail

Broken quoting of plain text mail
### Checklist

- [X] I have used the search function to see if someone else has already submitted the same bug report.
- [X] I will describe the problem with as much detail as possible.

### App version

6.000

### Where did you get the app from?

Google Play

### Android version

Android 12, One UI 4.1

### Device model

Samsung S21

### Steps to reproduce

Show mail. The mail is like this, it was a pure text mail:

```

Am 20.04.2022 um 04:16 schrieb XXXX YYYYYYYY:
>
> We also enjoy when people tell us that all their tests are green! It
> gives us confidence ;-) So regardless of the tests color (red or
> green), please do not hesitate to send me a short mail as both types
> of feedback are useful.

Apache PDFBox builds successfully with jdk19.

https://ci-builds.apache.org/job/PDFBox/job/PDFBox-Trunk-jdk19/

jacoco (tool for code coverage) had problems
("java.lang.instrument.IllegalClassFormatException: Error while
instrumenting sun/security/pkcs11/SunPKCS11") but it went away with
their latest version

https://github.com/jacoco/jacoco/issues/1296

Best regards

Tilman Hausherr

```

(I did make an issue search for "quote", but the issues looked different)

### Expected behavior

more like this (from thunderbird)

![grafik](https://user-images.githubusercontent.com/6665575/164276881-5599500c-6d61-493c-82dd-1d5bb66aa782.png)

### Actual behavior

![grafik](https://user-images.githubusercontent.com/6665575/164277096-009c7251-10e3-4664-8350-ed59f235b2fb.png)

### Logs

_No response_

I can't reproduce the issue with the text snippet you've provided. Can you attach the (redacted) message source (including headers)?
Here's the (redacted) EML file
[k9.zip](https://github.com/k9mail/k-9/files/8523702/k9.zip)

./gradlew :mail:common:testDebugUnitTest --tests "com.fsck.k9.mail.internet.FlowedMessageUtilsTest"

diff --git a/mail/common/src/main/java/com/fsck/k9/mail/internet/FlowedMessageUtils.java b/mail/common/src/main/java/com/fsck/k9/mail/internet/FlowedMessageUtils.java
index 25a10847b4d..580a5acaa2e 100644
--- a/mail/common/src/main/java/com/fsck/k9/mail/internet/FlowedMessageUtils.java
+++ b/mail/common/src/main/java/com/fsck/k9/mail/internet/FlowedMessageUtils.java
@@ -45,39 +45,54 @@ public static String deflow(String text, boolean delSp) {
             String line = i < lines.length ? lines[i] : null;
             int actualQuoteDepth = 0;

-            if (line != null && line.length() > 0) {
-                if (line.equals(RFC2646_SIGNATURE))
+            if (line != null) {
+                if (line.equals(RFC2646_SIGNATURE)) {
                     // signature handling (the previous line is not flowed)
                     resultLineFlowed = false;
-
-                else if (line.charAt(0) == RFC2646_QUOTE) {
+                } else if (line.length() > 0 && line.charAt(0) == RFC2646_QUOTE) {
                     // Quote
                     actualQuoteDepth = 1;
-                    while (actualQuoteDepth < line.length() && line.charAt(actualQuoteDepth) == RFC2646_QUOTE) actualQuoteDepth ++;
+                    while (actualQuoteDepth < line.length() && line.charAt(actualQuoteDepth) == RFC2646_QUOTE) {
+                        actualQuoteDepth++;
+                    }
                     // if quote-depth changes wrt the previous line then this is not flowed
-                    if (resultLineQuoteDepth != actualQuoteDepth) resultLineFlowed = false;
+                    if (resultLineQuoteDepth != actualQuoteDepth) {
+                        resultLineFlowed = false;
+                    }
                     line = line.substring(actualQuoteDepth);

                 } else {
-                    // id quote-depth changes wrt the first line then this is not flowed
-                    if (resultLineQuoteDepth > 0) resultLineFlowed = false;
+                    // if quote-depth changes wrt the first line then this is not flowed
+                    if (resultLineQuoteDepth > 0) {
+                        resultLineFlowed = false;
+                    }
                 }

-                if (line.length() > 0 && line.charAt(0) == RFC2646_SPACE)
+                if (line.length() > 0 && line.charAt(0) == RFC2646_SPACE) {
                     // Line space-stuffed
                     line = line.substring(1);
+                }

                 // if the previous was the last then it was not flowed
-            } else if (line == null) resultLineFlowed = false;
+            } else {
+                resultLineFlowed = false;
+            }

             // Add the PREVIOUS line.
             // This often will find the flow looking for a space as the last char of the line.
-            // With quote changes or signatures it could be the followinf line to void the flow.
+            // With quote changes or signatures it could be the following line to void the flow.
             if (!resultLineFlowed && i > 0) {
-                if (resultLineQuoteDepth > 0) resultLine.insert(0, RFC2646_SPACE);
-                for (int j = 0; j < resultLineQuoteDepth; j++) resultLine.insert(0, RFC2646_QUOTE);
-                if (result == null) result = new StringBuffer();
-                else result.append(RFC2646_CRLF);
+                if (resultLineQuoteDepth > 0) {
+                    resultLine.insert(0, RFC2646_SPACE);
+                }
+                for (int j = 0; j < resultLineQuoteDepth; j++) {
+                    resultLine.insert(0, RFC2646_QUOTE);
+                }
+                if (result == null) {
+                    result = new StringBuffer();
+                } else {
+                    result.append(RFC2646_CRLF);
+                }
                 result.append(resultLine.toString());
                 resultLine = new StringBuffer();
                 resultLineFlowed = false;
@@ -86,13 +101,16 @@ else if (line.charAt(0) == RFC2646_QUOTE) {

             if (line != null) {
                 if (!line.equals(RFC2646_SIGNATURE) && line.endsWith("" + RFC2646_SPACE) && i < lines.length - 1) {
-                    // Line flowed (NOTE: for the split operation the line having i == lines.length is the last that does not end with RFC2646_CRLF)
-                    if (delSp) line = line.substring(0, line.length() - 1);
+                    // Line flowed (NOTE: for the split operation the line having i == lines.length is the last that
+                    // does not end with RFC2646_CRLF)
+                    if (delSp) {
+                        line = line.substring(0, line.length() - 1);
+                    }
                     resultLineFlowed = true;
+                } else {
+                    resultLineFlowed = false;
                 }

-                else resultLineFlowed = false;
-
                 resultLine.append(line);
             }
         }

diff --git a/mail/common/src/test/java/com/fsck/k9/mail/internet/FlowedMessageUtilsTest.kt b/mail/common/src/test/java/com/fsck/k9/mail/internet/FlowedMessageUtilsTest.kt
new file mode 100644
index 00000000000..cf70f7145a4
--- /dev/null
+++ b/mail/common/src/test/java/com/fsck/k9/mail/internet/FlowedMessageUtilsTest.kt
@@ -0,0 +1,202 @@
+package com.fsck.k9.mail.internet
+
+import com.fsck.k9.mail.crlf
+import com.google.common.truth.Truth.assertThat
+import org.junit.Test
+
+private const val DEL_SP_NO = false
+private const val DEL_SP_YES = true
+
+class FlowedMessageUtilsTest {
+    @Test
+    fun `deflow() with simple text`() {
+        val input = "Text that should be \r\n" +
+            "displayed on one line"
+
+        val result = FlowedMessageUtils.deflow(input, DEL_SP_NO)
+
+        assertThat(result).isEqualTo("Text that should be displayed on one line")
+    }
+
+    @Test
+    fun `deflow() with only some lines ending in space`() {
+        val input = "Text that \r\n" +
+            "should be \r\n" +
+            "displayed on \r\n" +
+            "one line.\r\n" +
+            "Text that should retain\r\n" +
+            "its line break."
+
+        val result = FlowedMessageUtils.deflow(input, DEL_SP_NO)
+
+        assertThat(result).isEqualTo(
+            """
+            Text that should be displayed on one line.
+            Text that should retain
+            its line break.
+            """.trimIndent().crlf()
+        )
+    }
+
+    @Test
+    fun `deflow() with nothing to do`() {
+        val input = "Line one\r\nLine two\r\n"
+
+        val result = FlowedMessageUtils.deflow(input, DEL_SP_NO)
+
+        assertThat(result).isEqualTo(input)
+    }
+
+    @Test
+    fun `deflow() with quoted text`() {
+        val input = "On [date], [user] wrote:\r\n" +
+            "> Text that should be displayed \r\n" +
+            "> on one line\r\n" +
+            "\r\n" +
+            "Some more text that should be \r\n" +
+            "displayed on one line.\r\n"
+
+        val result = FlowedMessageUtils.deflow(input, DEL_SP_NO)
+
+        assertThat(result).isEqualTo(
+            """
+            |On [date], [user] wrote:
+            |> Text that should be displayed on one line
+            |
+            |Some more text that should be displayed on one line.
+            |
+            """.trimMargin().crlf()
+        )
+    }
+
+    @Test
+    fun `deflow() with quoted text ending in space`() {
+        val input = "> Quoted text \r\n" +
+            "Some other text"
+
+        val result = FlowedMessageUtils.deflow(input, DEL_SP_NO)
+
+        assertThat(result).isEqualTo("> Quoted text \r\nSome other text")
+    }
+
+    @Test
+    fun `deflow() with quoted text ending in space before quoted text of different quoting depth`() {
+        val input = ">> Depth 2 \r\n" +
+            "> Depth 1 \r\n" +
+            "> is here\r\n" +
+            "Some other text"
+
+        val result = FlowedMessageUtils.deflow(input, DEL_SP_NO)
+
+        assertThat(result).isEqualTo(
+            """
+            >> Depth 2${" "}
+            > Depth 1 is here
+            Some other text
+            """.trimIndent().crlf()
+        )
+    }
+
+    @Test
+    fun `deflow() with quoted text ending in space followed by empty line`() {
+        val input = "> Quoted \r\n" +
+            "\r\n" +
+            "Text"
+
+        val result = FlowedMessageUtils.deflow(input, DEL_SP_NO)
+
+        assertThat(result).isEqualTo(input)
+    }
+
+    @Test
+    fun `deflow() with delSp=true`() {
+        val input = "Text that is wrapped mid wo \r\nrd"
+
+        val result = FlowedMessageUtils.deflow(input, DEL_SP_YES)
+
+        assertThat(result).isEqualTo("Text that is wrapped mid word")
+    }
+
+    @Test
+    fun `deflow() with quoted text and space-stuffing and delSp=true`() {
+        val input = "> Quoted te \r\n" +
+            "> xt"
+
+        val result = FlowedMessageUtils.deflow(input, DEL_SP_YES)
+
+        assertThat(result).isEqualTo("> Quoted text")
+    }
+
+    @Test
+    fun `deflow() with space-stuffed second line`() {
+        val input = "Text that should be \r\n" +
+            " displayed on one line"
+
+        val result = FlowedMessageUtils.deflow(input, DEL_SP_NO)
+
+        assertThat(result).isEqualTo("Text that should be displayed on one line")
+    }
+
+    @Test
+    fun `deflow() with only space-stuffing`() {
+        val input = "Line 1\r\n" +
+            " Line 2\r\n" +
+            " Line 3\r\n"
+
+        val result = FlowedMessageUtils.deflow(input, DEL_SP_NO)
+
+        assertThat(result).isEqualTo("Line 1\r\nLine 2\r\nLine 3\r\n")
+    }
+
+    @Test
+    fun `deflow() with quoted space-stuffed second line`() {
+        val input = "> Text that should be \r\n" +
+            "> displayed on one line"
+
+        val result = FlowedMessageUtils.deflow(input, DEL_SP_NO)
+
+        assertThat(result).isEqualTo("> Text that should be displayed on one line")
+    }
+
+    @Test
+    fun `deflow() with text containing signature`() {
+        val input = "Text that should be \r\n" +
+            "displayed on one line.\r\n" +
+            "\r\n" +
+            "-- \r\n" +
+            "Signature \r\n" +
+            "text"
+
+        val result = FlowedMessageUtils.deflow(input, DEL_SP_NO)
+
+        assertThat(result).isEqualTo(
+            """
+            Text that should be displayed on one line.
+
+            --${" "}
+            Signature text
+            """.trimIndent().crlf()
+        )
+    }
+
+    @Test
+    fun `deflow() with quoted text containing signature`() {
+        val input = "> Text that should be \r\n" +
+            "> displayed on one line.\r\n" +
+            "> \r\n" +
+            "> -- \r\n" +
+            "> Signature \r\n" +
+            "> text"
+
+        val result = FlowedMessageUtils.deflow(input, DEL_SP_NO)
+
+        assertThat(result).isEqualTo(
+            """
+            > Text that should be displayed on one line.
+            >${" "}
+            > --${" "}
+            > Signature text
+            """.trimIndent().crlf()
+        )
+    }
+}


Truncate file when overwriting existing settings file

Fixes #5988

Impossible to import settings

Impossible to import settings
### Checklist

- [X] I have used the search function to see if someone else has already submitted the same bug report.
- [X] I will describe the problem with as much detail as possible.

### App version

5.913

### Where did you get the app from?

F-Droid

### Android version

11

### Device model

Xiaomi MI9

### Steps to reproduce

Parameters
Import parameters

### Expected behavior

Make it possible to import parameters

### Actual behavior

Hello,
i'm using K-9 mail 5.913 on Xiaomi MI9 (Android 11 - MIUI 12.5.1) and on a fresh reinstall i've got an error to import settings (all settings)
I've got this error message : "Echec de lecture du fichier de paramètres" (Failed to read parameter file)

### Logs

```
 Error reading settings file
 com.fsck.k9.preferences.SettingsImportExportException: org.xmlpull.v1.XmlPullParserException: Unexpected token (position:TEXT lue key="pushMod...@727:45 in java.io.InputStreamReader@e5a72f1)
 	at com.fsck.k9.preferences.SettingsImporter.parseSettings(SettingsImporter.java:685)
 	at com.fsck.k9.preferences.SettingsImporter.getImportStreamContents(SettingsImporter.java:129)
 	at com.fsck.k9.ui.settings.import.SettingsImportViewModel.readSettings(SettingsImportViewModel.kt:357)
 	at com.fsck.k9.ui.settings.import.SettingsImportViewModel.access$readSettings(SettingsImportViewModel.kt:28)
 	at com.fsck.k9.ui.settings.import.SettingsImportViewModel$startReadSettingsFile$1$1$1.invokeSuspend(SettingsImportViewModel.kt:238)
 	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
 	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
 	at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:39)
 	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95)
 	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
 Caused by: org.xmlpull.v1.XmlPullParserException: Unexpected token (position:TEXT lue key="pushMod...@727:45 in java.io.InputStreamReader@e5a72f1)
 	at com.android.org.kxml2.io.KXmlParser.next(KXmlParser.java:440)
 	at com.android.org.kxml2.io.KXmlParser.next(KXmlParser.java:321)
 	at com.fsck.k9.preferences.SettingsImporter.parseSettings(SettingsImporter.java:676)
 	... 12 more
```

I add here the part of my file concerned by the error (@727:45 )
I insist on the fact that I did not edit this file and that it is the result of the action "Export parameters"

```
726  </accounts>
727</k9settings>lue key="pushMode">SECOND_CLASS</value>
728        </folder>

```
Since a new reinstallation, the generated xml file seems to be correct,
I didn't dare to try a new import for fear of having to redo all my configuration.
It's a bit incomprehensible.
Thanks for reporting this.

Looks like this happens when overwriting a file and writing less data than there was before.

./gradlew :app:core:testDebugUnitTest --tests "com.fsck.k9.logging.LogcatLogFileWriterTest"

diff --git a/app/core/src/main/java/com/fsck/k9/logging/LogFileWriter.kt b/app/core/src/main/java/com/fsck/k9/logging/LogFileWriter.kt
index d81c9e16611..831637708c6 100644
--- a/app/core/src/main/java/com/fsck/k9/logging/LogFileWriter.kt
+++ b/app/core/src/main/java/com/fsck/k9/logging/LogFileWriter.kt
@@ -26,7 +26,7 @@ class LogcatLogFileWriter(
     private fun writeLogBlocking(contentUri: Uri) {
         Timber.v("Writing logcat output to content URI: %s", contentUri)

-        val outputStream = contentResolver.openOutputStream(contentUri)
+        val outputStream = contentResolver.openOutputStream(contentUri, "wt")
             ?: error("Error opening contentUri for writing")

         outputStream.use {
diff --git a/app/core/src/main/java/com/fsck/k9/preferences/SettingsExporter.kt b/app/core/src/main/java/com/fsck/k9/preferences/SettingsExporter.kt
index 39983b13918..08bc5bbefc0 100644
--- a/app/core/src/main/java/com/fsck/k9/preferences/SettingsExporter.kt
+++ b/app/core/src/main/java/com/fsck/k9/preferences/SettingsExporter.kt
@@ -33,7 +33,7 @@ class SettingsExporter(
         updateNotificationSettings(accountUuids)

         try {
-            contentResolver.openOutputStream(uri)!!.use { outputStream ->
+            contentResolver.openOutputStream(uri, "wt")!!.use { outputStream ->
                 exportPreferences(outputStream, includeGlobals, accountUuids)
             }
         } catch (e: Exception) {
diff --git a/app/ui/legacy/src/main/java/com/fsck/k9/ui/messageview/AttachmentController.java b/app/ui/legacy/src/main/java/com/fsck/k9/ui/messageview/AttachmentController.java
index 1248e19e462..0d90de27b98 100644
--- a/app/ui/legacy/src/main/java/com/fsck/k9/ui/messageview/AttachmentController.java
+++ b/app/ui/legacy/src/main/java/com/fsck/k9/ui/messageview/AttachmentController.java
@@ -117,7 +117,7 @@ private void writeAttachment(Uri documentUri) throws IOException {
         ContentResolver contentResolver = context.getContentResolver();
         InputStream in = contentResolver.openInputStream(attachment.internalUri);
         try {
-            OutputStream out = contentResolver.openOutputStream(documentUri);
+            OutputStream out = contentResolver.openOutputStream(documentUri, "wt");
             try {
                 IOUtils.copy(in, out);
                 out.flush();

diff --git a/app/core/src/test/java/com/fsck/k9/logging/LogcatLogFileWriterTest.kt b/app/core/src/test/java/com/fsck/k9/logging/LogcatLogFileWriterTest.kt
index 2fcc6e93ff9..a4229bbb0d8 100644
--- a/app/core/src/test/java/com/fsck/k9/logging/LogcatLogFileWriterTest.kt
+++ b/app/core/src/test/java/com/fsck/k9/logging/LogcatLogFileWriterTest.kt
@@ -57,13 +57,13 @@ class LogcatLogFileWriterTest {

     private fun createContentResolver(): ContentResolver {
         return mock {
-            on { openOutputStream(contentUri) } doReturn outputStream
+            on { openOutputStream(contentUri, "wt") } doReturn outputStream
         }
     }

     private fun createThrowingContentResolver(exception: Exception): ContentResolver {
         return mock {
-            on { openOutputStream(contentUri) } doAnswer { throw exception }
+            on { openOutputStream(contentUri, "wt") } doAnswer { throw exception }
         }
     }


Add support for duplicate "charset" parameters with matching values

- For extracting the `charset` parameter value, we only need basic (RFC 2045) parameter decoding.
- We typically ignore duplicate parameters (parameter name used more than once) because it's not specified which one to pick. We relax this rule to support the case where all `charset` values are equivalent, e.g. `Content-Type: text/html; charset=UTF-8; charset=utf-8`.

Fixes #5951

Incorrect encoding of the email text / Torrent tracker SoftPilot.win
### Checklist

- [X] I have used the search function to see if someone else has already submitted the same bug report.
- [X] I will describe the problem with as much detail as possible.

### App version

5.912

### Where did you get the app from?

F-Droid

### Android version

12 | KOSP 2.3 Unofficial

### Device model

Realme X2 CN (RMX1991)

### Steps to reproduce

1. Wait for an email from this tracker.
2. Open the email and see that the encoding of the text of the email is incorrect.
![Screenshot_20220226-175547_Nova Launcher](https://user-images.githubusercontent.com/47178587/157188524-d7328604-0aa4-4feb-b25b-a0b2f1ffa5d4.png)
![Screenshot_20220226-175559_Почта K-9](https://user-images.githubusercontent.com/47178587/157188568-430168f7-50c6-4acd-8315-1fc7997045b9.png)

### Expected behavior

Emails received from SoftPilot.win have the correct text encoding.

### Actual behavior

Emails received from SoftPilot.win have the incorrect text encoding.

### Logs

_No response_

Can you attach the message source?

Chances are the email is not properly declaring the character set that is used. That would be a bug in the software creating the message, not one in K-9 Mail.
Many email apps attempt to guess the character set in order to display the message as intended. K-9 Mail doesn't do that. So the message being displayed "correctly" in other apps is not an indication that the message isn't broken.
This ZIP only contains an HTML document. I was looking for the whole message source, including email headers.
[SoftPilot.win_Broken_Email.zip](https://github.com/k9mail/k-9/files/8224535/SoftPilot.win_Broken_Email.zip)

The problem is the following header line:

```text
Content-Type: text/html; charset=UTF-8; charset=utf-8
```

The relevant specifications only state that the ordering of parameters is not significant. It is not specified how duplicate parameter names should be interpreted. To avoid interoperability problems where one client prefers the first parameter value and another one the last parameter value, K-9 Mail ignores both `charset` parameter values (and the default charset US-ASCII is used).
This instance seems to be a special case where we could relax this rule. Because the charset is not case sensitive, both parameters are equivalent.

I'll update K-9 Mail to be able to handle this case. But you should still contact the authors of the newsletter and tell them to create emails with a proper `Content-Type` header.
Thanks.
Yes, I will inform the authors of the site about it.

./gradlew :mail:common:testDebugUnitTest --tests "com.fsck.k9.mail.internet.PartExtensionsTest"

diff --git a/mail/common/src/main/java/com/fsck/k9/mail/internet/MessageExtractor.java b/mail/common/src/main/java/com/fsck/k9/mail/internet/MessageExtractor.java
index e45aa18e34a..c54f469be8a 100644
--- a/mail/common/src/main/java/com/fsck/k9/mail/internet/MessageExtractor.java
+++ b/mail/common/src/main/java/com/fsck/k9/mail/internet/MessageExtractor.java
@@ -23,7 +23,6 @@
 import timber.log.Timber;

 import static com.fsck.k9.mail.internet.CharsetSupport.fixupCharset;
-import static com.fsck.k9.mail.internet.MimeUtility.getHeaderParameter;
 import static com.fsck.k9.mail.internet.MimeUtility.isSameMimeType;
 import static com.fsck.k9.mail.internet.Viewable.Alternative;
 import static com.fsck.k9.mail.internet.Viewable.Html;
@@ -77,7 +76,7 @@ private static String getTextFromTextPart(Part part, Body body, String mimeType,
         /*
          * We've got a text part, so let's see if it needs to be processed further.
          */
-        String charset = getHeaderParameter(part.getContentType(), "charset");
+        String charset = PartExtensions.getCharset(part);
         /*
          * determine the charset from HTML message.
          */
diff --git a/mail/common/src/main/java/com/fsck/k9/mail/internet/MimeParameterDecoder.kt b/mail/common/src/main/java/com/fsck/k9/mail/internet/MimeParameterDecoder.kt
index 7dcf29a6d6a..5f4fc4d4468 100644
--- a/mail/common/src/main/java/com/fsck/k9/mail/internet/MimeParameterDecoder.kt
+++ b/mail/common/src/main/java/com/fsck/k9/mail/internet/MimeParameterDecoder.kt
@@ -54,6 +54,26 @@ object MimeParameterDecoder {
         )
     }

+    fun decodeBasic(headerBody: String): MimeValue {
+        val parser = MimeHeaderParser(headerBody)
+
+        val value = parser.readHeaderValue()
+        parser.skipCFWS()
+        if (parser.endReached()) {
+            return MimeValue(value)
+        }
+
+        val (basicParameters, duplicateParameters, parserErrorIndex) = readBasicParameters(parser)
+        val parameters = basicParameters.mapValues { (_, parameterValue) -> parameterValue.value }
+
+        return MimeValue(
+            value = value,
+            parameters = parameters,
+            ignoredParameters = duplicateParameters,
+            parserErrorIndex = parserErrorIndex
+        )
+    }
+
     @JvmStatic
     fun extractHeaderValue(headerBody: String): String {
         val parser = MimeHeaderParser(headerBody)
diff --git a/mail/common/src/main/java/com/fsck/k9/mail/internet/PartExtensions.kt b/mail/common/src/main/java/com/fsck/k9/mail/internet/PartExtensions.kt
new file mode 100644
index 00000000000..a91d7db833c
--- /dev/null
+++ b/mail/common/src/main/java/com/fsck/k9/mail/internet/PartExtensions.kt
@@ -0,0 +1,24 @@
+@file:JvmName("PartExtensions")
+package com.fsck.k9.mail.internet
+
+import com.fsck.k9.mail.Part
+
+/**
+ * Return the `charset` parameter value of this [Part]'s `Content-Type` header.
+ */
+val Part.charset: String?
+    get() {
+        val contentTypeHeader = this.contentType ?: return null
+        val (_, parameters, duplicateParameters) = MimeParameterDecoder.decodeBasic(contentTypeHeader)
+        return parameters["charset"] ?: extractNonConflictingCharsetValue(duplicateParameters)
+    }
+
+// If there are multiple "charset" parameters, but they all agree on the value, we use that value.
+private fun extractNonConflictingCharsetValue(duplicateParameters: List<Pair<String, String>>): String? {
+    val charsets = duplicateParameters.asSequence()
+        .filter { (parameterName, _) -> parameterName == "charset" }
+        .map { (_, charset) -> charset.lowercase() }
+        .toSet()
+
+    return if (charsets.size == 1) charsets.first() else null
+}

diff --git a/mail/common/src/test/java/com/fsck/k9/mail/internet/PartExtensionsTest.kt b/mail/common/src/test/java/com/fsck/k9/mail/internet/PartExtensionsTest.kt
new file mode 100644
index 00000000000..f5afbdec568
--- /dev/null
+++ b/mail/common/src/test/java/com/fsck/k9/mail/internet/PartExtensionsTest.kt
@@ -0,0 +1,54 @@
+package com.fsck.k9.mail.internet
+
+import com.google.common.truth.Truth.assertThat
+import org.junit.Test
+
+class PartExtensionsTest {
+    @Test
+    fun `get charset without charset parameter`() {
+        assertGetCharset(headerValue = "text/plain", expectedCharset = null)
+    }
+
+    @Test
+    fun `get charset with single charset parameter`() {
+        assertGetCharset(headerValue = "text/plain; charset=UTF-8", expectedCharset = "utf-8")
+    }
+
+    @Test
+    fun `get charset with single quoted charset parameter`() {
+        assertGetCharset(headerValue = "text/plain; charset=\"iso-8859-1\"", expectedCharset = "ISO-8859-1")
+    }
+
+    @Test
+    fun `get charset with two charset parameters where values match exactly`() {
+        assertGetCharset(headerValue = "text/plain; charset=utf-8; charset=utf-8", expectedCharset = "utf-8")
+    }
+
+    @Test
+    fun `get charset with two charset parameters where values differ in case`() {
+        assertGetCharset(headerValue = "text/plain; charset=utf-8; charset=UTF-8", expectedCharset = "utf-8")
+    }
+
+    @Test
+    fun `get charset with two charset parameters where values differ in quoting`() {
+        assertGetCharset(headerValue = "text/plain; charset=utf-8; charset=\"utf-8\"", expectedCharset = "utf-8")
+    }
+
+    @Test
+    fun `get charset with two charset parameters with conflicting values`() {
+        assertGetCharset(headerValue = "text/plain; charset=utf-8; charset=iso-8859-1", expectedCharset = null)
+    }
+
+    @Test
+    fun `get charset with extended parameter syntax`() {
+        assertGetCharset(headerValue = "text/plain; charset*=us-ascii'en-us'utf-8", expectedCharset = null)
+    }
+
+    private fun assertGetCharset(headerValue: String, expectedCharset: String?) {
+        val part = MimeBodyPart.create(null, headerValue)
+
+        val charset = part.charset
+
+        assertThat(charset).ignoringCase().isEqualTo(expectedCharset)
+    }
+}


Add support for updating existing notifications

Fixes #5892

Two notifications for the same message

Two notifications for the same message
### Checklist

- [X] I have used the search function to see if someone else has already submitted the same bug report.
- [X] I will describe the problem with as much detail as possible.

### App version

5.910

### Where did you get the app from?

Other

### Android version

doesn't matter

### Device model

_No response_

### Steps to reproduce

Use an IMAP server that sometimes sends multiple fetch responses for a single message.

Example:
```text
RealImapConnection: conn66388302>>> 8 UID FETCH 98594 (UID FLAGS INTERNALDATE RFC822.SIZE BODY.PEEK[HEADER.FIELDS (date subject from content-type to cc reply-to message-id references in-reply-to X-K9mail-Identity Chat-Version)])
RealImapConnection: conn66388302<<<#null# [5320, FETCH, [UID, 98594, INTERNALDATE, 30-Jan-2022 18:02:23 +0100, RFC822.SIZE, 143729, BODY, [HEADER.FIELDS, [date, subject, from, content-type, to, cc, reply-to, message-id, references, in-reply-to, X-K9mail-Identity, Chat-Version]], {header fields} , FLAGS, [NonJunk]]]
RealImapConnection: conn66388302<<<#null# [5320, FETCH, [FLAGS, [NonJunk], UID, 98594]]
RealImapConnection: conn66388302<<<#8# [OK, FETCH complete]
```

### Expected behavior

The two FETCH responses should be combined and only be counted as one new message.

### Actual behavior

~~K-9 Mail's sync code currently counts this as 2 new messages and downloads the same message twice~~

~~The app will also create two notifications for that message.~~

The "new message" callback is invoked twice leading to two notifications being created.

### Logs

_No response_

I think I have the same issue using a POP3 server. If I swipe out the notification of a new message in the notification center, the same entry will show again some seconds later. After swiping out a second time, the notification disappears as expected.

Version: 5.910 from f-droid
Android: 8.1
@Showmaster3000: That's definitely a different problem. Please open a new issue and attach a debug log.

./gradlew :app:core:testDebugUnitTest --tests "com.fsck.k9.notification.NotificationDataStoreTest" --tests "com.fsck.k9.notification.NewMailNotificationManagerTest"

diff --git a/app/core/src/main/java/com/fsck/k9/notification/NewMailNotificationController.kt b/app/core/src/main/java/com/fsck/k9/notification/NewMailNotificationController.kt
index 19014486211..c29c7791f53 100644
--- a/app/core/src/main/java/com/fsck/k9/notification/NewMailNotificationController.kt
+++ b/app/core/src/main/java/com/fsck/k9/notification/NewMailNotificationController.kt
@@ -27,7 +27,9 @@ internal class NewMailNotificationController(
     fun addNewMailNotification(account: Account, message: LocalMessage, silent: Boolean) {
         val notificationData = newMailNotificationManager.addNewMailNotification(account, message, silent)

-        processNewMailNotificationData(notificationData)
+        if (notificationData != null) {
+            processNewMailNotificationData(notificationData)
+        }
     }

     fun removeNewMailNotifications(
diff --git a/app/core/src/main/java/com/fsck/k9/notification/NewMailNotificationManager.kt b/app/core/src/main/java/com/fsck/k9/notification/NewMailNotificationManager.kt
index 8e1a8c3fe26..3edfdb6abb4 100644
--- a/app/core/src/main/java/com/fsck/k9/notification/NewMailNotificationManager.kt
+++ b/app/core/src/main/java/com/fsck/k9/notification/NewMailNotificationManager.kt
@@ -38,10 +38,10 @@ internal class NewMailNotificationManager(
         )
     }

-    fun addNewMailNotification(account: Account, message: LocalMessage, silent: Boolean): NewMailNotificationData {
+    fun addNewMailNotification(account: Account, message: LocalMessage, silent: Boolean): NewMailNotificationData? {
         val content = contentCreator.createFromMessage(account, message)

-        val result = notificationRepository.addNotification(account, content, timestamp = now())
+        val result = notificationRepository.addNotification(account, content, timestamp = now()) ?: return null

         val singleNotificationData = createSingleNotificationData(
             account = account,
diff --git a/app/core/src/main/java/com/fsck/k9/notification/NotificationDataStore.kt b/app/core/src/main/java/com/fsck/k9/notification/NotificationDataStore.kt
index 50cd1be99e1..db99f40340e 100644
--- a/app/core/src/main/java/com/fsck/k9/notification/NotificationDataStore.kt
+++ b/app/core/src/main/java/com/fsck/k9/notification/NotificationDataStore.kt
@@ -2,7 +2,6 @@ package com.fsck.k9.notification

 import com.fsck.k9.Account
 import com.fsck.k9.controller.MessageReference
-import com.fsck.k9.core.BuildConfig

 internal const val MAX_NUMBER_OF_NEW_MESSAGE_NOTIFICATIONS = 8

@@ -30,15 +29,45 @@ internal class NotificationDataStore {
     }

     @Synchronized
-    fun addNotification(account: Account, content: NotificationContent, timestamp: Long): AddNotificationResult {
+    fun addNotification(account: Account, content: NotificationContent, timestamp: Long): AddNotificationResult? {
         val notificationData = getNotificationData(account)
         val messageReference = content.messageReference

-        if (BuildConfig.DEBUG && notificationData.contains(messageReference)) {
-            throw AssertionError("Notification for message $messageReference already exists")
+        val activeNotification = notificationData.activeNotifications.firstOrNull { notificationHolder ->
+            notificationHolder.content.messageReference == messageReference
         }
+        val inactiveNotification = notificationData.inactiveNotifications.firstOrNull { inactiveNotificationHolder ->
+            inactiveNotificationHolder.content.messageReference == messageReference
+        }
+
+        return if (activeNotification != null) {
+            val newActiveNotification = activeNotification.copy(content = content)
+            val notificationHolder = activeNotification.copy(
+                content = content
+            )
+
+            val operations = emptyList<NotificationStoreOperation>()
+
+            val newActiveNotifications = notificationData.activeNotifications
+                .replace(activeNotification, newActiveNotification)
+            val newNotificationData = notificationData.copy(
+                activeNotifications = newActiveNotifications
+            )
+            notificationDataMap[account.uuid] = newNotificationData
+
+            AddNotificationResult.newNotification(newNotificationData, operations, notificationHolder)
+        } else if (inactiveNotification != null) {
+            val newInactiveNotification = inactiveNotification.copy(content = content)
+            val newInactiveNotifications = notificationData.inactiveNotifications
+                .replace(inactiveNotification, newInactiveNotification)

-        return if (notificationData.isMaxNumberOfActiveNotificationsReached) {
+            val newNotificationData = notificationData.copy(
+                inactiveNotifications = newInactiveNotifications
+            )
+            notificationDataMap[account.uuid] = newNotificationData
+
+            null
+        } else if (notificationData.isMaxNumberOfActiveNotificationsReached) {
             val lastNotificationHolder = notificationData.activeNotifications.last()
             val inactiveNotificationHolder = lastNotificationHolder.toInactiveNotificationHolder()

@@ -175,14 +204,15 @@ internal class NotificationDataStore {
         throw AssertionError("getNewNotificationId() called with no free notification ID")
     }

-    private fun NotificationData.contains(messageReference: MessageReference): Boolean {
-        return activeNotifications.any { it.content.messageReference == messageReference } ||
-            inactiveNotifications.any { it.content.messageReference == messageReference }
-    }
-
     private fun NotificationHolder.toInactiveNotificationHolder() = InactiveNotificationHolder(timestamp, content)

     private fun InactiveNotificationHolder.toNotificationHolder(notificationId: Int): NotificationHolder {
         return NotificationHolder(notificationId, timestamp, content)
     }
+
+    private fun <T> List<T>.replace(old: T, new: T): List<T> {
+        return map { element ->
+            if (element === old) new else element
+        }
+    }
 }
diff --git a/app/core/src/main/java/com/fsck/k9/notification/NotificationRepository.kt b/app/core/src/main/java/com/fsck/k9/notification/NotificationRepository.kt
index f624810c990..ddad7bb965d 100644
--- a/app/core/src/main/java/com/fsck/k9/notification/NotificationRepository.kt
+++ b/app/core/src/main/java/com/fsck/k9/notification/NotificationRepository.kt
@@ -37,8 +37,8 @@ internal class NotificationRepository(
     }

     @Synchronized
-    fun addNotification(account: Account, content: NotificationContent, timestamp: Long): AddNotificationResult {
-        return notificationDataStore.addNotification(account, content, timestamp).also { result ->
+    fun addNotification(account: Account, content: NotificationContent, timestamp: Long): AddNotificationResult? {
+        return notificationDataStore.addNotification(account, content, timestamp)?.also { result ->
             persistNotificationDataStoreChanges(
                 account = account,
                 operations = result.notificationStoreOperations,

diff --git a/app/core/src/test/java/com/fsck/k9/notification/NewMailNotificationManagerTest.kt b/app/core/src/test/java/com/fsck/k9/notification/NewMailNotificationManagerTest.kt
index 72e8a193d58..105f49649b5 100644
--- a/app/core/src/test/java/com/fsck/k9/notification/NewMailNotificationManagerTest.kt
+++ b/app/core/src/test/java/com/fsck/k9/notification/NewMailNotificationManagerTest.kt
@@ -49,6 +49,7 @@ class NewMailNotificationManagerTest {

         val result = manager.addNewMailNotification(account, message, silent = false)

+        assertNotNull(result)
         assertThat(result.singleNotificationData.first().content).isEqualTo(
             NotificationContent(
                 messageReference = createMessageReference("msg-1"),
@@ -85,6 +86,7 @@ class NewMailNotificationManagerTest {

         val result = manager.addNewMailNotification(account, messageTwo, silent = false)

+        assertNotNull(result)
         assertThat(result.singleNotificationData.first().content).isEqualTo(
             NotificationContent(
                 messageReference = createMessageReference("msg-2"),
@@ -121,6 +123,7 @@ class NewMailNotificationManagerTest {

         val result = manager.addNewMailNotification(account, message, silent = false)

+        assertNotNull(result)
         val notificationId = NotificationIds.getSingleMessageNotificationId(account, index = 0)
         assertThat(result.cancelNotificationIds).isEqualTo(listOf(notificationId))
         assertThat(result.singleNotificationData.first().notificationId).isEqualTo(notificationId)
@@ -196,6 +199,7 @@ class NewMailNotificationManagerTest {
             messageUid = "msg-2"
         )
         val dataTwo = manager.addNewMailNotification(account, messageTwo, silent = true)
+        assertNotNull(dataTwo)
         val notificationIdTwo = dataTwo.singleNotificationData.first().notificationId
         val messageThree = addMessageToNotificationContentCreator(
             sender = "Alice",
diff --git a/app/core/src/test/java/com/fsck/k9/notification/NotificationDataStoreTest.kt b/app/core/src/test/java/com/fsck/k9/notification/NotificationDataStoreTest.kt
index a0746390964..57ffc8aba4f 100644
--- a/app/core/src/test/java/com/fsck/k9/notification/NotificationDataStoreTest.kt
+++ b/app/core/src/test/java/com/fsck/k9/notification/NotificationDataStoreTest.kt
@@ -3,12 +3,8 @@ package com.fsck.k9.notification
 import com.fsck.k9.Account
 import com.fsck.k9.RobolectricTest
 import com.fsck.k9.controller.MessageReference
-import com.fsck.k9.core.BuildConfig
 import com.google.common.truth.Truth.assertThat
 import kotlin.test.assertNotNull
-import org.junit.Assert.fail
-import org.junit.Assume.assumeFalse
-import org.junit.Assume.assumeTrue
 import org.junit.Test

 private const val ACCOUNT_UUID = "1-2-3"
@@ -26,6 +22,7 @@ class NotificationDataStoreTest : RobolectricTest() {

         val result = notificationDataStore.addNotification(account, content, TIMESTAMP)

+        assertNotNull(result)
         assertThat(result.shouldCancelNotification).isFalse()

         val holder = result.notificationHolder
@@ -48,6 +45,7 @@ class NotificationDataStoreTest : RobolectricTest() {

         val result = notificationDataStore.addNotification(account, createNotificationContent("9"), TIMESTAMP)

+        assertNotNull(result)
         assertThat(result.shouldCancelNotification).isTrue()
         assertThat(result.cancelNotificationId).isEqualTo(NotificationIds.getSingleMessageNotificationId(account, 0))
     }
@@ -107,19 +105,23 @@ class NotificationDataStoreTest : RobolectricTest() {
     fun testNewMessagesCount() {
         val contentOne = createNotificationContent("1")
         val resultOne = notificationDataStore.addNotification(account, contentOne, TIMESTAMP)
+        assertNotNull(resultOne)
         assertThat(resultOne.notificationData.newMessagesCount).isEqualTo(1)

         val contentTwo = createNotificationContent("2")
         val resultTwo = notificationDataStore.addNotification(account, contentTwo, TIMESTAMP)
+        assertNotNull(resultTwo)
         assertThat(resultTwo.notificationData.newMessagesCount).isEqualTo(2)
     }

     @Test
     fun testIsSingleMessageNotification() {
         val resultOne = notificationDataStore.addNotification(account, createNotificationContent("1"), TIMESTAMP)
+        assertNotNull(resultOne)
         assertThat(resultOne.notificationData.isSingleMessageNotification).isTrue()

         val resultTwo = notificationDataStore.addNotification(account, createNotificationContent("2"), TIMESTAMP)
+        assertNotNull(resultTwo)
         assertThat(resultTwo.notificationData.isSingleMessageNotification).isFalse()
     }

@@ -128,37 +130,58 @@ class NotificationDataStoreTest : RobolectricTest() {
         val content = createNotificationContent("1")
         val addResult = notificationDataStore.addNotification(account, content, TIMESTAMP)

+        assertNotNull(addResult)
         assertThat(addResult.notificationData.activeNotifications.first()).isEqualTo(addResult.notificationHolder)
     }

     @Test
-    fun `adding notification for same message twice should throw in debug build`() {
-        assumeTrue(BuildConfig.DEBUG)
-
+    fun `adding notification for message with active notification should update notification`() {
         val content1 = createNotificationContent("1")
         val content2 = createNotificationContent("1")
-        notificationDataStore.addNotification(account, content1, TIMESTAMP)

-        try {
-            notificationDataStore.addNotification(account, content2, TIMESTAMP)
-            fail("Exception expected")
-        } catch (e: AssertionError) {
-            assertThat(e).hasMessageThat().matches("Notification for message .+ already exists")
+        val resultOne = notificationDataStore.addNotification(account, content1, TIMESTAMP)
+        val resultTwo = notificationDataStore.addNotification(account, content2, TIMESTAMP)
+
+        assertNotNull(resultOne)
+        assertNotNull(resultTwo)
+        assertThat(resultTwo.notificationData.activeNotifications).hasSize(1)
+        assertThat(resultTwo.notificationData.activeNotifications.first().content).isSameInstanceAs(content2)
+        assertThat(resultTwo.notificationStoreOperations).isEmpty()
+        with(resultTwo.notificationHolder) {
+            assertThat(notificationId).isEqualTo(resultOne.notificationHolder.notificationId)
+            assertThat(timestamp).isEqualTo(resultOne.notificationHolder.timestamp)
+            assertThat(content).isSameInstanceAs(content2)
         }
+        assertThat(resultTwo.shouldCancelNotification).isFalse()
     }

     @Test
-    fun `adding notification for same message twice should add another notification in release build`() {
-        assumeFalse(BuildConfig.DEBUG)
+    fun `adding notification for message with inactive notification should update notificationData`() {
+        notificationDataStore.addNotification(account, createNotificationContent("1"), TIMESTAMP)
+        notificationDataStore.addNotification(account, createNotificationContent("2"), TIMESTAMP)
+        notificationDataStore.addNotification(account, createNotificationContent("3"), TIMESTAMP)
+        notificationDataStore.addNotification(account, createNotificationContent("4"), TIMESTAMP)
+        notificationDataStore.addNotification(account, createNotificationContent("5"), TIMESTAMP)
+        notificationDataStore.addNotification(account, createNotificationContent("6"), TIMESTAMP)
+        notificationDataStore.addNotification(account, createNotificationContent("7"), TIMESTAMP)
+        notificationDataStore.addNotification(account, createNotificationContent("8"), TIMESTAMP)
+        val latestNotificationContent = createNotificationContent("9")
+        notificationDataStore.addNotification(account, latestNotificationContent, TIMESTAMP)
+        val content = createNotificationContent("1")

-        val content1 = createNotificationContent("1")
-        val content2 = createNotificationContent("1")
+        val resultOne = notificationDataStore.addNotification(account, content, TIMESTAMP)

-        val addResult1 = notificationDataStore.addNotification(account, content1, TIMESTAMP)
-        val addResult2 = notificationDataStore.addNotification(account, content2, TIMESTAMP)
+        assertThat(resultOne).isNull()

-        assertThat(addResult1.notificationHolder.notificationId)
-            .isNotEqualTo(addResult2.notificationHolder.notificationId)
+        val resultTwo = notificationDataStore.removeNotifications(account) {
+            listOf(latestNotificationContent.messageReference)
+        }
+
+        assertNotNull(resultTwo)
+        val notificationHolder = resultTwo.notificationData.activeNotifications.first { notificationHolder ->
+            notificationHolder.content.messageReference == content.messageReference
+        }
+        assertThat(notificationHolder.content).isSameInstanceAs(content)
     }

     private fun createAccount(): Account {


Retain doctype when sanitizing HTML

Fixes #5882

broken display of

broken display of
### Checklist

- [X] I have used the search function to see if someone else has already submitted the same bug report.
- [X] I will describe the problem with as much detail as possible.

### App version

5.9.10

### Where did you get the app from?

Other

### Android version

Android 11, Havoc OS (custom)

### Device model

OnePlus 3T

### Steps to reproduce

Open mail by Mozilla

### Expected behavior

continous layout without gaps.

### Actual behavior

There is lots of blank space between paragraphs. (several times the device height between each paragraph is just white screen space so you need to scroll very far)
This might be due to the nested table layout but without a debugger it's hard for me to figure out which html tags K9 might have removed that broke the layout this way.

If you have time at some point, then please take a look.
No high priority, I just noticed the broken layout.

I attached the email as an html file. I modified three a href attributes, the rest is unmodified.
[mail.html zipped](https://github.com/k9mail/k-9/files/7936679/mail.zip)

### Logs

_No response_

I created a command line application that modifies HTML the same way that K-9 Mail does. See #5885

Could you use this to debug the issue yourself?
yes,
I finally had the time to look at this.

I tested this in Firefox and used the inspection tools in order to switch to a mobile layout (smartphone icon on the right)

The html mail starts with
```
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office">
<head>
    <!--[if gte mso 9]>
    <xml>
        <o:OfficeDocumentSettings>
            <o:AllowPNG/>
            <o:PixelsPerInch>96</o:PixelsPerInch>
        </o:OfficeDocumentSettings>
    </xml>
    <![endif]-->
```

Your scripts removes everything but keeps the naked html and head tags.
The actually relevant part of the code above seems to be the doctype.
Reintroducing it (html 4 strict)
`<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">`
or the html5 doctype instead:
`<!doctype html>`
both lines fix the layout.

The issue being:
The layout uses divs (one attribute only: height:100%) for nesting.
According to [Stackoverlow](https://stackoverflow.com/questions/1997842/why-certain-doctype-declarations-cause-100-height-tables-and-divs-to-stop-worki):
> If you have height: 100% and the height of the parent element is auto then [100% means auto](http://www.w3.org/TR/CSS21/visudet.html#the-height-property).

the new strict behavior in browser engines interpret the height differently and lets the element inherit from it's parent.
Removing the doctype makes the engine go into "quirks" mode and interprets the layout/css differently. (legacy mode)
Quirks mode then actually renders this height: 100% which in turn creates lots of blank space between each element of the layout.

I don't know why these nested divs (parent element: grid-div; child element: table) even exist in the first place (maybe legacy mail clients?), but removing them instead of the !doctype also fixes the layout.

But you might want to think about keeping !doctypes intact unless this was done to prevent other mail layouts from breaking

./gradlew :app:html-cleaner:test --tests "app.k9mail.html.cleaner.HtmlSanitizerTest"

diff --git a/app/html-cleaner/src/main/java/app/k9mail/html/cleaner/BodyCleaner.kt b/app/html-cleaner/src/main/java/app/k9mail/html/cleaner/BodyCleaner.kt
index 70045fe60bf..0720118d385 100644
--- a/app/html-cleaner/src/main/java/app/k9mail/html/cleaner/BodyCleaner.kt
+++ b/app/html-cleaner/src/main/java/app/k9mail/html/cleaner/BodyCleaner.kt
@@ -43,10 +43,17 @@ internal class BodyCleaner {

     fun clean(dirtyDocument: Document): Document {
         val cleanedDocument = cleaner.clean(dirtyDocument)
+        copyDocumentType(dirtyDocument, cleanedDocument)
         copyBodyAttributes(dirtyDocument, cleanedDocument)
         return cleanedDocument
     }

+    private fun copyDocumentType(dirtyDocument: Document, cleanedDocument: Document) {
+        dirtyDocument.documentType()?.let { documentType ->
+            cleanedDocument.insertChildren(0, documentType)
+        }
+    }
+
     private fun copyBodyAttributes(dirtyDocument: Document, cleanedDocument: Document) {
         val cleanedBody = cleanedDocument.body()
         for (attribute in dirtyDocument.body().attributes()) {

diff --git a/app/html-cleaner/src/test/java/app/k9mail/html/cleaner/HtmlSanitizerTest.kt b/app/html-cleaner/src/test/java/app/k9mail/html/cleaner/HtmlSanitizerTest.kt
index 7630fd56271..9ebc3857b5e 100644
--- a/app/html-cleaner/src/test/java/app/k9mail/html/cleaner/HtmlSanitizerTest.kt
+++ b/app/html-cleaner/src/test/java/app/k9mail/html/cleaner/HtmlSanitizerTest.kt
@@ -376,6 +376,37 @@ class HtmlSanitizerTest {
         )
     }

+    @Test
+    fun `should keep HTML 5 doctype`() {
+        val html =
+            """
+            <!doctype html>
+            text
+            """.trimIndent().trimLineBreaks()
+
+        val result = htmlSanitizer.sanitize(html)
+
+        assertThat(result.toCompactString()).isEqualTo(html)
+    }
+
+    @Test
+    fun `should keep HTML 4_01 doctype`() {
+        val html =
+            """
+            <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
+            text
+            """.trimIndent().trimLineBreaks()
+
+        val result = htmlSanitizer.sanitize(html)
+
+        assertThat(result.toCompactString()).isEqualTo(
+            """
+            <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
+            text
+            """.trimIndent().trimLineBreaks()
+        )
+    }
+
     private fun Document.toCompactString(): String {
         outputSettings()
             .prettyPrint(false)