Tags: mattfbacon/mox
Tags
v0.0.7
version: v0.0.7
date: 2023-09-24
Update instructions:
Due to a bug with "expunging" (deleting) messages that were
junk-filter-trained, messages that were removed from disk could be resurrected,
causing errors when mox would later try to open such messages again. Before
upgrading, you should first check and resolve this problem:
- Find missing files by running: mox-v0.0.6 fixmsgsize
- Create empty replacements for the missing files (use "touch data/accounts/...").
- Update the message metadata in the database for the newly created files:
mox-v0.0.6 fixmsgsize
(this will fix the previously missing files, but should no longer find new
missing files)
- Fix per-mailbox message/unread counts for each affected account, run:
mox-v0.0.6 recalculatemailboxcounts <account>
("account" is the name as it appears in the data/accounts/ directory,
not necessarily an email address)
- The affected accounts will see the recreated empty messages, which
can be deleted.
Now with the storage consistency resolved, you can start with the upgrade. Mox
v0.0.7 adds message threading, and all messages will be read and assigned a
message thread. This is done in the background, in two steps, and may take a
while. The first step adds Message-ID and a "thread base subject" to each
message in the database. The second step reads through all messages and uses
their References/In-Reply-To/Subject message headers to match threads. In
testing, upgrading took approximately 1 minute per 100k messages, but it will
depend on the hardware. Accounts are available for reading and delivery during
the upgrade, but the webmail may tell you that threading is not yet available.
You should do a dry-run of the upgrade first:
- Make a temporary backup with the old mox version:
mox-v0.0.6 backup data/tmp/testupgrade
- Verify that all is well with the old version:
mox-v0.0.6 verifydata data/tmp/testupgrade
- Trigger the threading upgrade using the new version:
mox-v0.0.7 openaccounts data/tmp/testupgrade <your-account-name>
(it prints nothing until done; the live upgrade prints progress)
- Verify the new state:
mox-v0.0.7 verifydata data/tmp/testupgrade
With a successful dry-run, the upgrade should go smoothly. Make a new backup
with `mox-v0.0.6 backup data/tmp/backup` (the previous backup used for the
dry-run has been modified, so couldn't be used to restore!), replace the binary
and restart.
After the upgrade, any accounts affected by the "missing file" problem should
get their "uid validity" increased, so IMAP clients will resynchronize. Run
"mox-v0.0.7 bumpuidvalidity <account>" for each affected account. The change is
made directly in the database file, so no IMAP/webmail sessions should be
active for the account. If any sessions are active, the database file is locked
and "bumpuidvalidity" will print a timeout error. Briefly shutting down mox is
an option, the bumpuidvalidity command operates quickly.
Apologies for the inconvenience, mox aims to make administrating a mail server
easier than this.
If you run into any problems, please create an issue.
New features:
- Keep track of message threading, and add a threaded view to the webmail. Newly
delivered messages are matched against threads, as are imported messages. The
message threading is currently only exposed through the webmail client, not yet
through IMAP. The webmail has new keyboard shortcuts to navigate between
threads, collapse/expand them (with state remembered), or mute them (so new
deliveries to the thread are marked as read).
- Add option to accept DMARC and TLS reports for other domains. You would add
the other domain in mox, and specify an alternative domain (of your regular
dmarc/tls reporting address) for the recipient of the reports. The webadmin
DNS-check page will show the required DNS records for delegated reporting.
Incoming reports will be delivered to the configured reporting address, and
processed by mox.
- Easier account setup on Apple devices without device management profiles
(.mobileconfig files). Download a profile from the account web interface, at an
email address. Or open the QR-code to easily get the profile on a mobile device.
For issue mjl-#65 by x8x.
- Transparent gzip compression when webserving files for selected content-types
that are likely compressible. Both for static files (with a cache of compressed
files, max 512MB stored in $datadir/tmp/httpstaticcompresscache), and for
forwarded requests (gzip level "fast"). Compression can be enabled per handler.
Improvements:
- In SMTP server, for submission (with authenticated clients), don't fail on a
bad domain/IP address in the EHLO command. With submission the domain/IP is
irrelevant, and clients often fill in something that isn't strictly correct. No
need to prevent those users from submitting email (except in pedantic mode).
For issue mjl-#55 reported by gimpf.
- At top of config files, mention the config file format, and hints to prevent
likely mistakes (given how sconf is different from what admins may be used to),
including a pointer to the sconf documentation page. For issue mjl-#56 reported by
kikoreis.
- Recognize when quickstart is probably run behind a NAT, possibly a container,
and set the "NATIPs" field of the "public" listener in mox.conf accordingly.
This triggers when you set up mox for a public domain, but only have
private/loopback IPs on the machine. For issue mjl-#59 reported by pmarini.
- When moving a message out of Rejects mailbox, mark it as unread. For issue mjl-#63
by x8x.
- The "mox setaccountpassword" subcommand now takes an account name as
parameter instead of an email address. The email address could be confusing
in the face of wildcard addresses and a typo/non-explicitly-created address:
The account holding the wildcard address would get a new password. For issue
mjl-#68 by x8x.
- Make Mac OS X Mail use the special-use mailbox attributes so it finds the
correct "Sent" and "Trash" mailboxes (instead of creating its own "Sent
Messages" and "Deleted Messages" mailboxes). The IMAP server now
unconditionally sends the special-use flags, even if the mail client doesn't
request it. For issue mjl-#66 by x8x.
Bug fixes:
- Fix expunging of messages marked junk/nonjunk. The messages would be marked
as expunged, then the junkfilter would retrain and clear the expunged field
again. This would cause the message to be resurrected while the on-disk message
file was already removed. Trying to read such messages would fail. The update
instructions should help fix the problem.
- Fix "mox sendmail" when submitting over a TLS connection by setting the remote
host name to verify the certificate of. Due to a logic bug the name wasn't set
and the connection would fail due to the missing setting.
- Don't generate duplicate suggested SPF record if hostname is equal to domain
name, e.g. postmaster@mail.domain.example. Fixes issue mjl-#46 reported by x8x.
- Fix showing attachments of type text/plain in webmail, they weren't shown at
all because they were skipped when parsing the message during webmail message
processing.
- Fix parsing the List-Post header in messages, for use in webmail when replying
to a mailing list message.
- When moving a message to the mailbox that has the special-use "junk" flag,
mark the message as junk and retrain. This should have already worked just like
the "AutomaticJunkFlags" config option, and the default account config already
handles marking messages as junk based on that option, but the special-use flag
should be recognized independently and now also takes precedence.
- Set the correct special-use mailbox flag "\Drafts" instead of "\Draft" on the
draft mailbox. Mail clients may have not found the correct drafts mailbox
before. For issue mjl-#66 by x8x.
- Fix "mox bumpuidvalidity" to not create a (mostly harmless) uidvalidity
inconsistency that "mox verifydata" will warn about. For issue mjl-#61 by x8x.
- And quite a few smaller tweaks/improvements/fixes.
Special thanks for contributions and/or feedback from: x8x, gimpf, kikoreis,
pmarini, fairking, gedw99, hmfaysal (and those I missed).
Feedback, requests, bug reports, contributions (start small!) are all welcome.
Good news: Mox is now being funded for a year of continued development through
the NGI0 Entrust Fund, a fund established by NLnet with financial support from
the European Commission's Next Generation Internet programme,
https://nlnet.nl/project/Mox/!
v0.0.6
Update instructions:
Make a backup, replace the binary and restart.
After the upgrade, the first time an account is opened with this new version,
new message indexes are created and mailbox message count statistics are
calculated. For large mailboxes, the time and memory this takes can be
noticable. The upgrade tests take about 15 seconds on 570k messages on a
thinkpad x1 from 2018, and run with a max memory data size of 768MB.
Don't forget to make a backup of the data directory with your currently running
mox before upgrading (e.g. "mox-v0.0.5 backup data/tmp/backup"). You can
dry-run the upgrade by making a separate backup ("mox-v0.0.5 backup
data/tmp/testupgrade") and running the "verifydata" command with the new mox
version ("mox-v0.0.6 verifydata data/tmp/testupgrade"). Running "verifydata"
with a newer mox will make changes to the database files, so don't run it on a
backup you may need to restore.
It is recommended to run the new "mox reparse" command after upgrading. It will
reparse all messages with the improved message parsing code.
For existing installations, the new webmail must be enabled manually in mox.conf
with config options "WebmailHTTP" and/or "WebmailHTTPS", similar to
"AccountHTTP(s)". See the example config printed by "mox config
describe-static".
If you are forwarding email to an address hosted with mox, you may want to
configure the new "IsForward" and possibly "AcceptRejectsToMailbox" options in
a delivery ruleset for better junk handling/analysis.
If your mox is behind a NAT, and your mox.conf uses config option IPsNATed, you
should switch to new config option NATIPs, and specify the public IPs that are
NATed.
New features:
- Webmail, for reading/writing messages. It is similar to other regular and
webmail clients, with a decent set of features for a first version (with more to
come in the future). Webmail is enabled for new installs created with the
quickstart. For more details about the implementation, see
mjl-@849b4ec.
- IMAP extensions CONDSTORE and QRESYNC have been implemented. With CONDSTORE,
changes to messages are tracked with a modification sequence, "modseq".
This is used by mail clients both to efficiently find changes when reconnecting,
and for conditionally storing updated message flags (only if modseq is still as
expected). QRESYNC is an additional extension for faster full mailbox
synchronization. These are useful by themselves, but keeping track of the
"modseq" will also help with JMAP.
Improvements:
- Faster IMAP "STATUS" command (which checks mailbox status), because mox now
keeps track of total/unseen/deleted number of messages in mailboxes. Noticable
in larger mailbox (>50k messages).
- Be less strict by default when parsing messages, and handle non-ascii/utf-8
encodings in message subjects. We already accepted messages with problems, but
stored them mostly unparsed. IMAP clients would parse the message themselves so
users wouldn't notice anything, but now that we have webmail it is more
important to have parsed forms of messages for problematic messages.
- Properly decode character encodings other than ASCII and UTF-8 when
returning/searching text in messages.
- Be more lenient for (authenticated) submission of email over SMTP (but not
during regular SMTP delivery). Before, we were strict about certain protocol
violations, but that wasn't helpful behaviour for legitimate users. In pedantic
mode, mox still rejects these violating commands.
- New config options "AcceptRejectsToMailbox" and "IsForward" in Rulesets, for
handling incoming messages that are forwards from another address. By setting
"AcceptRejectsToMailbox", if the junk analysis says a message is junk, the
message is accepted during SMTP instead of rejected, but delivered to the
configured mailbox. It isn't always a good idea to reject forwarded messages
that are junk. By setting "IsForward", the junk analysis and future
classification based on such a message is done differently. See the config
option for details. (PR mjl-#50 by bobobo1618)
- Add a KeepRejects options to account configurations, so the Rejects mailbox
isn't automatically cleaned up. (PR mjl-#49 by bobobo1618)
- Add option NATIPs to a listener in mox.conf for better handling when mox is
configured behind a NAT.
- "mox verifydata" now checks for more potential issues, and mox gained a few
subcommands to fix those issues.
Bug fixes:
- Fix showing the progress while importing messages. Messages were still
imported, but the SSE connection with progress updates was broken.
- Fix potential delays in propagating changes to IMAP IDLE connections.
- IMAP compliance: add missing space after continuation line ("+").
- IMAP compliance: add missing empty untagged response for SEARCH in case of no
matches.
- Fix the -existing-webserver flag for "mox quickstart". Its TLS key/cert check
tripped it up.
- And many more small fixes and improvements.
Special thanks for contributions and/or feedback from: Mendel, bobobo1618,
hmfaysal, x8x, kikoreis, gerben, andrii, liesbeth, morki, gedw99 and everyone I
forgot. More feedback/bug reports welcome!
v0.0.5 Update instructions: No special steps required other than updating the binary. Fixes/Improvements: - Fix "mox localserve", and some commands that talk to a running mox instance over the ctl socket. Broken in previous release. - During quickstart, warn if the host name resolves to a loopback IP, as it will likely prevent email delivery to local accounts from working. (mjl-#37 by dmikushin) - In case of DNS resolve errors on Linux likely due to missing name server in /etc/resolv.conf, point user to systemd-resolved manual page, the likely cause. (mjl-#38 by ArnoSen) - Redirect requests for /admin to /admin/, to prevent confusing users about failed credentials when they try to open the admin web interface. (mjl-#43 by FieryCod, lemos1235) - Fix race condition when handling forwarded websocket connections. Could show up as calling Close on a nil connection. (found by existing tests) - Fix parsing bugs in the IMAP server. One of them may cause correct commands to be rejected, but this probably isn't commonly triggered. (found by new tests) - Fix bug interpreting UID sets in the IMAP server, potentially causing a SEARCH not to return all matching messages, and potentially not EXPUNGE-ing (removing) all messages marked as deleted. (found by new tests) - Fix bug with assigning UIDs when renaming Inbox. Renaming Inbox is a special case and its special handling wasn't renumbering UIDs of moved messages, potentially leading to errors on later operations on the destination mailbox. Command "mox verifydata" has been updated to check for this inconsistency. Two new unlisted commands are introduced that can correct these inconsistencies: "mox reassignuids" and "mox fixuidmeta". This will only affect you if you have renamed your Inbox, not a common operation. (found by new tests) - Fix importing messages with a date with a timezone >= 24 hours. (found while importing old email archives) New features: - Implement user-defined flags/keywords in the IMAP server. Previously, only the standard keywords like \Seen, \Draft, etc were supported. - Configurable routes for outgoing email, through configurable transports. The default transport is regular direct delivery to MX hosts. Other transport options: 1. delivery through another mail server with SMTP (with various TLS/STARTTLS/authentication options), to a relay/smarthost or 3rd party account; 2. direct delivery but making connections through a SOCKS proxy. Routes can be configured for accounts, domains, or globally. A matching route selects the transport. If no route matches (the default), the default transport (direct delivery) is selected. Routes are evaluated for each delivery attempt. They are typically matched on destination address, and can also start matching after a configurable number of failed delivery attempts (with another transport). (mjl-#36 by dmikushin, and earlier discussions on HN) - Support more authentication mechanisms in sendmail: SCRAM-SHA-256, SCRAM-SHA-1, CRAM-MD5. Thanks to all who provided feedback, bug reports, patches. You've helped mox become a better mail server!
v0.0.4 Update instructions: No special steps required other than updating the binary. Changes: - Bugfix: When DKIM-signing messages, take the Date and Message-Id header into account if we added them just before. If you submit a message without these headers, they are added automatically, but the signature was calculated over the message as if those headers were not added. The receiving side would evaluate such DKIM signature as invalid. - Bugfix: SMTP transactions with SMTP from/to addresses with double quotes would fail. - Bugfix: Delivery of messages to addresses where the domain does not have an MX record would fail. In these cases, e.g. where CNAME records need to be followed or where we need to connect directly to the host, we were aborting immediately after starting because of a reused canceled Go context. - Bugfix: Use MTASTSHTTPS.NonTLS from the config file for the MTASTS config, not the similar field for Autoconfig. (mjl-#29) - Bugfix: If we encountered an error fetching an mta-sts policy as part of a delivery attempt, we weren't trying to continue in strict mode, which is safe to do in this case. - Bugfix: Dropping a message from the queue (with "mox queue drop <id>" or through the admin web interface) would be removed from the database, but left in the file system. If you still have old message files in data/queue/, they can be removed manually. - Update to latest bstore database library and add a new index on the Message database table, for fast indexed lookups of DKIM reputation for incoming messages. - Add "mox backup" and "mox verifydata" subcommands. Backup makes a consistent snapshot of the databases, message files, and the other files in the data directory. Simply copying the DB files while they are in use will lead to corrupted files sooner or later, so a consistent snapshot is important. The message files are hardlinked when possible, so a snapshot is fast and disk space consumption is limited to the database files. The verifydata subcommand verifies the database files and messages, useful to verify backups and restores. See documentation for these new commands for details. - Add support for websockets to the reverse proxy webserver. (mjl-#25) - After "logout" of an IMAP connection, close the connection. (mjl-#30) - Open pre existing TLS certificates/keys as the root user, not as the unprivileged user. Makes it easier to use these files when set up by other tools. (mjl-#30) - Various smaller technical and user-experience improvements. Thanks for the feedback, issues and PR's you sent in. naturalethic, 154pinkchairs, cuu508, inigoserna, kou029w, belst, and those I forgot. Keep it coming!
v0.0.3 Update instructions: No special steps required other than updating the binary. Important bug fixes: - When dkim-signing submitted messages, use the domain from the "message from header" instead of "smtp mail from". If you had an account with multiple addresses in different domains, then with some mail clients the DKIM signature on outgoing messages could be for the wrong domain. Changes: - Add "mox localserve", a subcommand for running mox locally for email-related testing/developing. Localserve creates a config for listening on localhost for imap/smtp/submission/http, on port numbers 1000 + the common service port numbers. All incoming email is accepted (if checks pass), and a few pattern in localparts are recognized and result in delivery errors. - Implement configurable (rate) limits on outgoing messages, per account. By default an account can send 1000 messages per day, to max 200 first-time receivers. The goal is to limit damage spammers can do with compromised accounts. - In domains.conf, for an account, the Destinations mapping will now always use full email addresses, no longer localparts relative to the Domain configured for the account. The old form with just a localpart is still accepted. When writing domains.conf through the cli commands or admin web pages, the destinations will automatically be written with full email addresses. In the future, support for the localpart-only form will be removed. - Implement a catchall address for a domain, an address without localpart (i.e. starting with "@<domain>"). If an incoming message does not match a configured address, it is delivered to the account configured that has the catchall address configured. - ACME TLS certificate validation can also be done with method http-01 (previously only with tls-alpn-01). - If you run mox behind a NAT, you can now specify "IPsNATed: true" in the SMTP listener to skip a few DNS checks that previously would always fail due to the IPs being NATed. - Add subcommand "mox dkim sign", useful for testing signatures. - For a "WebRedirect" rule, don't match a request if the destination URL would result in the same URL, causing a redirect. By not matching, the next configuration rules are evaluated for a match. - Various smaller technical and user-experience improvements. Thanks to everyone who sent in feedback (e.g. by creating an issue on github). It's much appreciated and essential for improving mox!
v0.0.2 - Start up as root, bind network sockets, then drop privileges to a process as user mox to do the actual work. Makes it easier to run mox on BSDs, and paves the way for future privilege separation work. - Add a webserver, with support for reverse proxying, serving static files, configurable redirects. With ACME of course. Takes away need to configure mox behind an existing webserver/reverse proxy, greatly simplifying configuration/operation. - Add docker images and script to build images. - Make it possible to run mox behind an existing webserver/reverse proxy too. Also add quickstart flag to make it easier to generate a config file for this case. - Quickstart improvements: Add flag to generate a config for a specified host name (instead of guessing). If we found public IPs, check them with popular DNS blocklists and warn user if they are listed. - Add encrypted cid to SMTP error response lines, for easier debugging. - Fix importing of junk messages that the junk filter cannot parse. - Fix bug verifying SPF. For IP lookups (e.g. for mechanism "a"), we would only look up with the same address family as the incoming connection. If there was no record, we counted a void lookup. This could lead to incorrect SPF failures, we reached the void lookup limit when we shouldn't. If the record exists for another address family (a vs aaaa), we shouldn't count a void lookup. - Fix reputation learning of messages that were initially delivered to the Rejects mailbox and then moved out. If you marked them as junk or nonjunk (e.g. by moving to Archive or Junk), we should use the message for reputation calculation of new incoming messages, but we weren't. - Lots of smaller improvements, bug fixes. Thanks for reporting issues: mteege, hismailbulut, belst, idnovic, Jens, the-solipsist, pexarkh, sherief, gedw99, agucova, lormayna, and everyone I forgot. Thanks for all your feedback, discussions, suggestions! It shaped these changes, and changes to come.