From 00018e7e2b226fafb3a5ab50db078885ca6c8ddc Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Wed, 24 Jan 2024 17:05:55 +0100 Subject: [PATCH 001/267] general: update base image to Debian 12 ("Bookworm") (#3403) Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Co-authored-by: Casper --- CHANGELOG.md | 27 +++ Dockerfile | 15 +- target/bin/debug-getmail | 2 +- target/bin/getmail-cron | 2 +- target/postfix/main.cf | 7 +- target/scripts/build/compile.sh | 2 +- target/scripts/build/packages.sh | 191 ++++++++---------- target/scripts/check-for-changes.sh | 2 +- .../scripts/startup/setup.d/security/misc.sh | 5 - target/supervisor/conf.d/supervisor-app.conf | 2 +- test/tests/parallel/set1/getmail.bats | 4 +- .../parallel/set1/spam_virus/amavis.bats | 4 +- .../disabled_clamav_spamassassin.bats | 6 +- .../set1/spam_virus/postgrey_enabled.bats | 8 +- .../parallel/set1/spam_virus/rspamd_dkim.bats | 4 +- .../process_check_restart.bats | 4 +- test/tests/serial/mail_with_oauth2.bats | 4 + test/tests/serial/open_dkim.bats | 4 +- 18 files changed, 145 insertions(+), 148 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a77b28f..35ee73a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,33 @@ All notable changes to this project will be documented in this file. The format > **Note**: Changes and additions listed here are contained in the `:edge` image tag. These changes may not be as stable as released changes. +The most noteworthy change of this release is the update of the container's base image from Debian 11 ("Bullseye") to Debian 12 ("Bookworm"). This update alone involves breaking changes and requires a careful update! + +### Breaking + +- **Updated base image to Debian 12** ([#3403](https://github.com/docker-mailserver/docker-mailserver/pull/3403)) + - Changed the default of `DOVECOT_COMMUNITY_REPO` to `0` (disabled) - the Dovecot community repo will (for now) not be the default when building the DMS. + - While Debian 12 (Bookworm) was released in June 2023 and the latest Dovecot `2.3.21` in Sep 2023, as of Jan 2024 there is no [Dovecot community repo available for Debian 12](https://repo.dovecot.org). + - This results in the Dovecot version being downgraded from `2.3.21` (DMS v13.3) to `2.3.19`, which [may affect functionality when you've explicitly configured for these features](https://github.com/dovecot/core/blob/30cde20f63650d8dcc4c7ad45418986f03159946/NEWS#L1-L158): + - OAuth2 (_mostly regarding JWT usage, or POST requests (`introspection_mode = post`) with `client_id` + `client_secret`_). + - Lua HTTP client (_DNS related_). + - Updated packages. For an overview, [we have a review comment on the PR that introduces Debian 12](https://github.com/docker-mailserver/docker-mailserver/pull/3403#issuecomment-1694563615) + - Notable major version bump: `openssl 3`, `clamav 1`, `spamassassin 4`, `redis-server 7`. + - Notable minor version bump: `postfix 3.5.23 => 3.7.9` + - Notable minor version bump + downgrade: `dovecot 2.3.13 => 2.3.19` (_Previous release provided `2.3.21` via community repo, `2.3.19` is now the default_) + - Updates to `packages.sh`: + - The script now uses `/etc/os-release` to determine the release name of Debian + - Removed custom installations of Fail2Ban, getmail6 and Rspamd + - Updated packages lists and added comments for maintainability +- **Postfix:** + - Postfix upgrade from 3.5 to 3.7 ([#3403](https://github.com/docker-mailserver/docker-mailserver/pull/3403)) + - `compatibility_level` was raised from `2` to `3.6` + - Postfix has deprecated the usage of `whitelist` / `blacklist` in config parameters and logging in favor of `allowlist` / `denylist` and similar variations. ([#3403](https://github.com/docker-mailserver/docker-mailserver/pull/3403/files#r1306356328)) + - This [may affect monitoring / analysis of logs output from Postfix](https://www.postfix.org/COMPATIBILITY_README.html#respectful_logging) that expects to match patterns on the prior terminology used. + - DMS `main.cf` has renamed `postscreen_dnsbl_whitelist_threshold` to `postscreen_dnsbl_allowlist_threshold` as part of this change. + - `smtpd_relay_restrictions` (relay policy) is now evaluated after `smtpd_recipient_restrictions` (spam policy). Previously it was evaluated before `smtpd_recipient_restrictions`. Mail to be relayed via DMS must now pass through the spam policy first. + - The TLS fingerprint policy has changed the default from MD5 to SHA256 (_DMS does not modify this Postfix parameter, but may affect any user customizations that do_). + ## [v13.3.1](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v13.3.1) ### Fixes diff --git a/Dockerfile b/Dockerfile index e822632a..b8bf89bc 100644 --- a/Dockerfile +++ b/Dockerfile @@ -4,10 +4,10 @@ # This is in preparation for more granular stages (eg ClamAV and Fail2Ban split into their own) ARG DEBIAN_FRONTEND=noninteractive -ARG DOVECOT_COMMUNITY_REPO=1 +ARG DOVECOT_COMMUNITY_REPO=0 ARG LOG_LEVEL=trace -FROM docker.io/debian:11-slim AS stage-base +FROM docker.io/debian:12-slim AS stage-base ARG DEBIAN_FRONTEND ARG DOVECOT_COMMUNITY_REPO @@ -30,8 +30,6 @@ COPY target/scripts/helpers/log.sh /usr/local/bin/helpers/log.sh RUN /bin/bash /build/packages.sh && rm -r /build - - # ----------------------------------------------- # --- Compile deb packages ---------------------- # ----------------------------------------------- @@ -130,7 +128,8 @@ COPY \ # hadolint ignore=SC2016 RUN </etc/default/spamassassin sedfile -i -r 's/^\$INIT restart/supervisorctl restart amavis/g' /etc/spamassassin/sa-update-hooks.d/amavisd-new mkdir /etc/spamassassin/kam/ curl -sSfLo /etc/spamassassin/kam/kam.sa-channels.mcgrail.com.key https://mcgrail.com/downloads/kam.sa-channels.mcgrail.com.key @@ -189,7 +188,6 @@ RUN < /dev/null|/usr/bin/supervisorctl signal hup rsyslog >/dev/null|g' /usr/lib/rsyslog/rsyslog-rotate + # this change is for our alternative process manager rather than part of + # a fix related to the change preceding it. + echo -e '\n/usr/bin/supervisorctl signal hup rsyslog >/dev/null' >>/usr/lib/rsyslog/rsyslog-rotate EOF # ----------------------------------------------- diff --git a/target/bin/debug-getmail b/target/bin/debug-getmail index c007ebd2..27834475 100644 --- a/target/bin/debug-getmail +++ b/target/bin/debug-getmail @@ -15,5 +15,5 @@ else fi for FILE in /etc/getmailrc.d/getmailrc*; do - /usr/local/bin/getmail --getmaildir "${GETMAILDIR}" --rcfile "${FILE}" --dump | tail -n +7 + getmail --getmaildir "${GETMAILDIR}" --rcfile "${FILE}" --dump | tail -n +6 done diff --git a/target/bin/getmail-cron b/target/bin/getmail-cron index 8f31e3b1..8e1e4749 100644 --- a/target/bin/getmail-cron +++ b/target/bin/getmail-cron @@ -2,6 +2,6 @@ for FILE in /etc/getmailrc.d/getmailrc*; do if ! pgrep -f "${FILE}$" &>/dev/null; then - /usr/local/bin/getmail --getmaildir /var/lib/getmail --rcfile "${FILE}" + getmail --getmaildir /var/lib/getmail --rcfile "${FILE}" fi done diff --git a/target/postfix/main.cf b/target/postfix/main.cf index 8c1d4c89..a0d805cc 100644 --- a/target/postfix/main.cf +++ b/target/postfix/main.cf @@ -86,7 +86,7 @@ postscreen_dnsbl_sites = list.dnswl.org=127.0.[0..255].1*-3 list.dnswl.org=127.0.[0..255].[2..3]*-4 postscreen_dnsbl_threshold = 3 -postscreen_dnsbl_whitelist_threshold = -1 +postscreen_dnsbl_allowlist_threshold = -1 postscreen_greet_action = enforce postscreen_bare_newline_action = enforce @@ -121,7 +121,4 @@ smtp_header_checks = pcre:/etc/postfix/maps/sender_header_filter.pcre # http://www.postfix.org/COMPATIBILITY_README.html # If backwards-compaitibilty log messages appear, fix them by explicitly adding # the legacy or new default value (alternatively raise the compatibility_level) -# -# TODO: The next compatibility_level is 3.6, when Postfix 3.6 is available consider -# bumping this value after taking the compaitibilty changes into account. -compatibility_level = 2 +compatibility_level = 3.6 diff --git a/target/scripts/build/compile.sh b/target/scripts/build/compile.sh index a7010ba5..56feea72 100644 --- a/target/scripts/build/compile.sh +++ b/target/scripts/build/compile.sh @@ -17,7 +17,7 @@ function _compile_dovecot_fts_xapian() { tar xzvf dovecot-fts-xapian.tar.gz cd fts-xapian-1.5.5 USER=root dh_make -p dovecot-fts-xapian-1.5.5 --single --native --copyright gpl2 -y - rm debian/*.ex debian/*.EX + rm debian/*.ex cp PACKAGES/DEB/control debian/ cp PACKAGES/DEB/changelog debian/ cp PACKAGES/DEB/compat debian/ diff --git a/target/scripts/build/packages.sh b/target/scripts/build/packages.sh index e57cfe07..24dae8a7 100644 --- a/target/scripts/build/packages.sh +++ b/target/scripts/build/packages.sh @@ -1,10 +1,13 @@ #!/bin/bash -# -eE :: exit on error (do this in functions as well) -# -u :: show (and exit) when using unset variables +# -e :: exit on error (do this in functions as well) +# -E :: inherit the ERR trap to functions, command substitutions and sub-shells +# -u :: show (and exit) when using unset variables # -o pipefail :: exit on error in pipes set -eE -u -o pipefail +VERSION_CODENAME='bookworm' + # shellcheck source=../helpers/log.sh source /usr/local/bin/helpers/log.sh @@ -17,17 +20,38 @@ function _pre_installation_steps() { _log 'trace' 'Updating package signatures' apt-get "${QUIET}" update - _log 'trace' 'Installing packages that are needed early' - apt-get "${QUIET}" install --no-install-recommends apt-utils 2>/dev/null - _log 'trace' 'Upgrading packages' apt-get "${QUIET}" upgrade + + _log 'trace' 'Installing packages that are needed early' + # add packages usually required by apt to + # - not log unnecessary warnings + # - be able to add PPAs early (e.g., Rspamd) + local EARLY_PACKAGES=( + apt-utils # avoid useless warnings + apt-transport-https ca-certificates curl gnupg # required for adding PPAs + systemd-standalone-sysusers # avoid problems with SA / Amavis (https://github.com/docker-mailserver/docker-mailserver/pull/3403#pullrequestreview-1596689953) + ) + apt-get "${QUIET}" install --no-install-recommends "${EARLY_PACKAGES[@]}" 2>/dev/null +} + +function _install_utils() { + _log 'debug' 'Installing utils sourced from Github' + _log 'trace' 'Installing jaq' + curl -sSfL "https://github.com/01mf02/jaq/releases/latest/download/jaq-v1.2.0-$(uname -m)-unknown-linux-gnu" -o /usr/bin/jaq && chmod +x /usr/bin/jaq + + _log 'trace' 'Installing swaks' + local SWAKS_VERSION='20240103.0' + local SWAKS_RELEASE="swaks-${SWAKS_VERSION}" + curl -sSfL "https://github.com/jetmore/swaks/releases/download/v${SWAKS_VERSION}/${SWAKS_RELEASE}.tar.gz" | tar -xz + mv "${SWAKS_RELEASE}/swaks" /usr/local/bin + rm -r "${SWAKS_RELEASE}" } function _install_postfix() { _log 'debug' 'Installing Postfix' - _log 'warn' 'Applying workaround for Postfix bug (see https://github.com//issues/2023#issuecomment-855326403)' + _log 'warn' 'Applying workaround for Postfix bug (see https://github.com/docker-mailserver/docker-mailserver/issues/2023#issuecomment-855326403)' # Debians postfix package has a post-install script that expects a valid FQDN hostname to work: mv /bin/hostname /bin/hostname.bak @@ -43,12 +67,17 @@ function _install_postfix() { function _install_packages() { _log 'debug' 'Installing all packages now' - ANTI_VIRUS_SPAM_PACKAGES=( - amavisd-new clamav clamav-daemon - pyzor razor spamassassin + local ANTI_VIRUS_SPAM_PACKAGES=( + clamav clamav-daemon + # spamassassin is used only with amavisd-new, while pyzor + razor are used by spamassasin + amavisd-new spamassassin pyzor razor + # the following packages are all for Fail2Ban + # https://github.com/docker-mailserver/docker-mailserver/pull/3403#discussion_r1306581431 + fail2ban python3-pyinotify python3-dnspython ) - CODECS_PACKAGES=( + # predominantly for Amavis support + local CODECS_PACKAGES=( altermime arj bzip2 cabextract cpio file gzip lhasa liblz4-tool @@ -57,26 +86,33 @@ function _install_packages() { unrar-free unzip xz-utils ) - MISCELLANEOUS_PACKAGES=( - apt-transport-https binutils bsd-mailx - ca-certificates curl dbconfig-no-thanks - dumb-init gnupg iproute2 libdate-manip-perl - libldap-common libmail-spf-perl - libnet-dns-perl locales logwatch - netcat-openbsd nftables rsyslog - supervisor uuid whois + local MISCELLANEOUS_PACKAGES=( + binutils bsd-mailx + dbconfig-no-thanks dumb-init iproute2 + libdate-manip-perl libldap-common libmail-spf-perl libnet-dns-perl + locales logwatch netcat-openbsd + nftables # primarily for Fail2Ban + rsyslog supervisor + uuid # used for file-locking + whois ) - POSTFIX_PACKAGES=( + local POSTFIX_PACKAGES=( pflogsumm postgrey postfix-ldap postfix-mta-sts-resolver postfix-pcre postfix-policyd-spf-python postsrsd ) - MAIL_PROGRAMS_PACKAGES=( - fetchmail opendkim opendkim-tools + local MAIL_PROGRAMS_PACKAGES=( + opendkim opendkim-tools opendmarc libsasl2-modules sasl2-bin ) + # These packages support community contributed features. + # If they cause too much maintenance burden in future, they are liable for removal. + local COMMUNITY_PACKAGES=( + fetchmail getmail6 + ) + # `bind9-dnsutils` provides the `dig` command # `iputils-ping` provides the `ping` command DEBUG_PACKAGES=( @@ -89,14 +125,12 @@ function _install_packages() { "${MISCELLANEOUS_PACKAGES[@]}" \ "${POSTFIX_PACKAGES[@]}" \ "${MAIL_PROGRAMS_PACKAGES[@]}" \ - "${DEBUG_PACKAGES[@]}" + "${DEBUG_PACKAGES[@]}" \ + "${COMMUNITY_PACKAGES[@]}" } function _install_dovecot() { - declare -a DOVECOT_PACKAGES - - # Dovecot packages for officially supported features. - DOVECOT_PACKAGES=( + local DOVECOT_PACKAGES=( dovecot-core dovecot-imapd dovecot-ldap dovecot-lmtpd dovecot-managesieved dovecot-pop3d dovecot-sieve dovecot-solr @@ -111,7 +145,8 @@ function _install_dovecot() { _log 'trace' 'Using Dovecot community repository' curl https://repo.dovecot.org/DOVECOT-REPO-GPG | gpg --import gpg --export ED409DA1 > /etc/apt/trusted.gpg.d/dovecot.gpg - echo "deb https://repo.dovecot.org/ce-2.3-latest/debian/bullseye bullseye main" > /etc/apt/sources.list.d/dovecot.list + # VERSION_CODENAME sourced from /etc/os-release + echo "deb https://repo.dovecot.org/ce-2.3-latest/debian/${VERSION_CODENAME} ${VERSION_CODENAME} main" > /etc/apt/sources.list.d/dovecot.list _log 'trace' 'Updating Dovecot package signatures' apt-get "${QUIET}" update @@ -128,47 +163,31 @@ function _install_dovecot() { } function _install_rspamd() { - _log 'trace' 'Adding Rspamd package signatures' - local DEB_FILE='/etc/apt/sources.list.d/rspamd.list' - - curl -sSfL https://rspamd.com/apt-stable/gpg.key | gpg --dearmor >/etc/apt/trusted.gpg.d/rspamd.gpg - local URL='[signed-by=/etc/apt/trusted.gpg.d/rspamd.gpg] http://rspamd.com/apt-stable/ bullseye main' - echo "deb ${URL}" >"${DEB_FILE}" - _log 'debug' 'Installing Rspamd' + _log 'trace' 'Adding Rspamd PPA' + curl -sSfL https://rspamd.com/apt-stable/gpg.key | gpg --dearmor >/etc/apt/trusted.gpg.d/rspamd.gpg + echo \ + "deb [signed-by=/etc/apt/trusted.gpg.d/rspamd.gpg] http://rspamd.com/apt-stable/ ${VERSION_CODENAME} main" \ + >/etc/apt/sources.list.d/rspamd.list + + _log 'trace' 'Updating package index after adding PPAs' apt-get "${QUIET}" update - apt-get "${QUIET}" --no-install-recommends install 'rspamd' 'redis-server' + + _log 'trace' 'Installing actual package' + apt-get "${QUIET}" install rspamd redis-server } -function _install_fail2ban() { - local FAIL2BAN_DEB_URL='https://github.com/fail2ban/fail2ban/releases/download/1.0.2/fail2ban_1.0.2-1.upstream1_all.deb' - local FAIL2BAN_DEB_ASC_URL="${FAIL2BAN_DEB_URL}.asc" - local FAIL2BAN_GPG_FINGERPRINT='8738 559E 26F6 71DF 9E2C 6D9E 683B F1BE BD0A 882C' - local FAIL2BAN_GPG_PUBLIC_KEY_ID='0x683BF1BEBD0A882C' - local FAIL2BAN_GPG_PUBLIC_KEY_SERVER='hkps://keyserver.ubuntu.com' +function _post_installation_steps() { + _log 'debug' 'Running post-installation steps (cleanup)' + _log 'debug' 'Deleting sensitive files (secrets)' + rm /etc/postsrsd.secret - _log 'debug' 'Installing Fail2ban' - apt-get "${QUIET}" --no-install-recommends install python3-pyinotify python3-dnspython + _log 'debug' 'Deleting default logwatch cronjob' + rm /etc/cron.daily/00logwatch - gpg --keyserver "${FAIL2BAN_GPG_PUBLIC_KEY_SERVER}" --recv-keys "${FAIL2BAN_GPG_PUBLIC_KEY_ID}" 2>&1 - - curl -Lkso fail2ban.deb "${FAIL2BAN_DEB_URL}" - curl -Lkso fail2ban.deb.asc "${FAIL2BAN_DEB_ASC_URL}" - - FINGERPRINT=$(LANG=C gpg --verify fail2ban.deb.asc fail2ban.deb |& sed -n 's#Primary key fingerprint: \(.*\)#\1#p') - - if [[ -z ${FINGERPRINT} ]]; then - echo 'ERROR: Invalid GPG signature!' >&2 - exit 1 - fi - - if [[ ${FINGERPRINT} != "${FAIL2BAN_GPG_FINGERPRINT}" ]]; then - echo "ERROR: Wrong GPG fingerprint!" >&2 - exit 1 - fi - - dpkg -i fail2ban.deb 2>&1 - rm fail2ban.deb fail2ban.deb.asc + _log 'trace' 'Removing leftovers from APT' + apt-get "${QUIET}" clean + rm -rf /var/lib/apt/lists/* _log 'debug' 'Patching Fail2ban to enable network bans' # Enable network bans @@ -176,56 +195,10 @@ function _install_fail2ban() { sedfile -i -r 's/^_nft_add_set = .+/_nft_add_set = add set \\{ type \\; flags interval\\; \\}/' /etc/fail2ban/action.d/nftables.conf } -# Presently the getmail6 package is v6.14, which is too old. -# v6.18 contains fixes for Google and Microsoft OAuth support. -# using pip to install getmail. -# TODO This can be removed when the base image is updated to Debian 12 (Bookworm) -function _install_getmail() { - _log 'debug' 'Installing getmail6' - apt-get "${QUIET}" --no-install-recommends install python3-pip - pip3 install --no-cache-dir 'getmail6~=6.18.12' - ln -s /usr/local/bin/getmail /usr/bin/getmail - ln -s /usr/local/bin/getmail-gmail-xoauth-tokens /usr/bin/getmail-gmail-xoauth-tokens - apt-get "${QUIET}" purge python3-pip - apt-get "${QUIET}" autoremove -} - -function _install_utils() { - _log 'debug' 'Installing utils sourced from Github' - _log 'trace' 'Installing jaq' - curl -sL "https://github.com/01mf02/jaq/releases/latest/download/jaq-v1.2.0-$(uname -m)-unknown-linux-gnu" -o /usr/bin/jaq && chmod +x /usr/bin/jaq - - _log 'trace' 'Installing swaks' - local SWAKS_VERSION='20240103.0' - local SWAKS_RELEASE="swaks-${SWAKS_VERSION}" - curl -sSfL "https://github.com/jetmore/swaks/releases/download/v${SWAKS_VERSION}/${SWAKS_RELEASE}.tar.gz" | tar -xz - mv "${SWAKS_RELEASE}/swaks" /usr/local/bin - rm -r "${SWAKS_RELEASE}" -} - -function _remove_data_after_package_installations() { - _log 'debug' 'Deleting sensitive files (secrets)' - rm /etc/postsrsd.secret - - _log 'debug' 'Deleting default logwatch cronjob' - rm /etc/cron.daily/00logwatch -} - -function _post_installation_steps() { - _log 'debug' 'Running post-installation steps (cleanup)' - apt-get "${QUIET}" clean - rm -rf /var/lib/apt/lists/* - - _log 'info' 'Finished installing packages' -} - _pre_installation_steps +_install_utils _install_postfix _install_packages _install_dovecot _install_rspamd -_install_fail2ban -_install_getmail -_install_utils -_remove_data_after_package_installations _post_installation_steps diff --git a/target/scripts/check-for-changes.sh b/target/scripts/check-for-changes.sh index 66417c09..2fa004b0 100755 --- a/target/scripts/check-for-changes.sh +++ b/target/scripts/check-for-changes.sh @@ -88,7 +88,7 @@ function _reload_amavis() { if [[ ${CHANGED} =~ ${DMS_DIR}/postfix-accounts.cf ]] || [[ ${CHANGED} =~ ${DMS_DIR}/postfix-virtual.cf ]]; then # /etc/postfix/vhost was updated, amavis must refresh it's config by # reading this file again in case of new domains, otherwise they will be ignored. - amavisd-new reload + amavisd reload fi } diff --git a/target/scripts/startup/setup.d/security/misc.sh b/target/scripts/startup/setup.d/security/misc.sh index 36d8905f..ec32a1e6 100644 --- a/target/scripts/startup/setup.d/security/misc.sh +++ b/target/scripts/startup/setup.d/security/misc.sh @@ -81,11 +81,6 @@ function __setup__security__spamassassin() { # shellcheck disable=SC2016 sed -i -r 's|^\$sa_kill_level_deflt (.*);|\$sa_kill_level_deflt = '"${SA_KILL}"';|g' /etc/amavis/conf.d/20-debian_defaults - # fix cron.daily for spamassassin - sed -i \ - 's|invoke-rc.d spamassassin reload|/etc/init\.d/spamassassin reload|g' \ - /etc/cron.daily/spamassassin - if [[ ${SA_SPAM_SUBJECT} == 'undef' ]]; then # shellcheck disable=SC2016 sed -i -r 's|^\$sa_spam_subject_tag (.*);|\$sa_spam_subject_tag = undef;|g' /etc/amavis/conf.d/20-debian_defaults diff --git a/target/supervisor/conf.d/supervisor-app.conf b/target/supervisor/conf.d/supervisor-app.conf index d64d3d72..50a7acc3 100644 --- a/target/supervisor/conf.d/supervisor-app.conf +++ b/target/supervisor/conf.d/supervisor-app.conf @@ -94,7 +94,7 @@ autostart=false autorestart=true stdout_logfile=/var/log/supervisor/%(program_name)s.log stderr_logfile=/var/log/supervisor/%(program_name)s.log -command=/usr/sbin/amavisd-new foreground +command=/usr/sbin/amavisd foreground [program:rspamd] startsecs=0 diff --git a/test/tests/parallel/set1/getmail.bats b/test/tests/parallel/set1/getmail.bats index 4b5c528a..ca1fdf87 100644 --- a/test/tests/parallel/set1/getmail.bats +++ b/test/tests/parallel/set1/getmail.bats @@ -25,9 +25,9 @@ function teardown_file() { _default_teardown ; } assert_line 'received = false' assert_line 'delivered_to = false' - _run_in_container stat /usr/local/bin/debug-getmail + _run_in_container_bash '[[ -f /usr/local/bin/debug-getmail ]]' assert_success - _run_in_container stat /usr/local/bin/getmail-cron + _run_in_container_bash '[[ -f /usr/local/bin/getmail-cron ]]' assert_success } diff --git a/test/tests/parallel/set1/spam_virus/amavis.bats b/test/tests/parallel/set1/spam_virus/amavis.bats index bc920234..d7a59cb2 100644 --- a/test/tests/parallel/set1/spam_virus/amavis.bats +++ b/test/tests/parallel/set1/spam_virus/amavis.bats @@ -51,10 +51,8 @@ function teardown_file() { export CONTAINER_NAME=${CONTAINER1_NAME} # give Amavis just a bit of time to print out its full debug log - run _repeat_in_container_until_success_or_timeout 5 "${CONTAINER_NAME}" grep 'ANTI-SPAM-SA' /var/log/mail/mail.log + run _repeat_in_container_until_success_or_timeout 20 "${CONTAINER_NAME}" grep 'SpamControl: init_pre_fork on SpamAssassin done' /var/log/mail/mail.log assert_success - assert_output --partial 'loaded' - refute_output --partial 'NOT loaded' } @test '(Amavis enabled) SA ENV should update Amavis config' { diff --git a/test/tests/parallel/set1/spam_virus/disabled_clamav_spamassassin.bats b/test/tests/parallel/set1/spam_virus/disabled_clamav_spamassassin.bats index 5ec28396..1711950f 100644 --- a/test/tests/parallel/set1/spam_virus/disabled_clamav_spamassassin.bats +++ b/test/tests/parallel/set1/spam_virus/disabled_clamav_spamassassin.bats @@ -30,8 +30,12 @@ function teardown_file() { _default_teardown ; } } @test "SA - Amavis integration should not be active" { - _run_in_container_bash "grep -i 'ANTI-SPAM-SA code' /var/log/mail/mail.log | grep 'NOT loaded'" + # Wait until Amavis has finished initializing: + run _repeat_in_container_until_success_or_timeout 20 "${CONTAINER_NAME}" grep 'Deleting db files in /var/lib/amavis/db' /var/log/mail/mail.log assert_success + # Amavis module for SA should not be loaded (`SpamControl: scanner SpamAssassin, module Amavis::SpamControl::SpamAssassin`): + _run_in_container grep 'scanner SpamAssassin' /var/log/mail/mail.log + assert_failure } @test "SA - should not have been called" { diff --git a/test/tests/parallel/set1/spam_virus/postgrey_enabled.bats b/test/tests/parallel/set1/spam_virus/postgrey_enabled.bats index 5538d3bf..2609812e 100644 --- a/test/tests/parallel/set1/spam_virus/postgrey_enabled.bats +++ b/test/tests/parallel/set1/spam_virus/postgrey_enabled.bats @@ -59,7 +59,7 @@ function teardown_file() { _default_teardown ; } _should_have_log_entry \ 'action=greylist' \ 'reason=new' \ - 'client_address=127.0.0.1/32, sender=user@external.tld, recipient=user1@localhost.localdomain' + 'client_address=127.0.0.1, sender=user@external.tld, recipient=user1@localhost.localdomain' } # NOTE: This test case depends on the previous one @@ -73,7 +73,7 @@ function teardown_file() { _default_teardown ; } _should_have_log_entry \ 'action=pass' \ 'reason=triplet found' \ - 'client_address=127.0.0.1/32, sender=user@external.tld, recipient=user1@localhost.localdomain' + 'client_address=127.0.0.1, sender=user@external.tld, recipient=user1@localhost.localdomain' } # NOTE: These two whitelist tests use `files/nc/` instead of `files/emails`. @@ -91,7 +91,7 @@ function teardown_file() { _default_teardown ; } _should_have_log_entry \ 'action=pass' \ 'reason=client whitelist' \ - 'client_address=127.0.0.1/32, sender=test@whitelist.tld, recipient=user1@localhost.localdomain' + 'client_address=127.0.0.1, sender=test@whitelist.tld, recipient=user1@localhost.localdomain' } @test "should whitelist recipient 'user2@otherdomain.tld'" { @@ -100,7 +100,7 @@ function teardown_file() { _default_teardown ; } _should_have_log_entry \ 'action=pass' \ 'reason=recipient whitelist' \ - 'client_address=127.0.0.1/32, sender=test@nonwhitelist.tld, recipient=user2@otherdomain.tld' + 'client_address=127.0.0.1, sender=test@nonwhitelist.tld, recipient=user2@otherdomain.tld' } function _should_have_log_entry() { diff --git a/test/tests/parallel/set1/spam_virus/rspamd_dkim.bats b/test/tests/parallel/set1/spam_virus/rspamd_dkim.bats index 215b334d..044e38fb 100644 --- a/test/tests/parallel/set1/spam_virus/rspamd_dkim.bats +++ b/test/tests/parallel/set1/spam_virus/rspamd_dkim.bats @@ -222,7 +222,7 @@ function __check_rsa_keys() { # Check the private key matches the specification _run_in_container_bash "openssl rsa -in '${BASE_FILE_NAME}.private.txt' -noout -text" assert_success - assert_line --index 0 "RSA Private-Key: (${KEYSIZE} bit, 2 primes)" + assert_line --index 0 "Private-Key: (${KEYSIZE} bit, 2 primes)" # Check the public key matches the specification # @@ -232,7 +232,7 @@ function __check_rsa_keys() { PUBKEY=$(_exec_in_container_bash "grep -o 'p=.*' ${BASE_FILE_NAME}.public.dns.txt") _run_in_container_bash "openssl enc -base64 -d <<< ${PUBKEY#p=} | openssl pkey -inform DER -pubin -noout -text" assert_success - assert_line --index 0 "RSA Public-Key: (${KEYSIZE} bit)" + assert_line --index 0 "Public-Key: (${KEYSIZE} bit)" } # Verify that all DKIM key files are present. diff --git a/test/tests/parallel/set3/container_configuration/process_check_restart.bats b/test/tests/parallel/set3/container_configuration/process_check_restart.bats index 0a54a60f..4f0fd90f 100644 --- a/test/tests/parallel/set3/container_configuration/process_check_restart.bats +++ b/test/tests/parallel/set3/container_configuration/process_check_restart.bats @@ -13,7 +13,7 @@ function teardown() { _default_teardown ; } # opendmarc (/usr/sbin/opendmarc) # postfix (/usr/lib/postfix/sbin/master) - Postfix main process (two ancestors, launched via pidproxy python3 script) # -# amavisd-new (usr/sbin/amavisd-new) +# amavisd (usr/sbin/amavisd) # clamd (/usr/sbin/clamd) # dovecot (/usr/sbin/dovecot) # fetchmail (/usr/bin/fetchmail) @@ -37,7 +37,7 @@ CORE_PROCESS_LIST=( # These processes can be toggled via ENV: # NOTE: clamd handled in separate test case ENV_PROCESS_LIST=( - amavisd-new + amavisd dovecot fail2ban-server fetchmail diff --git a/test/tests/serial/mail_with_oauth2.bats b/test/tests/serial/mail_with_oauth2.bats index 9b4d12fe..8da87980 100644 --- a/test/tests/serial/mail_with_oauth2.bats +++ b/test/tests/serial/mail_with_oauth2.bats @@ -59,6 +59,10 @@ function teardown_file() { } @test "should authenticate with XOAUTH2" { + # curl packaged in Debian 12 (and the latest release as of Jan 2024) broke XOAUTH2 support + # https://github.com/docker-mailserver/docker-mailserver/pull/3403#issuecomment-1907100624 + skip 'unable to test XOAUTH mechanism due to bug since curl 7.80' + __should_login_successfully_with 'XOAUTH2' } diff --git a/test/tests/serial/open_dkim.bats b/test/tests/serial/open_dkim.bats index b4950d1f..e51fdedb 100644 --- a/test/tests/serial/open_dkim.bats +++ b/test/tests/serial/open_dkim.bats @@ -226,7 +226,7 @@ function __should_have_expected_files() { # DKIM private key for signing, parse it to verify private key size is correct: _run_in_container_bash "openssl rsa -in '${TARGET_DIR}/mail.private' -noout -text" assert_success - assert_line --index 0 "RSA Private-Key: (${EXPECTED_KEYSIZE} bit, 2 primes)" + assert_line --index 0 "Private-Key: (${EXPECTED_KEYSIZE} bit, 2 primes)" # DKIM record, extract public key (base64 encoded, potentially multi-line) # - tail to exclude first line, @@ -240,7 +240,7 @@ function __should_have_expected_files() { ) | openssl enc -base64 -d | openssl pkey -inform DER -pubin -noout -text " assert_success - assert_line --index 0 "RSA Public-Key: (${EXPECTED_KEYSIZE} bit)" + assert_line --index 0 "Public-Key: (${EXPECTED_KEYSIZE} bit)" # Contents is for expected DKIM_DOMAIN and selector (mail): _run_in_container cat "${TARGET_DIR}/mail.txt" From ed1e1ebbd3eeb61b059967d94c54d8881112fba4 Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Wed, 24 Jan 2024 23:06:05 +0100 Subject: [PATCH 002/267] tests: new sending and filtering functions (#3786) * move log/filter functions into own file * add ShellCheck global directives * use new function for tracking logs The new function, called `_send_email_with_mid`, aligns with suggestions from @polarethene and is heavily simplified compared to its predecessor `_send_email_and_get_id`. New helpers will be introduced to filter logs according to the MID constructed in this function. * new filters for searching logs with MID * use new filters (and sending) functions * add new helper for asserting non-existence of log message * use new filters in tests * Apply suggestions from code review - `_mid` / `MID` => `_msgid` / `MSG_ID` - Revised documentation / tooltip comments * Apply suggestions from code review * fix tests * use more distinct names for MSG_ID headers * update `_filter_service_log` to not use `-i -E` Moreover, I added a function to print the whole mail log. Appropriate comments were added to this function to indicate that one should only use this function when necessary. * adjust helpers to new helper filter * follow-up of previous commit * add CHANGELOG entry * Apply suggestions from code review * chore: Update OAuth2 to use new log helper * Apply suggestions from code review Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> * added explicit `_regexp` filters for logs * Apply suggestions from code review --------- Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- CHANGELOG.md | 5 + test/helper/common.bash | 68 +--------- test/helper/log_and_filtering.bash | 119 ++++++++++++++++++ test/helper/sending.bash | 83 ++++-------- .../parallel/set1/spam_virus/clamav.bats | 7 +- .../disabled_clamav_spamassassin.bats | 10 +- .../parallel/set1/spam_virus/postscreen.bats | 9 +- .../parallel/set1/spam_virus/rspamd_full.bats | 46 ++++--- test/tests/parallel/set3/mta/dsn.bats | 10 +- .../parallel/set3/mta/smtp_delivery.bats | 26 ++-- test/tests/serial/mail_pop3.bats | 9 +- test/tests/serial/mail_with_oauth2.bats | 2 +- test/tests/serial/tests.bats | 30 ++--- test/tests/serial/vmail-id.bats | 30 ++--- 14 files changed, 227 insertions(+), 227 deletions(-) create mode 100644 test/helper/log_and_filtering.bash diff --git a/CHANGELOG.md b/CHANGELOG.md index 35ee73a9..12f07561 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -33,6 +33,11 @@ The most noteworthy change of this release is the update of the container's base - `smtpd_relay_restrictions` (relay policy) is now evaluated after `smtpd_recipient_restrictions` (spam policy). Previously it was evaluated before `smtpd_recipient_restrictions`. Mail to be relayed via DMS must now pass through the spam policy first. - The TLS fingerprint policy has changed the default from MD5 to SHA256 (_DMS does not modify this Postfix parameter, but may affect any user customizations that do_). +### Updates + +- **Tests:** + - Refactored helper methods for sending e-mails with specific `Message-ID` headers and the helpers for retrieving + filtering logs, which together help isolate logs relevant to specific mail when multiple mails have been processed within a single test. ([#3786](https://github.com/docker-mailserver/docker-mailserver/pull/3786)) + ## [v13.3.1](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v13.3.1) ### Fixes diff --git a/test/helper/common.bash b/test/helper/common.bash index 0891bf8c..3c54862c 100644 --- a/test/helper/common.bash +++ b/test/helper/common.bash @@ -18,6 +18,7 @@ function __load_bats_helper() { load "${REPOSITORY_ROOT}/test/test_helper/bats-support/load" load "${REPOSITORY_ROOT}/test/test_helper/bats-assert/load" load "${REPOSITORY_ROOT}/test/helper/sending" + load "${REPOSITORY_ROOT}/test/helper/log_and_filtering" } __load_bats_helper @@ -228,7 +229,6 @@ function _run_until_success_or_timeout() { # ! ------------------------------------------------------------------- # ? >> Functions to wait until a condition is met - # Wait until a port is ready. # # @param ${1} = port @@ -351,15 +351,6 @@ function _add_mail_account_then_wait_until_ready() { _wait_until_account_maildir_exists "${MAIL_ACCOUNT}" } -# Assert that the number of lines output by a previous command matches the given -# amount (${1}). `lines` is a special BATS variable updated via `run`. -# -# @param ${1} = number of lines that the output should have -function _should_output_number_of_lines() { - # shellcheck disable=SC2154 - assert_equal "${#lines[@]}" "${1:?Number of lines not provided}" -} - # Reloads the postfix service. # # @param ${1} = container name [OPTIONAL] @@ -372,7 +363,6 @@ function _reload_postfix() { _exec_in_container postfix reload } - # Get the IP of the container (${1}). # # @param ${1} = container name [OPTIONAL] @@ -413,62 +403,6 @@ function _should_have_content_in_directory() { assert_success } -# Filters a service's logs (under `/var/log/supervisor/.log`) given -# a specific string. -# -# @param ${1} = service name -# @param ${2} = string to filter by -# @param ${3} = container name [OPTIONAL] -# -# ## Attention -# -# The string given to this function is interpreted by `grep -E`, i.e. -# as a regular expression. In case you use characters that are special -# in regular expressions, you need to escape them! -function _filter_service_log() { - local SERVICE=${1:?Service name must be provided} - local STRING=${2:?String to match must be provided} - local CONTAINER_NAME=$(__handle_container_name "${3:-}") - local FILE="/var/log/supervisor/${SERVICE}.log" - - # Fallback to alternative log location: - [[ -f ${FILE} ]] || FILE="/var/log/mail/${SERVICE}.log" - _run_in_container grep -E "${STRING}" "${FILE}" -} - -# Like `_filter_service_log` but asserts that the string was found. -# -# @param ${1} = service name -# @param ${2} = string to filter by -# @param ${3} = container name [OPTIONAL] -# -# ## Attention -# -# The string given to this function is interpreted by `grep -E`, i.e. -# as a regular expression. In case you use characters that are special -# in regular expressions, you need to escape them! -function _service_log_should_contain_string() { - local SERVICE=${1:?Service name must be provided} - local STRING=${2:?String to match must be provided} - local CONTAINER_NAME=$(__handle_container_name "${3:-}") - - _filter_service_log "${SERVICE}" "${STRING}" - assert_success -} - -# Filters the mail log for lines that belong to a certain email identified -# by its ID. You can obtain the ID of an email you want to send by using -# `_send_email_and_get_id`. -# -# @param ${1} = email ID -# @param ${2} = container name [OPTIONAL] -function _print_mail_log_for_id() { - local MAIL_ID=${1:?Mail ID must be provided} - local CONTAINER_NAME=$(__handle_container_name "${2:-}") - - _run_in_container grep -E "${MAIL_ID}" /var/log/mail.log -} - # A simple wrapper for netcat (`nc`). This is useful when sending # "raw" e-mails or doing IMAP-related work. # diff --git a/test/helper/log_and_filtering.bash b/test/helper/log_and_filtering.bash new file mode 100644 index 00000000..415a203b --- /dev/null +++ b/test/helper/log_and_filtering.bash @@ -0,0 +1,119 @@ +#!/bin/bash + +# ? ABOUT: Functions defined here aid in working with logs and filtering them. + +# ! ATTENTION: This file is loaded by `common.sh` - do not load it yourself! +# ! ATTENTION: This file requires helper functions from `common.sh`! + +# shellcheck disable=SC2034,SC2155 + +# Assert that the number of lines output by a previous command matches the given amount (${1}). +# `lines` is a special BATS variable updated via `run`. +# +# @param ${1} = number of lines that the output should have +function _should_output_number_of_lines() { + # shellcheck disable=SC2154 + assert_equal "${#lines[@]}" "${1:?Number of lines not provided}" +} + +# Filters a service's logs (under `/var/log/supervisor/.log`) given a specific string. +# +# @param ${1} = service name +# @param ${2} = string to filter by +# @param ... = options given to `grep` (which is used to filter logs) +function _filter_service_log() { + local SERVICE=${1:?Service name must be provided} + local STRING=${2:?String to match must be provided} + shift 2 + + local FILE="/var/log/supervisor/${SERVICE}.log" + # Alternative log location fallback: + [[ -f ${FILE} ]] || FILE="/var/log/mail/${SERVICE}.log" + _run_in_container grep "${@}" "${STRING}" "${FILE}" +} + +# Prints the entirety of the primary mail log. +# Avoid using this method when you could filter more specific log lines with: +# +# 1. _filter_service_log +# 2. _service_log_should[_not]_contain_string +function _show_complete_mail_log() { + _run_in_container cat /var/log/mail/mail.log +} + +# Like `_filter_service_log` but asserts that the string was found. +# +# @param ${1} = service name +# @param ${2} = string to filter by +function _service_log_should_contain_string() { + _filter_service_log "${1}" "${2}" --fixed-strings + assert_success +} + +# Like `_filter_service_log` but asserts that the string was _not_ found. +# +# @param ${1} = service name +# @param ${2} = string to filter by +function _service_log_should_not_contain_string() { + _filter_service_log "${1}" "${2}" --fixed-strings + assert_failure +} + +# Like `_filter_service_log` but asserts that the string was found. +# Uses regular expressions under the hood for pattern matching. +# +# @param ${1} = service name +# @param ${2} = regular expression to filter by +function _service_log_should_contain_string_regexp() { + _filter_service_log "${1}" "${2}" --extended-regexp + assert_success +} + +# Like `_filter_service_log` but asserts that the string was _not_ found. +# Uses regular expressions under the hood for pattern matching. +# +# @param ${1} = service name +# @param ${2} = regular expression to filter by +function _service_log_should_not_contain_string_regexp() { + _filter_service_log "${1}" "${2}" --extended-regexp + assert_failure +} + +# Filters the mail log by the given MSG_ID (Message-ID) parameter, +# printing log lines which include the associated Postfix Queue ID. +# +# @param ${1} = The local-part of a Message-ID header value (``) +function _print_mail_log_of_queue_id_from_msgid() { + # A unique ID Postfix generates for tracking queued mail as it's processed. + # The length can vary (as per the postfix docs). Hence, we use a range to safely capture it. + # https://github.com/docker-mailserver/docker-mailserver/pull/3747#discussion_r1446679671 + local QUEUE_ID_REGEX='[A-Z0-9]{9,12}' + + local MSG_ID=$(__construct_msgid "${1:?The local-part for MSG_ID was not provided}") + shift 1 + + _wait_for_empty_mail_queue_in_container + + QUEUE_ID=$(_exec_in_container tac /var/log/mail.log \ + | grep -E "postfix/cleanup.*: ${QUEUE_ID_REGEX}:.*message-id=${MSG_ID}" \ + | grep -E --only-matching --max-count 1 "${QUEUE_ID_REGEX}" || :) + + # We perform plausibility checks on the IDs. + assert_not_equal "${QUEUE_ID}" '' + run echo "${QUEUE_ID}" + assert_line --regexp "^${QUEUE_ID_REGEX}$" + + # Postfix specific logs: + _filter_service_log 'mail' "${QUEUE_ID}" +} + +# A convenience method that filters for Dovecot specific logs with a `msgid` field that matches the MSG_ID input. +# +# @param ${1} = The local-part of a Message-ID header value (``) +function _print_mail_log_for_msgid() { + local MSG_ID=$(__construct_msgid "${1:?The local-part for MSG_ID was not provided}") + shift 1 + + # Dovecot specific logs: + _filter_service_log 'mail' "msgid=${MSG_ID}" +} diff --git a/test/helper/sending.bash b/test/helper/sending.bash index e18dc1ac..1c5d844a 100644 --- a/test/helper/sending.bash +++ b/test/helper/sending.bash @@ -1,11 +1,13 @@ #!/bin/bash -# shellcheck disable=SC2034,SC2155 - # ? ABOUT: Functions defined here help with sending emails in tests. # ! ATTENTION: This file is loaded by `common.sh` - do not load it yourself! # ! ATTENTION: This file requires helper functions from `common.sh`! +# ! ATTENTION: Functions prefixed with `__` are intended for internal use within +# ! this file (or other helpers) only, not in tests. + +# shellcheck disable=SC2034,SC2155 # Sends an e-mail from the container named by the environment variable `CONTAINER_NAME` # to the same or another container. @@ -100,74 +102,35 @@ function _send_email() { return "${RETURN_VALUE}" } -# Like `_send_email` with two major differences: +# Construct the value for a 'Message-ID' header. +# For tests we use only the local-part to identify mail activity in logs. The rest of the value is fixed. # -# 1. this function waits for the mail to be processed; there is no asynchronicity -# because filtering the logs in a synchronous way is easier and safer; -# 2. this function takes the name of a variable and inserts IDs one can later -# filter by to check logs. +# A Message-ID header value should be in the form of: `` +# https://en.wikipedia.org/wiki/Message-ID +# https://datatracker.ietf.org/doc/html/rfc5322#section-3.6.4 # -# No. 2 is especially useful in case you send more than one email in a single -# test file and need to assert certain log entries for each mail individually. +# @param ${1} = The local-part of a Message-ID header value (``) +function __construct_msgid() { + local MSG_ID_LOCALPART=${1:?The local-part for MSG_ID was not provided} + echo "<${MSG_ID_LOCALPART}@dms-tests>" +} + +# Like `_send_email` but adds a "Message-ID: ${1}@dms-tests>" header, +# which allows for filtering logs later. # -# The first argument has to be the name of the variable that the e-mail ID is stored in. -# The second argument **can** be the flag `--expect-rejection`. -# -# - If this flag is supplied, the function does not check whether the whole mail delivery -# transaction was successful. Additionally the queue ID will be retrieved differently. -# - CAUTION: It must still be possible to `grep` for the Message-ID that Postfix -# generated in the mail log; otherwise this function fails. -# -# The rest of the arguments are the same as `_send_email`. -# -# ## Attention -# -# This function assumes `CONTAINER_NAME` to be properly set (to the container -# name the command should be executed in)! -# -# ## Safety -# -# This functions assumes **no concurrent sending of emails to the same container**! -# If two clients send simultaneously, there is no guarantee the correct ID is -# chosen. Sending more than one mail at any given point in time with this function -# is UNDEFINED BEHAVIOR! -function _send_email_and_get_id() { - # Export the variable denoted by ${1} so everyone has access - export "${1:?Mail ID must be set for _send_email_and_get_id}" - # Get a "reference" to the content of the variable denoted by ${1} so we can manipulate the content - local -n ID_ENV_VAR_REF=${1:?} - # Prepare the message ID header here because we will shift away ${1} later - local MID="<${1}@dms-tests>" - # Get rid of ${1} so only the arguments for swaks remain +# @param ${1} = The local-part of a Message-ID header value (``) +function _send_email_with_msgid() { + local MSG_ID=$(__construct_msgid "${1:?The local-part for MSG_ID was not provided}") shift 1 - # The unique ID Postfix (and other services) use may be different in length - # on different systems. Hence, we use a range to safely capture it. - local QUEUE_ID_REGEX='[A-Z0-9]{9,12}' - - _wait_for_empty_mail_queue_in_container - _send_email "${@}" --header "Message-Id: ${MID}" - _wait_for_empty_mail_queue_in_container - - # We store Postfix's queue ID first - ID_ENV_VAR_REF=$(_exec_in_container tac /var/log/mail.log \ - | grep -E "postfix/cleanup.*: ${QUEUE_ID_REGEX}:.*message-id=${MID}" \ - | grep -E --only-matching --max-count 1 "${QUEUE_ID_REGEX}" || :) - # But we also requre potential Dovecot sieve output, which requires the mesage ID, - # so we need to provide the message ID too. - ID_ENV_VAR_REF+="|${MID}" - - # Last but not least, we perform plausibility checks on the IDs. - assert_not_equal "${ID_ENV_VAR_REF}" '' - run echo "${ID_ENV_VAR_REF}" - assert_line --regexp "^${QUEUE_ID_REGEX}\|${MID}$" + _send_email "${@}" --header "Message-ID: ${MSG_ID}" } # Send a spam e-mail by utilizing GTUBE. # -# Extra arguments given to this function will be supplied by `_send_email_and_get_id` directly. +# Extra arguments given to this function will be supplied by `_send_email_with_msgid` directly. function _send_spam() { - _send_email_and_get_id MAIL_ID_SPAM "${@}" \ + _send_email_with_msgid 'dms-test-email-spam' "${@}" \ --from 'spam@external.tld' \ --body 'XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X' } diff --git a/test/tests/parallel/set1/spam_virus/clamav.bats b/test/tests/parallel/set1/spam_virus/clamav.bats index a916896a..36149edc 100644 --- a/test/tests/parallel/set1/spam_virus/clamav.bats +++ b/test/tests/parallel/set1/spam_virus/clamav.bats @@ -38,8 +38,7 @@ function teardown_file() { _default_teardown ; } } @test 'should be identified by Amavis' { - _run_in_container grep -i 'Found secondary av scanner ClamAV-clamscan' /var/log/mail/mail.log - assert_success + _service_log_should_contain_string 'mail' 'Found secondary av scanner ClamAV-clamscan' } @test 'freshclam cron is enabled' { @@ -53,6 +52,6 @@ function teardown_file() { _default_teardown ; } } @test 'rejects virus' { - _run_in_container_bash "grep 'Blocked INFECTED' /var/log/mail/mail.log | grep ' -> '" - assert_success + _service_log_should_contain_string 'mail' 'Blocked INFECTED' + assert_output --partial ' -> ' } diff --git a/test/tests/parallel/set1/spam_virus/disabled_clamav_spamassassin.bats b/test/tests/parallel/set1/spam_virus/disabled_clamav_spamassassin.bats index 1711950f..0f3a7b5b 100644 --- a/test/tests/parallel/set1/spam_virus/disabled_clamav_spamassassin.bats +++ b/test/tests/parallel/set1/spam_virus/disabled_clamav_spamassassin.bats @@ -25,20 +25,18 @@ function setup_file() { function teardown_file() { _default_teardown ; } @test "ClamAV - Amavis integration should not be active" { - _run_in_container grep -i 'Found secondary av scanner ClamAV-clamscan' /var/log/mail/mail.log - assert_failure + _service_log_should_not_contain_string 'mail' 'Found secondary av scanner ClamAV-clamscan' } @test "SA - Amavis integration should not be active" { # Wait until Amavis has finished initializing: run _repeat_in_container_until_success_or_timeout 20 "${CONTAINER_NAME}" grep 'Deleting db files in /var/lib/amavis/db' /var/log/mail/mail.log assert_success + # Amavis module for SA should not be loaded (`SpamControl: scanner SpamAssassin, module Amavis::SpamControl::SpamAssassin`): - _run_in_container grep 'scanner SpamAssassin' /var/log/mail/mail.log - assert_failure + _service_log_should_not_contain_string 'mail' 'scanner SpamAssassin' } @test "SA - should not have been called" { - _run_in_container grep -i 'connect to /var/run/clamav/clamd.ctl failed' /var/log/mail/mail.log - assert_failure + _service_log_should_not_contain_string 'mail' 'connect to /var/run/clamav/clamd.ctl failed' } diff --git a/test/tests/parallel/set1/spam_virus/postscreen.bats b/test/tests/parallel/set1/spam_virus/postscreen.bats index 92333f98..3f93fe9d 100644 --- a/test/tests/parallel/set1/spam_virus/postscreen.bats +++ b/test/tests/parallel/set1/spam_virus/postscreen.bats @@ -56,12 +56,11 @@ function teardown_file() { @test "should successfully pass postscreen and get postfix greeting message (respecting postscreen_greet_wait time)" { # Configure `send_email()` to send from the mail client container (CONTAINER2_NAME) via ENV override, # mail is sent to the DMS server container (CONTAINER1_NAME) via `--server` parameter: - # TODO: Use _send_email_and_get_id when proper resolution of domain names is possible: CONTAINER_NAME=${CONTAINER2_NAME} _send_email --expect-rejection --server "${CONTAINER1_IP}" --port 25 --data 'postscreen.txt' - # CONTAINER_NAME=${CONTAINER2_NAME} _send_email_and_get_id MAIL_ID_POSTSCREEN --server "${CONTAINER1_IP}" --data 'postscreen.txt' - # _print_mail_log_for_id "${MAIL_ID_POSTSCREEN}" + # TODO: Use _send_email_with_msgid when proper resolution of domain names is possible: + # CONTAINER_NAME=${CONTAINER2_NAME} _send_email_with_msgid 'msgid-postscreen' --server "${CONTAINER1_IP}" --data 'postscreen.txt' + # _print_mail_log_for_msgid 'msgid-postscreen' # assert_output --partial "stored mail into mailbox 'INBOX'" - _run_in_container cat /var/log/mail.log - assert_output --partial 'PASS NEW' + _service_log_should_contain_string 'mail' 'PASS NEW' } diff --git a/test/tests/parallel/set1/spam_virus/rspamd_full.bats b/test/tests/parallel/set1/spam_virus/rspamd_full.bats index f66e9231..9e6d6222 100644 --- a/test/tests/parallel/set1/spam_virus/rspamd_full.bats +++ b/test/tests/parallel/set1/spam_virus/rspamd_full.bats @@ -45,16 +45,17 @@ function setup_file() { # We will send 4 emails: # 1. The first one should pass just fine - _send_email_and_get_id MAIL_ID_PASS + _send_email_with_msgid 'rspamd-test-email-pass' # 2. The second one should be rejected (Rspamd-specific GTUBE pattern for rejection) _send_spam --expect-rejection # 3. The third one should be rejected due to a virus (ClamAV EICAR pattern) # shellcheck disable=SC2016 - _send_email_and_get_id MAIL_ID_VIRUS --expect-rejection \ + _send_email_with_msgid 'rspamd-test-email-virus' --expect-rejection \ --body 'X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*' # 4. The fourth one will receive an added header (Rspamd-specific GTUBE pattern for adding a spam header) # ref: https://rspamd.com/doc/gtube_patterns.html - _send_email_and_get_id MAIL_ID_HEADER --body "YJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X" + _send_email_with_msgid 'rspamd-test-email-header' \ + --body "YJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X" _run_in_container cat /var/log/mail.log assert_success @@ -92,7 +93,7 @@ function teardown_file() { _default_teardown ; } } @test 'service log exist and contains proper content' { - _service_log_should_contain_string 'rspamd' 'rspamd .* is loading configuration' + _service_log_should_contain_string_regexp 'rspamd' 'rspamd .* is loading configuration' _service_log_should_contain_string 'rspamd' 'lua module clickhouse is disabled in the configuration' _service_log_should_contain_string 'rspamd' 'lua module elastic is disabled in the configuration' _service_log_should_contain_string 'rspamd' 'lua module neural is disabled in the configuration' @@ -108,33 +109,40 @@ function teardown_file() { _default_teardown ; } } @test 'normal mail passes fine' { - _service_log_should_contain_string 'rspamd' 'F \(no action\)' + _service_log_should_contain_string 'rspamd' 'F (no action)' - _print_mail_log_for_id "${MAIL_ID_PASS}" + _print_mail_log_for_msgid 'rspamd-test-email-pass' assert_output --partial "stored mail into mailbox 'INBOX'" _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 1 } @test 'detects and rejects spam' { - _service_log_should_contain_string 'rspamd' 'S \(reject\)' + _service_log_should_contain_string 'rspamd' 'S (reject)' _service_log_should_contain_string 'rspamd' 'reject "Gtube pattern"' - _print_mail_log_for_id "${MAIL_ID_SPAM}" + _print_mail_log_of_queue_id_from_msgid 'dms-test-email-spam' assert_output --partial 'milter-reject' assert_output --partial '5.7.1 Gtube pattern' + _print_mail_log_for_msgid 'dms-test-email-spam' + refute_output --partial "stored mail into mailbox 'INBOX'" + assert_failure + _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 1 } @test 'detects and rejects virus' { - _service_log_should_contain_string 'rspamd' 'T \(reject\)' + _service_log_should_contain_string 'rspamd' 'T (reject)' _service_log_should_contain_string 'rspamd' 'reject "ClamAV FOUND VIRUS "Eicar-Signature"' - _print_mail_log_for_id "${MAIL_ID_VIRUS}" + _print_mail_log_of_queue_id_from_msgid 'rspamd-test-email-virus' assert_output --partial 'milter-reject' assert_output --partial '5.7.1 ClamAV FOUND VIRUS "Eicar-Signature"' + + _print_mail_log_for_msgid 'dms-test-email-spam' refute_output --partial "stored mail into mailbox 'INBOX'" + assert_failure _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 1 } @@ -217,10 +225,10 @@ function teardown_file() { _default_teardown ; } _run_in_container_bash '[[ -f /usr/lib/dovecot/sieve-global/after/spam_to_junk.svbin ]]' assert_success - _service_log_should_contain_string 'rspamd' 'S \(add header\)' + _service_log_should_contain_string 'rspamd' 'S (add header)' _service_log_should_contain_string 'rspamd' 'add header "Gtube pattern"' - _print_mail_log_for_id "${MAIL_ID_HEADER}" + _print_mail_log_for_msgid 'rspamd-test-email-header' assert_output --partial "fileinto action: stored mail into mailbox 'Junk'" _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 1 @@ -265,10 +273,10 @@ function teardown_file() { _default_teardown ; } _nc_wrapper 'nc/rspamd_imap_move_to_junk.txt' '0.0.0.0 143' sleep 1 # wait for the transaction to finish - _run_in_container cat /var/log/mail/mail.log - assert_success - assert_output --partial 'imapsieve: Matched static mailbox rule [1]' - refute_output --partial 'imapsieve: Matched static mailbox rule [2]' + _service_log_should_contain_string 'mail' 'imapsieve: Matched static mailbox rule [1]' + _service_log_should_not_contain_string 'mail' 'imapsieve: Matched static mailbox rule [2]' + + _show_complete_mail_log for LINE in "${LEARN_SPAM_LINES[@]}"; do assert_output --partial "${LINE}" done @@ -279,9 +287,9 @@ function teardown_file() { _default_teardown ; } _nc_wrapper 'nc/rspamd_imap_move_to_inbox.txt' '0.0.0.0 143' sleep 1 # wait for the transaction to finish - _run_in_container cat /var/log/mail/mail.log - assert_success - assert_output --partial 'imapsieve: Matched static mailbox rule [2]' + _service_log_should_contain_string 'mail' 'imapsieve: Matched static mailbox rule [2]' + + _show_complete_mail_log for LINE in "${LEARN_HAM_LINES[@]}"; do assert_output --partial "${LINE}" done diff --git a/test/tests/parallel/set3/mta/dsn.bats b/test/tests/parallel/set3/mta/dsn.bats index 9a65b147..7a6af76c 100644 --- a/test/tests/parallel/set3/mta/dsn.bats +++ b/test/tests/parallel/set3/mta/dsn.bats @@ -54,7 +54,7 @@ function teardown_file() { _nc_wrapper 'emails/nc_raw/dsn/authenticated.txt' '0.0.0.0 587' _wait_for_empty_mail_queue_in_container - _run_in_container grep "${LOG_DSN}" /var/log/mail/mail.log + _filter_service_log 'mail' "${LOG_DSN}" _should_output_number_of_lines 3 } @@ -70,15 +70,14 @@ function teardown_file() { # # Although external requests are discarded, anyone who has requested a DSN # will still receive it, but it will come from the sending mail server, not this one. - _run_in_container grep "${LOG_DSN}" /var/log/mail/mail.log - assert_failure + _service_log_should_not_contain_string 'mail' "${LOG_DSN}" # These ports are excluded via master.cf. _nc_wrapper 'emails/nc_raw/dsn/authenticated.txt' '0.0.0.0 465' _nc_wrapper 'emails/nc_raw/dsn/authenticated.txt' '0.0.0.0 587' _wait_for_empty_mail_queue_in_container - _run_in_container grep "${LOG_DSN}" /var/log/mail/mail.log + _service_log_should_contain_string 'mail' "${LOG_DSN}" _should_output_number_of_lines 2 } @@ -92,6 +91,5 @@ function teardown_file() { # DSN requests are rejected regardless of origin. # This is usually a bad idea, as you won't get them either. - _run_in_container grep "${LOG_DSN}" /var/log/mail/mail.log - assert_failure + _service_log_should_not_contain_string 'mail' "${LOG_DSN}" } diff --git a/test/tests/parallel/set3/mta/smtp_delivery.bats b/test/tests/parallel/set3/mta/smtp_delivery.bats index e851d94e..d183694b 100644 --- a/test/tests/parallel/set3/mta/smtp_delivery.bats +++ b/test/tests/parallel/set3/mta/smtp_delivery.bats @@ -174,33 +174,28 @@ function _successful() { } @test "delivers mail to existing alias" { - _run_in_container grep 'to=, orig_to=' /var/log/mail/mail.log - assert_success + _service_log_should_contain_string 'mail' 'to=, orig_to=' assert_output --partial 'status=sent' _should_output_number_of_lines 1 } @test "delivers mail to existing alias with recipient delimiter" { - _run_in_container grep 'to=, orig_to=' /var/log/mail/mail.log - assert_success + _service_log_should_contain_string 'mail' 'to=, orig_to=' assert_output --partial 'status=sent' _should_output_number_of_lines 1 - _run_in_container grep 'to=' /var/log/mail/mail.log - assert_success + _service_log_should_contain_string 'mail' 'to=' refute_output --partial 'status=bounced' } @test "delivers mail to existing catchall" { - _run_in_container grep 'to=, orig_to=' /var/log/mail/mail.log - assert_success + _service_log_should_contain_string 'mail' 'to=, orig_to=' assert_output --partial 'status=sent' _should_output_number_of_lines 1 } @test "delivers mail to regexp alias" { - _run_in_container grep 'to=, orig_to=' /var/log/mail/mail.log - assert_success + _service_log_should_contain_string 'mail' 'to=, orig_to=' assert_output --partial 'status=sent' _should_output_number_of_lines 1 } @@ -227,23 +222,20 @@ function _successful() { } @test "rejects mail to unknown user" { - _run_in_container grep ': Recipient address rejected: User unknown in virtual mailbox table' /var/log/mail/mail.log - assert_success + _service_log_should_contain_string 'mail' ': Recipient address rejected: User unknown in virtual mailbox table' _should_output_number_of_lines 1 } @test "redirects mail to external aliases" { - _run_in_container_bash "grep 'Passed CLEAN {RelayedInbound}' /var/log/mail/mail.log | grep -- '-> '" - assert_success - assert_output --partial ' -> ' + _service_log_should_contain_string 'mail' 'Passed CLEAN {RelayedInbound}' + run bash -c "grep ' -> ' <<< '${output}'" _should_output_number_of_lines 2 # assert_output --partial 'external.tld=user@example.test> -> ' } # TODO: Add a test covering case SPAMASSASSIN_SPAM_TO_INBOX=1 (default) @test "rejects spam" { - _run_in_container grep 'Blocked SPAM {NoBounceInbound,Quarantined}' /var/log/mail/mail.log - assert_success + _service_log_should_contain_string 'mail' 'Blocked SPAM {NoBounceInbound,Quarantined}' assert_output --partial ' -> ' _should_output_number_of_lines 1 diff --git a/test/tests/serial/mail_pop3.bats b/test/tests/serial/mail_pop3.bats index d815e1ee..95ae14aa 100644 --- a/test/tests/serial/mail_pop3.bats +++ b/test/tests/serial/mail_pop3.bats @@ -33,11 +33,10 @@ function teardown_file() { _default_teardown ; } assert_success } -@test '/var/log/mail/mail.log is error-free' { - _run_in_container grep 'non-null host address bits in' /var/log/mail/mail.log - assert_failure - _run_in_container grep ': error:' /var/log/mail/mail.log - assert_failure +# TODO: Remove in favor of a common helper method, as described in vmail-id.bats equivalent test-case +@test 'Mail log is error free' { + _service_log_should_not_contain_string 'mail' 'non-null host address bits in' + _service_log_should_not_contain_string 'mail' ': Error:' } @test '(Manage Sieve) disabled per default' { diff --git a/test/tests/serial/mail_with_oauth2.bats b/test/tests/serial/mail_with_oauth2.bats index 8da87980..c9a02eea 100644 --- a/test/tests/serial/mail_with_oauth2.bats +++ b/test/tests/serial/mail_with_oauth2.bats @@ -110,7 +110,7 @@ function __verify_auth_with_smtp() { function __dovecot_logs_should_verify_success() { # Inspect the relevant Dovecot logs to catch failure / success: - _run_in_container grep 'dovecot:' /var/log/mail.log + _service_log_should_contain_string 'mail' 'dovecot:' refute_output --partial 'oauth2 failed: Introspection failed' assert_output --partial "dovecot: imap-login: Login: user=<${USER_ACCOUNT}>, method=${AUTH_METHOD}" diff --git a/test/tests/serial/tests.bats b/test/tests/serial/tests.bats index a344bd8d..eff151df 100644 --- a/test/tests/serial/tests.bats +++ b/test/tests/serial/tests.bats @@ -182,23 +182,16 @@ function teardown_file() { _default_teardown ; } assert_success } -@test "system: /var/log/mail/mail.log is error free" { - _run_in_container grep 'non-null host address bits in' /var/log/mail/mail.log - assert_failure - _run_in_container grep 'mail system configuration error' /var/log/mail/mail.log - assert_failure - _run_in_container grep ': error:' /var/log/mail/mail.log - assert_failure - _run_in_container grep -i 'is not writable' /var/log/mail/mail.log - assert_failure - _run_in_container grep -i 'permission denied' /var/log/mail/mail.log - assert_failure - _run_in_container grep -i '(!)connect' /var/log/mail/mail.log - assert_failure - _run_in_container grep -i 'using backwards-compatible default setting' /var/log/mail/mail.log - assert_failure - _run_in_container grep -i 'connect to 127.0.0.1:10023: Connection refused' /var/log/mail/mail.log - assert_failure +# TODO: Remove in favor of a common helper method, as described in vmail-id.bats equivalent test-case +@test "system: Mail log is error free" { + _service_log_should_not_contain_string 'mail' 'non-null host address bits in' + _service_log_should_not_contain_string 'mail' 'mail system configuration error' + _service_log_should_not_contain_string 'mail' ': Error:' + _service_log_should_not_contain_string 'mail' 'is not writable' + _service_log_should_not_contain_string 'mail' 'Permission denied' + _service_log_should_not_contain_string 'mail' '(!)connect' + _service_log_should_not_contain_string 'mail' 'using backwards-compatible default setting' + _service_log_should_not_contain_string 'mail' 'connect to 127.0.0.1:10023: Connection refused' } @test "system: /var/log/auth.log is error free" { @@ -212,7 +205,8 @@ function teardown_file() { _default_teardown ; } } @test "system: amavis decoders installed and available" { - _run_in_container_bash "grep -E '.*(Internal decoder|Found decoder) for\s+\..*' /var/log/mail/mail.log*|grep -Eo '(mail|Z|gz|bz2|xz|lzma|lrz|lzo|lz4|rpm|cpio|tar|deb|rar|arj|arc|zoo|doc|cab|tnef|zip|kmz|7z|jar|swf|lha|iso|exe)' | sort | uniq" + _service_log_should_contain_string_regexp 'mail' '.*(Internal decoder|Found decoder) for\s+\..*' + run bash -c "grep -Eo '(mail|Z|gz|bz2|xz|lzma|lrz|lzo|lz4|rpm|cpio|tar|deb|rar|arj|arc|zoo|doc|cab|tnef|zip|kmz|7z|jar|swf|lha|iso|exe)' <<< '${output}' | sort | uniq" assert_success # Support for doc and zoo removed in buster cat <<'EOF' | assert_output diff --git a/test/tests/serial/vmail-id.bats b/test/tests/serial/vmail-id.bats index 2541f4f8..a0fb8537 100644 --- a/test/tests/serial/vmail-id.bats +++ b/test/tests/serial/vmail-id.bats @@ -24,8 +24,7 @@ function teardown_file() { _default_teardown ; } _wait_for_empty_mail_queue_in_container # Should be successfully sent (received) by Postfix: - _run_in_container grep 'to=' /var/log/mail/mail.log - assert_success + _service_log_should_contain_string 'mail' 'to=' assert_output --partial 'status=sent' _should_output_number_of_lines 1 @@ -41,35 +40,28 @@ function teardown_file() { _default_teardown ; } # This test case is shared with tests.bats, but provides context on errors + some minor edits # TODO: Could improve in future with keywords from https://github.com/docker-mailserver/docker-mailserver/pull/3550#issuecomment-1738509088 # Potentially via a helper that allows an optional fixed number of errors to be present if they were intentional -@test '/var/log/mail/mail.log is error free' { +@test 'Mail log is error free' { # Postfix: https://serverfault.com/questions/934703/postfix-451-4-3-0-temporary-lookup-failure - _run_in_container grep 'non-null host address bits in' /var/log/mail/mail.log - assert_failure + _service_log_should_not_contain_string 'mail' 'non-null host address bits in' # Postfix delivery failure: https://github.com/docker-mailserver/docker-mailserver/issues/230 - _run_in_container grep 'mail system configuration error' /var/log/mail/mail.log - assert_failure + _service_log_should_not_contain_string 'mail' 'mail system configuration error' # Unknown error source: https://github.com/docker-mailserver/docker-mailserver/pull/85 - _run_in_container grep -i ': error:' /var/log/mail/mail.log - assert_failure + _service_log_should_not_contain_string 'mail' ': Error:' # Unknown error source: https://github.com/docker-mailserver/docker-mailserver/pull/320 - _run_in_container grep -i 'not writable' /var/log/mail/mail.log - assert_failure - _run_in_container grep -i 'permission denied' /var/log/mail/mail.log - assert_failure + _service_log_should_not_contain_string 'mail' 'not writable' + _service_log_should_not_contain_string 'mail' 'Permission denied' # Amavis: https://forum.howtoforge.com/threads/postfix-smtp-error-caused-by-clamav-cant-connect-to-a-unix-socket-var-run-clamav-clamd-ctl.81002/ - _run_in_container grep -i '(!)connect' /var/log/mail/mail.log - assert_failure + _service_log_should_not_contain_string 'mail' '(!)connect' # Postfix: https://github.com/docker-mailserver/docker-mailserver/pull/2597 - _run_in_container grep -i 'using backwards-compatible default setting' /var/log/mail/mail.log - assert_failure + # Log line match example: https://github.com/docker-mailserver/docker-mailserver/pull/2598#issuecomment-1141176633 + _service_log_should_not_contain_string 'mail' 'using backwards-compatible default setting' # Postgrey: https://github.com/docker-mailserver/docker-mailserver/pull/612#discussion_r117635774 - _run_in_container grep -i 'connect to 127.0.0.1:10023: Connection refused' /var/log/mail/mail.log - assert_failure + _service_log_should_not_contain_string 'mail' 'connect to 127.0.0.1:10023: Connection refused' } From 303ca82fb925ab9a0e0aa8bf68a2caa402afe0aa Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Fri, 26 Jan 2024 01:02:19 +1300 Subject: [PATCH 003/267] docs(fix): New external link icon workaround for mkdocs-material 9.5.5 (#3823) This is the easiest to maintain workaround now available. Upstream continues to reject the value such a feature for accessibility. --- docs/content/assets/css/customizations.css | 8 ++++++-- docs/mkdocs.yml | 4 ++-- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/content/assets/css/customizations.css b/docs/content/assets/css/customizations.css index 49ede009..320da152 100644 --- a/docs/content/assets/css/customizations.css +++ b/docs/content/assets/css/customizations.css @@ -16,12 +16,16 @@ If you want to append instead, switch `::before` to `::after`. src: url('../fonts/external-link.woff') format('woff'); } -/* Matches the two nav link classes that start with `http` `href` values, regular docs pages use relative URLs instead. */ -.md-tabs__link[href^="http"]::before, .md-nav__link[href^="http"]::before { +/* + Since mkdocs-material 9.5.5 broke support in our docs from DMS v13.3.1, we now use our own class name, + which has been included for the two external nav links in mkdocs.yml via workaround (insert HTML). +*/ +.icon-external-link::before { display: inline-block; /* treat similar to text */ font-family: 'external-link'; content:'\0041'; /* represents "A" which our font renders as an icon instead of the "A" glyph */ font-size: 80%; /* icon is a little too big by default, scale it down */ + margin-right: 4px; } /* ============================================================================================================= */ diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 0d34b407..15d2c3b2 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -173,5 +173,5 @@ nav: - 'General Information': contributing/general.md - 'Tests': contributing/tests.md - 'Issues and Pull Requests': contributing/issues-and-pull-requests.md - - 'DockerHub': https://hub.docker.com/r/mailserver/docker-mailserver/ - - 'GHCR': https://github.com/docker-mailserver/docker-mailserver/pkgs/container/docker-mailserver + - 'DockerHub': https://hub.docker.com/r/mailserver/docker-mailserver/ + - 'GHCR': https://github.com/docker-mailserver/docker-mailserver/pkgs/container/docker-mailserver From 22c6daee32ad0769002564054654a4e989c9eb10 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Fri, 26 Jan 2024 01:21:24 +1300 Subject: [PATCH 004/267] chore: Revise improper restart message (#3826) Improved guidance. --- target/scripts/startup/check-stack.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/target/scripts/startup/check-stack.sh b/target/scripts/startup/check-stack.sh index 61f21d1c..cf1c40f5 100644 --- a/target/scripts/startup/check-stack.sh +++ b/target/scripts/startup/check-stack.sh @@ -19,7 +19,7 @@ function _check_improper_restart() { if [[ -f /CONTAINER_START ]]; then _log 'warn' 'This container was (likely) improperly restarted which can result in undefined behavior' - _log 'warn' 'Please destroy the container properly and then start DMS again' + _log 'warn' "Please use 'docker compose up --force-recreate' or equivalent (view our troubleshooting docs)" fi } From 47f8d50bebaa0b1f1cb85d45c1d9210d9ce80a49 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Fri, 26 Jan 2024 10:28:26 +1300 Subject: [PATCH 005/267] fix: Ensure configs are sanitized for parsing (#3819) * chore: Detect missing final newline in configs read These lines will be not be processed by `read`, emit a warning to raise awareness. * fix: Ensure parsed config has final newline appended (when possible) This functionality was handled in `accounts.sh` via a similar sed command (that the linked references also offer). `printf` is better for this, no shellcheck comment required either. We additionally don't attempt to modify files that are read-only. * fix: Ensure parsed configs have CRLF to LF corrected (where possible) Likewise, this runtime fix was only covering two config files. It now applies to all callers of this method. * fix: Sanitize `postfix-master.cf` via helper This feature should have been using the helper to avoid user error from their config updates accidentally introducing subtle breakage implicitly (due to CRLF or missing final newline). * tests: Add test cases for new helpers * tests: `rm` is redundant when using `BATS_TEST_TMPDIR` This temporary directory is created and removed implicitly. Even after a test failure. * chore: Remove old `postfix-virtual.cf` migration logic This was introduced in 2018, there should be no one needing to rely on this anymore? * tests: Remove comment on sed failure concern * chore: Add entry to `CHANGELOG.md` * Apply suggestions from code review Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> --------- Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> --- CHANGELOG.md | 4 ++ target/scripts/helpers/accounts.sh | 14 ------- target/scripts/helpers/aliases.sh | 5 --- target/scripts/helpers/utils.sh | 35 ++++++++++++++++ target/scripts/startup/setup.d/postfix.sh | 22 +++++----- test/helper/setup.bash | 1 - .../set3/scripts/helper_functions.bats | 42 +++++++++++++++++++ 7 files changed, 92 insertions(+), 31 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 12f07561..9f440e31 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,6 +38,10 @@ The most noteworthy change of this release is the update of the container's base - **Tests:** - Refactored helper methods for sending e-mails with specific `Message-ID` headers and the helpers for retrieving + filtering logs, which together help isolate logs relevant to specific mail when multiple mails have been processed within a single test. ([#3786](https://github.com/docker-mailserver/docker-mailserver/pull/3786)) +### Fixes + +- DMS config files that are parsed line by line are now more robust to parse by detecting and fixing line-endings ([#3819](https://github.com/docker-mailserver/docker-mailserver/pull/3819)) + ## [v13.3.1](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v13.3.1) ### Fixes diff --git a/target/scripts/helpers/accounts.sh b/target/scripts/helpers/accounts.sh index 7499a2ec..31ded04a 100644 --- a/target/scripts/helpers/accounts.sh +++ b/target/scripts/helpers/accounts.sh @@ -19,16 +19,9 @@ function _create_accounts() { _create_masters if [[ -f ${DATABASE_ACCOUNTS} ]]; then - _log 'trace' "Checking file line endings" - sed -i 's|\r||g' "${DATABASE_ACCOUNTS}" - _log 'trace' "Regenerating postfix user list" echo "# WARNING: this file is auto-generated. Modify ${DATABASE_ACCOUNTS} to edit the user list." > /etc/postfix/vmailbox - # checking that ${DATABASE_ACCOUNTS} ends with a newline - # shellcheck disable=SC1003 - sed -i -e '$a\' "${DATABASE_ACCOUNTS}" - chown dovecot:dovecot "${DOVECOT_USERDB_FILE}" chmod 640 "${DOVECOT_USERDB_FILE}" @@ -158,15 +151,8 @@ function _create_masters() { local DATABASE_DOVECOT_MASTERS='/tmp/docker-mailserver/dovecot-masters.cf' if [[ -f ${DATABASE_DOVECOT_MASTERS} ]]; then - _log 'trace' "Checking file line endings" - sed -i 's|\r||g' "${DATABASE_DOVECOT_MASTERS}" - _log 'trace' "Regenerating dovecot masters list" - # checking that ${DATABASE_DOVECOT_MASTERS} ends with a newline - # shellcheck disable=SC1003 - sed -i -e '$a\' "${DATABASE_DOVECOT_MASTERS}" - chown dovecot:dovecot "${DOVECOT_MASTERDB_FILE}" chmod 640 "${DOVECOT_MASTERDB_FILE}" diff --git a/target/scripts/helpers/aliases.sh b/target/scripts/helpers/aliases.sh index 04a56da3..b0f2fa1a 100644 --- a/target/scripts/helpers/aliases.sh +++ b/target/scripts/helpers/aliases.sh @@ -12,11 +12,6 @@ function _handle_postfix_virtual_config() { local DATABASE_VIRTUAL=/tmp/docker-mailserver/postfix-virtual.cf if [[ -f ${DATABASE_VIRTUAL} ]]; then - # fixing old virtual user file - if grep -q ",$" "${DATABASE_VIRTUAL}"; then - sed -i -e "s|, |,|g" -e "s|,$||g" "${DATABASE_VIRTUAL}" - fi - cp -f "${DATABASE_VIRTUAL}" /etc/postfix/virtual else _log 'debug' "'${DATABASE_VIRTUAL}' not provided - no mail alias/forward created" diff --git a/target/scripts/helpers/utils.sh b/target/scripts/helpers/utils.sh index f7095bf3..c848e13e 100644 --- a/target/scripts/helpers/utils.sh +++ b/target/scripts/helpers/utils.sh @@ -17,9 +17,44 @@ function _escape_for_sed() { # Returns input after filtering out lines that are: # empty, white-space, comments (`#` as the first non-whitespace character) function _get_valid_lines_from_file() { + _convert_crlf_to_lf_if_necessary "${1}" + _append_final_newline_if_missing "${1}" + grep --extended-regexp --invert-match "^\s*$|^\s*#" "${1}" || true } +# This is to sanitize configs from users that unknowingly introduced CRLF: +function _convert_crlf_to_lf_if_necessary() { + if [[ $(file "${1}") =~ 'CRLF' ]]; then + _log 'warn' "File '${1}' contains CRLF line-endings" + + if [[ -w ${1} ]]; then + _log 'debug' 'Converting CRLF to LF' + sed -i 's|\r||g' "${1}" + else + _log 'warn' "File '${1}' is not writable - cannot change CRLF to LF" + fi + fi +} + +# This is to sanitize configs from users that unknowingly removed the end-of-file LF: +function _append_final_newline_if_missing() { + # Correctly detect a missing final newline and fix it: + # https://stackoverflow.com/questions/38746/how-to-detect-file-ends-in-newline#comment82380232_25749716 + # https://unix.stackexchange.com/questions/31947/how-to-add-a-newline-to-the-end-of-a-file/441200#441200 + # https://unix.stackexchange.com/questions/159557/how-to-non-invasively-test-for-write-access-to-a-file + if [[ $(tail -c1 "${1}" | wc -l) -eq 0 ]]; then + # Avoid fixing when the destination is read-only: + if [[ -w ${1} ]]; then + printf '\n' >> "${1}" + + _log 'info' "File '${1}' was missing a final newline - this has been fixed" + else + _log 'warn' "File '${1}' is missing a final newline - it is not writable, hence it was not fixed - the last line will not be processed!" + fi + fi +} + # Provide the name of an environment variable to this function # and it will return its value stored in /etc/dms-settings function _get_dms_env_value() { diff --git a/target/scripts/startup/setup.d/postfix.sh b/target/scripts/startup/setup.d/postfix.sh index 5aec8636..05052faa 100644 --- a/target/scripts/startup/setup.d/postfix.sh +++ b/target/scripts/startup/setup.d/postfix.sh @@ -109,8 +109,9 @@ function _setup_postfix_late() { function __postfix__setup_override_configuration() { __postfix__log 'debug' 'Overriding / adjusting configuration with user-supplied values' - if [[ -f /tmp/docker-mailserver/postfix-main.cf ]]; then - cat /tmp/docker-mailserver/postfix-main.cf >>/etc/postfix/main.cf + local OVERRIDE_CONFIG_POSTFIX_MAIN='/tmp/docker-mailserver/postfix-main.cf' + if [[ -f ${OVERRIDE_CONFIG_POSTFIX_MAIN} ]]; then + cat "${OVERRIDE_CONFIG_POSTFIX_MAIN}" >>/etc/postfix/main.cf _adjust_mtime_for_postfix_maincf # do not directly output to 'main.cf' as this causes a read-write-conflict @@ -118,20 +119,19 @@ function __postfix__setup_override_configuration() { mv /tmp/postfix-main-new.cf /etc/postfix/main.cf _adjust_mtime_for_postfix_maincf - __postfix__log 'trace' "Adjusted '/etc/postfix/main.cf' according to '/tmp/docker-mailserver/postfix-main.cf'" + __postfix__log 'trace' "Adjusted '/etc/postfix/main.cf' according to '${OVERRIDE_CONFIG_POSTFIX_MAIN}'" else - __postfix__log 'trace' "No extra Postfix settings loaded because optional '/tmp/docker-mailserver/postfix-main.cf' was not provided" + __postfix__log 'trace' "No extra Postfix settings loaded because optional '${OVERRIDE_CONFIG_POSTFIX_MAIN}' was not provided" fi - if [[ -f /tmp/docker-mailserver/postfix-master.cf ]]; then + local OVERRIDE_CONFIG_POSTFIX_MASTER='/tmp/docker-mailserver/postfix-master.cf' + if [[ -f ${OVERRIDE_CONFIG_POSTFIX_MASTER} ]]; then while read -r LINE; do - if [[ ${LINE} =~ ^[0-9a-z] ]]; then - postconf -P "${LINE}" - fi - done < /tmp/docker-mailserver/postfix-master.cf - __postfix__log 'trace' "Adjusted '/etc/postfix/master.cf' according to '/tmp/docker-mailserver/postfix-master.cf'" + [[ ${LINE} =~ ^[0-9a-z] ]] && postconf -P "${LINE}" + done < <(_get_valid_lines_from_file "${OVERRIDE_CONFIG_POSTFIX_MASTER}") + __postfix__log 'trace' "Adjusted '/etc/postfix/master.cf' according to '${OVERRIDE_CONFIG_POSTFIX_MASTER}'" else - __postfix__log 'trace' "No extra Postfix settings loaded because optional '/tmp/docker-mailserver/postfix-master.cf' was not provided" + __postfix__log 'trace' "No extra Postfix settings loaded because optional '${OVERRIDE_CONFIG_POSTFIX_MASTER}' was not provided" fi } diff --git a/test/helper/setup.bash b/test/helper/setup.bash index 0dd57bd6..ed2e4e32 100644 --- a/test/helper/setup.bash +++ b/test/helper/setup.bash @@ -102,7 +102,6 @@ function _init_with_defaults() { # The config volume cannot be read-only as some data needs to be written at container startup # - # - two sed failures (unknown lines) # - dovecot-quotas.cf (setup-stack.sh:_setup_dovecot_quotas) # - postfix-aliases.cf (setup-stack.sh:_setup_postfix_aliases) # TODO: Check how many tests need write access. Consider using `docker create` + `docker cp` for easier cleanup. diff --git a/test/tests/parallel/set3/scripts/helper_functions.bats b/test/tests/parallel/set3/scripts/helper_functions.bats index 332de448..518f8717 100644 --- a/test/tests/parallel/set3/scripts/helper_functions.bats +++ b/test/tests/parallel/set3/scripts/helper_functions.bats @@ -70,3 +70,45 @@ SOURCE_BASE_PATH="${REPOSITORY_ROOT:?Expected REPOSITORY_ROOT to be set}/target/ assert_failure assert_output --partial "ENV var name must be provided to _env_var_expect_integer" } + +@test '(utils.sh) _convert_crlf_to_lf_if_necessary' { + # shellcheck source=../../../../../target/scripts/helpers/log.sh + source "${SOURCE_BASE_PATH}/log.sh" + # shellcheck source=../../../../../target/scripts/helpers/utils.sh + source "${SOURCE_BASE_PATH}/utils.sh" + + # Create a temporary file in the BATS test-case folder: + local TMP_DMS_CONFIG=$(mktemp -p "${BATS_TEST_TMPDIR}" -t 'dms_XXX.cf') + # A file with mixed line-endings including CRLF: + echo -en 'line one\nline two\r\n' > "${TMP_DMS_CONFIG}" + + # Confirm CRLF detected: + run file "${TMP_DMS_CONFIG}" + assert_output --partial 'CRLF' + + # Helper method detects and fixes: + _convert_crlf_to_lf_if_necessary "${TMP_DMS_CONFIG}" + run file "${TMP_DMS_CONFIG}" + refute_output --partial 'CRLF' +} + +@test '(utils.sh) _append_final_newline_if_missing' { + # shellcheck source=../../../../../target/scripts/helpers/log.sh + source "${SOURCE_BASE_PATH}/log.sh" + # shellcheck source=../../../../../target/scripts/helpers/utils.sh + source "${SOURCE_BASE_PATH}/utils.sh" + + # Create a temporary file in the BATS test-case folder: + local TMP_DMS_CONFIG=$(mktemp -p "${BATS_TEST_TMPDIR}" -t 'dms_XXX.cf') + # A file missing a final newline: + echo -en 'line one\nline two' > "${TMP_DMS_CONFIG}" + + # Confirm missing newline: + run bash -c "tail -c 1 '${TMP_DMS_CONFIG}' | wc -l" + assert_output '0' + + # Helper method detects and fixes: + _append_final_newline_if_missing "${TMP_DMS_CONFIG}" + run bash -c "tail -c 1 '${TMP_DMS_CONFIG}' | wc -l" + assert_output '1' +} From 487867285bd1c7c6d3157afde45adce08d3c0409 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Fri, 26 Jan 2024 23:32:18 +1300 Subject: [PATCH 006/267] docs: UX Improvement - Better distinguish side nav page categories (#3835) --- docs/content/assets/css/customizations.css | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/content/assets/css/customizations.css b/docs/content/assets/css/customizations.css index 320da152..25cb0274 100644 --- a/docs/content/assets/css/customizations.css +++ b/docs/content/assets/css/customizations.css @@ -102,3 +102,8 @@ div.md-content article.md-content__inner a.toclink code { .highlight.no-copy .md-clipboard { display: none; } /* ============================================================================================================= */ + +/* Make the left-sidebar nav categories better distinguished from page links (bold text) */ +.md-nav__item--nested > .md-nav__link { + font-weight: 700; +} From 7d9eb1e4a7ddd7b78b7bda2928b802af990af5ec Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Fri, 26 Jan 2024 23:32:49 +1300 Subject: [PATCH 007/267] docs: Add context to `sender-cleanup` in Postfix `master.cf` (#3834) Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> --- target/postfix/master.cf | 2 ++ 1 file changed, 2 insertions(+) diff --git a/target/postfix/master.cf b/target/postfix/master.cf index e5b955a4..7d2b21d6 100644 --- a/target/postfix/master.cf +++ b/target/postfix/master.cf @@ -46,6 +46,8 @@ pickup fifo n - n 60 1 pickup -o content_filter= -o receive_override_options=no_header_body_checks +# This relates to submission(s) services defined above: +# https://www.postfix.org/BUILTIN_FILTER_README.html#mx_submission sender-cleanup unix n - n - 0 cleanup -o syslog_name=postfix/sender-cleanup -o header_checks=pcre:/etc/postfix/maps/sender_header_filter.pcre From ba27edc801c0b94cfb7d975dd2fb918a75ac9ea3 Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Fri, 26 Jan 2024 14:07:46 +0100 Subject: [PATCH 008/267] Rspamd: only declare Rspamd variables when not already declared (#3837) * only declare Rspamd vars when not already declared * update CHANGELOG * Update CHANGELOG.md --------- Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- CHANGELOG.md | 1 + target/scripts/helpers/rspamd.sh | 17 +++++++++++------ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f440e31..a9c03f79 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -41,6 +41,7 @@ The most noteworthy change of this release is the update of the container's base ### Fixes - DMS config files that are parsed line by line are now more robust to parse by detecting and fixing line-endings ([#3819](https://github.com/docker-mailserver/docker-mailserver/pull/3819)) +- Variables related to Rspamd are declared as `readonly`, which would cause warnings in the log when being re-declared; we now guard against this issue ([#3837](https://github.com/docker-mailserver/docker-mailserver/pull/3837)) ## [v13.3.1](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v13.3.1) diff --git a/target/scripts/helpers/rspamd.sh b/target/scripts/helpers/rspamd.sh index 2f4dcc46..8d1fd668 100644 --- a/target/scripts/helpers/rspamd.sh +++ b/target/scripts/helpers/rspamd.sh @@ -15,14 +15,19 @@ function __do_as_rspamd_user() { # they cannot be modified. Use this function when you require common directory # names, file names, etc. function _rspamd_get_envs() { - readonly RSPAMD_LOCAL_D='/etc/rspamd/local.d' - readonly RSPAMD_OVERRIDE_D='/etc/rspamd/override.d' + # If the variables are already set, we cannot set them again as they are declared + # with `readonly`. Checking whether one is declared suffices, because either all + # are declared at once, or none. + if [[ ! -v RSPAMD_LOCAL_D ]]; then + readonly RSPAMD_LOCAL_D='/etc/rspamd/local.d' + readonly RSPAMD_OVERRIDE_D='/etc/rspamd/override.d' - readonly RSPAMD_DMS_D='/tmp/docker-mailserver/rspamd' - readonly RSPAMD_DMS_DKIM_D="${RSPAMD_DMS_D}/dkim" - readonly RSPAMD_DMS_OVERRIDE_D="${RSPAMD_DMS_D}/override.d" + readonly RSPAMD_DMS_D='/tmp/docker-mailserver/rspamd' + readonly RSPAMD_DMS_DKIM_D="${RSPAMD_DMS_D}/dkim" + readonly RSPAMD_DMS_OVERRIDE_D="${RSPAMD_DMS_D}/override.d" - readonly RSPAMD_DMS_CUSTOM_COMMANDS_F="${RSPAMD_DMS_D}/custom-commands.conf" + readonly RSPAMD_DMS_CUSTOM_COMMANDS_F="${RSPAMD_DMS_D}/custom-commands.conf" + fi } # Parses `RSPAMD_DMS_CUSTOM_COMMANDS_F` and executed the directives given by the file. From 9ac11021e1c33d87ec706f3f76b6c3197f1f000d Mon Sep 17 00:00:00 2001 From: Andreas Perhab Date: Fri, 26 Jan 2024 14:40:29 +0100 Subject: [PATCH 009/267] setup-stack: fix error when RSPAMD_DMS_DKIM_D is not set (#3827) * setup-stack: fix error when RSPAMD_DMS_DKIM_D is not set prevent messages like this chown: cannot access '': No such file or directory when RSPAMD_DMS_DKIM_D has no value * Update target/scripts/startup/setup-stack.sh --------- Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> --- target/scripts/startup/setup-stack.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/target/scripts/startup/setup-stack.sh b/target/scripts/startup/setup-stack.sh index c3c54cc3..24ef1581 100644 --- a/target/scripts/startup/setup-stack.sh +++ b/target/scripts/startup/setup-stack.sh @@ -95,8 +95,9 @@ function _setup_apply_fixes_after_configuration() { _log 'debug' 'Removing files and directories from older versions' rm -rf /var/mail-state/spool-postfix/{dev,etc,lib,pid,usr,private/auth} + _rspamd_get_envs # /tmp/docker-mailserver/rspamd/dkim - _log 'debug' "Ensuring ${RSPAMD_DMS_DKIM_D} is owned by '_rspamd:_rspamd'" + _log 'debug' "Ensuring '${RSPAMD_DMS_DKIM_D}' is owned by '_rspamd:_rspamd'" chown -R _rspamd:_rspamd "${RSPAMD_DMS_DKIM_D}" } From a8ccd54da57111f6e619c987da1ca9d807fa4fcc Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Sun, 28 Jan 2024 01:50:01 +1300 Subject: [PATCH 010/267] ci: `docs-preview-deploy.yml` - Switch to official `download-artifact` action (#3838) v4 of the official action now supports this use-case. --- .github/workflows/docs-preview-deploy.yml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/.github/workflows/docs-preview-deploy.yml b/.github/workflows/docs-preview-deploy.yml index 93819efa..6f6904d5 100644 --- a/.github/workflows/docs-preview-deploy.yml +++ b/.github/workflows/docs-preview-deploy.yml @@ -23,14 +23,13 @@ jobs: # Restore workflow context # # ======================== # - # The official Github Action for downloading artifacts does not support multi-workflow + # Retrieve the artifact uploaded from `docs-preview-prepare.yml` workflow run that triggered this deployment - name: 'Download build artifact' - uses: dawidd6/action-download-artifact@v3 + uses: actions/download-artifact@v4 with: + name: preview-build github_token: ${{ secrets.GITHUB_TOKEN }} run_id: ${{ github.event.workflow_run.id }} - workflow: docs-preview-prepare.yml - name: preview-build - name: 'Extract build artifact' run: tar -xf artifact.tar.zst From f27629be4e24a1fc288e947662f8294743063d5f Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Sun, 28 Jan 2024 01:51:37 +1300 Subject: [PATCH 011/267] docs: Minor revisions to `README.md` (#3839) --- README.md | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index e81d21ad..cfd453ff 100644 --- a/README.md +++ b/README.md @@ -4,22 +4,27 @@ [ci::status]: https://img.shields.io/github/actions/workflow/status/docker-mailserver/docker-mailserver/default_on_push.yml?branch=master&color=blue&label=CI&logo=github&logoColor=white&style=for-the-badge [ci::github]: https://github.com/docker-mailserver/docker-mailserver/actions -[docker::pulls]: https://img.shields.io/docker/pulls/mailserver/docker-mailserver.svg?style=for-the-badge&logo=docker&logoColor=white +[docker::pulls]: https://img.shields.io/docker/pulls/mailserver/docker-mailserver.svg?style=for-the-badge&logo=docker&logoColor=white&color=blue [docker::hub]: https://hub.docker.com/r/mailserver/docker-mailserver/ -[documentation::badge]: https://img.shields.io/badge/DOCUMENTATION-GH%20PAGES-0078D4?style=for-the-badge&logo=git&logoColor=white +[documentation::badge]: https://img.shields.io/badge/DOCUMENTATION-GH%20PAGES-0078D4?style=for-the-badge&logo=googledocs&logoColor=white [documentation::web]: https://docker-mailserver.github.io/docker-mailserver/latest/ ## :page_with_curl: About -A production-ready fullstack but simple containerized mail server (SMTP, IMAP, LDAP, Anti-spam, Anti-virus, etc.). Only configuration files, no SQL database. Keep it simple and versioned. Easy to deploy and upgrade. Originally created by @tomav, this project is now maintained by volunteers since January 2021. +A production-ready fullstack but simple containerized mail server (SMTP, IMAP, LDAP, Anti-spam, Anti-virus, etc.). +- Only configuration files, no SQL database. Keep it simple and versioned. Easy to deploy and upgrade. +- Originally created by [@tomav](https://github.com/tomav), this project is now maintained by volunteers since January 2021. -## :bulb: Documentation +## -We provide a [dedicated documentation][documentation::web] hosted on GitHub Pages. Make sure to read it as it contains all the information necessary to set up and configure your mail server. The documentation is crafted with Markdown & [MkDocs Material](https://squidfunk.github.io/mkdocs-material/). +> [!TIP] +> Be sure to read [our documentation][documentation::web]. It provides guidance on initial setup of your mail server. -## :boom: Issues - -If you have issues, please search through [the documentation][documentation::web] **for your version** before opening an issue. The issue tracker is for issues, not for personal support. Make sure the version of the documentation matches the image version you're using! +> [!IMPORTANT] +> If you have issues, please search through [the documentation][documentation::web] **for your version** before opening an issue. +> +> The issue tracker is for issues, not for personal support. +> Make sure the version of the documentation matches the image version you're using! ## :link: Links to Useful Resources From 4a05d7bb7caeb67347034f5280f41f2a42170410 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Sun, 28 Jan 2024 10:23:49 +1300 Subject: [PATCH 012/267] docs: Add Debian 12 breaking change for `opendmarc` package (#3841) --- CHANGELOG.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a9c03f79..e785cac0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,11 @@ The most noteworthy change of this release is the update of the container's base - The script now uses `/etc/os-release` to determine the release name of Debian - Removed custom installations of Fail2Ban, getmail6 and Rspamd - Updated packages lists and added comments for maintainability +- OpenDMARC upgrade: `v1.4.0` => `v1.4.2` ([#3841](https://github.com/docker-mailserver/docker-mailserver/pull/3841)) + - Previous versions of OpenDMARC would place incoming mail from domains announcing `p=quarantaine` (_that fail the DMARC check_) into the [Postfix "hold" queue](https://www.postfix.org/QSHAPE_README.html#hold_queue) until administrative intervention. + - [OpenDMARC v1.4.2 has disabled that feature by default](https://github.com/trusteddomainproject/OpenDMARC/issues/105), but it can be enabled again by adding the setting `HoldQuarantinedMessages true` to [`/etc/opendmarc.conf`](https://github.com/docker-mailserver/docker-mailserver/blob/v13.3.1/target/opendmarc/opendmarc.conf) (_provided from DMS_). + - [Our `user-patches.sh` feature](https://docker-mailserver.github.io/docker-mailserver/latest/config/advanced/override-defaults/user-patches/) provides a convenient approach to updating that config file. + - Please let us know if you disagree with the upstream default being carried with DMS, or the value of providing alternative configuration support within DMS. - **Postfix:** - Postfix upgrade from 3.5 to 3.7 ([#3403](https://github.com/docker-mailserver/docker-mailserver/pull/3403)) - `compatibility_level` was raised from `2` to `3.6` From 204825fa5a2476a0b185df85365b5cad6b73b570 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Sun, 28 Jan 2024 20:41:19 +1300 Subject: [PATCH 013/267] ci(fix): `docs-preview-deploy.yml` - Use the correct setting names (#3843) --- .github/workflows/docs-preview-deploy.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/docs-preview-deploy.yml b/.github/workflows/docs-preview-deploy.yml index 6f6904d5..3b6f3c67 100644 --- a/.github/workflows/docs-preview-deploy.yml +++ b/.github/workflows/docs-preview-deploy.yml @@ -28,8 +28,8 @@ jobs: uses: actions/download-artifact@v4 with: name: preview-build - github_token: ${{ secrets.GITHUB_TOKEN }} - run_id: ${{ github.event.workflow_run.id }} + github-token: ${{ secrets.GITHUB_TOKEN }} + run-id: ${{ github.event.workflow_run.id }} - name: 'Extract build artifact' run: tar -xf artifact.tar.zst From 11c508cd11c9c850527c54b49631374052c67303 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sun, 28 Jan 2024 11:15:47 +0000 Subject: [PATCH 014/267] docs: update `CONTRIBUTORS.md` (#3844) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Casper --- CONTRIBUTORS.md | 183 +++++++++++++++++++++++++----------------------- 1 file changed, 95 insertions(+), 88 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index f86435f4..72784ab0 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -134,13 +134,6 @@ Thanks goes to these wonderful people ✨ - + + + + - + + + - + - + + + - + + + - + + + - + + + - + + + - + + + - + + + - + + + - + + + - + + + - + - + + + - + + + - + + + - + - + + + - + + + - + - - + + - - - + + + - - + + + + - - - - - - - - - @@ -873,10 +830,53 @@ Thanks goes to these wonderful people ✨ + + + + + + + + + + + - + + + - + + + - + - + + - + + + - + + + - + + + - + + + - + + + - + + + - + + + - + + + - + + + - + + + - + -
- - alinmear -
- alinmear - - 		ap-wtioit @@ -148,6 +141,13 @@ Thanks goes to these wonderful people ✨ ap-wtioit + + alinmear +
+ alinmear + + 		dominikwinter @@ -815,13 +815,20 @@ Thanks goes to these wonderful people ✨ - - nilshoell + + auchri
- nilshoell + auchri
+ + stephan-devop +
+ stephan-devop + + 		stigok @@ -856,15 +863,15 @@ Thanks goes to these wonderful people ✨
TechnicLab -
thomasschmit
thomasschmit -
Thiritin @@ -899,15 +906,15 @@ Thanks goes to these wonderful people ✨
Twist235 -
k3it
k3it -
Drakulix @@ -942,15 +949,15 @@ Thanks goes to these wonderful people ✨
Zepmann - 		- - allddd -
- allddd -
+ + nilshoell +
+ nilshoell + + 		nknapp @@ -985,15 +992,15 @@ Thanks goes to these wonderful people ✨
mrPjer -
p3dda
p3dda -
peter-hartmann @@ -1028,15 +1035,15 @@ Thanks goes to these wonderful people ✨
norrs -
MightySCollins
MightySCollins -
501st-alpha1 @@ -1071,15 +1078,15 @@ Thanks goes to these wonderful people ✨
shyim -
sjmudd
sjmudd -
simonsystem @@ -1088,10 +1095,10 @@ Thanks goes to these wonderful people ✨ - - stephan-devop + + allddd
- stephan-devop + allddd 		@@ -1114,15 +1121,15 @@ Thanks goes to these wonderful people ✨
odinis -
okamidash
okamidash -
olaf-mandel @@ -1157,15 +1164,15 @@ Thanks goes to these wonderful people ✨
rhyst -
rmlhuk
rmlhuk -
rriski @@ -1200,15 +1207,15 @@ Thanks goes to these wonderful people ✨
squash -
strarsis
strarsis -
tamueller @@ -1243,15 +1250,15 @@ Thanks goes to these wonderful people ✨
worldworm -
arcaine2
arcaine2 -
awb99 @@ -1286,15 +1293,15 @@ Thanks goes to these wonderful people ✨
dimalo -
eleith
eleith -
ghnp5 @@ -1329,15 +1336,15 @@ Thanks goes to these wonderful people ✨
idaadi -
ixeft
ixeft -
jjtt @@ -1372,15 +1379,15 @@ Thanks goes to these wonderful people ✨
callmemagnus -
marios88
marios88 -
matrixes @@ -1395,13 +1402,6 @@ Thanks goes to these wonderful people ✨ mchamplain - - auchri -
- auchri - - 		arkanovicz @@ -1539,10 +1539,10 @@ Thanks goes to these wonderful people ✨ - - thechubbypanda + + froks
- thechubbypanda + froks 		@@ -1696,6 +1696,13 @@ Thanks goes to these wonderful people ✨ crash7 + + thechubbypanda +
+ thechubbypanda + + 		KCrawley @@ -1716,15 +1723,15 @@ Thanks goes to these wonderful people ✨
JustAnother1 -
LeoWinterDE
LeoWinterDE -
linhandev @@ -1759,15 +1766,15 @@ Thanks goes to these wonderful people ✨
madmath03 -
maxemann96
maxemann96 -
dragetd @@ -1802,15 +1809,15 @@ Thanks goes to these wonderful people ✨
mcchots -
MohammedNoureldin
MohammedNoureldin -
mpldr @@ -1839,13 +1846,6 @@ Thanks goes to these wonderful people ✨ radicand - - froks -
- froks - - 		fkefer @@ -1911,6 +1911,13 @@ Thanks goes to these wonderful people ✨ Influencer + + JamBalaya56562 +
+ JamBalaya56562 + + 		jcalfee @@ -1931,15 +1938,15 @@ Thanks goes to these wonderful people ✨
init-js -
Jeidnx
Jeidnx -
JiLleON @@ -1974,15 +1981,15 @@ Thanks goes to these wonderful people ✨
JOduMonT -
Kaan88
Kaan88 -
akkumar From 3b11a8305ee2e303b7e14faeb942e140d0434b3b Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Mon, 29 Jan 2024 10:35:19 +1300 Subject: [PATCH 015/267] docs: Remove ENV `ONE_DIR` (#3840) * docs: Better document DMS volumes * docs: Remove any mention of `ONE_DIR` ENV * chore: Remove `ONE_DIR` ENV from scripts Only `ONE_DIR=0` has any effect. As the actual feature is now dependent upon the `/var/mail-state` location existing. It is advised not mounting anything there instead if wanting to avoid runtime state consolidation. * docs: Adjust link ref convention This is more search friendly / organized to find references to all DMS volumes. * lint: Ensure final newline is present VSCode by default excludes this if the last line rendered is removed (rendered as a separate blank line). A separate setting can enforce adding the final newline upon save regardless. --- CHANGELOG.md | 3 + docs/content/config/advanced/kubernetes.md | 1 - .../config/advanced/optional-config.md | 62 +++++++++++++++++-- .../config/best-practices/dkim_dmarc_spf.md | 2 +- docs/content/config/environment.md | 13 ++-- docs/content/config/security/fail2ban.md | 4 +- docs/content/config/security/rspamd.md | 18 ++++-- docs/content/config/security/ssl.md | 4 +- .../tutorials/mailserver-behind-proxy.md | 2 - docs/content/faq.md | 39 ++++-------- docs/content/usage.md | 4 +- mailserver.env | 4 -- target/scripts/startup/setup.d/mail_state.sh | 6 +- .../startup/setup.d/security/rspamd.sh | 6 +- target/scripts/startup/variables-stack.sh | 1 - 15 files changed, 102 insertions(+), 67 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e785cac0..41e6d5e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,6 +40,9 @@ The most noteworthy change of this release is the update of the container's base ### Updates +- **Environment Variables:** + - `ONE_DIR` has been removed (legacy ENV) ([#3840](https://github.com/docker-mailserver/docker-mailserver/pull/3840)) + - It's only functionality remaining was to opt-out of run-time state consolidation with `ONE_DIR=0` (_when a volume was already mounted to `/var/mail-state`_). - **Tests:** - Refactored helper methods for sending e-mails with specific `Message-ID` headers and the helpers for retrieving + filtering logs, which together help isolate logs relevant to specific mail when multiple mails have been processed within a single test. ([#3786](https://github.com/docker-mailserver/docker-mailserver/pull/3786)) diff --git a/docs/content/config/advanced/kubernetes.md b/docs/content/config/advanced/kubernetes.md index 8a47bffc..c277852a 100644 --- a/docs/content/config/advanced/kubernetes.md +++ b/docs/content/config/advanced/kubernetes.md @@ -40,7 +40,6 @@ data: POSTMASTER_ADDRESS: postmaster@example.com UPDATE_CHECK_INTERVAL: 10d POSTFIX_INET_PROTOCOLS: ipv4 - ONE_DIR: '1' ENABLE_CLAMAV: '1' ENABLE_POSTGREY: '0' ENABLE_FAIL2BAN: '1' diff --git a/docs/content/config/advanced/optional-config.md b/docs/content/config/advanced/optional-config.md index 31090050..fb1ab29d 100644 --- a/docs/content/config/advanced/optional-config.md +++ b/docs/content/config/advanced/optional-config.md @@ -4,9 +4,64 @@ hide: - toc # Hide Table of Contents for this page --- -This is a list of all configuration files and directories which are optional or automatically generated in your [`docker-data/dms/config/`][docs-dms-config-volume] directory. +## Volumes -## Directories +DMS has several locations in the container which may be worth persisting externally via [Docker Volumes][docker-docs::volumes]. + +- Often you will want to prefer [bind mount volumes][docker-docs::volumes::bind-mount] for easy access to files at a local location on your filesystem. +- As a convention for our docs and example configs, the local location has the common prefix `docker-data/dms/` for grouping these related volumes. + +!!! info "Reference - Volmes for DMS" + + Our docs may refer to these DMS specific volumes only by name, or the host/container path for brevity. + + - [Config](#volumes-config): `docker-data/dms/config/` => `/tmp/docker-mailserver/` + - [Mail Storage](#volumes-mail): `docker-data/dms/mail-data/` => `/var/mail/` + - [State](#volumes-state): `docker-data/dms/mail-state/` => `/var/mail-state/` + - [Logs](#volumes-logs): `docker-data/dms/mail-logs/` => `/var/log/mail/` + +[docker-docs::volumes]: https://docs.docker.com/storage/volumes/ +[docker-docs::volumes::bind-mount]: https://docs.docker.com/storage/bind-mounts/ + +### Mail Storage Volume { #volumes-mail } + +This is the location where mail is delivered to your mailboxes. + +### State Volume { #volumes-state } + +Run-time specific state lives here, but so does some data you may want to keep if a failure event occurs (_crash, power loss_). + +!!! example "Examples of relevant data" + + - The Postfix queue (eg: mail pending delivery attempt) + - Fail2Ban blocks. + - ClamAV signature updates. + - Redis storage for Rspamd. + +!!! info "When a volume is mounted to `/var/mail-state/`" + + - Service run-time data is [consolidated into the `/var/mail-state/` directory][mail-state-folders]. Otherwise the original locations vary and would need to be mounted individually. + - The original locations are updated with symlinks to redirect to their new path in `/var/mail-state/` (_eg: `/var/lib/redis` => `/var/mail-state/lib-redis/`_). + + Supported services: Postfix, Dovecot, Fail2Ban, Amavis, PostGrey, ClamAV, SpamAssassin, Rspamd & Redis, Fetchmail, Getmail, LogRotate, PostSRSd, MTA-STS. + +!!! tip + + Sometimes it is helpful to disable this volume when troubleshooting to verify if the data stored here is in a bad state (_eg: caused by a failure event_). + +[mail-state-folders]: https://github.com/docker-mailserver/docker-mailserver/blob/v13.3.1/target/scripts/startup/setup.d/mail_state.sh#L13-L33 + +### Logs Volume { #volumes-log } + +This can be a useful volume to persist for troubleshooting needs for the full set of log files. + +### Config Volume { #volumes-config } + +Most configuration files for Postfix, Dovecot, etc. are persisted here. + +This is a list of all configuration files and directories which are optional, automatically generated / updated by our `setup` CLI, or other internal scripts. + +#### Directories - **sieve-filter:** directory for sieve filter scripts. (Docs: [Sieve][docs-sieve]) - **sieve-pipe:** directory for sieve pipe scripts. (Docs: [Sieve][docs-sieve]) @@ -14,7 +69,7 @@ This is a list of all configuration files and directories which are optional or - **ssl:** SSL Certificate directory if `SSL_TYPE` is set to `self-signed` or `custom`. (Docs: [SSL][docs-ssl]) - **rspamd:** Override directory for custom settings when using Rspamd (Docs: [Rspamd][docs-rspamd-override-d]) -## Files +#### Files - **{user_email_address}.dovecot.sieve:** User specific Sieve filter file. (Docs: [Sieve][docs-sieve]) - **before.dovecot.sieve:** Global Sieve filter file, applied prior to the `${login}.dovecot.sieve` filter. (Docs: [Sieve][docs-sieve]) @@ -42,7 +97,6 @@ This is a list of all configuration files and directories which are optional or - **user-patches.sh:** this file will be run after all configuration files are set up, but before the postfix, amavis and other daemons are started. (Docs: [FAQ - How to adjust settings with the `user-patches.sh` script][docs-faq-userpatches]) - **rspamd/custom-commands.conf:** list of simple commands to adjust Rspamd modules in an easy way (Docs: [Rspamd][docs-rspamd-commands]) -[docs-dms-config-volume]: ../../faq.md#what-about-the-docker-datadmsconfig-directory [docs-accounts-quota]: ../../config/user-management.md#quotas [docs-aliases-regex]: ../../config/user-management.md#configuring-regexp-aliases [docs-dkim]: ../../config/best-practices/dkim_dmarc_spf.md#dkim diff --git a/docs/content/config/best-practices/dkim_dmarc_spf.md b/docs/content/config/best-practices/dkim_dmarc_spf.md index ed56504c..290ac5cd 100644 --- a/docs/content/config/best-practices/dkim_dmarc_spf.md +++ b/docs/content/config/best-practices/dkim_dmarc_spf.md @@ -345,7 +345,7 @@ volumes: ``` [docs-accounts-add]: ../user-management.md#adding-a-new-account -[docs-volumes-config]: ../advanced/optional-config.md +[docs-volumes-config]: ../advanced/optional-config.md#volumes-config [docs-env-opendkim]: ../environment.md#enable_opendkim [docs-env-rspamd]: ../environment.md#enable_rspamd [docs-rspamd-config-dropin]: ../security/rspamd.md#manually diff --git a/docs/content/config/environment.md b/docs/content/config/environment.md index 1aa6799d..b1c3e2b6 100644 --- a/docs/content/config/environment.md +++ b/docs/content/config/environment.md @@ -45,11 +45,6 @@ Default: 5000 The Group ID assigned to the static vmail group for `/var/mail` (_Mail storage managed by Dovecot_). -##### ONE_DIR - -- 0 => state in default directories. -- **1** => consolidate all states into a single directory (`/var/mail-state`) to allow persistence using docker volumes. See the [related FAQ entry][docs-faq-onedir] for more information. - ##### ACCOUNT_PROVISIONER Configures the provisioning source of user accounts (including aliases) for user queries and authentication by services managed by DMS (_Postfix and Dovecot_). @@ -648,10 +643,10 @@ Controls the spam score threshold for triggering an action on mail that has a hi - [It will be quarantined][amavis-docs::quarantine] regardless of the `SA_KILL` action to perform. - With `D_PASS` the delivered mail also appends an `X-Quarantine-ID` mail header. The ID value of this header is part of the quarantined file name. - If emails are quarantined, they are compressed and stored at a location dependent on the [`ONE_DIR`](#one_dir) setting: + If emails are quarantined, they are compressed and stored at a location: - - `ONE_DIR=1` (default): `/var/mail-state/lib-amavis/virusmails/` - - `ONE_DIR=0`: `/var/lib/amavis/virusmails/` + - Default: `/var/lib/amavis/virusmails/` + - When the [`/var/mail-state/` volume][docs::dms-volumes-state] is present: `/var/mail-state/lib-amavis/virusmails/` !!! tip @@ -1056,9 +1051,9 @@ you to replace both instead of just the envelope sender. - password for default relay user [docs-rspamd]: ./security/rspamd.md -[docs-faq-onedir]: ../faq.md#what-about-docker-datadmsmail-state-folder-varmail-state-internally [docs-tls]: ./security/ssl.md [docs-tls-letsencrypt]: ./security/ssl.md#lets-encrypt-recommended [docs-tls-manual]: ./security/ssl.md#bring-your-own-certificates [docs-tls-selfsigned]: ./security/ssl.md#self-signed-certificates [docs-accounts-quota]: ./user-management.md#quotas +[docs::dms-volumes-state]: ./advanced/optional-config.md#volumes-state diff --git a/docs/content/config/security/fail2ban.md b/docs/content/config/security/fail2ban.md index 04e9a681..375f440c 100644 --- a/docs/content/config/security/fail2ban.md +++ b/docs/content/config/security/fail2ban.md @@ -33,7 +33,7 @@ DMS will automatically ban IP addresses of hosts that have generated 6 failed at ### Custom Files -!!! question "What is [`docker-data/dms/config/`][docs-dms-config-volume]?" +!!! question "What is [`docker-data/dms/config/`][docs::dms-volumes-config]?" This following configuration files inside the `docker-data/dms/config/` volume will be copied inside the container during startup @@ -44,7 +44,7 @@ This following configuration files inside the `docker-data/dms/config/` volume w - with this file, you can adjust F2B behavior in general - there is an example provided [in our repository on GitHub][github-file-f2bconfig] -[docs-dms-config-volume]: ../../faq.md#what-about-the-docker-datadmsconfig-directory +[docs::dms-volumes-config]: ../advanced/optional-config.md#volumes-config [github-file-f2bjail]: https://github.com/docker-mailserver/docker-mailserver/blob/master/config-examples/fail2ban-jail.cf [github-file-f2bconfig]: https://github.com/docker-mailserver/docker-mailserver/blob/master/config-examples/fail2ban-fail2ban.cf diff --git a/docs/content/config/security/rspamd.md b/docs/content/config/security/rspamd.md index 283c2a95..0c457ec3 100644 --- a/docs/content/config/security/rspamd.md +++ b/docs/content/config/security/rspamd.md @@ -83,9 +83,15 @@ DMS does not set a default password for the controller worker. You may want to d ### Persistence with Redis -When Rspamd is enabled, we implicitly also start an instance of Redis in the container. Redis is configured to persist its data via RDB snapshots to disk in the directory `/var/lib/redis` (_which is a symbolic link to `/var/mail-state/lib-redis/` when [`ONE_DIR=1`](../environment.md#one_dir) and a volume is mounted to `/var/mail-state/`_). With the volume mount the snapshot will restore the Redis data across container restarts, and provide a way to keep backup. +When Rspamd is enabled, we implicitly also start an instance of Redis in the container: -Redis uses `/etc/redis/redis.conf` for configuration. We adjust this file when enabling the internal Redis service. If you have an external instance of Redis to use, the internal Redis service can be opt-out via setting the ENV [`ENABLE_RSPAMD_REDIS=0`](../environment.md#enable_rspamd_redis) (_link also details required changes to the DMS Rspamd config_). +- Redis is configured to persist its data via RDB snapshots to disk in the directory `/var/lib/redis` (_or the [`/var/mail-state/`][docs::dms-volumes-state] volume when present_). +- With the volume mount the snapshot will restore the Redis data across container restarts, and provide a way to keep backup. + +Redis uses `/etc/redis/redis.conf` for configuration: + +- We adjust this file when enabling the internal Redis service. +- If you have an external instance of Redis to use, the internal Redis service can be opt-out via setting the ENV [`ENABLE_RSPAMD_REDIS=0`](../environment.md#enable_rspamd_redis) (_link also details required changes to the DMS Rspamd config_). ### Web Interface @@ -145,7 +151,7 @@ DMS brings sane default settings for Rspamd. They are located at `/etc/rspamd/lo ### Manually -!!! question "What is [`docker-data/dms/config/`][docs-dms-config-volume]?" +!!! question "What is [`docker-data/dms/config/`][docs::dms-volumes-config]?" If you want to overwrite the default settings and / or provide your own settings, you can place files at `docker-data/dms/config/rspamd/override.d/`. Files from this directory are copied to `/etc/rspamd/override.d/` during startup. These files [forcibly override][rspamd-docs-override-dir] Rspamd and DMS default settings. @@ -156,7 +162,6 @@ If you want to overwrite the default settings and / or provide your own settings Note that when also [using the `custom-commands.conf` file](#with-the-help-of-a-custom-file), files in `override.d` may be overwritten in case you adjust them manually and with the help of the file. [rspamd-docs-override-dir]: https://www.rspamd.com/doc/faq.html#what-are-the-locald-and-overrided-directories -[docs-dms-config-volume]: ../../faq.md#what-about-the-docker-datadmsconfig-directory [rspamd-docs-config-directories]: https://rspamd.com/doc/faq.html#what-are-the-locald-and-overrided-directories ### With the Help of a Custom File @@ -197,7 +202,7 @@ You can also have comments (the line starts with `#`) and blank lines in `custom You want to start using Rspamd? Rspamd is disabled by default, so you need to set the following environment variables: -```cf +```env ENABLE_RSPAMD=1 ENABLE_OPENDKIM=0 ENABLE_OPENDMARC=0 @@ -252,3 +257,6 @@ While _Abusix_ can be integrated into Postfix, Postscreen and a multitude of oth [Abusix]: https://abusix.com/ [abusix-rspamd-integration]: https://docs.abusix.com/abusix-mail-intelligence/gbG8EcJ3x3fSUv8cMZLiwA/getting-started/dmw9dcwSGSNQiLTssFAnBW#rspamd + +[docs::dms-volumes-config]: ../advanced/optional-config.md#volumes-config +[docs::dms-volumes-state]: ../advanced/optional-config.md#volumes-state diff --git a/docs/content/config/security/ssl.md b/docs/content/config/security/ssl.md index c0c615cc..10f7adc4 100644 --- a/docs/content/config/security/ssl.md +++ b/docs/content/config/security/ssl.md @@ -634,7 +634,7 @@ This setup only comes with one caveat: The domain has to be configured on anothe Use self-signed certificates only for testing purposes! -This feature requires you to provide the following files into your [`docker-data/dms/config/ssl/` directory][docs-optional-config] (_internal location: `/tmp/docker-mailserver/ssl/`_): +This feature requires you to provide the following files into your [`docker-data/dms/config/ssl/` directory][docs::dms-volumes-config] (_internal location: `/tmp/docker-mailserver/ssl/`_): - `-key.pem` - `-cert.pem` @@ -876,7 +876,7 @@ By default DMS uses [`ffdhe4096`][ffdhe4096-src] from [IETF RFC 7919][ietf::rfc: Despite this, if you must use non-standard DH parameters or you would like to swap `ffdhe4096` for a different group (eg `ffdhe2048`); Add your own PEM encoded DH params file via a volume to `/tmp/docker-mailserver/dhparams.pem`. This will replace DH params for both Dovecot and Postfix services during container startup. [docs-env::ssl-type]: ../environment.md#ssl_type -[docs-optional-config]: ../advanced/optional-config.md +[docs::dms-volumes-config]: ../advanced/optional-config.md#volumes-config [docs-faq-baredomain]: ../../faq.md#can-i-use-a-nakedbare-domain-ie-no-hostname [github-file-compose]: https://github.com/docker-mailserver/docker-mailserver/blob/master/compose.yaml diff --git a/docs/content/examples/tutorials/mailserver-behind-proxy.md b/docs/content/examples/tutorials/mailserver-behind-proxy.md index 7bdef5d6..99939542 100644 --- a/docs/content/examples/tutorials/mailserver-behind-proxy.md +++ b/docs/content/examples/tutorials/mailserver-behind-proxy.md @@ -125,5 +125,3 @@ service imap-login { !!! note Port `10993` is used here to avoid conflicts with internal systems like `postscreen` and `amavis` as they will exchange messages on the default port and obviously have a different origin then compared to the proxy. - -[docs-optionalconfig]: ../../config/advanced/optional-config.md diff --git a/docs/content/faq.md b/docs/content/faq.md index 0985c0a9..6b1782e3 100644 --- a/docs/content/faq.md +++ b/docs/content/faq.md @@ -360,20 +360,6 @@ DMS does not manage those concerns, verify they are not causing your delivery pr - [mail-tester](https://www.mail-tester.com/) can test your deliverability. - [helloinbox](https://www.helloinbox.email/) provides a checklist of things to improve your deliverability. -### Special Directories - -#### What About the `docker-data/dms/config/` Directory? - -This documentation and all example configuration files in the GitHub repository use `docker-data/dms/config/` to refer to the directory in the host that is mounted (e.g. via a bind mount) to `/tmp/docker-mailserver/` inside the container. - -Most configuration files for Postfix, Dovecot, etc. are persisted here. [Optional configuration][docs-optional-configuration] is stored here as well. - -#### What About the `docker-data/dms/mail-state/` Directory? - -This documentation and all example configuration files in the GitHub repository use `docker-data/dms/mail-state/` to refer to the directory in the host that is mounted (e.g. via a bind mount) to `/var/mail-state/` inside the container. - -When you run DMS with the ENV variable `ONE_DIR=1` (default), this directory will provide support to persist Fail2Ban blocks, ClamAV signature updates, and the like when the container is restarted or recreated. Service data is [relocated to the `mail-state` folder][mail-state-folders] for the following services: Postfix, Dovecot, Fail2Ban, Amavis, PostGrey, ClamAV, SpamAssassin, Rspamd & Redis. - ### SpamAssasin #### How can I manage my custom SpamAssassin rules? @@ -390,14 +376,15 @@ The default setup `@local_domains_acl = ( ".$mydomain" );` does not match subdom Put received spams in `.Junk/` imap folder using `SPAMASSASSIN_SPAM_TO_INBOX=1` and `MOVE_SPAM_TO_JUNK=1` and add a _user_ cron like the following: -```conf -# This assumes you're having `environment: ONE_DIR=1` in the `mailserver.env`, -# with a consolidated config in `/var/mail-state` -# -# m h dom mon dow command -# Everyday 2:00AM, learn spam from a specific user -0 2 * * * docker exec mailserver sa-learn --spam /var/mail/example.com/username/.Junk --dbpath /var/mail-state/lib-amavis/.spamassassin -``` +!!! example + + **NOTE:** This example assumes you have a [`/var/mail-state` volume][docs::dms-volumes-state] mounted. + + ```conf + # m h dom mon dow command + # Everyday 2:00AM, learn spam from a specific user + 0 2 * * * docker exec mailserver sa-learn --spam /var/mail/example.com/username/.Junk --dbpath /var/mail-state/lib-amavis/.spamassassin + ``` With `docker-compose` you can more easily use the internal instance of `cron` within DMS. This is less problematic than the simple solution shown above, because it decouples the learning from the host on which DMS is running, and avoids errors if the mail server is not running. @@ -405,6 +392,8 @@ The following configuration works nicely: ??? example + **NOTE:** This example assumes you have a [`/var/mail-state` volume][docs::dms-volumes-state] mounted. + Create a _system_ cron file: ```sh @@ -418,9 +407,6 @@ The following configuration works nicely: Edit the system cron file `nano ./docker-data/dms/cron/sa-learn`, and set an appropriate configuration: ```conf - # This assumes you're having `environment: ONE_DIR=1` in the env-mailserver, - # with a consolidated config in `/var/mail-state` - # # '> /dev/null' to send error notifications from 'stderr' to 'postmaster@example.com' # # m h dom mon dow user command @@ -495,10 +481,10 @@ $spam_quarantine_to = "quarantine\@example.com"; ``` [fail2ban-customize]: ./config/security/fail2ban.md +[docs::dms-volumes-state]: ./config/advanced/optional-config.md#volumes-state [docs-maintenance]: ./config/advanced/maintenance/update-and-cleanup.md [docs-override-postfix]: ./config/advanced/override-defaults/postfix.md [docs-userpatches]: ./config/advanced/override-defaults/user-patches.md -[docs-optional-configuration]: ./config/advanced/optional-config.md [docs::env::sa_env]: ./config/environment.md#spamassassin [docs::env::sa_kill]: ./config/environment.md#sa_kill [github-comment-baredomain]: https://github.com/docker-mailserver/docker-mailserver/issues/3048#issuecomment-1432358353 @@ -510,4 +496,3 @@ $spam_quarantine_to = "quarantine\@example.com"; [github-issue-1639]: https://github.com/docker-mailserver/docker-mailserver/issues/1639 [github-issue-1792]: https://github.com/docker-mailserver/docker-mailserver/pull/1792 [hanscees-userpatches]: https://github.com/hanscees/dockerscripts/blob/master/scripts/tomav-user-patches.sh -[mail-state-folders]: https://github.com/docker-mailserver/docker-mailserver/blob/c7e498194546416fb7231cb03254e77e085d18df/target/scripts/startup/misc-stack.sh#L24-L33 diff --git a/docs/content/usage.md b/docs/content/usage.md index 0550c24e..30f6e822 100644 --- a/docs/content/usage.md +++ b/docs/content/usage.md @@ -2,9 +2,9 @@ title: Usage --- -This pages explains how to get started with DMS. The guide uses Docker Compose as a reference. In our examples, a volume mounts the host location [`docker-data/dms/config/`][docs-dms-config-volume] to `/tmp/docker-mailserver/` inside the container. +This pages explains how to get started with DMS. The guide uses Docker Compose as a reference. In our examples, a volume mounts the host location [`docker-data/dms/config/`][docs::dms-volumes-config] to `/tmp/docker-mailserver/` inside the container. -[docs-dms-config-volume]: ./faq.md#what-about-the-docker-datadmsconfig-directory +[docs::dms-volumes-config]: ./config/advanced/optional-config.md#volumes-config ## Preliminary Steps diff --git a/mailserver.env b/mailserver.env index 1d131696..c171faed 100644 --- a/mailserver.env +++ b/mailserver.env @@ -30,10 +30,6 @@ LOG_LEVEL=info # debug => Also show debug messages SUPERVISOR_LOGLEVEL= -# 0 => mail state in default directories -# 1 => consolidate all states into a single directory (`/var/mail-state`) to allow persistence using docker volumes -ONE_DIR=1 - # Support for deployment where these defaults are not compatible (eg: some NAS appliances): # /var/mail vmail User ID (default: 5000) DMS_VMAIL_UID= diff --git a/target/scripts/startup/setup.d/mail_state.sh b/target/scripts/startup/setup.d/mail_state.sh index 9963bbcc..9c43fea4 100644 --- a/target/scripts/startup/setup.d/mail_state.sh +++ b/target/scripts/startup/setup.d/mail_state.sh @@ -7,7 +7,7 @@ function _setup_save_states() { STATEDIR='/var/mail-state' - if [[ ${ONE_DIR} -eq 1 ]] && [[ -d ${STATEDIR} ]]; then + if [[ -d ${STATEDIR} ]]; then _log 'debug' "Consolidating all state onto ${STATEDIR}" # Always enabled features: @@ -111,9 +111,7 @@ function _setup_save_states() { # Ref: https://github.com/docker-mailserver/docker-mailserver/pull/3625 chmod 730 "${STATEDIR}/spool-postfix/maildrop" chmod 710 "${STATEDIR}/spool-postfix/public" - elif [[ ${ONE_DIR} -eq 1 ]]; then - _log 'warn' "'ONE_DIR=1' but no volume was mounted to '${STATEDIR}'" else - _log 'debug' 'Not consolidating state (because it has been disabled)' + _log 'debug' "'${STATEDIR}' is not present; Not consolidating state" fi } diff --git a/target/scripts/startup/setup.d/security/rspamd.sh b/target/scripts/startup/setup.d/security/rspamd.sh index 86786932..a05a798e 100644 --- a/target/scripts/startup/setup.d/security/rspamd.sh +++ b/target/scripts/startup/setup.d/security/rspamd.sh @@ -127,9 +127,9 @@ expand_keys = true; EOF - # Here we adjust the Redis default configuration that we supply to Redis - # when starting it. Note that `/var/lib/redis/` is linked to - # `/var/mail-state/redis/` (for persisting it) if `ONE_DIR=1`. + # Here we adjust the Redis default configuration that we supply to Redis when starting it. + # NOTE: `/var/lib/redis/` is symlinked to `/var/mail-state/redis/` when DMS is started + # with a volume mounted to `/var/mail-state/` for data persistence. sedfile -i -E \ -e 's|^(bind).*|\1 127.0.0.1|g' \ -e 's|^(daemonize).*|\1 no|g' \ diff --git a/target/scripts/startup/variables-stack.sh b/target/scripts/startup/variables-stack.sh index 0b351a9e..d14aeb3f 100644 --- a/target/scripts/startup/variables-stack.sh +++ b/target/scripts/startup/variables-stack.sh @@ -141,7 +141,6 @@ function __environment_variables_general_setup() { VARS[LOGWATCH_INTERVAL]="${LOGWATCH_INTERVAL:=none}" VARS[LOGWATCH_RECIPIENT]="${LOGWATCH_RECIPIENT:=${REPORT_RECIPIENT}}" VARS[LOGWATCH_SENDER]="${LOGWATCH_SENDER:=${REPORT_SENDER}}" - VARS[ONE_DIR]="${ONE_DIR:=1}" VARS[PERMIT_DOCKER]="${PERMIT_DOCKER:=none}" VARS[PFLOGSUMM_RECIPIENT]="${PFLOGSUMM_RECIPIENT:=${REPORT_RECIPIENT}}" VARS[PFLOGSUMM_SENDER]="${PFLOGSUMM_SENDER:=${REPORT_SENDER}}" From afb00939391e409a7bd9759d7346256198262144 Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Mon, 29 Jan 2024 13:38:01 +0100 Subject: [PATCH 016/267] spam: use Sieve for rewriting subject with Rspamd & SA/Amavis (#3820) --- CHANGELOG.md | 10 ++ docs/content/config/environment.md | 35 +++--- docs/content/config/security/rspamd.md | 5 +- mailserver.env | 13 ++- target/rspamd/local.d/actions.conf | 8 +- target/scripts/start-mailserver.sh | 2 + target/scripts/startup/check-stack.sh | 8 ++ .../scripts/startup/setup.d/security/misc.sh | 61 +++++++++-- target/scripts/startup/variables-stack.sh | 8 +- test/helper/common.bash | 16 +++ .../parallel/set1/spam_virus/amavis.bats | 101 ++++++++++++------ .../parallel/set1/spam_virus/rspamd_dkim.bats | 9 +- .../parallel/set1/spam_virus/rspamd_full.bats | 89 +++++++++------ .../set1/spam_virus/rspamd_partly.bats | 9 +- .../set1/spam_virus/spam_junk_folder.bats | 6 +- .../set1/spam_virus/undef_spam_subject.bats | 61 ----------- 16 files changed, 262 insertions(+), 179 deletions(-) delete mode 100644 test/tests/parallel/set1/spam_virus/undef_spam_subject.bats diff --git a/CHANGELOG.md b/CHANGELOG.md index 41e6d5e9..d16c03ec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,14 @@ The most noteworthy change of this release is the update of the container's base - DMS `main.cf` has renamed `postscreen_dnsbl_whitelist_threshold` to `postscreen_dnsbl_allowlist_threshold` as part of this change. - `smtpd_relay_restrictions` (relay policy) is now evaluated after `smtpd_recipient_restrictions` (spam policy). Previously it was evaluated before `smtpd_recipient_restrictions`. Mail to be relayed via DMS must now pass through the spam policy first. - The TLS fingerprint policy has changed the default from MD5 to SHA256 (_DMS does not modify this Postfix parameter, but may affect any user customizations that do_). +- **Environment Variables**: + - `SA_SPAM_SUBJECT` has been renamed into `SPAM_SUBJECT` to become anti-spam service agnostic. ([3820](https://github.com/docker-mailserver/docker-mailserver/pull/3820)) + - As this functionality is now handled in Dovecot via a Sieve script instead of the respective anti-spam service during Postfix processing, this feature will only apply to mail stored in Dovecot. If you have relied on this feature in a different context, it will no longer be available. + - Rspamd previously handled this functionality via the `rewrite_subject` action which as now been disabled by default in favor of the new approach with `SPAM_SUBJECT`. + - `SA_SPAM_SUBJECT` is now deprecated and will log a warning if used. The value is copied as a fallback to `SPAM_SUBJECT`. + - The default has changed to not prepend any prefix to the subject unless configured to do so. If you relied on the implicit prefix, you will now need to provide one explicitly. + - `undef` was previously supported as an opt-out with `SA_SPAM_SUBJECT`. This is no longer valid, the equivalent opt-out value is now an empty value (_or rather the omission of this ENV being configured_). + - The feature to include [`_SCORE_` tag](https://spamassassin.apache.org/full/4.0.x/doc/Mail_SpamAssassin_Conf.html#rewrite_header-subject-from-to-STRING) in your value to be replaced by the associated spam score is no longer available. ### Updates @@ -45,6 +53,8 @@ The most noteworthy change of this release is the update of the container's base - It's only functionality remaining was to opt-out of run-time state consolidation with `ONE_DIR=0` (_when a volume was already mounted to `/var/mail-state`_). - **Tests:** - Refactored helper methods for sending e-mails with specific `Message-ID` headers and the helpers for retrieving + filtering logs, which together help isolate logs relevant to specific mail when multiple mails have been processed within a single test. ([#3786](https://github.com/docker-mailserver/docker-mailserver/pull/3786)) +- **Rspamd**: + - The `rewrite_subject` action, is now disabled by default. It has been replaced with the new `SPAM_SUBJECT` environment variable, which implements the functionality via a Sieve script instead in favor of being anti-spam service agnostic ([3820](https://github.com/docker-mailserver/docker-mailserver/pull/3820)) ### Fixes diff --git a/docs/content/config/environment.md b/docs/content/config/environment.md index b1c3e2b6..c98016d1 100644 --- a/docs/content/config/environment.md +++ b/docs/content/config/environment.md @@ -51,7 +51,7 @@ Configures the provisioning source of user accounts (including aliases) for user !!! tip "OAuth2 Support" - Presently DMS supports OAuth2 only as an supplementary authentication method. + Presently DMS supports OAuth2 only as an supplementary authentication method. - A third-party service must provide a valid token for the user which Dovecot validates with the authentication service provider. To enable this feature reference the [OAuth2 configuration example guide][docs::auth::oauth2-config-guide]. - User accounts must be provisioned to receive mail via one of the supported `ACCOUNT_PROVISIONER` providers. @@ -354,6 +354,16 @@ Enable to treat received spam as "read" (_avoids notification to MUA client of n - `X-Spam: Yes` (_added by Rspamd_) - `X-Spam-Flag: YES` (_added by SpamAssassin - requires [`SPAMASSASSIN_SPAM_TO_INBOX=1`](#spamassassin_spam_to_inbox)_) +##### SPAM_SUBJECT + +This variable defines a prefix for e-mails tagged with the `X-Spam: Yes` (Rspamd) or `X-Spam-Flag: YES` (SpamAssassin/Amavis) header. + +Default: empty (no prefix will be added to e-mails) + +??? example "Including trailing white-space" + + Add trailing white-space by quote wrapping the value: `SPAM_SUBJECT='[SPAM] '` + #### Rspamd ##### ENABLE_RSPAMD @@ -562,7 +572,7 @@ Changes the interval in which log files are rotated. - [`MOVE_SPAM_TO_JUNK=1`](#move_spam_to_junk) - [`MARK_SPAM_AS_READ=1`](#mark_spam_as_read) - - [`SA_SPAM_SUBJECT`](#sa_spam_subject) + - [`SPAM_SUBJECT`](#spam_subject) ##### SA_TAG @@ -602,8 +612,8 @@ When a spam score is high enough, mark mail as spam (_Appends the mail header: ` !!! info "Interaction with other ENV" - - [`SA_SPAM_SUBJECT`](#sa_spam_subject) modifies the mail subject to better communicate spam mail to the user. - - [`MOVE_SPAM_TO_JUNK=1`](#move_spam_to_junk): The mail is still delivered, but to the recipient(s) junk folder instead. This feature reduces the usefulness of `SA_SPAM_SUBJECT`. + - [`SPAM_SUBJECT`](#spam_subject) modifies the mail subject to better communicate spam mail to the user. + - [`MOVE_SPAM_TO_JUNK=1`](#move_spam_to_junk): The mail is still delivered, but to the recipient(s) junk folder instead. This feature reduces the usefulness of `SPAM_SUBJECT`. ##### SA_KILL @@ -659,23 +669,6 @@ Controls the spam score threshold for triggering an action on mail that has a hi [amavis-docs::actions]: https://www.ijs.si/software/amavisd/amavisd-new-docs.html#actions [amavis-docs::quarantine]: https://www.ijs.si/software/amavisd/amavisd-new-docs.html#quarantine -##### SA_SPAM_SUBJECT - -Adds a prefix to the subject header when mail is marked as spam (_via [`SA_TAG2`](#sa_tag2)_). - -- **`'***SPAM*** '`** => A string value to use as a mail subject prefix. -- `undef` => Opt-out of modifying the subject for mail marked as spam. - -??? example "Including trailing white-space" - - Add trailing white-space by quote wrapping the value: `SA_SPAM_SUBJECT='[SPAM] '` - -??? example "Including the associated spam score" - - The [`_SCORE_` tag][sa-docs::score-tag] will be substituted with the SpamAssassin score: `SA_SPAM_SUBJECT=***SPAM(_SCORE_)***`. - -[sa-docs::score-tag]: https://spamassassin.apache.org/full/4.0.x/doc/Mail_SpamAssassin_Conf.html#rewrite_header-subject-from-to-STRING - ##### SA_SHORTCIRCUIT_BAYES_SPAM - **1** => will activate SpamAssassin short circuiting for bayes spam detection. diff --git a/docs/content/config/security/rspamd.md b/docs/content/config/security/rspamd.md index 0c457ec3..4c466c5d 100644 --- a/docs/content/config/security/rspamd.md +++ b/docs/content/config/security/rspamd.md @@ -22,8 +22,9 @@ The following environment variables are related to Rspamd: 5. [`RSPAMD_HFILTER`](../environment.md#rspamd_hfilter) 6. [`RSPAMD_HFILTER_HOSTNAME_UNKNOWN_SCORE`](../environment.md#rspamd_hfilter_hostname_unknown_score) 7. [`RSPAMD_LEARN`](../environment.md#rspamd_learn) -8. [`MOVE_SPAM_TO_JUNK`][docs-spam-to-junk] -9. [`MARK_SPAM_AS_READ`](../environment.md#mark_spam_as_read) +8. [`SPAM_SUBJECT`](../environment.md#spam_subject) +9. [`MOVE_SPAM_TO_JUNK`][docs-spam-to-junk] +10. [`MARK_SPAM_AS_READ`](../environment.md#mark_spam_as_read) With these variables, you can enable Rspamd itself, and you can enable / disable certain features related to Rspamd. diff --git a/mailserver.env b/mailserver.env index c171faed..38b9adca 100644 --- a/mailserver.env +++ b/mailserver.env @@ -130,6 +130,11 @@ ENABLE_IMAP=1 # **0** => Disabled ENABLE_CLAMAV=0 +# Add the value as a prefix to the mail subject when spam is detected. +# NOTE: By default spam is delivered to a junk folder, reducing the value of a subject prefix for spam. +# NOTE: When not using Docker Compose, other CRI may not support quote-wrapping the value here to preserve any trailing white-space. +SPAM_SUBJECT= + # Enables Rspamd # **0** => Disabled # 1 => Enabled @@ -378,7 +383,7 @@ ENABLE_SPAMASSASSIN=0 # Note: only has an effect if `ENABLE_SPAMASSASSIN=1` ENABLE_SPAMASSASSIN_KAM=0 -# deliver spam messages to the inbox (tagged using SA_SPAM_SUBJECT) +# deliver spam messages to the inbox (tagged using SPAM_SUBJECT) SPAMASSASSIN_SPAM_TO_INBOX=1 # spam messages will be moved in the Junk folder (SPAMASSASSIN_SPAM_TO_INBOX=1 required) @@ -396,12 +401,6 @@ SA_TAG2=6.31 # triggers spam evasive actions SA_KILL=10.0 -# add tag to subject if spam detected -# The value `undef` opts-out of this feature. The value shown below is the default. -# NOTE: By default spam is delivered to a junk folder, reducing the value of adding a subject prefix. -# NOTE: If not using Docker Compose, other CRI may require the single quotes removed. -#SA_SPAM_SUBJECT='***SPAM*** ' - # ----------------------------------------------- # --- Fetchmail Section ------------------------- # ----------------------------------------------- diff --git a/target/rspamd/local.d/actions.conf b/target/rspamd/local.d/actions.conf index fb4c15b9..6ecb6d4b 100644 --- a/target/rspamd/local.d/actions.conf +++ b/target/rspamd/local.d/actions.conf @@ -6,7 +6,11 @@ # and to be able to explain the impact on the whole system. greylist = 4; add_header = 6; -rewrite_subject = 7; reject = 11; -subject = "***SPAM*** %s" +# The value `null` disabled the action. A subject rewrite is handled by `SPAM_SUBJECT`: +# https://docker-mailserver.github.io/docker-mailserver/latest/config/environment/#spam_subject +# +# The reasoning for this can be found in +# https://github.com/docker-mailserver/docker-mailserver/issues/3804 +rewrite_subject = null; diff --git a/target/scripts/start-mailserver.sh b/target/scripts/start-mailserver.sh index 56dfa1fb..f18abdf6 100755 --- a/target/scripts/start-mailserver.sh +++ b/target/scripts/start-mailserver.sh @@ -36,6 +36,7 @@ function _register_functions() { _register_check_function '_check_improper_restart' _register_check_function '_check_hostname' _register_check_function '_check_log_level' + _register_check_function '_check_spam_prefix' # ? >> Setup @@ -48,6 +49,7 @@ function _register_functions() { _register_setup_function '_setup_dovecot_sieve' _register_setup_function '_setup_dovecot_dhparam' _register_setup_function '_setup_dovecot_quota' + _register_setup_function '_setup_spam_subject' _register_setup_function '_setup_spam_to_junk' _register_setup_function '_setup_spam_mark_as_read' fi diff --git a/target/scripts/startup/check-stack.sh b/target/scripts/startup/check-stack.sh index cf1c40f5..cc98cfe2 100644 --- a/target/scripts/startup/check-stack.sh +++ b/target/scripts/startup/check-stack.sh @@ -52,3 +52,11 @@ function _check_log_level() { LOG_LEVEL="${DEFAULT_LOG_LEVEL}" fi } + +function _check_spam_prefix() { + # This check should be independent of ENABLE_POP3 and ENABLE_IMAP + if [[ ${MOVE_SPAM_TO_JUNK} -eq 0 ]] \ + && [[ -z ${SPAM_SUBJECT} ]]; then + _log 'warn' "'MOVE_SPAM_TO_JUNK=0' and 'SPAM_SUBJECT' is empty - make sure this is intended: spam e-mails might not be immediately recognizable in this configuration" + fi +} diff --git a/target/scripts/startup/setup.d/security/misc.sh b/target/scripts/startup/setup.d/security/misc.sh index ec32a1e6..aefeba20 100644 --- a/target/scripts/startup/setup.d/security/misc.sh +++ b/target/scripts/startup/setup.d/security/misc.sh @@ -81,13 +81,9 @@ function __setup__security__spamassassin() { # shellcheck disable=SC2016 sed -i -r 's|^\$sa_kill_level_deflt (.*);|\$sa_kill_level_deflt = '"${SA_KILL}"';|g' /etc/amavis/conf.d/20-debian_defaults - if [[ ${SA_SPAM_SUBJECT} == 'undef' ]]; then - # shellcheck disable=SC2016 - sed -i -r 's|^\$sa_spam_subject_tag (.*);|\$sa_spam_subject_tag = undef;|g' /etc/amavis/conf.d/20-debian_defaults - else - # shellcheck disable=SC2016 - sed -i -r 's|^\$sa_spam_subject_tag (.*);|\$sa_spam_subject_tag = '"'${SA_SPAM_SUBJECT}'"';|g' /etc/amavis/conf.d/20-debian_defaults - fi + # disable rewriting the subject as this is handles by _setup_spam_subject (which uses Dovecot Sieve) + # shellcheck disable=SC2016 + sed -i -r 's|^\$sa_spam_subject_tag (.*);|\$sa_spam_subject_tag = undef;|g' /etc/amavis/conf.d/20-debian_defaults # activate short circuits when SA BAYES is certain it has spam or ham. if [[ ${SA_SHORTCIRCUIT_BAYES_SPAM} -eq 1 ]]; then @@ -245,6 +241,57 @@ function __setup__security__amavis() { fi } +# If `SPAM_SUBJECT` is not empty, we create a Sieve script that alters the `Subject` +# header, in order to prepend a user-defined string. +function _setup_spam_subject() { + if [[ -z ${SPAM_SUBJECT} ]] + then + _log 'debug' 'Spam subject is not set - no prefix will be added to spam e-mails' + else + _log 'debug' "Spam subject is set - the prefix '${SPAM_SUBJECT}' will be added to spam e-mails" + + _log 'trace' "Enabling '+editheader' Sieve extension" + # check whether sieve_global_extensions is disabled (and enabled it if so) + sed -i -E 's|#(sieve_global_extensions.*)|\1|' /etc/dovecot/conf.d/90-sieve.conf + # then append the extension + sedfile -i -E 's|(sieve_global_extensions.*)|\1 +editheader|' /etc/dovecot/conf.d/90-sieve.conf + + _log 'trace' "Adding global (before) Sieve script for subject rewrite" + # This directory contains Sieve scripts that are executed before user-defined Sieve + # scripts run. + local DOVECOT_SIEVE_GLOBAL_BEFORE_DIR='/usr/lib/dovecot/sieve-global/before' + local DOVECOT_SIEVE_FILE='spam_subject' + readonly DOVECOT_SIEVE_GLOBAL_BEFORE_DIR DOVECOT_SIEVE_FILE + + mkdir -p "${DOVECOT_SIEVE_GLOBAL_BEFORE_DIR}" + # ref: https://superuser.com/a/1502589 + cat >"${DOVECOT_SIEVE_GLOBAL_BEFORE_DIR}/${DOVECOT_SIEVE_FILE}.sieve" << EOF +require ["editheader","variables"]; + +if anyof (header :contains "X-Spam-Flag" "YES", + header :contains "X-Spam" "Yes") +{ + # Match the entire subject ... + if header :matches "Subject" "*" { + # ... to get it in a match group that can then be stored in a variable: + set "subject" "\${1}"; + } + + # We can't "replace" a header, but we can delete (all instances of) it and + # re-add (a single instance of) it: + deleteheader "Subject"; + + # Note that the header is added ":last" (so it won't appear before possible + # "Received" headers). + addheader :last "Subject" "${SPAM_SUBJECT}\${subject}"; +} +EOF + + sievec "${DOVECOT_SIEVE_GLOBAL_BEFORE_DIR}/${DOVECOT_SIEVE_FILE}.sieve" + chown dovecot:root "${DOVECOT_SIEVE_GLOBAL_BEFORE_DIR}/${DOVECOT_SIEVE_FILE}."{sieve,svbin} + fi +} + # We can use Sieve to move spam emails to the "Junk" folder. function _setup_spam_to_junk() { if [[ ${MOVE_SPAM_TO_JUNK} -eq 1 ]]; then diff --git a/target/scripts/startup/variables-stack.sh b/target/scripts/startup/variables-stack.sh index d14aeb3f..219ce6c8 100644 --- a/target/scripts/startup/variables-stack.sh +++ b/target/scripts/startup/variables-stack.sh @@ -25,6 +25,12 @@ function __environment_variables_backwards_compatibility() { _log 'error' "The ENV for which LDAP host to connect to must include the URI scheme ('ldap://', 'ldaps://', 'ldapi://')" fi + if [[ -n ${SA_SPAM_SUBJECT:-} ]]; then + _log 'warn' "'SA_SPAM_SUBJECT' has been renamed to 'SPAM_SUBJECT' - this warning will block startup on v15.0.0" + _log 'info' "Copying value of 'SA_SPAM_SUBJECT' into 'SPAM_SUBJECT' if 'SPAM_SUBJECT' has not been set explicitly" + SPAM_SUBJECT=${SPAM_SUBJECT:-${SA_SPAM_SUBJECT}} + fi + # TODO this can be uncommented in a PR handling the HOSTNAME/DOMAINNAME issue # TODO see check_for_changes.sh and dns.sh # if [[ -n ${OVERRIDE_HOSTNAME:-} ]] @@ -67,7 +73,7 @@ function __environment_variables_general_setup() { VARS[RSPAMD_HFILTER_HOSTNAME_UNKNOWN_SCORE]="${RSPAMD_HFILTER_HOSTNAME_UNKNOWN_SCORE:=6}" VARS[RSPAMD_LEARN]="${RSPAMD_LEARN:=0}" VARS[SA_KILL]=${SA_KILL:="10.0"} - VARS[SA_SPAM_SUBJECT]=${SA_SPAM_SUBJECT:="***SPAM*** "} + VARS[SPAM_SUBJECT]=${SPAM_SUBJECT:=} VARS[SA_TAG]=${SA_TAG:="2.0"} VARS[SA_TAG2]=${SA_TAG2:="6.31"} VARS[SPAMASSASSIN_SPAM_TO_INBOX]="${SPAMASSASSIN_SPAM_TO_INBOX:=1}" diff --git a/test/helper/common.bash b/test/helper/common.bash index 3c54862c..35f41283 100644 --- a/test/helper/common.bash +++ b/test/helper/common.bash @@ -417,5 +417,21 @@ function _nc_wrapper() { _run_in_container_bash "nc ${NC_PARAMETERS} < /tmp/docker-mailserver-test/${FILE}" } +# A simple wrapper for a test that checks whether a file exists. +# +# @param ${1} = the path to the file inside the container +function _file_exists_in_container() { + _run_in_container_bash "[[ -f ${1} ]]" + assert_success +} + +# A simple wrapper for a test that checks whether a file does not exist. +# +# @param ${1} = the path to the file (that should not exists) inside the container +function _file_does_not_exist_in_container() { + _run_in_container_bash "[[ -f ${1} ]]" + assert_failure +} + # ? << Miscellaneous helper functions # ! ------------------------------------------------------------------- diff --git a/test/tests/parallel/set1/spam_virus/amavis.bats b/test/tests/parallel/set1/spam_virus/amavis.bats index d7a59cb2..73c5b2ae 100644 --- a/test/tests/parallel/set1/spam_virus/amavis.bats +++ b/test/tests/parallel/set1/spam_virus/amavis.bats @@ -2,8 +2,9 @@ load "${REPOSITORY_ROOT}/test/helper/common" load "${REPOSITORY_ROOT}/test/helper/setup" BATS_TEST_NAME_PREFIX='[Amavis + SA] ' -CONTAINER1_NAME='dms-test_amavis_enabled' -CONTAINER2_NAME='dms-test_amavis_disabled' +CONTAINER1_NAME='dms-test_amavis-enabled-default' +CONTAINER2_NAME='dms-test_amavis-enabled-custom' +CONTAINER3_NAME='dms-test_amavis-disabled' function setup_file() { export CONTAINER_NAME @@ -12,13 +13,25 @@ function setup_file() { _init_with_defaults local CUSTOM_SETUP_ARGUMENTS=( --env ENABLE_AMAVIS=1 - --env AMAVIS_LOGLEVEL=2 --env ENABLE_SPAMASSASSIN=1 ) _common_container_setup 'CUSTOM_SETUP_ARGUMENTS' CONTAINER_NAME=${CONTAINER2_NAME} _init_with_defaults + local CUSTOM_SETUP_ARGUMENTS=( + --env ENABLE_AMAVIS=1 + --env AMAVIS_LOGLEVEL=2 + --env ENABLE_SPAMASSASSIN=1 + --env SA_TAG=-5.0 + --env SA_TAG2=2.0 + --env SA_KILL=3.0 + --env SPAM_SUBJECT='***SPAM*** ' + ) + _common_container_setup 'CUSTOM_SETUP_ARGUMENTS' + + CONTAINER_NAME=${CONTAINER3_NAME} + _init_with_defaults local CUSTOM_SETUP_ARGUMENTS=( --env ENABLE_AMAVIS=0 --env ENABLE_SPAMASSASSIN=0 @@ -27,11 +40,37 @@ function setup_file() { } function teardown_file() { - docker rm -f "${CONTAINER1_NAME}" "${CONTAINER2_NAME}" + docker rm -f "${CONTAINER1_NAME}" "${CONTAINER2_NAME}" "${CONTAINER3_NAME}" } -@test '(Amavis enabled) configuration should be correct' { +@test '(Amavis enabled - defaults) default Amavis config is correct' { export CONTAINER_NAME=${CONTAINER1_NAME} + local AMAVIS_DEFAULTS_FILE='/etc/amavis/conf.d/20-debian_defaults' + + _run_in_container grep 'sa_tag_level_deflt' "${AMAVIS_DEFAULTS_FILE}" + assert_success + assert_output --partial 'sa_tag_level_deflt = 2.0;' + + _run_in_container grep 'sa_tag2_level_deflt' "${AMAVIS_DEFAULTS_FILE}" + assert_success + # shellcheck disable=SC2016 + assert_output --partial '$sa_tag2_level_deflt = 6.31;' + + _run_in_container grep 'sa_kill_level_deflt' "${AMAVIS_DEFAULTS_FILE}" + assert_success + # shellcheck disable=SC2016 + assert_output --partial '$sa_kill_level_deflt = 10.0;' + + # This feature is handled by our SPAM_SUBJECT ENV through a sieve script instead. + # Thus the feature here should always be disabled via the 'undef' value. + _run_in_container grep 'sa_spam_subject_tag' "${AMAVIS_DEFAULTS_FILE}" + assert_success + # shellcheck disable=SC2016 + assert_output --partial '$sa_spam_subject_tag = undef;' +} + +@test '(Amavis enabled - custom) configuration should be correct' { + export CONTAINER_NAME=${CONTAINER2_NAME} _run_in_container postconf -h content_filter assert_success @@ -41,47 +80,47 @@ function teardown_file() { _run_in_container grep -F '127.0.0.1:10025' /etc/postfix/master.cf assert_success - _run_in_container_bash '[[ -f /etc/cron.d/amavisd-new.disabled ]]' - assert_failure - _run_in_container_bash '[[ -f /etc/cron.d/amavisd-new ]]' - assert_success + _file_does_not_exist_in_container /etc/cron.d/amavisd-new.disabled + _file_exists_in_container /etc/cron.d/amavisd-new } -@test '(Amavis enabled) SA integration should be active' { - export CONTAINER_NAME=${CONTAINER1_NAME} +@test '(Amavis enabled - custom) SA integration should be active' { + export CONTAINER_NAME=${CONTAINER2_NAME} # give Amavis just a bit of time to print out its full debug log run _repeat_in_container_until_success_or_timeout 20 "${CONTAINER_NAME}" grep 'SpamControl: init_pre_fork on SpamAssassin done' /var/log/mail/mail.log assert_success } -@test '(Amavis enabled) SA ENV should update Amavis config' { - export CONTAINER_NAME=${CONTAINER1_NAME} - +@test '(Amavis enabled - custom) ENV should update Amavis config' { + export CONTAINER_NAME=${CONTAINER2_NAME} local AMAVIS_DEFAULTS_FILE='/etc/amavis/conf.d/20-debian_defaults' - # shellcheck disable=SC2016 - _run_in_container grep '\$sa_tag_level_deflt' "${AMAVIS_DEFAULTS_FILE}" - assert_success - assert_output --partial '= 2.0' - # shellcheck disable=SC2016 - _run_in_container grep '\$sa_tag2_level_deflt' "${AMAVIS_DEFAULTS_FILE}" + _run_in_container grep 'sa_tag_level_deflt' "${AMAVIS_DEFAULTS_FILE}" assert_success - assert_output --partial '= 6.31' + # shellcheck disable=SC2016 + assert_output --partial '$sa_tag_level_deflt = -5.0;' - # shellcheck disable=SC2016 - _run_in_container grep '\$sa_kill_level_deflt' "${AMAVIS_DEFAULTS_FILE}" + _run_in_container grep 'sa_tag2_level_deflt' "${AMAVIS_DEFAULTS_FILE}" assert_success - assert_output --partial '= 10.0' + # shellcheck disable=SC2016 + assert_output --partial '$sa_tag2_level_deflt = 2.0;' - # shellcheck disable=SC2016 - _run_in_container grep '\$sa_spam_subject_tag' "${AMAVIS_DEFAULTS_FILE}" + _run_in_container grep 'sa_kill_level_deflt' "${AMAVIS_DEFAULTS_FILE}" assert_success - assert_output --partial "= '***SPAM*** ';" + # shellcheck disable=SC2016 + assert_output --partial '$sa_kill_level_deflt = 3.0;' + + # This feature is handled by our SPAM_SUBJECT ENV through a sieve script instead. + # Thus the feature here should always be disabled via the 'undef' value. + _run_in_container grep 'sa_spam_subject_tag' "${AMAVIS_DEFAULTS_FILE}" + assert_success + # shellcheck disable=SC2016 + assert_output --partial '$sa_spam_subject_tag = undef;' } @test '(Amavis disabled) configuration should be correct' { - export CONTAINER_NAME=${CONTAINER2_NAME} + export CONTAINER_NAME=${CONTAINER3_NAME} _run_in_container postconf -h content_filter assert_success @@ -91,8 +130,6 @@ function teardown_file() { _run_in_container grep -F '127.0.0.1:10025' /etc/postfix/master.cf assert_failure - _run_in_container_bash '[[ -f /etc/cron.d/amavisd-new.disabled ]]' - assert_success - _run_in_container_bash '[[ -f /etc/cron.d/amavisd-new ]]' - assert_failure + _file_exists_in_container /etc/cron.d/amavisd-new.disabled + _file_does_not_exist_in_container /etc/cron.d/amavisd-new } diff --git a/test/tests/parallel/set1/spam_virus/rspamd_dkim.bats b/test/tests/parallel/set1/spam_virus/rspamd_dkim.bats index 044e38fb..20939f03 100644 --- a/test/tests/parallel/set1/spam_virus/rspamd_dkim.bats +++ b/test/tests/parallel/set1/spam_virus/rspamd_dkim.bats @@ -62,8 +62,7 @@ function teardown_file() { _default_teardown ; } assert_output --partial "Supplying a default configuration (to '${SIGNING_CONF_FILE}')" refute_output --partial "'${SIGNING_CONF_FILE}' exists, not supplying a default" assert_output --partial "Finished DKIM key creation" - _run_in_container_bash "[[ -f ${SIGNING_CONF_FILE} ]]" - assert_success + _file_exists_in_container "${SIGNING_CONF_FILE}" _exec_in_container_bash "echo 'blabla' >${SIGNING_CONF_FILE}" local INITIAL_SHA512_SUM=$(_exec_in_container sha512sum "${SIGNING_CONF_FILE}") @@ -87,8 +86,7 @@ function teardown_file() { _default_teardown ; } assert_success _count_files_in_directory_in_container /tmp/docker-mailserver/rspamd/dkim/ 3 - _run_in_container_bash "[[ -f ${SIGNING_CONF_FILE} ]]" - assert_success + _file_exists_in_container "${SIGNING_CONF_FILE}" __check_path_in_signing_config "/tmp/docker-mailserver/rspamd/dkim/rsa-2048-mail-${DOMAIN_NAME}.private.txt" __check_selector_in_signing_config 'mail' @@ -241,8 +239,7 @@ function __check_rsa_keys() { function __check_key_files_are_present() { local BASE_FILE_NAME="${1:?Base file name must be supplied to __check_key_files_are_present}" for FILE in ${BASE_FILE_NAME}.{public.txt,public.dns.txt,private.txt}; do - _run_in_container_bash "[[ -f ${FILE} ]]" - assert_success + _file_exists_in_container "${FILE}" done } diff --git a/test/tests/parallel/set1/spam_virus/rspamd_full.bats b/test/tests/parallel/set1/spam_virus/rspamd_full.bats index 9e6d6222..9b14860b 100644 --- a/test/tests/parallel/set1/spam_virus/rspamd_full.bats +++ b/test/tests/parallel/set1/spam_virus/rspamd_full.bats @@ -29,6 +29,7 @@ function setup_file() { --env RSPAMD_GREYLISTING=1 --env RSPAMD_HFILTER=1 --env RSPAMD_HFILTER_HOSTNAME_UNKNOWN_SCORE=7 + --env SPAM_SUBJECT='[POTENTIAL SPAM] ' ) cp -r "${TEST_TMP_CONFIG}"/rspamd_full/* "${TEST_TMP_CONFIG}/" @@ -43,7 +44,7 @@ function setup_file() { _wait_for_service postfix _wait_for_smtp_port_in_container - # We will send 4 emails: + # We will send 5 emails: # 1. The first one should pass just fine _send_email_with_msgid 'rspamd-test-email-pass' # 2. The second one should be rejected (Rspamd-specific GTUBE pattern for rejection) @@ -56,6 +57,9 @@ function setup_file() { # ref: https://rspamd.com/doc/gtube_patterns.html _send_email_with_msgid 'rspamd-test-email-header' \ --body "YJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X" + # 5. The fifth one will have its subject rewritten, but now spam header is applied. + _send_email_with_msgid 'rspamd-test-email-rewrite_subject' \ + --body "ZJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X" _run_in_container cat /var/log/mail.log assert_success @@ -73,11 +77,18 @@ function teardown_file() { _default_teardown ; } assert_output 'rspamd_milter = inet:localhost:11332' } +@test 'Rspamd base configuration is correct' { + _run_in_container rspamadm configdump actions + assert_success + assert_line 'greylist = 4;' + assert_line 'reject = 11;' + assert_line 'add_header = 6;' + refute_line --regexp 'rewrite_subject = [0-9]+;' +} + @test "contents of '/etc/rspamd/override.d/' are copied" { local OVERRIDE_D='/etc/rspamd/override.d' - - _run_in_container_bash "[[ -f ${OVERRIDE_D}/testmodule_complicated.conf ]]" - assert_success + _file_exists_in_container "${OVERRIDE_D}/testmodule_complicated.conf" } @test 'startup log shows all features as properly enabled' { @@ -89,6 +100,7 @@ function teardown_file() { _default_teardown ; } assert_line --partial 'Enabling greylisting' assert_line --partial 'Hfilter (group) module is enabled' assert_line --partial "Adjusting score for 'HFILTER_HOSTNAME_UNKNOWN' in Hfilter group module to" + assert_line --partial "Spam subject is set - the prefix '[POTENTIAL SPAM] ' will be added to spam e-mails" assert_line --partial "Found file '/tmp/docker-mailserver/rspamd/custom-commands.conf' - parsing and applying it" } @@ -114,7 +126,7 @@ function teardown_file() { _default_teardown ; } _print_mail_log_for_msgid 'rspamd-test-email-pass' assert_output --partial "stored mail into mailbox 'INBOX'" - _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 1 + _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 2 } @test 'detects and rejects spam' { @@ -129,7 +141,7 @@ function teardown_file() { _default_teardown ; } refute_output --partial "stored mail into mailbox 'INBOX'" assert_failure - _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 1 + _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 2 } @test 'detects and rejects virus' { @@ -144,14 +156,13 @@ function teardown_file() { _default_teardown ; } refute_output --partial "stored mail into mailbox 'INBOX'" assert_failure - _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 1 + _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 2 } @test 'custom commands work correctly' { # check `testmodule1` which should be disabled local MODULE_PATH='/etc/rspamd/override.d/testmodule1.conf' - _run_in_container_bash "[[ -f ${MODULE_PATH} ]]" - assert_success + _file_exists_in_container "${MODULE_PATH}" _run_in_container grep -F '# documentation: https://rspamd.com/doc/modules/testmodule1.html' "${MODULE_PATH}" assert_success _run_in_container grep -F 'enabled = false;' "${MODULE_PATH}" @@ -161,8 +172,7 @@ function teardown_file() { _default_teardown ; } # check `testmodule2` which should be enabled and it should have extra options set MODULE_PATH='/etc/rspamd/override.d/testmodule2.conf' - _run_in_container_bash "[[ -f ${MODULE_PATH} ]]" - assert_success + _file_exists_in_container "${MODULE_PATH}" _run_in_container grep -F '# documentation: https://rspamd.com/doc/modules/testmodule2.html' "${MODULE_PATH}" assert_success _run_in_container grep -F 'enabled = true;' "${MODULE_PATH}" @@ -181,8 +191,7 @@ function teardown_file() { _default_teardown ; } # check whether adding a single line writes the line properly in `testmodule4.something` MODULE_PATH='/etc/rspamd/override.d/testmodule4.something' - _run_in_container_bash "[[ -f ${MODULE_PATH} ]]" - assert_success + _file_exists_in_container "${MODULE_PATH}" # shellcheck disable=SC2016 _run_in_container grep -F 'some very long line with "weird $charact"ers' "${MODULE_PATH}" assert_success @@ -193,37 +202,31 @@ function teardown_file() { _default_teardown ; } # check whether spaces in front of options are handles properly in `testmodule_complicated` MODULE_PATH='/etc/rspamd/override.d/testmodule_complicated.conf' - _run_in_container_bash "[[ -f ${MODULE_PATH} ]]" - assert_success + _file_exists_in_container "${MODULE_PATH}" _run_in_container grep -F ' anOption = anotherValue;' "${MODULE_PATH}" # check whether controller option was written properly MODULE_PATH='/etc/rspamd/override.d/worker-controller.inc' - _run_in_container_bash "[[ -f ${MODULE_PATH} ]]" - assert_success + _file_exists_in_container "${MODULE_PATH}" _run_in_container grep -F 'someOption = someValue42;' "${MODULE_PATH}" assert_success # check whether controller option was written properly MODULE_PATH='/etc/rspamd/override.d/worker-proxy.inc' - _run_in_container_bash "[[ -f ${MODULE_PATH} ]]" - assert_success + _file_exists_in_container "${MODULE_PATH}" _run_in_container grep -F 'abcdefg71 = RAAAANdooM;' "${MODULE_PATH}" assert_success # check whether basic options are written properly MODULE_PATH='/etc/rspamd/override.d/options.inc' - _run_in_container_bash "[[ -f ${MODULE_PATH} ]]" - assert_success + _file_exists_in_container "${MODULE_PATH}" _run_in_container grep -F 'OhMy = "PraiseBeLinters !";' "${MODULE_PATH}" assert_success } @test 'MOVE_SPAM_TO_JUNK works for Rspamd' { - _run_in_container_bash '[[ -f /usr/lib/dovecot/sieve-global/after/spam_to_junk.sieve ]]' - assert_success - _run_in_container_bash '[[ -f /usr/lib/dovecot/sieve-global/after/spam_to_junk.svbin ]]' - assert_success + _file_exists_in_container /usr/lib/dovecot/sieve-global/after/spam_to_junk.sieve + _file_exists_in_container /usr/lib/dovecot/sieve-global/after/spam_to_junk.svbin _service_log_should_contain_string 'rspamd' 'S (add header)' _service_log_should_contain_string 'rspamd' 'add header "Gtube pattern"' @@ -231,14 +234,38 @@ function teardown_file() { _default_teardown ; } _print_mail_log_for_msgid 'rspamd-test-email-header' assert_output --partial "fileinto action: stored mail into mailbox 'Junk'" - _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 1 + _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 2 _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/.Junk/new/ 1 } +@test 'Rewriting subject works when enforcing it via GTUBE' { + _service_log_should_contain_string 'rspamd' 'S (rewrite subject)' + _service_log_should_contain_string 'rspamd' 'rewrite subject "Gtube pattern"' + + _print_mail_log_for_msgid 'rspamd-test-email-rewrite_subject' + assert_output --partial "stored mail into mailbox 'INBOX'" + + # check that the inbox contains the subject-rewritten e-mail + _run_in_container_bash "grep --fixed-strings 'Subject: *** SPAM ***' /var/mail/localhost.localdomain/user1/new/*" + assert_success + + # check that the inbox contains the normal e-mail (that passes just fine) + _run_in_container_bash "grep --fixed-strings 'Subject: test' /var/mail/localhost.localdomain/user1/new/*" + assert_success +} + +@test 'SPAM_SUBJECT works' { + _file_exists_in_container /usr/lib/dovecot/sieve-global/before/spam_subject.sieve + _file_exists_in_container /usr/lib/dovecot/sieve-global/before/spam_subject.svbin + + # we only have one e-mail in the junk folder, hence using '*' is fine + _run_in_container_bash "grep --fixed-strings 'Subject: [POTENTIAL SPAM]' /var/mail/localhost.localdomain/user1/.Junk/new/*" + assert_success +} + @test 'RSPAMD_LEARN works' { for FILE in learn-{ham,spam}.{sieve,svbin}; do - _run_in_container_bash "[[ -f /usr/lib/dovecot/sieve-pipe/${FILE} ]]" - assert_success + _file_exists_in_container "/usr/lib/dovecot/sieve-pipe/${FILE}" done _run_in_container grep 'mail_plugins.*imap_sieve' /etc/dovecot/conf.d/20-imap.conf @@ -305,8 +332,7 @@ function teardown_file() { _default_teardown ; } @test 'hfilter group module is configured correctly' { local MODULE_FILE='/etc/rspamd/local.d/hfilter_group.conf' - _run_in_container_bash "[[ -f ${MODULE_FILE} ]]" - assert_success + _file_exists_in_container "${MODULE_FILE}" _run_in_container grep '__TAG__HFILTER_HOSTNAME_UNKNOWN' "${MODULE_FILE}" assert_success @@ -315,8 +341,7 @@ function teardown_file() { _default_teardown ; } @test 'checks on authenticated users are disabled' { local MODULE_FILE='/etc/rspamd/local.d/settings.conf' - _run_in_container_bash "[[ -f ${MODULE_FILE} ]]" - assert_success + _file_exists_in_container "${MODULE_FILE}" _run_in_container grep -E -A 6 'authenticated \{' "${MODULE_FILE}" assert_success diff --git a/test/tests/parallel/set1/spam_virus/rspamd_partly.bats b/test/tests/parallel/set1/spam_virus/rspamd_partly.bats index 9fc8af31..b647f562 100644 --- a/test/tests/parallel/set1/spam_virus/rspamd_partly.bats +++ b/test/tests/parallel/set1/spam_virus/rspamd_partly.bats @@ -55,6 +55,7 @@ function teardown_file() { _default_teardown ; } assert_line --partial 'Intelligent learning of spam and ham is disabled' assert_line --partial 'Greylisting is disabled' assert_line --partial 'Disabling Hfilter (group) module' + assert_line --partial 'Spam subject is not set' } @test 'antivirus maximum size was not adjusted unnecessarily' { @@ -64,8 +65,7 @@ function teardown_file() { _default_teardown ; } @test 'learning is properly disabled' { for FILE in learn-{ham,spam}.{sieve,svbin}; do - _run_in_container_bash "[[ -f /usr/lib/dovecot/sieve-pipe/${FILE} ]]" - assert_failure + _file_does_not_exist_in_container "/usr/lib/dovecot/sieve-pipe/${FILE}" done _run_in_container grep 'mail_plugins.*imap_sieve' /etc/dovecot/conf.d/20-imap.conf @@ -83,14 +83,13 @@ function teardown_file() { _default_teardown ; } } @test 'hfilter group module configuration is deleted' { - _run_in_container_bash '[[ -f /etc/rspamd/local.d/hfilter_group.conf ]]' + _file_does_not_exist_in_container /etc/rspamd/local.d/hfilter_group.conf assert_failure } @test 'checks on authenticated users are enabled' { local MODULE_FILE='/etc/rspamd/local.d/settings.conf' - _run_in_container_bash "[[ -f ${MODULE_FILE} ]]" - assert_success + _file_exists_in_container "${MODULE_FILE}" _run_in_container grep -E 'authenticated \{' "${MODULE_FILE}" assert_failure diff --git a/test/tests/parallel/set1/spam_virus/spam_junk_folder.bats b/test/tests/parallel/set1/spam_virus/spam_junk_folder.bats index 15ec4fe1..0c873382 100644 --- a/test/tests/parallel/set1/spam_virus/spam_junk_folder.bats +++ b/test/tests/parallel/set1/spam_virus/spam_junk_folder.bats @@ -34,7 +34,7 @@ function teardown() { _default_teardown ; } local CUSTOM_SETUP_ARGUMENTS=( --env ENABLE_AMAVIS=1 --env ENABLE_SPAMASSASSIN=1 - --env SA_SPAM_SUBJECT="SPAM: " + --env SPAM_SUBJECT="SPAM: " --env SPAMASSASSIN_SPAM_TO_INBOX=1 --env MOVE_SPAM_TO_JUNK=0 --env PERMIT_DOCKER=container @@ -55,7 +55,7 @@ function teardown() { _default_teardown ; } local CUSTOM_SETUP_ARGUMENTS=( --env ENABLE_AMAVIS=1 --env ENABLE_SPAMASSASSIN=1 - --env SA_SPAM_SUBJECT="SPAM: " + --env SPAM_SUBJECT="SPAM: " --env SPAMASSASSIN_SPAM_TO_INBOX=1 --env MOVE_SPAM_TO_JUNK=1 --env PERMIT_DOCKER=container @@ -77,7 +77,7 @@ function teardown() { _default_teardown ; } local CUSTOM_SETUP_ARGUMENTS=( --env ENABLE_AMAVIS=1 --env ENABLE_SPAMASSASSIN=1 - --env SA_SPAM_SUBJECT="SPAM: " + --env SPAM_SUBJECT="SPAM: " --env SPAMASSASSIN_SPAM_TO_INBOX=1 --env MARK_SPAM_AS_READ=1 --env PERMIT_DOCKER=container diff --git a/test/tests/parallel/set1/spam_virus/undef_spam_subject.bats b/test/tests/parallel/set1/spam_virus/undef_spam_subject.bats deleted file mode 100644 index 35260d51..00000000 --- a/test/tests/parallel/set1/spam_virus/undef_spam_subject.bats +++ /dev/null @@ -1,61 +0,0 @@ -load "${REPOSITORY_ROOT}/test/helper/setup" -load "${REPOSITORY_ROOT}/test/helper/common" - -BATS_TEST_NAME_PREFIX='[Spam] (undefined subject) ' - -CONTAINER1_NAME='dms-test_spam-undef-subject_1' -CONTAINER2_NAME='dms-test_spam-undef-subject_2' -CONTAINER_NAME=${CONTAINER2_NAME} - -function teardown() { _default_teardown ; } - -@test "'SA_SPAM_SUBJECT=undef' should update Amavis config" { - export CONTAINER_NAME=${CONTAINER1_NAME} - local CUSTOM_SETUP_ARGUMENTS=( - --env ENABLE_AMAVIS=1 - --env ENABLE_SPAMASSASSIN=1 - --env SA_SPAM_SUBJECT='undef' - ) - _init_with_defaults - _common_container_setup 'CUSTOM_SETUP_ARGUMENTS' - - _run_in_container_bash "grep '\$sa_spam_subject_tag' /etc/amavis/conf.d/20-debian_defaults | grep '= undef'" - assert_success -} - -# TODO: Unclear why some of these ENV are relevant for the test? -@test "Docker env variables are set correctly (custom)" { - export CONTAINER_NAME=${CONTAINER2_NAME} - - local CUSTOM_SETUP_ARGUMENTS=( - --env ENABLE_CLAMAV=1 - --env SPOOF_PROTECTION=1 - --env ENABLE_SPAMASSASSIN=1 - --env REPORT_RECIPIENT=user1@localhost.localdomain - --env REPORT_SENDER=report1@mail.my-domain.com - --env SA_TAG=-5.0 - --env SA_TAG2=2.0 - --env SA_KILL=3.0 - --env SA_SPAM_SUBJECT="SPAM: " - --env VIRUSMAILS_DELETE_DELAY=7 - --env ENABLE_SRS=1 - --env ENABLE_MANAGESIEVE=1 - --env PERMIT_DOCKER=host - # NOTE: ulimit required for `ENABLE_SRS=1` until running a newer `postsrsd` - --ulimit "nofile=$(ulimit -Sn):$(ulimit -Hn)" - ) - _init_with_defaults - _common_container_setup 'CUSTOM_SETUP_ARGUMENTS' - - _run_in_container_bash "grep '\$sa_tag_level_deflt' /etc/amavis/conf.d/20-debian_defaults | grep '= -5.0'" - assert_success - - _run_in_container_bash "grep '\$sa_tag2_level_deflt' /etc/amavis/conf.d/20-debian_defaults | grep '= 2.0'" - assert_success - - _run_in_container_bash "grep '\$sa_kill_level_deflt' /etc/amavis/conf.d/20-debian_defaults | grep '= 3.0'" - assert_success - - _run_in_container_bash "grep '\$sa_spam_subject_tag' /etc/amavis/conf.d/20-debian_defaults | grep '= .SPAM: .'" - assert_success -} From 2018be7fdc04ce313667d365427fa9e79b4db2eb Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 29 Jan 2024 14:08:59 +0100 Subject: [PATCH 017/267] chore(deps): Bump anchore/scan-action from 3.6.0 to 3.6.1 (#3848) Bumps [anchore/scan-action](https://github.com/anchore/scan-action) from 3.6.0 to 3.6.1. - [Release notes](https://github.com/anchore/scan-action/releases) - [Changelog](https://github.com/anchore/scan-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/anchore/scan-action/compare/v3.6.0...v3.6.1) --- updated-dependencies: - dependency-name: anchore/scan-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> --- .github/workflows/generic_vulnerability-scan.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index b261de91..cb1a5994 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -55,7 +55,7 @@ jobs: provenance: false - name: 'Run the Anchore Grype scan action' - uses: anchore/scan-action@v3.6.0 + uses: anchore/scan-action@v3.6.1 id: scan with: image: mailserver-testing:ci From 4162d608e40370198662e7006ce6d5a81b94671c Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Mon, 29 Jan 2024 22:10:03 +0100 Subject: [PATCH 018/267] Rspamd scripts: only correct permissions when directory exists (#3849) --- target/scripts/startup/setup-stack.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/target/scripts/startup/setup-stack.sh b/target/scripts/startup/setup-stack.sh index 24ef1581..81e1a98e 100644 --- a/target/scripts/startup/setup-stack.sh +++ b/target/scripts/startup/setup-stack.sh @@ -97,8 +97,10 @@ function _setup_apply_fixes_after_configuration() { _rspamd_get_envs # /tmp/docker-mailserver/rspamd/dkim - _log 'debug' "Ensuring '${RSPAMD_DMS_DKIM_D}' is owned by '_rspamd:_rspamd'" - chown -R _rspamd:_rspamd "${RSPAMD_DMS_DKIM_D}" + if [[ -d ${RSPAMD_DMS_DKIM_D} ]]; then + _log 'debug' "Ensuring '${RSPAMD_DMS_DKIM_D}' is owned by '_rspamd:_rspamd'" + chown -R _rspamd:_rspamd "${RSPAMD_DMS_DKIM_D}" + fi } function _run_user_patches() { From 244c455ca1b92371a5c29a47ce28588f3eb5baf7 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Tue, 30 Jan 2024 11:11:45 +1300 Subject: [PATCH 019/267] fix: `packages.sh` - Download `jaq` via release `tag` not `latest` (#3852) As the filename includes the version / tag, we cannot rely on the latest URL to be stable. --- target/scripts/build/packages.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/target/scripts/build/packages.sh b/target/scripts/build/packages.sh index 24dae8a7..afff4cad 100644 --- a/target/scripts/build/packages.sh +++ b/target/scripts/build/packages.sh @@ -38,7 +38,8 @@ function _pre_installation_steps() { function _install_utils() { _log 'debug' 'Installing utils sourced from Github' _log 'trace' 'Installing jaq' - curl -sSfL "https://github.com/01mf02/jaq/releases/latest/download/jaq-v1.2.0-$(uname -m)-unknown-linux-gnu" -o /usr/bin/jaq && chmod +x /usr/bin/jaq + local JAQ_TAG='v1.3.0' + curl -sSfL "https://github.com/01mf02/jaq/releases/download/${JAQ_TAG}/jaq-${JAQ_TAG}-$(uname -m)-unknown-linux-gnu" -o /usr/bin/jaq && chmod +x /usr/bin/jaq _log 'trace' 'Installing swaks' local SWAKS_VERSION='20240103.0' From 23705e6712cd5b9cacf4cf8b5e7bfc3375e72e2e Mon Sep 17 00:00:00 2001 From: Casper Date: Tue, 30 Jan 2024 07:34:26 +0100 Subject: [PATCH 020/267] fix: abort when (jaq) curl fails (#3853) --- target/scripts/build/packages.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/target/scripts/build/packages.sh b/target/scripts/build/packages.sh index afff4cad..1f769739 100644 --- a/target/scripts/build/packages.sh +++ b/target/scripts/build/packages.sh @@ -39,7 +39,8 @@ function _install_utils() { _log 'debug' 'Installing utils sourced from Github' _log 'trace' 'Installing jaq' local JAQ_TAG='v1.3.0' - curl -sSfL "https://github.com/01mf02/jaq/releases/download/${JAQ_TAG}/jaq-${JAQ_TAG}-$(uname -m)-unknown-linux-gnu" -o /usr/bin/jaq && chmod +x /usr/bin/jaq + curl -sSfL "https://github.com/01mf02/jaq/releases/download/${JAQ_TAG}/jaq-${JAQ_TAG}-$(uname -m)-unknown-linux-gnu" -o /usr/bin/jaq + chmod +x /usr/bin/jaq _log 'trace' 'Installing swaks' local SWAKS_VERSION='20240103.0' @@ -144,7 +145,7 @@ function _install_dovecot() { # when building for another architecture. if [[ ${DOVECOT_COMMUNITY_REPO} -eq 1 ]] && [[ "$(uname --machine)" == "x86_64" ]]; then _log 'trace' 'Using Dovecot community repository' - curl https://repo.dovecot.org/DOVECOT-REPO-GPG | gpg --import + curl -sSfL https://repo.dovecot.org/DOVECOT-REPO-GPG | gpg --import gpg --export ED409DA1 > /etc/apt/trusted.gpg.d/dovecot.gpg # VERSION_CODENAME sourced from /etc/os-release echo "deb https://repo.dovecot.org/ce-2.3-latest/debian/${VERSION_CODENAME} ${VERSION_CODENAME} main" > /etc/apt/sources.list.d/dovecot.list From dfd5edc000f41ba46b39c469e4b5075b7a4ef1a9 Mon Sep 17 00:00:00 2001 From: Andreas Perhab Date: Tue, 30 Jan 2024 11:17:58 +0100 Subject: [PATCH 021/267] docs: Add new local dependency (`file`) for running tests (#3856) --- docs/content/contributing/tests.md | 2 +- target/postfix/main.cf | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/content/contributing/tests.md b/docs/content/contributing/tests.md index 8816a228..dcd73505 100644 --- a/docs/content/contributing/tests.md +++ b/docs/content/contributing/tests.md @@ -51,7 +51,7 @@ Parallel tests are further partitioned into smaller sets. If your system has the To run the test suite, you will need to: 1. [Install Docker][get-docker] -2. Install `jq` and (GNU) `parallel` (under Ubuntu, use `sudo apt-get -y install jq parallel`) +2. Install `jq` , (GNU) `parallel` and `file` (under Ubuntu, use `sudo apt-get -y install jq parallel file`) 3. Execute `git submodule update --init --recursive` if you haven't already initialized the git submodules ### Executing Test(s) diff --git a/target/postfix/main.cf b/target/postfix/main.cf index a0d805cc..d501eec0 100644 --- a/target/postfix/main.cf +++ b/target/postfix/main.cf @@ -119,6 +119,6 @@ smtp_header_checks = pcre:/etc/postfix/maps/sender_header_filter.pcre # The default compatibility_level is 0 - which retains legacy settings defaults: # http://www.postfix.org/COMPATIBILITY_README.html -# If backwards-compaitibilty log messages appear, fix them by explicitly adding +# If backwards-compatibility log messages appear, fix them by explicitly adding # the legacy or new default value (alternatively raise the compatibility_level) compatibility_level = 3.6 From 5b54d1d32ee1f786c6c9a7f6bf4db07ceea871e2 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Wed, 31 Jan 2024 10:24:43 +1300 Subject: [PATCH 022/267] refactor: `relay.sh` (#3845) * chore: `relay.sh` helper - Reference user config paths via variables * chore: Better document postfix helper `_vhost_collect_postfix_domains()` The functionality is effectively the same for the two configs for the most part when it comes to parsing out a domain from the target value. Virtual aliases is more flexible in value, which may not have a domain-part present (manual user edit). * chore: `check-for-change.sh` - Support VHOST change visibility - Moves the "handle changes" logic into it's own scoped function, out of the main change detection loop logic. - This will be benefit a future commit change that will rely on `VHOST_UPDATED=1`. * chore: `relay.sh` - Minor revisions to minimize diff noise - Better phrasing of the current logic comments. - Regex patterns assigned to variables (easier to grok intention) - Bulk of the logic for generating `/etc/postfix/relayhost_map` wrapped into a separate function with Postfix config setting handled separately. * refactor: `relay.sh` opt-out logic - Split the two distinct features that configure `/etc/postfix/relayhost_map` into separate functions (_`MATCH_VALID` var no longer needed for legacy support_). - Instead of extracting domains from `postfix-accounts.cf` + `postfix-virtual.cf`, this has already been handled at `/etc/postfix/vhost`, sourcing from there is far less complicated. - Rename loop var `DOMAIN_PART`to `SENDER_DOMAIN` for better context of what it represents when appended to the config file. - Revised maintenance notes + guidance towards a future refactor of this relayhost feature support. * docs: `relay.sh` - Additional comment revisions * feat: `DEFAULT_RELAY_HOST` can now also use relay credentials ENV - Remove comment regarding `smtp_sasl_password_maps = static:${RELAY_USER}:${RELAY_PASSWORD}`, it could be used but `main.cf` presently has `644` permissions vs the `sasl_passwd` file permissions of `600`, less secure at preventing leaking of secrets (ignoring the ENV exposure itself). - Move the `main.cf` settings specific to relayhost credentials support / security into to the relevant function scope instead. This also allows for the configuration to be applied by a change detection event without container restart requirement. - Outer functions for setup and change detection to call have a clearer config dependency guard, as does the `_legacy_support()`. - These changes now support `DEFAULT_RELAY_HOST` to leverage the relay credentials ENV as well. - `DATABASE_RELAYHOSTS` is available in scope to the functions called here that reference it. * docs: Revised ENV docs on relay host config Better quality guidance on configuring relay hosts. * chore: Add entry to `CHANGELOG.md` * fix: `relay.sh` - `grep` regex compatibility with `+` requires `-E` * chore: `postfix.sh` - `FIRST_FIELD` => More descriptive field name --- CHANGELOG.md | 11 ++ docs/content/config/environment.md | 87 +++++++++--- mailserver.env | 16 ++- target/scripts/check-for-changes.sh | 55 ++++---- target/scripts/helpers/postfix.sh | 23 ++-- target/scripts/helpers/relay.sh | 199 ++++++++++++++-------------- 6 files changed, 239 insertions(+), 152 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d16c03ec..1f426a5b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,13 @@ The most noteworthy change of this release is the update of the container's base - DMS `main.cf` has renamed `postscreen_dnsbl_whitelist_threshold` to `postscreen_dnsbl_allowlist_threshold` as part of this change. - `smtpd_relay_restrictions` (relay policy) is now evaluated after `smtpd_recipient_restrictions` (spam policy). Previously it was evaluated before `smtpd_recipient_restrictions`. Mail to be relayed via DMS must now pass through the spam policy first. - The TLS fingerprint policy has changed the default from MD5 to SHA256 (_DMS does not modify this Postfix parameter, but may affect any user customizations that do_). +- **Features:** + - The relay host feature was refactored ([#3845](https://github.com/docker-mailserver/docker-mailserver/pull/3845)) + - The only breaking change this should introduce is with the Change Detection service (`check-for-changes.sh`). + - When credentials are configured for relays, change events that trigger the relayhost logic now reapply the relevant Postfix settings: + - `smtp_sasl_auth_enable = yes` (_SASL auth to outbound MTA connections is enabled_) + - `smtp_sasl_security_options = noanonymous` (_credentials are mandatory for outbound mail delivery_) + - `smtp_tls_security_level = encrypt` (_the outbound MTA connection must always be secure due to credentials sent_) - **Environment Variables**: - `SA_SPAM_SUBJECT` has been renamed into `SPAM_SUBJECT` to become anti-spam service agnostic. ([3820](https://github.com/docker-mailserver/docker-mailserver/pull/3820)) - As this functionality is now handled in Dovecot via a Sieve script instead of the respective anti-spam service during Postfix processing, this feature will only apply to mail stored in Dovecot. If you have relied on this feature in a different context, it will no longer be available. @@ -60,6 +67,10 @@ The most noteworthy change of this release is the update of the container's base - DMS config files that are parsed line by line are now more robust to parse by detecting and fixing line-endings ([#3819](https://github.com/docker-mailserver/docker-mailserver/pull/3819)) - Variables related to Rspamd are declared as `readonly`, which would cause warnings in the log when being re-declared; we now guard against this issue ([#3837](https://github.com/docker-mailserver/docker-mailserver/pull/3837)) +- Relay host feature refactored ([#3845](https://github.com/docker-mailserver/docker-mailserver/pull/3845)) + - `DEFAULT_RELAY_HOST` ENV can now also use the `RELAY_USER` + `RELAY_PASSWORD` ENV for supplying credentials. + - `RELAY_HOST` ENV no longer enforces configuring outbound SMTP to require credentials. Like `DEFAULT_RELAY_HOST` it can now configure a relay where credentials are optional. + - Restarting DMS should not be required when configuring relay hosts without these ENV, but solely via `setup relay ...`, as change detection events now apply relevant Postfix setting changes for supporting credentials too. ## [v13.3.1](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v13.3.1) diff --git a/docs/content/config/environment.md b/docs/content/config/environment.md index c98016d1..5db1c2c3 100644 --- a/docs/content/config/environment.md +++ b/docs/content/config/environment.md @@ -1012,36 +1012,90 @@ you to replace both instead of just the envelope sender. - **empty** => Derived from [`OVERRIDE_HOSTNAME`](#override_hostname), `$DOMAINNAME` (internal), or the container's hostname - Set this if auto-detection fails, isn't what you want, or you wish to have a separate container handle DSNs -#### Default Relay Host +#### Relay Host + +!!! tip "`RELAY_HOST` vs `DEFAULT_RELAY_HOST`" + + `DEFAULT_RELAY_HOST` is encouraged, but presently does not support sender domain opt-out (`setup relay exclude-domain`). + +!!! tip "Opt-in for relay host support" + + If you only want to enable relay for specific sender domains, use can use opt-in via `setup relay add-domain`. ##### DEFAULT_RELAY_HOST -- **empty** => don't set default relayhost setting in main.cf -- default host and port to relay all mail through. - Format: `[example.com]:587` (don't forget the brackets if you need this to - be compatible with `$RELAY_USER` and `$RELAY_PASSWORD`, explained below). +Configures a default relay host. -#### Multi-domain Relay Hosts +!!! info + + - All mail sent outbound from DMS will be relayed through the configured host, unless sender-dependent relayhost maps have been configured (_which have precedence_). + - The host value may optionally be wrapped in brackets (_skips DNS query for MX record_): `[mail.example.com]:587` vs `example.com:587` + +!!! abstract "Technical Details" + + Configures the Postfix `main.cf` setting: [`relayhost`][postfix-config::relayhost] ##### RELAY_HOST -- **empty** => don't configure relay host -- default host to relay mail through +Configures a default relay host. + +!!! info + + - This is a legacy ENV. It is however required for the opt-out feature of `postfix-relaymap.cf` to work. + - When configured, all known mail domains managed by DMS will be configured to relay outbound mail, just like `DEFAULT_RELAY_HOST`. + +!!! note + + Expects a value like `mail.example.com`. Internally this will be wrapped to `[mail.example.com]`, so it should resolve to the MTA directly. + + Do not use with `DEFAULT_RELAY_HOST`. `RELAY_HOST` has precedence as it is configured with `sender_dependent_relayhost_maps`. + +!!! abstract "Technical Details" + + This feature is configured internally using the: + + - Postfix setting with config: [`sender_dependent_relayhost_maps = texthash:/etc/postfix/relayhost_map`][postfix-config::relayhost_maps] + - DMS Config volume support via: `postfix-relaymap.cf` (_generates `/etc/postfix/relayhost_map`_) ##### RELAY_PORT -- **empty** => 25 -- default port to relay mail through +Default => 25 + +Support for configuring a different port than 25 for `RELAY_HOST` to use. + +!!! note + + Requires `RELAY_HOST`. + +#### Relay Host Credentials + +!!! warning "Configuring relay host credentials make outbound authentication mandatory" + + Presently when `RELAY_USER` + `RELAY_PASSWORD` or `postfix-sasl-password.cf` are configured, all outbound mail traffic is configured to require a secure connection established and forbids the omission of credentials. + + Additional feature work is required to only enforce these requirements on mail sent through a configured relay host. ##### RELAY_USER -- **empty** => no default -- default relay username (if no specific entry exists in postfix-sasl-password.cf) - ##### RELAY_PASSWORD -- **empty** => no default -- password for default relay user +Provide the credentials to use with `RELAY_HOST` or `DEFAULT_RELAY_HOST`. + +!!! tip "Alternative credentials config" + + You may prefer to use `setup relay add-auth` to avoid exposure of secrets in ENV. + + - With the CLI command you must provide each sender domain relay credentials. + - Alternatively manually edit `postfix-sasl-password.cf` with the correct relayhost entry (_`DEFAULT_RELAY_HOST` value or as defined in `/etc/postfix/relayhost_map`_) to provide credentials per relayhost configured. + +!!! abstract "Technical Details" + + Credentials for relay hosts are configured internally using the: + + - Postfix setting with config: [`smtp_sasl_password_maps = texthash:/etc/postfix/sasl_passwd`][postfix-config::sasl_passwd] + - DMS Config volume support via: `postfix-sasl-password.cf` (_generates `/etc/postfix/sasl_passwd`_) + + This file has relay hosts that must match the `host:port` of `/etc/postfix/relayhost_map` or `main.cf:relayhost`. DMS support handles this for you. [docs-rspamd]: ./security/rspamd.md [docs-tls]: ./security/ssl.md @@ -1050,3 +1104,6 @@ you to replace both instead of just the envelope sender. [docs-tls-selfsigned]: ./security/ssl.md#self-signed-certificates [docs-accounts-quota]: ./user-management.md#quotas [docs::dms-volumes-state]: ./advanced/optional-config.md#volumes-state +[postfix-config::relayhost]: https://www.postfix.org/postconf.5.html#relayhost +[postfix-config::relayhost_maps]: https://www.postfix.org/postconf.5.html#sender_dependent_relayhost_maps +[postfix-config::sasl_passwd]: https://www.postfix.org/postconf.5.html#smtp_sasl_password_maps diff --git a/mailserver.env b/mailserver.env index 38b9adca..6fc9c7c2 100644 --- a/mailserver.env +++ b/mailserver.env @@ -619,8 +619,8 @@ SRS_SECRET= # Setup relaying all mail through a default relay host # -# empty => don't configure default relay host -# default host and optional port to relay all mail through +# Set a default host to relay all mail through (optionally include a port) +# Example: [mail.example.com]:587 DEFAULT_RELAY_HOST= # ----------------------------------------------- @@ -630,18 +630,22 @@ DEFAULT_RELAY_HOST= # Setup relaying for multiple domains based on the domain name of the sender # optionally uses usernames and passwords in postfix-sasl-password.cf and relay host mappings in postfix-relaymap.cf # -# empty => don't configure relay host -# default host to relay mail through +# Set a default host to relay mail through +# Example: mail.example.com RELAY_HOST= # empty => 25 # default port to relay mail RELAY_PORT=25 +# ----------------------------------------------- +# --- Relay Host Credentials Section ------------ +# ----------------------------------------------- + +# Configure a relay user and password to use with RELAY_HOST / DEFAULT_RELAY_HOST + # empty => no default -# default relay username (if no specific entry exists in postfix-sasl-password.cf) RELAY_USER= # empty => no default -# password for default relay user RELAY_PASSWORD= diff --git a/target/scripts/check-for-changes.sh b/target/scripts/check-for-changes.sh index 2fa004b0..3f268710 100755 --- a/target/scripts/check-for-changes.sh +++ b/target/scripts/check-for-changes.sh @@ -49,17 +49,7 @@ function _check_for_changes() { local CHANGED CHANGED=$(_get_changed_files "${CHKSUM_FILE}" "${CHKSUM_FILE}.new") - - # Handle any changes - _ssl_changes - _postfix_dovecot_changes - _rspamd_changes - - _log_with_date 'debug' 'Reloading services due to detected changes' - - [[ ${ENABLE_AMAVIS} -eq 1 ]] && _reload_amavis - _reload_postfix - [[ ${SMTP_ONLY} -ne 1 ]] && dovecot reload + _handle_changes _remove_lock _log_with_date 'debug' 'Completed handling of detected change' @@ -69,6 +59,29 @@ function _check_for_changes() { fi } +function _handle_changes() { + # Variable to identify any config updates dependent upon vhost changes. + local VHOST_UPDATED=0 + # These two configs are the source for /etc/postfix/vhost (managed mail domains) + if [[ ${CHANGED} =~ ${DMS_DIR}/postfix-(accounts|virtual).cf ]]; then + _log_with_date 'trace' 'Regenerating vhosts (Postfix)' + # Regenerate via `helpers/postfix.sh`: + _create_postfix_vhost + + VHOST_UPDATED=1 + fi + + _ssl_changes + _postfix_dovecot_changes + _rspamd_changes + + _log_with_date 'debug' 'Reloading services due to detected changes' + + [[ ${ENABLE_AMAVIS} -eq 1 ]] && _reload_amavis + _reload_postfix + [[ ${SMTP_ONLY} -ne 1 ]] && dovecot reload +} + function _get_changed_files() { local CHKSUM_CURRENT=${1} local CHKSUM_NEW=${2} @@ -85,9 +98,9 @@ function _get_changed_files() { } function _reload_amavis() { - if [[ ${CHANGED} =~ ${DMS_DIR}/postfix-accounts.cf ]] || [[ ${CHANGED} =~ ${DMS_DIR}/postfix-virtual.cf ]]; then - # /etc/postfix/vhost was updated, amavis must refresh it's config by - # reading this file again in case of new domains, otherwise they will be ignored. + # /etc/postfix/vhost was updated, amavis must refresh it's config by + # reading this file again in case of new domains, otherwise they will be ignored. + if [[ ${VHOST_UPDATED} -eq 1 ]]; then amavisd reload fi } @@ -114,14 +127,12 @@ function _postfix_dovecot_changes() { # - postfix-sasl-password.cf used by _relayhost_sasl # - _populate_relayhost_map relies on: # - postfix-relaymap.cf - # - postfix-accounts.cf + postfix-virtual.cf (both will be dropped in future) - if [[ ${CHANGED} =~ ${DMS_DIR}/postfix-accounts.cf ]] \ - || [[ ${CHANGED} =~ ${DMS_DIR}/postfix-virtual.cf ]] \ + if [[ ${VHOST_UPDATED} -eq 1 ]] \ || [[ ${CHANGED} =~ ${DMS_DIR}/postfix-relaymap.cf ]] \ || [[ ${CHANGED} =~ ${DMS_DIR}/postfix-sasl-password.cf ]] then _log_with_date 'trace' 'Regenerating relay config (Postfix)' - _rebuild_relayhost + _process_relayhost_configs fi # Regenerate system + virtual account aliases via `helpers/aliases.sh`: @@ -129,14 +140,6 @@ function _postfix_dovecot_changes() { [[ ${CHANGED} =~ ${DMS_DIR}/postfix-regexp.cf ]] && _handle_postfix_regexp_config [[ ${CHANGED} =~ ${DMS_DIR}/postfix-aliases.cf ]] && _handle_postfix_aliases_config - # Regenerate `/etc/postfix/vhost` (managed mail domains) via `helpers/postfix.sh`: - if [[ ${CHANGED} =~ ${DMS_DIR}/postfix-accounts.cf ]] \ - || [[ ${CHANGED} =~ ${DMS_DIR}/postfix-virtual.cf ]] - then - _log_with_date 'trace' 'Regenerating vhosts (Postfix)' - _create_postfix_vhost - fi - # Legacy workaround handled here, only seems necessary for _create_accounts: # - `helpers/accounts.sh` logic creates folders/files with wrong ownership. _chown_var_mail_if_necessary diff --git a/target/scripts/helpers/postfix.sh b/target/scripts/helpers/postfix.sh index 5fa4fa83..0bb1c158 100644 --- a/target/scripts/helpers/postfix.sh +++ b/target/scripts/helpers/postfix.sh @@ -43,21 +43,28 @@ function _vhost_collect_postfix_domains() { local DATABASE_VIRTUAL='/tmp/docker-mailserver/postfix-virtual.cf' local DOMAIN UNAME - # getting domains FROM mail accounts + # Extract domains from mail accounts: if [[ -f ${DATABASE_ACCOUNTS} ]]; then - while IFS=$'|' read -r LOGIN _; do - DOMAIN=$(echo "${LOGIN}" | cut -d @ -f2) + while IFS=$'|' read -r MAIL_ACCOUNT _; do + # It is expected valid lines have the format local-part@domain-part: + DOMAIN=$(cut -d '@' -f 2 <<< "${MAIL_ACCOUNT}") + echo "${DOMAIN}" >>"${TMP_VHOST}" done < <(_get_valid_lines_from_file "${DATABASE_ACCOUNTS}") fi - # getting domains FROM mail aliases + # TODO: Consider if virtual aliases should be configured to the same vhost file: + # https://github.com/docker-mailserver/docker-mailserver/issues/2813#issuecomment-1272394563 + # Extract domains from virtual alias config: + # Aliases may have the forms: 'local-part@domain-part', only 'local-part', or '@domain-part' (wildcard catch-all) if [[ -f ${DATABASE_VIRTUAL} ]]; then - while read -r FROM _; do - UNAME=$(echo "${FROM}" | cut -d @ -f1) - DOMAIN=$(echo "${FROM}" | cut -d @ -f2) + while read -r ALIAS_FIELD _; do + UNAME=$(cut -d '@' -f 1 <<< "${ALIAS_FIELD}") + DOMAIN=$(cut -d '@' -f 2 <<< "${ALIAS_FIELD}") - # if they are equal it means the line looks like: "user1 other@domain.tld" + # Only add valid domain-parts found: + # The '@' is optional for an alias key (eg: "user1 other@domain.tld"), + # but cut with -f2 would output the same value as it would -f1 when '@' is missing. [[ ${UNAME} != "${DOMAIN}" ]] && echo "${DOMAIN}" >>"${TMP_VHOST}" done < <(_get_valid_lines_from_file "${DATABASE_VIRTUAL}") fi diff --git a/target/scripts/helpers/relay.sh b/target/scripts/helpers/relay.sh index de29dddc..b92da3b4 100644 --- a/target/scripts/helpers/relay.sh +++ b/target/scripts/helpers/relay.sh @@ -4,9 +4,9 @@ # Description: # This helper is responsible for configuring outbound SMTP (delivery) through relay-hosts. # -# When mail is sent from Postfix, it is considered relaying to that destination (or the next hop). -# By default delivery external of the container would be direct to the MTA of the recipient address (destination). -# Alternatively mail can be indirectly delivered to the destination by routing through a different MTA (relay-host service). +# When mail is sent to Postfix and the destination is not a domain DMS manages, this requires relaying to that destination (or the next hop). +# By default outbound mail delivery would be direct to the MTA of the recipient address (destination). +# Alternatively mail can be delivered indirectly to that destination by routing through a different MTA (relay-host service). # # This helper is only concerned with relaying mail from authenticated submission (ports 587 + 465). # Thus it does not deal with `relay_domains` (which routes through `relay_transport` transport, default: `master.cf:relay`), @@ -37,22 +37,23 @@ # `postfix reload` or `supervisorctl restart postfix` should be run to properly apply config (which it is). # Otherwise use another table type such as `hash` and run `postmap` on the table after modification. # -# WARNING: Databases (tables above) are rebuilt during change detection. There is a minor chance of -# a lookup occurring during a rebuild of these files that may affect or delay delivery? +# WARNING: Databases (tables above) are rebuilt during change detection. +# There is a minor chance of a lookup occurring during a rebuild of these files that may affect or delay delivery? # TODO: Should instead perform an atomic operation with a temporary file + `mv` to replace? # Or switch back to using `hash` table type if plaintext access is not needed (unless retaining file for postmap). # Either way, plaintext copy is likely accessible if using our supported configs for providing them to the container. -# NOTE: Present support has enforced wrapping the relay host with `[]` (prevents DNS MX record lookup), -# which restricts what is supported by RELAY_HOST, although you usually do want to provide MX host directly. -# NOTE: Present support expects to always append a port with an implicit default of `25`. -# NOTE: DEFAULT_RELAY_HOST imposes neither restriction. +# NOTE: Present support has enforced wrapping the `RELAY_HOST` value with `[]` (prevents DNS MX record lookup), +# shouldn't be an issue as you typically do want to provide the MX host directly? This was presumably for config convenience. +# NOTE: Present support expects to always append a port (_with an implicit default of `25`_). +# NOTE: The `DEFAULT_RELAY_HOST` ENV imposes neither restriction. # -# TODO: RELAY_PORT should be optional, it will use the transport default port (`postconf smtp_tcp_port`), +# TODO: `RELAY_PORT` should be optional (Postfix would fallback to the transports default port (`postconf smtp_tcp_port`), # That shouldn't be a breaking change, as long as the mapping is maintained correctly. -# TODO: RELAY_HOST should consider dropping `[]` and require the user to include that? -# Future refactor for _populate_relayhost_map may warrant dropping these two ENV in favor of DEFAULT_RELAY_HOST? +# TODO: `RELAY_HOST` should consider dropping the implicit `[]` and require the user to include that? +# +# A future refactor of `_populate_relayhost_map()` may warrant dropping those two ENV in favor of `DEFAULT_RELAY_HOST`? function _env_relay_host() { echo "[${RELAY_HOST}]:${RELAY_PORT:-25}" } @@ -60,10 +61,12 @@ function _env_relay_host() { # Responsible for `postfix-sasl-password.cf` support: # `/etc/postfix/sasl_passwd` example at end of file. function _relayhost_sasl() { - if [[ ! -f /tmp/docker-mailserver/postfix-sasl-password.cf ]] \ - && [[ -z ${RELAY_USER} || -z ${RELAY_PASSWORD} ]] - then - _log 'warn' "Missing relay-host mapped credentials provided via ENV, or from postfix-sasl-password.cf" + local DATABASE_SASL_PASSWD='/tmp/docker-mailserver/postfix-sasl-password.cf' + + # Only relevant when required credential sources are provided: + if [[ ! -f ${DATABASE_SASL_PASSWD} ]] \ + && [[ -z ${RELAY_USER} || -z ${RELAY_PASSWORD} ]]; then + _log 'warn' "Missing relay-host mapped credentials provided via ENV, or from ${DATABASE_SASL_PASSWD}" return 1 fi @@ -74,7 +77,6 @@ function _relayhost_sasl() { chown root:root /etc/postfix/sasl_passwd chmod 0600 /etc/postfix/sasl_passwd - local DATABASE_SASL_PASSWD='/tmp/docker-mailserver/postfix-sasl-password.cf' if [[ -f ${DATABASE_SASL_PASSWD} ]]; then # Add domain-specific auth from config file: _get_valid_lines_from_file "${DATABASE_SASL_PASSWD}" >> /etc/postfix/sasl_passwd @@ -83,90 +85,93 @@ function _relayhost_sasl() { postconf 'smtp_sender_dependent_authentication = yes' fi - # Add an authenticated relay host defined via ENV config: - if [[ -n ${RELAY_USER} ]] && [[ -n ${RELAY_PASSWORD} ]]; then - echo "$(_env_relay_host) ${RELAY_USER}:${RELAY_PASSWORD}" >> /etc/postfix/sasl_passwd + # Support authentication to a primary relayhost (when configured with credentials via ENV): + if [[ -n ${DEFAULT_RELAY_HOST} || -n ${RELAY_HOST} ]] \ + && [[ -n ${RELAY_USER} && -n ${RELAY_PASSWORD} ]]; then + echo "${DEFAULT_RELAY_HOST:-$(_env_relay_host)} ${RELAY_USER}:${RELAY_PASSWORD}" >>/etc/postfix/sasl_passwd fi - # Technically if only a single relay host is configured, a `static` lookup table could be used instead?: - # postconf "smtp_sasl_password_maps = static:${RELAY_USER}:${RELAY_PASSWORD}" - postconf 'smtp_sasl_password_maps = texthash:/etc/postfix/sasl_passwd' + # Enable credential lookup + SASL authentication to relayhost: + # - `noanonymous` enforces authentication requirement + # - `encrypt` enforces requirement for a secure connection (prevents sending credentials over cleartext, aka mandatory TLS) + postconf \ + 'smtp_sasl_password_maps = texthash:/etc/postfix/sasl_passwd' \ + 'smtp_sasl_auth_enable = yes' \ + 'smtp_sasl_security_options = noanonymous' \ + 'smtp_tls_security_level = encrypt' } # Responsible for `postfix-relaymap.cf` support: # `/etc/postfix/relayhost_map` example at end of file. # -# Present support uses a table lookup for sender address or domain mapping to relay-hosts, -# Populated via `postfix-relaymap.cf `, which also features a non-standard way to exclude implicitly added internal domains from the feature. -# It also maps all known sender domains (from configs postfix-accounts + postfix-virtual.cf) to the same ENV configured relay-host. +# `postfix-relaymap.cf` represents table syntax expected for `/etc/postfix/relayhost_map`, except that it adds an opt-out parsing feature. +# All known mail domains managed by DMS (/etc/postfix/vhost) are implicitly configured to use `RELAY_HOST` + `RELAY_PORT` as the default relay. +# This approach is effectively equivalent to using `main.cf:relayhost`, but with an excessive workaround to support the explicit opt-out feature. # -# TODO: The account + virtual config parsing and appending to /etc/postfix/relayhost_map seems to be an excessive `main.cf:relayhost` -# implementation, rather than leveraging that for the same purpose and selectively overriding only when needed with `/etc/postfix/relayhost_map`. -# If the issue was to opt-out select domains, if avoiding a default relay-host was not an option, then mapping those sender domains or addresses -# to a separate transport (which can drop the `relayhost` setting) would be more appropriate. -# TODO: With `sender_dependent_default_transport_maps`, we can extract out the excluded domains and route them through a separate transport. -# while deprecating that support in favor of a transport config, similar to what is offered currently via sasl_passwd and relayhost_map. +# TODO: Refactor this feature support so that in `main.cf`: +# - Relay all outbound mail through an external MTA by default (works without credentials): +# `relayhost = ${DEFAULT_RELAY_HOST}` +# - Opt-in to relaying - Selectively relay outbound mail by sender/domain to an external MTA (relayhost can vary): +# `sender_dependent_relayhost_maps = texthash:/etc/postfix/relayhost_map` +# - Opt-out from relaying - Selectively prevent outbound mail from relaying via separate transport mappings (where relayhost is not configured): +# By sender: `sender_dependent_default_transport_maps = texthash:/etc/postfix/sender_transport_map` (the current opt-out feature could utilize this instead) +# By recipient (has precedence): `transport_maps = texthash:/etc/postfix/recipient_transport_map` +# +# Support for relaying via port 465 or equivalent requires additional config support (as needed for 465 vs 587 transports extending smtpd) +# - Default relay transport is configured by `relay_transport`, with default transport port configured by `smtp_tcp_port`. +# - The `relay` transport itself extends from `smtp` transport. More than one can be configured with separate settings via `master.cf`. + function _populate_relayhost_map() { # Create the relayhost_map config file: : >/etc/postfix/relayhost_map chown root:root /etc/postfix/relayhost_map chmod 0600 /etc/postfix/relayhost_map - # Matches lines that are not comments or only white-space: - local MATCH_VALID='^\s*[^#[:space:]]' - - # This config is mostly compatible with `/etc/postfix/relayhost_map`, but additionally supports - # not providing a relay host for a sender domain to opt-out of RELAY_HOST? (2nd half of function) - if [[ -f /tmp/docker-mailserver/postfix-relaymap.cf ]]; then - _log 'trace' "Adding relay mappings from postfix-relaymap.cf" - - # Match two values with some white-space between them (eg: `@example.test [relay.service.test]:465`): - local MATCH_VALUE_PAIR='\S*\s+\S' - - # Copy over lines which are not a comment *and* have a destination. - sed -n -r "/${MATCH_VALID}${MATCH_VALUE_PAIR}/p" /tmp/docker-mailserver/postfix-relaymap.cf >>/etc/postfix/relayhost_map - fi - - # Everything below here is to parse `postfix-accounts.cf` and `postfix-virtual.cf`, - # extracting out the domain parts (value of email address after `@`), and then - # adding those as mappings to ENV configured RELAY_HOST for lookup in `/etc/postfix/relayhost_map`. - # Provided `postfix-relaymap.cf` didn't exclude any of the domains, - # and they don't already exist within `/etc/postfix/relayhost_map`. - # - # TODO: Breaking change. Replace this lower half and remove the opt-out feature from `postfix-relaymap.cf`. - # Leverage `main.cf:relayhost` for setting a default relayhost as it was prior to this feature addition. - # Any sender domains or addresses that need to opt-out of that default relay-host can either - # map to a different relay-host, or use a separate transport (needs feature support added). - - # Args: - function _list_domain_parts() { - [[ -f $2 ]] && sed -n -r "/${MATCH_VALID}/ ${1}" "${2}" - } - # Matches and outputs (capture group via `/\1/p`) the domain part (value of address after `@`) in the config file. - local PRINT_DOMAIN_PART_ACCOUNTS='s/^[^@|]*@([^\|]+)\|.*$/\1/p' - local PRINT_DOMAIN_PART_VIRTUAL='s/^\s*[^@[:space:]]*@(\S+)\s.*/\1/p' - - { - _list_domain_parts "${PRINT_DOMAIN_PART_ACCOUNTS}" /tmp/docker-mailserver/postfix-accounts.cf - _list_domain_parts "${PRINT_DOMAIN_PART_VIRTUAL}" /tmp/docker-mailserver/postfix-virtual.cf - } | sort -u | while read -r DOMAIN_PART; do - # DOMAIN_PART not already present in `/etc/postfix/relayhost_map`, and not listed as a relay opt-out domain in `postfix-relaymap.cf` - # `^@${DOMAIN_PART}\b` - To check for existing entry, the `\b` avoids accidental partial matches on similar domain parts. - # `^\s*@${DOMAIN_PART}\s*$` - Matches line with only a domain part (eg: @example.test) to avoid including a mapping for those domains to the RELAY_HOST. - if ! grep -q -e "^@${DOMAIN_PART}\b" /etc/postfix/relayhost_map && ! grep -qs -e "^\s*@${DOMAIN_PART}\s*$" /tmp/docker-mailserver/postfix-relaymap.cf; then - _log 'trace' "Adding relay mapping for ${DOMAIN_PART}" - echo "@${DOMAIN_PART} $(_env_relay_host)" >> /etc/postfix/relayhost_map - fi - done + _multiple_relayhosts + _legacy_support postconf 'sender_dependent_relayhost_maps = texthash:/etc/postfix/relayhost_map' } -function _relayhost_configure_postfix() { - postconf \ - 'smtp_sasl_auth_enable = yes' \ - 'smtp_sasl_security_options = noanonymous' \ - 'smtp_tls_security_level = encrypt' +function _multiple_relayhosts() { + if [[ -f ${DATABASE_RELAYHOSTS} ]]; then + _log 'trace' "Adding relay mappings from ${DATABASE_RELAYHOSTS}" + + # Matches lines that are not comments or only white-space: + local MATCH_VALID='^\s*[^#[:space:]]' + # Match two values with some white-space between them (eg: `@example.test [relay.service.test]:465`): + local MATCH_VALUE_PAIR='\S*\s+\S' + + # Copy over lines which are not a comment *and* have a relay destination. + # Extra condition is due to legacy support (due to opt-out feature), otherwise `_get_valid_lines_from_file()` would be valid. + sed -n -r "/${MATCH_VALID}${MATCH_VALUE_PAIR}/p" "${DATABASE_RELAYHOSTS}" >> /etc/postfix/relayhost_map + fi +} + +# Implicitly force configure all domains DMS manages to be relayed that haven't yet been configured or provided an explicit opt-out. +# This would normally be handled via an opt-in approach, or through `main.cf:relayhost` with an opt-out approach (sender_dependent_default_transport_maps) +function _legacy_support() { + local DATABASE_VHOST='/etc/postfix/vhost' + + # Only relevant when `RELAY_HOST` is configured: + [[ -z ${RELAY_HOST} ]] && return 1 + + # Configures each `SENDER_DOMAIN` to send outbound mail through the default `RELAY_HOST` + `RELAY_PORT` + # (by adding an entry in `/etc/postfix/relayhost_map`) provided it: + # - `/etc/postfix/relayhost_map` doesn't already have it as an existing entry. + # - `postfix-relaymap.cf` has no explicit opt-out (SENDER_DOMAIN key exists, but with no relayhost value assigned) + # + # NOTE: /etc/postfix/vhost represents managed mail domains sourced from `postfix-accounts.cf` and `postfix-virtual.cf`. + while read -r SENDER_DOMAIN; do + local MATCH_EXISTING_ENTRY="^@${SENDER_DOMAIN}\s+" + local MATCH_OPT_OUT_LINE="^\s*@${SENDER_DOMAIN}\s*$" + + # NOTE: `-E` is required for `\s+` syntax to avoid escaping `+` + if ! grep -q -E "${MATCH_EXISTING_ENTRY}" /etc/postfix/relayhost_map && ! grep -qs "${MATCH_OPT_OUT_LINE}" "${DATABASE_RELAYHOSTS}"; then + _log 'trace' "Configuring '${SENDER_DOMAIN}' for the default relayhost '${RELAY_HOST}'" + echo "@${SENDER_DOMAIN} $(_env_relay_host)" >> /etc/postfix/relayhost_map + fi + done < <(_get_valid_lines_from_file "${DATABASE_VHOST}") } function _setup_relayhost() { @@ -177,23 +182,23 @@ function _setup_relayhost() { postconf "relayhost = ${DEFAULT_RELAY_HOST}" fi - if [[ -n ${RELAY_HOST} ]]; then - _log 'trace' "Setting up relay hosts (default: ${RELAY_HOST})" - - _relayhost_sasl - _populate_relayhost_map - - _relayhost_configure_postfix - fi + _process_relayhost_configs } -function _rebuild_relayhost() { - if [[ -n ${RELAY_HOST} ]]; then - _relayhost_sasl - _populate_relayhost_map - fi -} +# Called during initial container setup, or by change detection event: +function _process_relayhost_configs() { + local DATABASE_RELAYHOSTS='/tmp/docker-mailserver/postfix-relaymap.cf' + # One of these must configure a relayhost for the feature to relevant: + if [[ ! -f ${DATABASE_RELAYHOSTS} ]] \ + && [[ -z ${DEFAULT_RELAY_HOST} ]] \ + && [[ -z ${RELAY_HOST} ]]; then + return 1 + fi + + _relayhost_sasl + _populate_relayhost_map +} # # Config examples for reference From d65b2f35a7122ce2a9620c6179e59e540e2ff347 Mon Sep 17 00:00:00 2001 From: Casper Date: Tue, 30 Jan 2024 23:04:42 +0100 Subject: [PATCH 023/267] chore: `CHANGELOG.md` - Add `rsyslog` breaking changes for v14 (#3854) Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1f426a5b..29da7935 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,14 @@ The most noteworthy change of this release is the update of the container's base - DMS `main.cf` has renamed `postscreen_dnsbl_whitelist_threshold` to `postscreen_dnsbl_allowlist_threshold` as part of this change. - `smtpd_relay_restrictions` (relay policy) is now evaluated after `smtpd_recipient_restrictions` (spam policy). Previously it was evaluated before `smtpd_recipient_restrictions`. Mail to be relayed via DMS must now pass through the spam policy first. - The TLS fingerprint policy has changed the default from MD5 to SHA256 (_DMS does not modify this Postfix parameter, but may affect any user customizations that do_). +- **rsyslog:** + - Debian 12 adjusted the `rsyslog` configuration for the default file template from `RSYSLOG_TraditionalFileFormat` to `RSYSLOG_FileFormat` (_upstream default since 2012_). This change may affect you if you have any monitoring / analysis of log output (_eg: `mail.log` / `docker logs`_). + - The two formats are roughly equivalent to [RFC 3164](https://www.rfc-editor.org/rfc/rfc3164)) and [RFC 5424](https://datatracker.ietf.org/doc/html/rfc5424#section-1) respectively. + - A notable difference is the change to [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html#appendix-A) timestamps (_a strict subset of ISO 8601_). The [previous non-standardized timestamp format was defined in RFC 3164](https://www.rfc-editor.org/rfc/rfc3164.html#section-4.1.2) as `Mmm dd hh:mm:ss`. + - To revert this change you can add `sedfile -i '1i module(load="builtin:omfile" template="RSYSLOG_TraditionalFileFormat")' /etc/rsyslog.conf` via [our `user-patches.sh` feature](https://docker-mailserver.github.io/docker-mailserver/v14.0/config/advanced/override-defaults/user-patches/). + - Rsyslog now creates fewer log files: + - The files `/var/log/mail/mail.{info,warn,err}` are no longer created. These files represented `/var/log/mail.log` filtered into separate priority levels. As `/var/log/mail.log` contains all mail related messages, these files (_and their rotated counterparts_) can be deleted safely. + - `/var/log/messages`, `/var/log/debug` and several other log files not relevant to DMS were configured by default by Debian previously. These are not part of the `/var/log/mail/` volume mount, so should not impact anyone. - **Features:** - The relay host feature was refactored ([#3845](https://github.com/docker-mailserver/docker-mailserver/pull/3845)) - The only breaking change this should introduce is with the Change Detection service (`check-for-changes.sh`). From d426f724cd1f2a7474ad46233eea13df5fe5ea42 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Wed, 31 Jan 2024 23:11:19 +1300 Subject: [PATCH 024/267] docs: Complete rewrite of Relay Host pages (#3861) * docs: Complete rewrite on relay host docs - Both relay docs pages have had heavy refactor / rewrite. - ENV docs page relay host section revised. * docs: Revise relay host page with technical details section * docs: Add LDAP compatibility caveat for `RELAY_HOST` --- .../advanced/mail-forwarding/aws-ses.md | 53 ++++-- .../advanced/mail-forwarding/relay-hosts.md | 175 +++++++++++++----- docs/content/config/environment.md | 52 ++++-- docs/mkdocs.yml | 10 +- 4 files changed, 203 insertions(+), 87 deletions(-) diff --git a/docs/content/config/advanced/mail-forwarding/aws-ses.md b/docs/content/config/advanced/mail-forwarding/aws-ses.md index 2a4dab17..00ca2a13 100644 --- a/docs/content/config/advanced/mail-forwarding/aws-ses.md +++ b/docs/content/config/advanced/mail-forwarding/aws-ses.md @@ -2,29 +2,46 @@ title: 'Mail Forwarding | AWS SES' --- -[Amazon SES (Simple Email Service)](https://aws.amazon.com/ses/) is intended to provide a simple way for cloud based applications to send email and receive email. For the purposes of this project only sending email via SES is supported. Older versions of docker-mailserver used `AWS_SES_HOST` and `AWS_SES_USERPASS` to configure sending, this has changed and the setup is managed through [Configure Relay Hosts][docs-relay]. +[Amazon SES (Simple Email Service)][aws-ses] provides a simple way for cloud based applications to send and receive email. -You will need to create some [Amazon SES SMTP credentials](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html). The SMTP credentials you create will be used to populate the `RELAY_USER` and `RELAY_PASSWORD` environment variables. +!!! example "Configuration via ENV" -The `RELAY_HOST` should match your [AWS SES region](https://docs.aws.amazon.com/general/latest/gr/ses.html), the `RELAY_PORT` will be 587. + [Configure a relay host in DMS][docs::relay] to forward all your mail through AWS SES: -If all of your email is being forwarded through AWS SES, `DEFAULT_RELAY_HOST` should be set accordingly. + - `RELAY_HOST` should match your [AWS SES region][aws-ses::region]. + - `RELAY_PORT` should be set to [one of the supported AWS SES SMTP ports][aws-ses::smtp-ports] (_eg: 587 for STARTTLS_). + - `RELAY_USER` and `RELAY_PASSWORD` should be set to your [Amazon SES SMTP credentials][aws-ses::credentials]. -Example: -``` -DEFAULT_RELAY_HOST=[email-smtp.us-west-2.amazonaws.com]:587 -``` + ```env + RELAY_HOST=email-smtp.us-west-2.amazonaws.com + RELAY_PORT=587 + # Alternative to RELAY_HOST + RELAY_PORT which is compatible with LDAP: + DEFAULT_RELAY_HOST=[email-smtp.us-west-2.amazonaws.com]:587 -!!! note - If you set up [AWS Easy DKIM](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-authentication-dkim-easy.html) you can safely skip setting up DKIM as the AWS SES will take care of signing your outgoing email. + RELAY_USER=aws-user + RELAY_PASSWORD=secret + ``` -To verify proper operation, send an email to some external account of yours and inspect the mail headers. You will also see the connection to SES in the mail logs. For example: +!!! tip -```log -May 23 07:09:36 mail postfix/smtp[692]: Trusted TLS connection established to email-smtp.us-east-1.amazonaws.com[107.20.142.169]:25: -TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) -May 23 07:09:36 mail postfix/smtp[692]: 8C82A7E7: to=, relay=email-smtp.us-east-1.amazonaws.com[107.20.142.169]:25, -delay=0.35, delays=0/0.02/0.13/0.2, dsn=2.0.0, status=sent (250 Ok 01000154dc729264-93fdd7ea-f039-43d6-91ed-653e8547867c-000000) -``` + If you have set up [AWS Easy DKIM][aws-ses::easy-dkim], you can safely skip setting up DKIM as AWS SES will take care of signing your outbound mail. -[docs-relay]: ./relay-hosts.md +!!! note "Verify the relay host is configured correctly" + + To verify proper operation, send an email to some external account of yours and inspect the mail headers. + + You will also see the connection to SES in the mail logs: + + ```log + postfix/smtp[692]: Trusted TLS connection established to email-smtp.us-west-1.amazonaws.com[107.20.142.169]:25: + TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) + postfix/smtp[692]: 8C82A7E7: to=, relay=email-smtp.us-west-1.amazonaws.com[107.20.142.169]:25, + delay=0.35, delays=0/0.02/0.13/0.2, dsn=2.0.0, status=sent (250 Ok 01000154dc729264-93fdd7ea-f039-43d6-91ed-653e8547867c-000000) + ``` + +[docs::relay]: ./relay-hosts.md +[aws-ses]: https://aws.amazon.com/ses/ +[aws-ses::credentials]: https://docs.aws.amazon.com/ses/latest/dg/smtp-credentials.html +[aws-ses::smtp-ports]: https://docs.aws.amazon.com/ses/latest/dg/smtp-connect.html +[aws-ses::region]: https://docs.aws.amazon.com/general/latest/gr/ses.html +[aws-ses::easy-dkim]: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-authentication-dkim-easy.html diff --git a/docs/content/config/advanced/mail-forwarding/relay-hosts.md b/docs/content/config/advanced/mail-forwarding/relay-hosts.md index db2528e5..7ef8238c 100644 --- a/docs/content/config/advanced/mail-forwarding/relay-hosts.md +++ b/docs/content/config/advanced/mail-forwarding/relay-hosts.md @@ -2,82 +2,155 @@ title: 'Mail Forwarding | Relay Hosts' --- -## Introduction +## What is a Relay Host? -Rather than having Postfix deliver mail directly, you can configure Postfix to send mail via another mail relay (smarthost). Examples include [Mailgun](https://www.mailgun.com/), [Sendgrid](https://sendgrid.com/) and [AWS SES](https://aws.amazon.com/ses/). +An SMTP relay service (_aka relay host / [smarthost][wikipedia::smarthost]_) is an MTA that relays (_forwards_) mail on behalf of third-parties (_it does not manage the mail domains_). -Depending on the domain of the sender, you may want to send via a different relay, or authenticate in a different way. +- Instead of DMS handling SMTP delivery directly itself (_via Postfix_), it can be configured to delegate delivery by sending all outbound mail through a relay service. +- Examples of popular mail relay services: [AWS SES][smarthost::aws-ses], [Mailgun][smarthost::mailgun], [Mailjet][smarthost::mailjet], [SendGrid][smarthost::sendgrid] -## Basic Configuration +!!! info "When can a relay service can be helpful?" -Basic configuration is done via environment variables: + - Your network provider has blocked outbound connections on port 25 (_required for direct delivery_). + - To improve delivery success via better established reputation (trust) of a relay service. -- `RELAY_HOST`: _default host to relay mail through, `empty` (aka '', or no ENV set) will disable this feature_ -- `RELAY_PORT`: _port on default relay, defaults to port 25_ -- `RELAY_USER`: _username for the default relay_ -- `RELAY_PASSWORD`: _password for the default user_ +## Configuration -Setting these environment variables will cause mail for all sender domains to be routed via the specified host, authenticating with the user/password combination. +All mail sent outbound from DMS (_where the sender address is a DMS account or a virtual alias_) will be relayed through the configured relay host. -!!! warning - For users of the previous `AWS_SES_*` variables: please update your configuration to use these new variables, no other configuration is required. +!!! info "Configuration via ENV" -## Advanced Configuration + Configure the default relayhost with either of these ENV: -### Sender-dependent Authentication + - Preferable (_LDAP compatible_): `DEFAULT_RELAY_HOST` (eg: `[mail.relay-service.com]:25`) + - `RELAY_HOST` (eg: `mail.relay-service.com`) + `RELAY_PORT` (default: 25) -Sender dependent authentication is done in `docker-data/dms/config/postfix-sasl-password.cf`. You can create this file manually, or use: + Most relay services also require authentication configured: -```sh -setup.sh relay add-auth [] -``` + - `RELAY_USER` + `RELAY_PASSWORD` provides credentials for authenticating with the default relayhost. -An example configuration file looks like this: + !!! warning "Providing secrets via ENV" -```txt -@domain1.com relay_user_1:password_1 -@domain2.com relay_user_2:password_2 -``` + While ENV is convenient, the risk of exposing secrets is higher. -If there is no other configuration, this will cause Postfix to deliver email through the relay specified in `RELAY_HOST` env variable, authenticating as `relay_user_1` when sent from `domain1.com` and authenticating as `relay_user_2` when sending from `domain2.com`. + `setup relay add-auth` is a better alternative, which manages the credentials via a config file. -!!! note - To activate the configuration you must either restart the container, or you can also trigger an update by modifying a mail account. +??? tip "Excluding specific sender domains from relay" -### Sender-dependent Relay Host + You can opt-out with: `setup relay exclude-domain ` -Sender dependent relay hosts are configured in `docker-data/dms/config/postfix-relaymap.cf`. You can create this file manually, or use: + Outbound mail from senders of that domain will be sent normally (_instead of through the configured `RELAY_HOST`_). -```sh -setup.sh relay add-domain [] -``` + !!! warning "When any relay host credentials are configured" -An example configuration file looks like this: + It will still be expected that mail is sent over a secure connection with credentials provided. -```txt -@domain1.com [relay1.org]:587 -@domain2.com [relay2.org]:2525 -``` + Thus this opt-out feature is rarely practical. -Combined with the previous configuration in `docker-data/dms/config/postfix-sasl-password.cf`, this will cause Postfix to deliver mail sent from `domain1.com` via `relay1.org:587`, authenticating as `relay_user_1`, and mail sent from `domain2.com` via `relay2.org:2525` authenticating as `relay_user_2`. +### Advanced Configuration -!!! note - You still have to define `RELAY_HOST` to activate the feature +When mail is sent, there is support to change the relay service or the credentials configured based on the sender address domain used. -### Excluding Sender Domains +We provide this support via two config files: -If you want mail sent from some domains to be delivered directly, you can exclude them from being delivered via the default relay by adding them to `docker-data/dms/config/postfix-relaymap.cf` with no destination. You can also do this via: +- Sender-dependent Relay Host: `docker-data/dms/config/postfix-relaymap.cf` +- Sender-dependent Authentication: `docker-data/dms/config/postfix-sasl-password.cf` -```sh -setup.sh relay exclude-domain -``` +!!! tip "Configure with our `setup relay` commands" -Extending the configuration file from above: + While you can edit those configs directly, DMS provides these helpful config management commands: -```txt -@domain1.com [relay1.org]:587 -@domain2.com [relay2.org]:2525 -@domain3.com -``` + ```cli-syntax + # Configure a sender domain to use a specific relay host: + setup relay add-domain [] -This will cause email sent from `domain3.com` to be delivered directly. + # Configure relay host credentials for a sender domain to use: + setup relay add-auth [] + + # Optionally avoid relaying from senders of this domain: + # NOTE: Only supported when configured with the `RELAY_HOST` ENV! + setup relay exclude-domain + ``` + +!!! example "Config file: `postfix-sasl-password.cf`" + + ```cf-extra title="docker-data/dms/config/postfix-sasl-password.cf" + @domain1.com mailgun-user:secret + @domain2.com sendgrid-user:secret + + # NOTE: This must have an exact match with the relay host in `postfix-relaymap.cf`, + # `/etc/postfix/relayhost_map`, or the `DEFAULT_RELAY_HOST` ENV. + # NOTE: Not supported via our setup CLI, but valid config for Postfix. + [email-smtp.us-west-2.amazonaws.com]:2587 aws-user:secret + ``` + + When Postfix needs to lookup credentials for mail sent outbound, the above config will: + + - Authenticate as `mailgun-user` for mail sent with a sender belonging to `@domain1.com` + - Authenticate as `sendgrid-user` for mail sent with a sender belonging to `@domain2.com` + - Authenticate as `aws-user` for mail sent through a configured AWS SES relay host (any sender domain). + +!!! example "Config file: `postfix-relaymap.cf`" + + ```cf-extra title="docker-data/dms/config/postfix-relaymap.cf" + @domain1.com [smtp.mailgun.org]:587 + @domain2.com [smtp.sendgrid.net]:2525 + + # Opt-out of relaying: + @domain3.com + ``` + + When Postfix sends mail outbound from these sender domains, the above config will: + + - Relay mail through `[smtp.mailgun.org]:587` when mail is sent from a sender of `@domain1.com` + - Relay mail through `[smtp.sendgrid.net]:2525` when mail is sent from a sender of `@domain1.com` + - Mail with a sender from `@domain3.com` is not sent through a relay (_**Only applicable** when using `RELAY_HOST`_) + +### Technical Details + +- Both the supported ENV and config files for this feature have additional details covered in our ENV docs [Relay Host section][docs::env-relay]. +- For troubleshooting, a [minimal `compose.yaml` config with several DMS instances][dms-gh::relay-example] demonstrates this feature for local testing. +- [Subscribe to this tracking issue][dms-gh::pr-3607] for future improvements intended for this feature. + +!!! abstract "Postfix Settings" + + Internally this feature is implemented in DMS by [`relay.sh`][dms-repo::helpers-relay]. + + The `relay.sh` script manages configuring these Postfix settings: + + ```cf-extra + # Send all outbound mail through this relay service: + relayhost = [smtp.relay-service.com]:587 + + # Credentials to use: + smtp_sasl_password_maps = texthash:/etc/postfix/sasl_passwd + # Alternative table type examples which do not require a separate file: + #smtp_sasl_password_maps = static:john.doe@relay-service.com:secret + #smtp_sasl_password_maps = inline:{ [smtp.relay-service.com]:587=john.doe@relay-service.com:secret } + + ## Authentication support: + # Required to provide credentials to the relay service: + smtp_sasl_auth_enable = yes + # Enforces requiring credentials when sending mail outbound: + smtp_sasl_security_options = noanonymous + # Enforces a secure connection (TLS required) to the relay service: + smtp_tls_security_level = encrypt + + ## Support for advanced requirements: + # Relay service(s) to use instead of direct delivery for specific sender domains: + sender_dependent_relayhost_maps = texthash:/etc/postfix/relayhost_map + # Support credentials to a relay service(s) that vary by relay host used or sender domain: + smtp_sender_dependent_authentication = yes + ``` + + +[smarthost::mailgun]: https://www.mailgun.com/ +[smarthost::mailjet]: https://www.mailjet.com +[smarthost::sendgrid]: https://sendgrid.com/ +[smarthost::aws-ses]: https://aws.amazon.com/ses/ +[wikipedia::smarthost]: https://en.wikipedia.org/wiki/Smart_host + +[docs::env-relay]: ../../environment.md#relay-host +[dms-repo::helpers-relay]: https://github.com/docker-mailserver/docker-mailserver/blob/v14.0.0/target/scripts/helpers/relay.sh +[dms-gh::pr-3607]: https://github.com/docker-mailserver/docker-mailserver/issues/3607 +[dms-gh::relay-example]: https://github.com/docker-mailserver/docker-mailserver/issues/3842#issuecomment-1913380639 diff --git a/docs/content/config/environment.md b/docs/content/config/environment.md index 5db1c2c3..d04da022 100644 --- a/docs/content/config/environment.md +++ b/docs/content/config/environment.md @@ -1014,13 +1014,19 @@ you to replace both instead of just the envelope sender. #### Relay Host -!!! tip "`RELAY_HOST` vs `DEFAULT_RELAY_HOST`" +Supported ENV for the [Relay Host][docs::relay-host] feature. - `DEFAULT_RELAY_HOST` is encouraged, but presently does not support sender domain opt-out (`setup relay exclude-domain`). +!!! note "Prefer `DEFAULT_RELAY_HOST` instead of `RELAY_HOST`" + + This is advised unless you need support for sender domain opt-out (via `setup relay exclude-domain`). + + The implementation for `RELAY_HOST` is not compatible with LDAP. !!! tip "Opt-in for relay host support" - If you only want to enable relay for specific sender domains, use can use opt-in via `setup relay add-domain`. + Enable relaying only for specific sender domains instead by using `setup relay add-domain`. + + **NOTE:** Presently there is a caveat when relay host credentials are configured (_which is incompatible with opt-in_). ##### DEFAULT_RELAY_HOST @@ -1033,22 +1039,24 @@ Configures a default relay host. !!! abstract "Technical Details" - Configures the Postfix `main.cf` setting: [`relayhost`][postfix-config::relayhost] + This ENV internally configures the Postfix `main.cf` setting: [`relayhost`][postfix-config::relayhost] ##### RELAY_HOST Configures a default relay host. -!!! info - - - This is a legacy ENV. It is however required for the opt-out feature of `postfix-relaymap.cf` to work. - - When configured, all known mail domains managed by DMS will be configured to relay outbound mail, just like `DEFAULT_RELAY_HOST`. - !!! note Expects a value like `mail.example.com`. Internally this will be wrapped to `[mail.example.com]`, so it should resolve to the MTA directly. - Do not use with `DEFAULT_RELAY_HOST`. `RELAY_HOST` has precedence as it is configured with `sender_dependent_relayhost_maps`. + !!! warning "Do not use with `DEFAULT_RELAY_HOST`" + + `RELAY_HOST` has precedence as it is configured with `sender_dependent_relayhost_maps`. + +!!! info + + - This is a legacy ENV. It is however required for the opt-out feature of `postfix-relaymap.cf` to work. + - Internal configuration however differs from `DEFAULT_RELAY_HOST`. !!! abstract "Technical Details" @@ -1057,6 +1065,8 @@ Configures a default relay host. - Postfix setting with config: [`sender_dependent_relayhost_maps = texthash:/etc/postfix/relayhost_map`][postfix-config::relayhost_maps] - DMS Config volume support via: `postfix-relaymap.cf` (_generates `/etc/postfix/relayhost_map`_) + All known mail domains managed by DMS will be configured to relay outbound mail to `RELAY_HOST` by adding them implicitly to `/etc/postfix/relayhost_map`, except for domains using the opt-out feature of `postfix-relaymap.cf`. + ##### RELAY_PORT Default => 25 @@ -1069,10 +1079,10 @@ Support for configuring a different port than 25 for `RELAY_HOST` to use. #### Relay Host Credentials -!!! warning "Configuring relay host credentials make outbound authentication mandatory" +!!! warning "Configuring relay host credentials enforces outbound authentication" Presently when `RELAY_USER` + `RELAY_PASSWORD` or `postfix-sasl-password.cf` are configured, all outbound mail traffic is configured to require a secure connection established and forbids the omission of credentials. - + Additional feature work is required to only enforce these requirements on mail sent through a configured relay host. ##### RELAY_USER @@ -1083,10 +1093,10 @@ Provide the credentials to use with `RELAY_HOST` or `DEFAULT_RELAY_HOST`. !!! tip "Alternative credentials config" - You may prefer to use `setup relay add-auth` to avoid exposure of secrets in ENV. + You may prefer to use `setup relay add-auth` to avoid risking ENV exposing secrets. - - With the CLI command you must provide each sender domain relay credentials. - - Alternatively manually edit `postfix-sasl-password.cf` with the correct relayhost entry (_`DEFAULT_RELAY_HOST` value or as defined in `/etc/postfix/relayhost_map`_) to provide credentials per relayhost configured. + - With the CLI command, you must provide relay credentials for each of your sender domains. + - Alternatively manually edit `postfix-sasl-password.cf` with the correct relayhost entry (_`DEFAULT_RELAY_HOST` value, or as defined in `/etc/postfix/relayhost_map`_) to provide credentials per relayhost configured. !!! abstract "Technical Details" @@ -1095,7 +1105,16 @@ Provide the credentials to use with `RELAY_HOST` or `DEFAULT_RELAY_HOST`. - Postfix setting with config: [`smtp_sasl_password_maps = texthash:/etc/postfix/sasl_passwd`][postfix-config::sasl_passwd] - DMS Config volume support via: `postfix-sasl-password.cf` (_generates `/etc/postfix/sasl_passwd`_) - This file has relay hosts that must match the `host:port` of `/etc/postfix/relayhost_map` or `main.cf:relayhost`. DMS support handles this for you. + --- + + When `postfix-sasl-password.cf` is present, DMS will copy it internally to `/etc/postfix/sasl_passwd`. + + - DMS provides support for mapping credentials by sender domain: + - Explicitly via `setup relay add-auth` (_creates / updates `postfix-sasl-password.cf`_). + - Implicitly via the relay ENV support (_configures all known DMS managed domains to use the relay ENV_). + - Credentials can be explicitly configured for specific relay hosts instead of sender domains: + - Add the exact relayhost value (`host:port` / `[host]:port`) from the generated `/etc/postfix/relayhost_map`, or `main.cf:relayhost` (`DEFAULT_RELAY_HOST`). + - `setup relay ...` is missing support, you must instead add these manually to `postfix-sasl-password.cf`. [docs-rspamd]: ./security/rspamd.md [docs-tls]: ./security/ssl.md @@ -1103,6 +1122,7 @@ Provide the credentials to use with `RELAY_HOST` or `DEFAULT_RELAY_HOST`. [docs-tls-manual]: ./security/ssl.md#bring-your-own-certificates [docs-tls-selfsigned]: ./security/ssl.md#self-signed-certificates [docs-accounts-quota]: ./user-management.md#quotas +[docs::relay-host]: ./advanced/mail-forwarding/relay-hosts.md [docs::dms-volumes-state]: ./advanced/optional-config.md#volumes-state [postfix-config::relayhost]: https://www.postfix.org/postconf.5.html#relayhost [postfix-config::relayhost_maps]: https://www.postfix.org/postconf.5.html#sender_dependent_relayhost_maps diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 15d2c3b2..84366423 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -89,6 +89,7 @@ markdown_extensions: emoji_index: !!python/name:material.extensions.emoji.twemoji emoji_generator: !!python/name:material.extensions.emoji.to_svg - pymdownx.highlight: + # Configures an alias (name) to a supported syntax (lang): extend_pygments_lang: - name: yml lang: yaml @@ -98,8 +99,13 @@ markdown_extensions: lang: cfg - name: env lang: properties - # Not helpful with Python Pygments lexer highlighting, but we might change to a JS highlighter in future - # Ideally, this type of codefence might also have word-wrap enabled (CSS: {white-space: pre-wrap}) + # A variant that sometimes has nicer syntax highlighting: + - name: cf-extra + lang: linuxconfig + - name: cli-syntax + lang: linuxconfig + # These formats aren't supported by Python Pygments lexer, + # but we use them when the context is appropriate. - name: log lang: shell-session - name: fetchmailrc From 05fbcf68890bbf8e3acd3214457af8f55f9d125e Mon Sep 17 00:00:00 2001 From: Aaron Spettl Date: Wed, 31 Jan 2024 11:50:58 +0100 Subject: [PATCH 025/267] fix(rspamd): Add missing comma to `local_networks` setting (#3862) Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- CHANGELOG.md | 1 + target/rspamd/local.d/options.inc | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 29da7935..d80bbf8a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -79,6 +79,7 @@ The most noteworthy change of this release is the update of the container's base - `DEFAULT_RELAY_HOST` ENV can now also use the `RELAY_USER` + `RELAY_PASSWORD` ENV for supplying credentials. - `RELAY_HOST` ENV no longer enforces configuring outbound SMTP to require credentials. Like `DEFAULT_RELAY_HOST` it can now configure a relay where credentials are optional. - Restarting DMS should not be required when configuring relay hosts without these ENV, but solely via `setup relay ...`, as change detection events now apply relevant Postfix setting changes for supporting credentials too. +- Rspamd configuration: Add a missing comma in `local_networks` so that all internal IP addresses are actually considered as internal ([#3862](https://github.com/docker-mailserver/docker-mailserver/pull/3862)) ## [v13.3.1](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v13.3.1) diff --git a/target/rspamd/local.d/options.inc b/target/rspamd/local.d/options.inc index 8755cfad..8d67fe74 100644 --- a/target/rspamd/local.d/options.inc +++ b/target/rspamd/local.d/options.inc @@ -1,3 +1,3 @@ pidfile = false; soft_reject_on_timeout = true; -local_networks = "127.0.0.1/8, 10.0.0.0/8, 172.16.0.0/12 192.168.0.0/16"; +local_networks = "127.0.0.1/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16"; From 45935f5fb81733ddf7a6afd1503e5b8fce30ef89 Mon Sep 17 00:00:00 2001 From: Hans-Cees Speel Date: Thu, 1 Feb 2024 17:34:33 +0100 Subject: [PATCH 026/267] rspamd: add neural module config (#3833) Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- CHANGELOG.md | 1 + docs/content/config/environment.md | 11 +++++ mailserver.env | 6 +++ target/rspamd/local.d/neural.conf | 42 +++++++++++++++++++ target/rspamd/local.d/neural_group.conf | 26 ++++++++++++ .../startup/setup.d/security/rspamd.sh | 19 ++++++++- target/scripts/startup/variables-stack.sh | 1 + 7 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 target/rspamd/local.d/neural.conf create mode 100644 target/rspamd/local.d/neural_group.conf diff --git a/CHANGELOG.md b/CHANGELOG.md index d80bbf8a..09afaba4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -70,6 +70,7 @@ The most noteworthy change of this release is the update of the container's base - Refactored helper methods for sending e-mails with specific `Message-ID` headers and the helpers for retrieving + filtering logs, which together help isolate logs relevant to specific mail when multiple mails have been processed within a single test. ([#3786](https://github.com/docker-mailserver/docker-mailserver/pull/3786)) - **Rspamd**: - The `rewrite_subject` action, is now disabled by default. It has been replaced with the new `SPAM_SUBJECT` environment variable, which implements the functionality via a Sieve script instead in favor of being anti-spam service agnostic ([3820](https://github.com/docker-mailserver/docker-mailserver/pull/3820)) + - `RSPAMD_NEURAL` was added and is disabled by default. If switched on it wil enable the experimental Rspamd Neural network module to add a layer of analysis to spam detection using neural network technology. ([3833](https://github.com/docker-mailserver/docker-mailserver/pull/3833)) ### Fixes diff --git a/docs/content/config/environment.md b/docs/content/config/environment.md index d04da022..323b6321 100644 --- a/docs/content/config/environment.md +++ b/docs/content/config/environment.md @@ -451,6 +451,17 @@ Can be used to control the score when the [`HFILTER_HOSTNAME_UNKNOWN` symbol](#r Default: 6 (which corresponds to the `add_header` action) + +##### RSPAMD_NEURAL + +Can be used to enable or disable the [Neural network module][rspamd-docs-neural-network]. This is an experimental anti-spam weigh method using three neuaral networks in the configuration added here. As far as we can tell it trains itsself by using other modules to find out what spam is. It will take a while (a week or more) to train its first neural network. The config trains new networks all the time and discards of old networks. +Since it is experimental it is switched of by default. + +- **0** => Disabled +- 1 => Enabled + +[rspamd-docs-neural-network]: https://www.rspamd.com/doc/modules/neural.html + #### Reports ##### PFLOGSUMM_TRIGGER diff --git a/mailserver.env b/mailserver.env index 6fc9c7c2..7f493a3a 100644 --- a/mailserver.env +++ b/mailserver.env @@ -183,6 +183,12 @@ RSPAMD_HFILTER=1 # Default: 6 RSPAMD_HFILTER_HOSTNAME_UNKNOWN_SCORE=6 +# Can be used to enable or disable the (still experimental) neural module. +# +# - **0** => Disabled +# - 1 => Enabled +RSPAMD_NEURAL=0 + # Amavis content filter (used for ClamAV & SpamAssassin) # 0 => Disabled # 1 => Enabled diff --git a/target/rspamd/local.d/neural.conf b/target/rspamd/local.d/neural.conf new file mode 100644 index 00000000..bfc22bef --- /dev/null +++ b/target/rspamd/local.d/neural.conf @@ -0,0 +1,42 @@ +#https://github.com/rspamd/rspamd/issues/3099 +rules { + "NEURAL_WEEK_1000" { + train { + max_trains = 1000; + max_usages = 50; + max_iterations = 25; + learning_rate = 0.01, + spam_score = 8; + ham_score = -2; + } + symbol_spam = "NEURAL_WEEK_SPAM"; + symbol_ham = "NEURAL_WEEK_HAM"; + ann_expire = 300d; + } + "NEURAL_DAYS_200" { + train { + max_trains = 200; + max_usages = 10; + max_iterations = 25; + learning_rate = 0.01, + spam_score = 8; + ham_score = -2; + } + symbol_spam = "NEURAL_DAYS_SPAM"; + symbol_ham = "NEURAL_DAYS_HAM"; + ann_expire = 100d; + } + "NEURAL_HALF_DAY_50" { + train { + max_trains = 50; + max_usages = 4; + max_iterations = 25; + learning_rate = 0.01, + spam_score = 8; + ham_score = -2; + } + symbol_spam = "NEURAL_HALF_DAY_SPAM"; + symbol_ham = "NEURAL_HALF_DAY_HAM"; + ann_expire = 13d; + } +} diff --git a/target/rspamd/local.d/neural_group.conf b/target/rspamd/local.d/neural_group.conf new file mode 100644 index 00000000..90ba6871 --- /dev/null +++ b/target/rspamd/local.d/neural_group.conf @@ -0,0 +1,26 @@ +symbols = { + "NEURAL_WEEK_SPAM" { + weight = 3.0; # sample weight + description = "Neural network spam (long)"; + } + "NEURAL_WEEK_HAM" { + weight = -3.0; # sample weight + description = "Neural network ham (long)"; + } + "NEURAL_DAYS_SPAM" { + weight = 2.5; # sample weight + description = "Neural network spam (medium)"; + } + "NEURAL_DAYS_HAM" { + weight = -1.5; # sample weight + description = "Neural network ham (medium)"; + } + "NEURAL_HALF_DAY_SPAM" { + weight = 2.0; # sample weight + description = "Neural network spam (short)"; + } + "NEURAL_HALF_DAY_HAM" { + weight = -1.0; # sample weight + description = "Neural network ham (short)"; + } +} diff --git a/target/scripts/startup/setup.d/security/rspamd.sh b/target/scripts/startup/setup.d/security/rspamd.sh index a05a798e..98f83de6 100644 --- a/target/scripts/startup/setup.d/security/rspamd.sh +++ b/target/scripts/startup/setup.d/security/rspamd.sh @@ -20,6 +20,7 @@ function _setup_rspamd() { __rspamd__setup_learning __rspamd__setup_greylisting __rspamd__setup_hfilter_group + __rspamd__setup_neural __rspamd__setup_check_authenticated _rspamd_handle_user_modules_adjustments # must run last @@ -186,7 +187,6 @@ function __rspamd__setup_default_modules() { local DISABLE_MODULES=( clickhouse elastic - neural reputation spamassassin url_redirector @@ -283,6 +283,23 @@ function __rspamd__setup_hfilter_group() { fi } + +# This function handles setup of the neural module (see +# https://www.rspamd.com/doc/modules/neural.html). This module is experimental +# but can enhance anti-spam scoring possibly. +function __rspamd__setup_neural() { + if _env_var_expect_zero_or_one 'RSPAMD_NEURAL' && [[ ${RSPAMD_NEURAL} -eq 1 ]]; then + __rspamd__log 'debug' 'Enabling Neural module' + __rspamd__log 'warn' 'The Neural module is still experimental (in Rspamd) and hence not tested in DMS' + else + __rspamd__log 'debug' 'Neural module is disabled' + rm -f "${RSPAMD_LOCAL_D}/neural.conf" + rm -f "${RSPAMD_LOCAL_D}/neural_group.conf" + __rspamd__helper__enable_disable_module 'neural' 'false' + fi +} + + # If 'RSPAMD_CHECK_AUTHENTICATED' is enabled, then content checks for all users, i.e. # also for authenticated users, are performed. # diff --git a/target/scripts/startup/variables-stack.sh b/target/scripts/startup/variables-stack.sh index 219ce6c8..eb5bf149 100644 --- a/target/scripts/startup/variables-stack.sh +++ b/target/scripts/startup/variables-stack.sh @@ -71,6 +71,7 @@ function __environment_variables_general_setup() { VARS[RSPAMD_GREYLISTING]="${RSPAMD_GREYLISTING:=0}" VARS[RSPAMD_HFILTER]="${RSPAMD_HFILTER:=1}" VARS[RSPAMD_HFILTER_HOSTNAME_UNKNOWN_SCORE]="${RSPAMD_HFILTER_HOSTNAME_UNKNOWN_SCORE:=6}" + VARS[RSPAMD_NEURAL]="${RSPAMD_NEURAL:=0}" VARS[RSPAMD_LEARN]="${RSPAMD_LEARN:=0}" VARS[SA_KILL]=${SA_KILL:="10.0"} VARS[SPAM_SUBJECT]=${SPAM_SUBJECT:=} From db661bf3ac4a8b61bd92fbeb45466d7ab62af42a Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Fri, 2 Feb 2024 18:38:22 +0100 Subject: [PATCH 027/267] docs: misc improvements (but mostly related to Rspamd) (#3858) * remove leftover statement on `/etc/os-release` * update wording on the PR template * add section about other services to Rspamd docs * remove more outdated information from Rspamd docs * moved links and minor rewording in Rspamd docs --------- Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- .github/pull_request_template.md | 6 +- CHANGELOG.md | 1 - docs/content/config/security/rspamd.md | 111 +++++++++++++------------ 3 files changed, 60 insertions(+), 58 deletions(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 040d67dd..104f9433 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -18,12 +18,12 @@ Fixes # - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] This change requires a documentation update -## Checklist: +## Checklist - [ ] My code follows the style guidelines of this project -- [ ] I have performed a self-review of my own code +- [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (README.md or the documentation under `docs/`) -- [ ] If necessary I have added tests that prove my fix is effective or that my feature works +- [ ] If necessary, I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] **I have added information about changes made in this PR to `CHANGELOG.md`** diff --git a/CHANGELOG.md b/CHANGELOG.md index 09afaba4..7918d810 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,7 +21,6 @@ The most noteworthy change of this release is the update of the container's base - Notable minor version bump: `postfix 3.5.23 => 3.7.9` - Notable minor version bump + downgrade: `dovecot 2.3.13 => 2.3.19` (_Previous release provided `2.3.21` via community repo, `2.3.19` is now the default_) - Updates to `packages.sh`: - - The script now uses `/etc/os-release` to determine the release name of Debian - Removed custom installations of Fail2Ban, getmail6 and Rspamd - Updated packages lists and added comments for maintainability - OpenDMARC upgrade: `v1.4.0` => `v1.4.2` ([#3841](https://github.com/docker-mailserver/docker-mailserver/pull/3841)) diff --git a/docs/content/config/security/rspamd.md b/docs/content/config/security/rspamd.md index 4c466c5d..6dc5f202 100644 --- a/docs/content/config/security/rspamd.md +++ b/docs/content/config/security/rspamd.md @@ -4,12 +4,9 @@ title: 'Security | Rspamd' ## About -Rspamd is a ["fast, free and open-source spam filtering system"][rspamd-homepage]. DMS integrates Rspamd like any other service. We provide a very simple but easy to maintain setup of Rspamd. +Rspamd is a ["fast, free and open-source spam filtering system"][www::rspamd-homepage]. DMS integrates Rspamd like any other service. We provide a basic but easy to maintain setup of Rspamd. -If you want to have a look at the default configuration files for Rspamd that DMS packs, navigate to [`target/rspamd/` inside the repository][dms-default-configuration]. Please consult the [section "The Default Configuration"](#the-default-configuration) section down below for a written overview. - -[rspamd-homepage]: https://rspamd.com/ -[dms-default-configuration]: https://github.com/docker-mailserver/docker-mailserver/tree/master/target/rspamd +If you want to take a look at the default configuration files for Rspamd that DMS packs, navigate to [`target/rspamd/` inside the repository][repo::dms-default-configuration]. Please consult the [section "The Default Configuration"](#the-default-configuration) section down below for a written overview. ## Related Environment Variables @@ -23,15 +20,17 @@ The following environment variables are related to Rspamd: 6. [`RSPAMD_HFILTER_HOSTNAME_UNKNOWN_SCORE`](../environment.md#rspamd_hfilter_hostname_unknown_score) 7. [`RSPAMD_LEARN`](../environment.md#rspamd_learn) 8. [`SPAM_SUBJECT`](../environment.md#spam_subject) -9. [`MOVE_SPAM_TO_JUNK`][docs-spam-to-junk] +9. [`MOVE_SPAM_TO_JUNK`][docs::spam-to-junk] 10. [`MARK_SPAM_AS_READ`](../environment.md#mark_spam_as_read) With these variables, you can enable Rspamd itself, and you can enable / disable certain features related to Rspamd. -[docs-spam-to-junk]: ../environment.md#move_spam_to_junk - ## The Default Configuration +### Other Anti-Spam-Services + +DMS packs other anti-spam services, like SpamAssassin or Amavis, next to Rspamd. There exist services, like ClamAV (`ENABLE_CLAMAV`), that Rspamd can utilize to improve the scanning. Except for ClamAV, we recommend disabling **all other** anti-spam services when using Rspamd. The [basic configuration shown below](#a-very-basic-configuration) provides a good starting point. + ### Mode of Operation !!! tip "Attention" @@ -40,12 +39,12 @@ With these variables, you can enable Rspamd itself, and you can enable / disable Rspamd is integrated as a milter into DMS. When enabled, Postfix's `main.cf` configuration file includes the parameter `rspamd_milter = inet:localhost:11332`, which is added to `smtpd_milters`. As a milter, Rspamd can inspect incoming and outgoing e-mails. -Each mail is assigned what Rspamd calls symbols: when an e-mail matches a specific criterion, the mail receives a symbol. Afterwards, Rspamd applies a _spam score_ (as usual with anti-spam software) to the e-mail. +Each mail is assigned what Rspamd calls symbols: when an e-mail matches a specific criterion, the e-mail receives a symbol. Afterward, Rspamd applies a _spam score_ (as usual with anti-spam software) to the e-mail. - The score itself is calculated by adding the values of the individual symbols applied earlier. The higher the spam score is, the more likely the e-mail is spam. -- Symbol values can be negative (i.e., these symbols indicate the mail is legitimate, maybe because [SPF and DKIM][docs-dkim-dmarc-spf] are verified successfully) or the symbol can be positive (i.e., these symbols indicate the e-mail is spam, maybe because the e-mail contains a lot of links). +- Symbol values can be negative (i.e., these symbols indicate the mail is legitimate, maybe because [SPF and DKIM][docs::dkim-dmarc-spf] are verified successfully). On the other hand, symbol scores can be positive (i.e., these symbols indicate the e-mail is spam, perhaps because the e-mail contains numerous links). -Rspamd then adds (a few) headers to the e-mail based on the spam score. Most important are `X-Spamd-Result`, which contains an overview of which symbols were applied. It could look like this: +Rspamd then adds (a few) headers to the e-mail based on the spam score. Most important is `X-Spamd-Result`, which contains an overview of which symbols were applied. It could look like this: ```txt X-Spamd-Result default: False [-2.80 / 11.00]; R_SPF_NA(1.50)[no SPF record]; R_DKIM_ALLOW(-1.00)[example.com:s=dtag1]; DWL_DNSWL_LOW(-1.00)[example.com:dkim]; RWL_AMI_LASTHOP(-1.00)[192.0.2.42:from]; DMARC_POLICY_ALLOW(-1.00)[example.com,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[192.0.2.42:from]; FORGED_SENDER(0.30)[noreply@example.com,some-reply-address@bounce.example.com]; RCVD_IN_DNSWL_LOW(-0.10)[192.0.2.42:from]; MIME_GOOD(-0.10)[multipart/mixed,multipart/related,multipart/alternative,text/plain]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:+,4:~,5:~,6:~]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; REPLYTO_DN_EQ_FROM_DN(0.00)[]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[example.com:+]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; FROM_NEQ_ENVFROM(0.00)[noreply@example.com,some-reply-address@bounce.example.com]; FROM_HAS_DN(0.00)[]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[receiver@anotherexample.com]; ASN(0.00)[asn:3320, ipnet:192.0.2.0/24, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; MISSING_XM_UA(0.00)[]; HAS_REPLYTO(0.00)[some-reply-address@dms-reply.example.com] @@ -57,37 +56,30 @@ And then there is a corresponding `X-Rspamd-Action` header, which shows the over X-Rspamd-Action no action ``` -Since the score is `-2.80`, nothing will happen and the e-mail is not classified as spam. Our custom [`actions.conf`][rspamd-actions-config] defines what to do at certain scores: +Since the score is `-2.80`, nothing will happen and the e-mail is not classified as spam. Our custom [`actions.conf`][www::rspamd-actions-config] defines what to do at certain scores: 1. At a score of 4, the e-mail is to be _greylisted_; 2. At a score of 6, the e-mail is _marked with a header_ (`X-Spam: Yes`); -3. At a score of 7, the e-mail will additionally have their _subject re-written_ (appending a prefix like `[SPAM]`); -4. At a score of 11, the e-mail is outright _rejected_. +3. At a score of 11, the e-mail is outright _rejected_. --- -There is more to spam analysis than meets the eye: we have not covered the [Bayes training and filters][rspamc-docs-bayes] here, nor have we talked about [Sieve rules for e-mails that are marked as spam][docs-spam-to-junk]. +There is more to spam analysis than meets the eye: we have not covered the [Bayes training and filters][www::rspamd-docs-bayes] here, nor have we discussed [Sieve rules for e-mails that are marked as spam][docs::spam-to-junk]. Even the calculation of the score with the individual symbols has been presented to you in a simplified manner. But with the knowledge from above, you're equipped to read on and use Rspamd confidently. Keep on reading to understand the integration even better - you will want to know about your anti-spam software, not only to keep the bad e-mail out, but also to make sure the good e-mail arrive properly! -[docs-dkim-dmarc-spf]: ../best-practices/dkim_dmarc_spf.md -[rspamd-actions-config]: https://github.com/docker-mailserver/docker-mailserver/blob/master/target/rspamd/local.d/actions.conf -[rspamc-docs-bayes]: https://rspamd.com/doc/configuration/statistic.html - ### Workers -The proxy worker operates in [self-scan mode][rspamd-docs-proxy-self-scan-mode]. This simplifies the setup as we do not require a normal worker. You can easily change this though by [overriding the configuration by DMS](#providing-custom-settings-overriding-settings). +The proxy worker operates in [self-scan mode][www::rspamd-docs-proxy-self-scan-mode]. This simplifies the setup as we do not require a normal worker. You can easily change this though by [overriding the configuration by DMS](#providing-custom-settings-overriding-settings). DMS does not set a default password for the controller worker. You may want to do that yourself. In setups where you already have an authentication provider in front of the Rspamd webpage, you may want to [set the `secure_ip ` option to `"0.0.0.0/0"` for the controller worker](#with-the-help-of-a-custom-file) to disable password authentication inside Rspamd completely. -[rspamd-docs-proxy-self-scan-mode]: https://rspamd.com/doc/workers/rspamd_proxy.html#self-scan-mode - ### Persistence with Redis When Rspamd is enabled, we implicitly also start an instance of Redis in the container: - Redis is configured to persist its data via RDB snapshots to disk in the directory `/var/lib/redis` (_or the [`/var/mail-state/`][docs::dms-volumes-state] volume when present_). -- With the volume mount the snapshot will restore the Redis data across container restarts, and provide a way to keep backup. +- With the volume mount, the snapshot will restore the Redis data across container restarts, and provide a way to keep backup. Redis uses `/etc/redis/redis.conf` for configuration: @@ -96,15 +88,13 @@ Redis uses `/etc/redis/redis.conf` for configuration: ### Web Interface -Rspamd provides a [web interface][rspamc-docs-web-interface], which contains statistics and data Rspamd collects. The interface is enabled by default and reachable on port 11334. +Rspamd provides a [web interface][www::rspamd-docs-web-interface], which contains statistics and data Rspamd collects. The interface is enabled by default and reachable on port 11334. ![Rspamd Web Interface](https://rspamd.com/img/webui.png) -[rspamc-docs-web-interface]: https://rspamd.com/webui/ - ### DNS -DMS does not supply custom values for DNS servers to Rspamd. If you need to use custom DNS servers, which could be required when using [DNS-based black/whitelists](#rbls-realtime-blacklists-dnsbls-dns-based-blacklists), you need to adjust [`options.inc`][rspamd-docs-basic-options] yourself. +DMS does not supply custom values for DNS servers to Rspamd. If you need to use custom DNS servers, which could be required when using [DNS-based black/whitelists](#rbls-realtime-blacklists-dnsbls-dns-based-blacklists), you need to adjust [`options.inc`][www::rspamd-docs-basic-options] yourself. !!! tip "Making DNS Servers Configurable" @@ -122,9 +112,7 @@ You can find the Rspamd logs at `/var/log/mail/rspamd.log`, and the correspondin ### Modules -You can find a list of all Rspamd modules [on their website][rspamd-docs-modules]. - -[rspamd-docs-modules]: https://rspamd.com/doc/modules/ +You can find a list of all Rspamd modules [on their website][www::rspamd-docs-modules]. #### Disabled By Default @@ -136,7 +124,7 @@ You can choose to enable ClamAV, and Rspamd will then use it to check for viruse #### RBLs (Real-time Blacklists) / DNSBLs (DNS-based Blacklists) -The [RBL module](https://rspamd.com/doc/modules/rbl.html) is enabled by default. As a consequence, Rspamd will perform DNS lookups to a variety of blacklists. Whether an RBL or a DNSBL is queried depends on where the domain name was obtained: RBL servers are queried with IP addresses extracted from message headers, DNSBL server are queried with domains and IP addresses extracted from the message body \[[source][rbl-vs-dnsbl]\]. +The [RBL module](https://rspamd.com/doc/modules/rbl.html) is enabled by default. As a consequence, Rspamd will perform DNS lookups to various blacklists. Whether an RBL or a DNSBL is queried depends on where the domain name was obtained: RBL servers are queried with IP addresses extracted from message headers, DNSBL server are queried with domains and IP addresses extracted from the message body \[[source][www::rbl-vs-dnsbl]\]. !!! danger "Rspamd and DNS Block Lists" @@ -144,8 +132,6 @@ The [RBL module](https://rspamd.com/doc/modules/rbl.html) is enabled by default. If you want to use DNSBLs, **try to use your own DNS resolver** and make sure it is set up correctly, i.e. it should be a non-public & **recursive** resolver. Otherwise, you might not be able ([see this Spamhaus post](https://www.spamhaus.org/faq/section/DNSBL%20Usage#365)) to make use of the block lists. -[rbl-vs-dnsbl]: https://forum.eset.com/topic/25277-dnsbl-vs-rbl-mail-security/?do=findComment&comment=119818 - ## Providing Custom Settings & Overriding Settings DMS brings sane default settings for Rspamd. They are located at `/etc/rspamd/local.d/` inside the container (or `target/rspamd/local.d/` in the repository). @@ -154,20 +140,17 @@ DMS brings sane default settings for Rspamd. They are located at `/etc/rspamd/lo !!! question "What is [`docker-data/dms/config/`][docs::dms-volumes-config]?" -If you want to overwrite the default settings and / or provide your own settings, you can place files at `docker-data/dms/config/rspamd/override.d/`. Files from this directory are copied to `/etc/rspamd/override.d/` during startup. These files [forcibly override][rspamd-docs-override-dir] Rspamd and DMS default settings. +If you want to overwrite the default settings or provide your settings, you can place files at `docker-data/dms/config/rspamd/override.d/`. Files from this directory are copied to `/etc/rspamd/override.d/` during startup. These files [forcibly override][www::rspamd-docs-override-dir] Rspamd and DMS default settings. -!!! question "What is the [`local.d` directory and how does it compare to `override.d`][rspamd-docs-config-directories]?" +!!! question "What is the [`local.d` directory and how does it compare to `override.d`][www::rspamd-docs-config-directories]?" !!! warning "Clashing Overrides" Note that when also [using the `custom-commands.conf` file](#with-the-help-of-a-custom-file), files in `override.d` may be overwritten in case you adjust them manually and with the help of the file. -[rspamd-docs-override-dir]: https://www.rspamd.com/doc/faq.html#what-are-the-locald-and-overrided-directories -[rspamd-docs-config-directories]: https://rspamd.com/doc/faq.html#what-are-the-locald-and-overrided-directories - ### With the Help of a Custom File -DMS provides the ability to do simple adjustments to Rspamd modules with the help of a single file. Just place a file called `custom-commands.conf` into `docker-data/dms/config/rspamd/`. If this file is present, DMS will evaluate it. The structure is _very_ simple. Each line in the file looks like this: +DMS provides the ability to do simple adjustments to Rspamd modules with the help of a single file. Just place a file called `custom-commands.conf` into `docker-data/dms/config/rspamd/`. If this file is present, DMS will evaluate it. The structure is simple, as each line in the file looks like this: ```txt COMMAND ARGUMENT1 ARGUMENT2 ARGUMENT3 @@ -180,7 +163,7 @@ where `COMMAND` can be: 3. `set-option-for-module`: sets the value for option `ARGUMENT2` to `ARGUMENT3` inside module `ARGUMENT1` 4. `set-option-for-controller`: set the value of option `ARGUMENT1` to `ARGUMENT2` for the controller worker 5. `set-option-for-proxy`: set the value of option `ARGUMENT1` to `ARGUMENT2` for the proxy worker -6. `set-common-option`: set the option `ARGUMENT1` that [defines basic Rspamd behavior][rspamd-docs-basic-options] to value `ARGUMENT2` +6. `set-common-option`: set the option `ARGUMENT1` that [defines basic Rspamd behavior][www::rspamd-docs-basic-options] to value `ARGUMENT2` 7. `add-line`: this will add the complete line after `ARGUMENT1` (with all characters) to the file `/etc/rspamd/override.d/` !!! example "An Example Is [Shown Down Below](#adjusting-and-extending-the-very-basic-configuration)" @@ -195,31 +178,36 @@ You can also have comments (the line starts with `#`) and blank lines in `custom These simple commands are meant to give users the ability to _easily_ alter modules and their options. As a consequence, they are not powerful enough to enable multi-line adjustments. If you need to do something more complex, we advise to do that [manually](#manually)! -[rspamd-docs-basic-options]: https://rspamd.com/doc/configuration/options.html - ## Examples & Advanced Configuration ### A Very Basic Configuration -You want to start using Rspamd? Rspamd is disabled by default, so you need to set the following environment variables: +Do you want to start using Rspamd? Rspamd is disabled by default, so you need to set the following environment variables: ```env ENABLE_RSPAMD=1 +# ClamAV is compatible with Rspamd. Optionally enable it for anti-virus support: +ENABLE_CLAMAV=1 + +# Rspamd replaces the functionality of all these anti-spam services, disable them: ENABLE_OPENDKIM=0 ENABLE_OPENDMARC=0 ENABLE_POLICYD_SPF=0 ENABLE_AMAVIS=0 ENABLE_SPAMASSASSIN=0 + +# Provided you've set `RSPAMD_GREYLISTING=1`, also disable Postgrey: +ENABLE_POSTGREY=0 ``` This will enable Rspamd and disable services you don't need when using Rspamd. ### Adjusting and Extending The Very Basic Configuration -Rspamd is running, but you want or need to adjust it? First, create a file named `custom-commands.conf` under `docker-data/dms/config/rspamd` (which translates to `/tmp/docker-mailserver/rspamd/` inside the container). Then add you changes: +Rspamd is running, but you want or need to adjust it? First, create a file named `custom-commands.conf` under `docker-data/dms/config/rspamd` (which translates to `/tmp/docker-mailserver/rspamd/` inside the container). Then add your changes: -1. Say you want to be able to easily look at the frontend Rspamd provides on port 11334 (default) without the need to enter a password (maybe because you already provide authorization and authentication). You will need to adjust the controller worker: `set-option-for-controller secure_ip "0.0.0.0/0"`. -2. You additionally want to enable the auto-spam-learning for the Bayes module? No problem: `set-option-for-module classifier-bayes autolearn true`. +1. Say you want to be able to easily look at the frontend Rspamd provides on port 11334 (default) without the need to enter a password (maybe because you already provide authorization and authentication). You will have to adjust the controller worker: `set-option-for-controller secure_ip "0.0.0.0/0"`. +2. Do you additionally want to enable the auto-spam-learning for the Bayes module? No problem: `set-option-for-module classifier-bayes autolearn true`. 3. But the chartable module gets on your nerves? Easy: `disable-module chartable`. ??? example "What Does the Result Look Like?" @@ -239,25 +227,40 @@ Rspamd is running, but you want or need to adjust it? First, create a file named ### DKIM Signing -There is a dedicated [section for setting up DKIM with Rspamd in our documentation][docs-dkim-with-rspamd]. - -[docs-dkim-with-rspamd]: ../best-practices/dkim_dmarc_spf.md#dkim +There is a dedicated [section for setting up DKIM with Rspamd in our documentation][docs::dkim-with-rspamd]. ### _Abusix_ Integration -This subsection gives information about the integration of [Abusix], "a set of blocklists that work as an additional email security layer for your existing mail environment". The setup is straight-forward and well documented: +This subsection provides information about the integration of [Abusix][www::abusix], "a set of blocklists that work as an additional email security layer for your existing mail environment". The setup is straight-forward and well documented: 1. [Create an account](https://app.abusix.com/signup) 2. Retrieve your API key -3. Navigate to the ["Getting Started" documentation for Rspamd][abusix-rspamd-integration] and follow the steps described there +3. Navigate to the ["Getting Started" documentation for Rspamd][www::abusix-rspamd-integration] and follow the steps described there 4. Make sure to change `` to your private API key -We recommend mounting the files directly into the container, as they are rather big and not manageable with the [modules script](#with-the-help-of-a-custom-file). If mounted to the correct location, Rspamd will automatically pick them up. +We recommend mounting the files directly into the container, as they are rather big and not manageable with our [`custom-command.conf` script](#with-the-help-of-a-custom-file). If mounted to the correct location, Rspamd will automatically pick them up. While _Abusix_ can be integrated into Postfix, Postscreen and a multitude of other software, we recommend integrating _Abusix_ only into a single piece of software running in your mail server - everything else would be excessive and wasting queries. Moreover, we recommend the integration into suitable filtering software and not Postfix itself, as software like Postscreen or Rspamd can properly evaluate the return codes and other configuration. -[Abusix]: https://abusix.com/ -[abusix-rspamd-integration]: https://docs.abusix.com/abusix-mail-intelligence/gbG8EcJ3x3fSUv8cMZLiwA/getting-started/dmw9dcwSGSNQiLTssFAnBW#rspamd +[rspamd-web]: https://rspamd.com/ +[rspamd-docs::bayes]: https://rspamd.com/doc/configuration/statistic.html +[rspamd-docs::proxy-self-scan-mode]: https://rspamd.com/doc/workers/rspamd_proxy.html#self-scan-mode +[rspamd-docs::web-interface]: https://rspamd.com/webui/ +[rspamd-docs::modules]: https://rspamd.com/doc/modules/ +[rspamd-docs::override-dir]: https://www.rspamd.com/doc/faq.html#what-are-the-locald-and-overrided-directories +[rspamd-docs::config-directories]: https://rspamd.com/doc/faq.html#what-are-the-locald-and-overrided-directories +[rspamd-docs::basic-options]: https://rspamd.com/doc/configuration/options.html + +[www::rbl-vs-dnsbl]: https://forum.eset.com/topic/25277-dnsbl-vs-rbl-mail-security/?do=findComment&comment=119818 +[abusix-web]: https://abusix.com/ +[abusix-docs::rspamd-integration]: https://docs.abusix.com/abusix-mail-intelligence/gbG8EcJ3x3fSUv8cMZLiwA/getting-started/dmw9dcwSGSNQiLTssFAnBW#rspamd + +[dms-repo::rspamd-actions-config]: https://github.com/docker-mailserver/docker-mailserver/blob/v14.0.0/target/rspamd/local.d/actions.conf +[dms-repo::default-rspamd-configuration]: https://github.com/docker-mailserver/docker-mailserver/tree/v14.0.0/target/rspamd + +[docs::spam-to-junk]: ../environment.md#move_spam_to_junk +[docs::dkim-dmarc-spf]: ../best-practices/dkim_dmarc_spf.md +[docs::dkim-with-rspamd]: ../best-practices/dkim_dmarc_spf.md#dkim [docs::dms-volumes-config]: ../advanced/optional-config.md#volumes-config [docs::dms-volumes-state]: ../advanced/optional-config.md#volumes-state From 2c026715874c9240a1e0c4f34c38f9f612eceb5f Mon Sep 17 00:00:00 2001 From: Jackson Zheng <60581068+JacksonZ03@users.noreply.github.com> Date: Sun, 4 Feb 2024 10:04:07 +0000 Subject: [PATCH 028/267] Minor spelling correction (#3870) --- docs/content/config/best-practices/dkim_dmarc_spf.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/config/best-practices/dkim_dmarc_spf.md b/docs/content/config/best-practices/dkim_dmarc_spf.md index 290ac5cd..52d01dea 100644 --- a/docs/content/config/best-practices/dkim_dmarc_spf.md +++ b/docs/content/config/best-practices/dkim_dmarc_spf.md @@ -82,7 +82,7 @@ You should have: When the DMS FQDN is `mail.example.com` or `example.com`, by default this command will generate DKIM keys for `example.com` as the primary domain for your users mail accounts (eg: `hello@example.com`). - The DKIM generation does not have support to query LDAP for additionanl mail domains it should know about. If the primary mail domain is not sufficient, then you must explicitly specify any extra domains via the `domain` option: + The DKIM generation does not have support to query LDAP for additional mail domains it should know about. If the primary mail domain is not sufficient, then you must explicitly specify any extra domains via the `domain` option: ```sh # ENABLE_OPENDKIM=1 (default): From 32dcabe82684e527fd705d469c7a3c91140e097c Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sun, 4 Feb 2024 10:05:24 +0000 Subject: [PATCH 029/267] docs: update `CONTRIBUTORS.md` (#3869) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- CONTRIBUTORS.md | 383 ++++++++++++++++++++++++------------------------ 1 file changed, 195 insertions(+), 188 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 72784ab0..a7c4e661 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -126,20 +126,20 @@ Thanks goes to these wonderful people ✨ gmasse - - 00angus -
- 00angus - -
ap-wtioit
ap-wtioit +
+ + 00angus +
+ 00angus + 		@@ -212,33 +212,26 @@ Thanks goes to these wonderful people ✨ tyranron - - mindrunner -
- mindrunner - -
- - MichaelSp -
- MichaelSp - - 		KyleOndy
KyleOndy +
+ + mindrunner +
+ mindrunner + 		- - bilak + + MichaelSp
- bilak + MichaelSp 		@@ -248,6 +241,13 @@ Thanks goes to these wonderful people ✨ m-a-v + + bilak +
+ bilak + + 		vortex852456 @@ -255,20 +255,20 @@ Thanks goes to these wonderful people ✨ vortex852456 - - chris54721 -
- chris54721 - -
hanscees
hanscees +
+ + chris54721 +
+ chris54721 + 		@@ -657,6 +657,20 @@ Thanks goes to these wonderful people ✨ mpanneck + + andrewlow +
+ andrewlow + + 		+ + abh +
+ abh + + 		aminvakil @@ -671,20 +685,6 @@ Thanks goes to these wonderful people ✨ elbracht - - abh -
- abh - - 		- - andrewlow -
- andrewlow - - 		ubenmackin @@ -737,38 +737,38 @@ Thanks goes to these wonderful people ✨
- - nueaf + + fl42
- nueaf + fl42 		- - martinwepner + + ipernet
- martinwepner + ipernet 		- - artonge + + H4R0
- artonge + H4R0 		- - spacecowboy + + eltociear
- spacecowboy + eltociear 		- - jedateach + + jamesfryer
- jedateach + jamesfryer 		@@ -780,88 +780,45 @@ Thanks goes to these wonderful people ✨
- - jamesfryer + + artonge
- jamesfryer + artonge 		- - eltociear + + jedateach
- eltociear + jedateach 		- - H4R0 + + spacecowboy
- H4R0 + spacecowboy 		- - ipernet + + martinwepner
- ipernet + martinwepner 		- - fl42 + + nueaf
- fl42 + nueaf 		- - auchri + + Thiritin
- auchri - -
- - stephan-devop -
- stephan-devop - - 		- - stigok -
- stigok - - 		- - 5ven -
- 5ven - - 		- - syl20bnr -
- syl20bnr - - 		- - sylvaindumont -
- sylvaindumont - - 		- - TechnicLab -
- TechnicLab + Thiritin
- - Thiritin + + TechnicLab
- Thiritin + TechnicLab + + 		+ + sylvaindumont +
+ sylvaindumont + + 		+ + syl20bnr +
+ syl20bnr + + 		+ + 5ven +
+ 5ven + + 		+ + stigok +
+ stigok + +
+ + stephan-devop +
+ stephan-devop + + 		+ + radicand +
+ radicand 		@@ -1402,13 +1402,28 @@ Thanks goes to these wonderful people ✨ mchamplain + + crash7 +
+ crash7 + + 		+ + auchri +
+ auchri + + 		arkanovicz
arkanovicz -
CBeerta @@ -1422,8 +1437,7 @@ Thanks goes to these wonderful people ✨
damianmoore -
espitall @@ -1451,7 +1465,8 @@ Thanks goes to these wonderful people ✨
danielvandenberg95 -
denisix @@ -1465,8 +1480,7 @@ Thanks goes to these wonderful people ✨
mlatorre31 -
mazzz1y @@ -1494,7 +1508,8 @@ Thanks goes to these wonderful people ✨
edvorg -
eliroca @@ -1508,8 +1523,7 @@ Thanks goes to these wonderful people ✨
ekkis -
ErikEngerd @@ -1537,22 +1551,15 @@ Thanks goes to these wonderful people ✨
flole - 		- - froks -
- froks - -
0xflotus
0xflotus -
ifokeev @@ -1581,6 +1588,14 @@ Thanks goes to these wonderful people ✨ askz + + aspettl +
+ aspettl + +
acch @@ -1594,8 +1609,7 @@ Thanks goes to these wonderful people ✨
vifino -
kachkaev @@ -1623,7 +1637,8 @@ Thanks goes to these wonderful people ✨
eglia -
groupmsl @@ -1637,8 +1652,7 @@ Thanks goes to these wonderful people ✨
green-anger -
iRhonin @@ -1666,7 +1680,8 @@ Thanks goes to these wonderful people ✨
astrocket -
baxerus @@ -1680,8 +1695,7 @@ Thanks goes to these wonderful people ✨
spock -
erdos4d @@ -1690,10 +1704,10 @@ Thanks goes to these wonderful people ✨ - - crash7 + + akkumar
- crash7 + akkumar 		@@ -1709,7 +1723,8 @@ Thanks goes to these wonderful people ✨
KCrawley -
khuedoan @@ -1723,8 +1738,7 @@ Thanks goes to these wonderful people ✨
JustAnother1 -
LeoWinterDE @@ -1752,7 +1766,8 @@ Thanks goes to these wonderful people ✨
LucidityCrash -
MadsRC @@ -1766,8 +1781,7 @@ Thanks goes to these wonderful people ✨
madmath03 -
maxemann96 @@ -1795,7 +1809,8 @@ Thanks goes to these wonderful people ✨
exhuma -
milas @@ -1809,8 +1824,7 @@ Thanks goes to these wonderful people ✨
mcchots -
MohammedNoureldin @@ -1838,12 +1852,13 @@ Thanks goes to these wonderful people ✨
neuralp -
- - radicand + + froks
- radicand + froks 		@@ -1852,8 +1867,7 @@ Thanks goes to these wonderful people ✨
fkefer -
frugan-dev @@ -1881,7 +1895,8 @@ Thanks goes to these wonderful people ✨
GiovanH -
harryyoud @@ -1895,8 +1910,7 @@ Thanks goes to these wonderful people ✨
HeySora -
sirgantrithon @@ -1924,7 +1938,8 @@ Thanks goes to these wonderful people ✨
jcalfee -
mivek @@ -1938,8 +1953,7 @@ Thanks goes to these wonderful people ✨
init-js -
Jeidnx @@ -1967,7 +1981,8 @@ Thanks goes to these wonderful people ✨
jmccl -
jurekbarth @@ -1981,21 +1996,13 @@ Thanks goes to these wonderful people ✨
JOduMonT -
Kaan88
Kaan88 - 		- - akkumar -
- akkumar -
From 9fc7f979507a4045b60627a985a26487b7b2de74 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 6 Feb 2024 01:34:20 +1300 Subject: [PATCH 030/267] chore(deps): Bump docker/metadata-action from 5.5.0 to 5.5.1 (#3878) Bumps [docker/metadata-action](https://github.com/docker/metadata-action) from 5.5.0 to 5.5.1. - [Release notes](https://github.com/docker/metadata-action/releases) - [Commits](https://github.com/docker/metadata-action/compare/v5.5.0...v5.5.1) --- updated-dependencies: - dependency-name: docker/metadata-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_publish.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index d7a791c5..83357062 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -23,7 +23,7 @@ jobs: - name: 'Prepare tags' id: prep - uses: docker/metadata-action@v5.5.0 + uses: docker/metadata-action@v5.5.1 with: images: | ${{ secrets.DOCKER_REPOSITORY }} From d5efaf95c3a9239fa5ad3728be1525f505cc38b9 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 5 Feb 2024 12:35:57 +0000 Subject: [PATCH 031/267] chore(deps): Bump anchore/scan-action from 3.6.1 to 3.6.4 (#3877) Bumps [anchore/scan-action](https://github.com/anchore/scan-action) from 3.6.1 to 3.6.4. - [Release notes](https://github.com/anchore/scan-action/releases) - [Changelog](https://github.com/anchore/scan-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/anchore/scan-action/compare/v3.6.1...v3.6.4) --- updated-dependencies: - dependency-name: anchore/scan-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- .github/workflows/generic_vulnerability-scan.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index cb1a5994..78af5ded 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -55,7 +55,7 @@ jobs: provenance: false - name: 'Run the Anchore Grype scan action' - uses: anchore/scan-action@v3.6.1 + uses: anchore/scan-action@v3.6.4 id: scan with: image: mailserver-testing:ci From 51a391525775058b0f0344f417da4f0c4c8d1ccc Mon Sep 17 00:00:00 2001 From: Rahil Bhimjiani Date: Tue, 6 Feb 2024 04:26:47 +0530 Subject: [PATCH 032/267] docs: fix 404 in mailserver.env and default to RSA 2048 for TLS certs (#3875) * fix 404: broken MTA-STS link in comment of mailserver.env Signed-off-by: Rahil Bhimjiani * docs: recommend and default to RSA 2048 for ssl certs Signed-off-by: Rahil Bhimjiani --------- Signed-off-by: Rahil Bhimjiani --- docs/content/config/security/ssl.md | 8 ++++---- mailserver.env | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/content/config/security/ssl.md b/docs/content/config/security/ssl.md index 10f7adc4..1b84c4df 100644 --- a/docs/content/config/security/ssl.md +++ b/docs/content/config/security/ssl.md @@ -408,7 +408,7 @@ The following example is the [basic setup][acme-companion::basic-setup] you need - `LETSENCRYPT_TEST=true`: _Recommended during initial setup_. Otherwise the default production endpoint has a [rate limit of 5 duplicate certificates per week][letsencrypt::limits]. Overrides `ACME_CA_URI` to use the _Let's Encrypt_ staging endpoint. - `LETSENCRYPT_EMAIL`: For when you don't use `DEFAULT_EMAIL` on `acme-companion`, or want to assign a different email contact for this container. - - `LETSENCRYPT_KEYSIZE`: Allows you to configure the type (RSA or ECDSA) and size of the private key for your certificate. Default is RSA 4096. + - `LETSENCRYPT_KEYSIZE`: Allows you to configure the type (RSA or ECDSA) and size of the private key for your certificate. Default is RSA 4096, but RSA 2048 is recommended. - `LETSENCRYPT_RESTART_CONTAINER=true`: When the certificate is renewed, the entire container will be restarted to ensure the new certificate is used. [`acme-companion` ENV for default settings][acme-companion::env-config] that apply to all containers using `LETSENCRYPT_HOST`: @@ -450,8 +450,8 @@ The following example is the [basic setup][acme-companion::basic-setup] you need # Optional variables: LETSENCRYPT_mail_TEST=true LETSENCRYPT_mail_EMAIL='admin@example.com' - # RSA-4096 => `4096`, ECDSA-256 => `ec-256`: - LETSENCRYPT_mail_KEYSIZE=4096 + # Supported values are `2048`, `3072` and `4096` for RSA keys, and `ec-256` or `ec-384` for elliptic curve keys. + LETSENCRYPT_mail_KEYSIZE=2048 ``` Unlike with the equivalent ENV for containers, [changes to this file will **not** be detected automatically][acme-companion::standalone-changes]. You would need to wait until the next renewal check by `acme-companion` (_every hour by default_), restart `acme-companion`, or [manually invoke the _service loop_][acme-companion::service-loop]: @@ -488,7 +488,7 @@ For Caddy v2 you can specify the `key_type` in your server's global settings, wh http_port 80 https_port 443 default_sni example.com - key_type rsa4096 + key_type rsa2048 } ``` diff --git a/mailserver.env b/mailserver.env index 7f493a3a..44a9a484 100644 --- a/mailserver.env +++ b/mailserver.env @@ -362,7 +362,7 @@ POSTFIX_REJECT_UNKNOWN_CLIENT_HOSTNAME=0 POSTFIX_INET_PROTOCOLS=all # Enables MTA-STS support for outbound mail. -# More details: https://docker-mailserver.github.io/docker-mailserver/latest/config/advanced/mail-mta-sts/ +# More details: https://docker-mailserver.github.io/docker-mailserver/v13.3/config/best-practices/mta-sts/ # - **0** ==> MTA-STS disabled # - 1 => MTA-STS enabled ENABLE_MTA_STS=0 From 4f222fe256edc25509b053545b8848c073b535dd Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Tue, 6 Feb 2024 00:30:22 +0100 Subject: [PATCH 033/267] Rspamd: improve DKIM key generation (#3876) * correct removal of old files with `--force` `rm` would fail when one of the files is not present, which is quite undesirable log (not harmful until `set -e` is introduced). * use tmp log file ref: https://github.com/docker-mailserver/docker-mailserver/issues/3873#issuecomment-1926736020 * correct indentation --- target/bin/rspamd-dkim | 16 +++++++---- target/scripts/helpers/rspamd.sh | 46 ++++++++++++++++++++++++++++++-- 2 files changed, 55 insertions(+), 7 deletions(-) diff --git a/target/bin/rspamd-dkim b/target/bin/rspamd-dkim index 6dfcc1a0..689aa7d5 100755 --- a/target/bin/rspamd-dkim +++ b/target/bin/rspamd-dkim @@ -177,10 +177,14 @@ function _create_keys() { exit 1 else _log 'info' "Overwriting existing files as the '--force' option was supplied" - rm "${PUBLIC_KEY_FILE}" "${PUBLIC_KEY_DNS_FILE}" "${PRIVATE_KEY_FILE}" + [[ -f ${PUBLIC_KEY_FILE} ]] && rm "${PUBLIC_KEY_FILE}" + [[ -f ${PUBLIC_KEY_DNS_FILE} ]] && rm "${PUBLIC_KEY_DNS_FILE}" + [[ -f ${PRIVATE_KEY_FILE} ]] && rm "${PRIVATE_KEY_FILE}" fi fi + __create_rspamd_err_log + # shellcheck disable=SC2310 if __do_as_rspamd_user rspamadm \ dkim_keygen \ @@ -188,12 +192,14 @@ function _create_keys() { -d "${DOMAIN}" \ "${KEYTYPE_OPTIONS[@]}" \ -k "${PRIVATE_KEY_FILE}" \ - >"${PUBLIC_KEY_FILE}" + >"${PUBLIC_KEY_FILE}" \ + && ! __filter_rspamd_err_log 'Permission denied' # we also need to check the log for error messages then - _log 'info' 'Successfully created DKIM keys' - _log 'debug' "Public key written to '${PUBLIC_KEY_FILE}'" - _log 'debug' "Private key written to '${PRIVATE_KEY_FILE}'" + _log 'info' 'Successfully created DKIM keys' + _log 'debug' "Public key written to '${PUBLIC_KEY_FILE}'" + _log 'debug' "Private key written to '${PRIVATE_KEY_FILE}'" else + __print_rspamd_err_log _exit_with_error 'Creating keys failed' fi } diff --git a/target/scripts/helpers/rspamd.sh b/target/scripts/helpers/rspamd.sh index 8d1fd668..1d3e1417 100644 --- a/target/scripts/helpers/rspamd.sh +++ b/target/scripts/helpers/rspamd.sh @@ -5,9 +5,51 @@ # Perform a specific command as the Rspamd user (`_rspamd`). This is useful # in case you want to have correct permissions on newly created files or if # you want to check whether Rspamd can perform a specific action. +# +# @flag ${1} = '--quiet' to indicate whether log should be disabled [OPTIONAL] function __do_as_rspamd_user() { - _log 'trace' "Running '${*}' as user '_rspamd'" - su _rspamd -s /bin/bash -c "${*}" + if [[ ${1:-} != '--quiet' ]]; then + _log 'trace' "Running '${*}' as user '_rspamd'" + else + shift 1 + fi + + su _rspamd -s /bin/bash -c "${*} 2>${__RSPAMD_ERR_LOG_FILE:-/dev/null}" +} + +# Create a temporary log file (with `mktemp`) that one can filter to search +# for error messages. This is required as `rspamadm` sometimes prints an error +# but does not exit with an error. +# +# The file created is managed in the ENV `__RSPAMD_ERR_LOG_FILE`. This ENV is +# meant for internal usage; do not use it on your scripts. The log file is cleaned +# up when the script exits. +function __create_rspamd_err_log() { + _log 'trace' "Creating Rspamd error log" + trap 'rm -f "${__RSPAMD_ERR_LOG_FILE}"' EXIT # cleanup when we exit + __RSPAMD_ERR_LOG_FILE=$(__do_as_rspamd_user --quiet mktemp) +} + +# Print the Rspamd temporary error log. This will succeed only when the log has been +# created before. +function __print_rspamd_err_log() { + [[ -v __RSPAMD_ERR_LOG_FILE ]] && __do_as_rspamd_user cat "${__RSPAMD_ERR_LOG_FILE}" +} + +# Print the Rspamd temporary error log. We use `grep` but with "fixed strings", which +# means the message you provide is evaluated as-is, not as a regular expression. This +# will succeed only when the log has been created before. +# +# @param ${1} = message to filter by +function __filter_rspamd_err_log() { + if [[ -v __RSPAMD_ERR_LOG_FILE ]]; then + __do_as_rspamd_user grep \ + --quiet \ + --ignore-case \ + --fixed-strings \ + "${1:?A message for filtering is required}" \ + "${__RSPAMD_ERR_LOG_FILE}" + fi } # Calling this function brings common Rspamd-related environment variables From 34654c7e20ccae05a4299669aa395afebea68f40 Mon Sep 17 00:00:00 2001 From: Frugan <7957714+frugan-dev@users.noreply.github.com> Date: Thu, 8 Feb 2024 22:23:58 +0100 Subject: [PATCH 034/267] chore: Source Postgrey `whitelist_clients` config from Github (#3879) Use a more updated list for Postgrey `whitelist_clients` --- CHANGELOG.md | 2 ++ Dockerfile | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7918d810..c8030f43 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -65,6 +65,8 @@ The most noteworthy change of this release is the update of the container's base - **Environment Variables:** - `ONE_DIR` has been removed (legacy ENV) ([#3840](https://github.com/docker-mailserver/docker-mailserver/pull/3840)) - It's only functionality remaining was to opt-out of run-time state consolidation with `ONE_DIR=0` (_when a volume was already mounted to `/var/mail-state`_). +- **Internal:** + - Changed the Postgrey whitelist retrieved during build to [source directly from Github](https://github.com/schweikert/postgrey/blob/master/postgrey_whitelist_clients) as the list is updated more frequently than the [author publishes at their website](https://postgrey.schweikert.ch) ([#3879](https://github.com/docker-mailserver/docker-mailserver/pull/3879)) - **Tests:** - Refactored helper methods for sending e-mails with specific `Message-ID` headers and the helpers for retrieving + filtering logs, which together help isolate logs relevant to specific mail when multiple mails have been processed within a single test. ([#3786](https://github.com/docker-mailserver/docker-mailserver/pull/3786)) - **Rspamd**: diff --git a/Dockerfile b/Dockerfile index b8bf89bc..8110cebf 100644 --- a/Dockerfile +++ b/Dockerfile @@ -144,7 +144,7 @@ COPY target/postgrey/postgrey /etc/default/postgrey RUN < Date: Tue, 13 Feb 2024 19:42:17 +1300 Subject: [PATCH 035/267] docs: Complete rewrite of PROXY protocol guide (#3882) --- .../tutorials/mailserver-behind-proxy.md | 466 ++++++++++++++---- 1 file changed, 375 insertions(+), 91 deletions(-) diff --git a/docs/content/examples/tutorials/mailserver-behind-proxy.md b/docs/content/examples/tutorials/mailserver-behind-proxy.md index 99939542..47116fde 100644 --- a/docs/content/examples/tutorials/mailserver-behind-proxy.md +++ b/docs/content/examples/tutorials/mailserver-behind-proxy.md @@ -2,126 +2,410 @@ title: 'Tutorials | Mail Server behind a Proxy' --- -## Using DMS behind a Proxy +## Using a Reverse Proxy -### Information +Guidance is provided via a Traefik config example, however if you're only familiar with configuring a reverse proxy for web services there are some differences to keep in mind. -If you are hiding your container behind a proxy service you might have discovered that the proxied requests from now on contain the proxy IP as the request origin. Whilst this behavior is technical correct it produces certain problems on the containers behind the proxy as they cannot distinguish the real origin of the requests anymore. +- A security concern where preserving the client IP is important but needs to be handled at Layer 4 (TCP). +- TLS will be handled differently due protocols like STARTTLS and the need to comply with standards for interoperability with other MTAs. +- The ability to route the same port to different containers by FQDN can be limited. -To solve this problem on TCP connections we can make use of the [proxy protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt). Compared to other workarounds that exist (`X-Forwarded-For` which only works for HTTP requests or `Tproxy` that requires you to recompile your kernel) the proxy protocol: +This reduces many of the benefits for why you might use a reverse proxy, but they can still be useful. -- It is protocol agnostic (can work with any layer 7 protocols, even when encrypted). -- It does not require any infrastructure changes. -- NAT-ing firewalls have no impact it. -- It is scalable. +Some deployments may require a service to route traffic (kubernetes) when deploying, in which case the below advice is important to understand well. -There is only one condition: **both endpoints** of the connection MUST be compatible with proxy protocol. +## What can go wrong? -Luckily `dovecot` and `postfix` are both Proxy-Protocol ready softwares so it depends only on your used reverse-proxy / loadbalancer. +Without a reverse proxy involved, a service is typically aware of the client IP for a connection. -### Configuration of the used Proxy Software +However when a reverse proxy routes the connection this information can be lost, and the proxied service mistakenly treats the client IP as the reverse proxy handling the connection. -The configuration depends on the used proxy system. I will provide the configuration examples of [traefik v2](https://traefik.io/) using IMAP and SMTP with implicit TLS. +- That can be problematic when the client IP is meaningful information for the proxied service to act upon, especially when it [impacts security](#security-concerns). +- The [PROXY protocol][networking::spec:proxy-protocol] is a well established solution to preserve the client IP when both the proxy and service have enabled the support. -Feel free to add your configuration if you achieved the same goal using different proxy software below: +??? abstract "Technical Details - HTTP vs TCP proxying" -??? "Traefik v2" + A key difference for how the network is proxied relates to the [OSI Model][networking::osi-model]: - Truncated configuration of traefik itself: + - Layer 7 (_Application layer protocols: SMTP / IMAP / HTTP / etc_) + - Layer 4 (_Transport layer protocols: TCP / UDP_) - ```yaml + When working with Layer 7 and a protocol like HTTP, it is possible to inspect a protocol header like [`Forwarded`][networking::http-header::forwarded] (_or it's predecessor: [`X-Forwarded-For`][networking::http-header::x-forwarded-for]_). At a lower level with Layer 4, that information is not available and we are routing traffic agnostic to the application protocol being proxied. + + A proxy can prepend the [PROXY protocol][networking::spec:proxy-protocol] header to the TCP/UDP connection as it is routed to the service, which must be configured to be compatible with PROXY protocol (_often this adds a restriction that connections must provide the header, otherwise they're rejected_). + + Beyond your own proxy, traffic may be routed in the network by other means that would also rewrite this information such as Docker's own network management via `iptables` and `userland-proxy` (NAT). The PROXY header ensures the original source and destination IP addresses, along with their ports is preserved across transit. + +## Configuration + +### Reverse Proxy + +The below guidance is focused on configuring [Traefik][traefik-web], but the advice should be roughly applicable elsewhere (_eg: [NGINX][nginx-docs::proxyprotocol], [Caddy][caddy::plugin::l4]_). + +- Support requires the capability to proxy TCP (Layer 4) connections with PROXY protocol enabled for the upstream (DMS). The upstream must also support enabling PROXY protocol (_which for DMS services rejects any connection not using the protocol_). +- TLS should not be terminated at the proxy, that should be delegated to DMS (_which should be configured with the TLS certs_). Reasoning is covered under the [ports section](#ports). + +???+ example "Traefik service" + + The Traefik service config is fairly standard, just define the necessary entrypoints: + + ```yaml title="compose.yaml" services: reverse-proxy: - image: docker.io/traefik:latest # v2.5 - container_name: docker-traefik - restart: always + image: docker.io/traefik:latest # 2.10 / 3.0 + # CAUTION: In production you should configure the Docker API endpoint securely: + # https://doc.traefik.io/traefik/providers/docker/#docker-api-access + volumes: + - /var/run/docker.sock:/var/run/docker.sock command: - - "--providers.docker" - - "--providers.docker.exposedbydefault=false" - - "--providers.docker.network=proxy" - - "--entrypoints.web.address=:80" - - "--entryPoints.websecure.address=:443" - - "--entryPoints.smtp.address=:25" - - "--entryPoints.smtp-ssl.address=:465" - - "--entryPoints.imap-ssl.address=:993" - - "--entryPoints.sieve.address=:4190" + # Docker provider config: + - --providers.docker=true + - --providers.docker.exposedbydefault=false + # DMS ports you want to proxy: + - --entryPoints.mail-smtp.address=:25 + - --entryPoints.mail-submission.address=:587 + - --entryPoints.mail-submissions.address=:465 + - --entryPoints.mail-imap.address=:143 + - --entryPoints.mail-imaps.address=:993 + - --entryPoints.mail-pop3.address=:110 + - --entryPoints.mail-pop3s.address=:995 + - --entryPoints.mail-managesieve.address=:4190 + # Publish external access ports mapped to traefik entrypoint ports: ports: - "25:25" + - "587:587" - "465:465" + - "143:143" - "993:993" + - "110:110" + - "995:995" - "4190:4190" - [...] - ``` - - Truncated list of necessary labels on the DMS container: - - ```yaml - services: - mailserver: - image: ghcr.io/docker-mailserver/docker-mailserver:latest - container_name: mailserver - hostname: mail.example.com - restart: always + # An IP is assigned here for other services (Dovecot) to trust for PROXY protocol: networks: - - proxy - labels: - - "traefik.enable=true" - - "traefik.tcp.routers.smtp.rule=HostSNI(`*`)" - - "traefik.tcp.routers.smtp.entrypoints=smtp" - - "traefik.tcp.routers.smtp.service=smtp" - - "traefik.tcp.services.smtp.loadbalancer.server.port=25" - - "traefik.tcp.services.smtp.loadbalancer.proxyProtocol.version=1" - - "traefik.tcp.routers.smtp-ssl.rule=HostSNI(`*`)" - - "traefik.tcp.routers.smtp-ssl.entrypoints=smtp-ssl" - - "traefik.tcp.routers.smtp-ssl.tls.passthrough=true" - - "traefik.tcp.routers.smtp-ssl.service=smtp-ssl" - - "traefik.tcp.services.smtp-ssl.loadbalancer.server.port=465" - - "traefik.tcp.services.smtp-ssl.loadbalancer.proxyProtocol.version=1" - - "traefik.tcp.routers.imap-ssl.rule=HostSNI(`*`)" - - "traefik.tcp.routers.imap-ssl.entrypoints=imap-ssl" - - "traefik.tcp.routers.imap-ssl.service=imap-ssl" - - "traefik.tcp.routers.imap-ssl.tls.passthrough=true" - - "traefik.tcp.services.imap-ssl.loadbalancer.server.port=10993" - - "traefik.tcp.services.imap-ssl.loadbalancer.proxyProtocol.version=2" - - "traefik.tcp.routers.sieve.rule=HostSNI(`*`)" - - "traefik.tcp.routers.sieve.entrypoints=sieve" - - "traefik.tcp.routers.sieve.service=sieve" - - "traefik.tcp.services.sieve.loadbalancer.server.port=4190" - [...] + default: + ipv4_address: 172.16.42.2 + + # Specifying a subnet to assign a fixed container IP to the reverse proxy: + networks: + default: + name: my-network + ipam: + config: + - subnet: "172.16.42.0/24" ``` - Keep in mind that it is necessary to use port `10993` here. More information below at `dovecot` configuration. + !!! note "Extra considerations" -### Configuration of the Backend (`dovecot` and `postfix`) + - [`--providers.docker.network=my-network`][traefik-docs::provider-docker::network] is useful when there is more than one network to consider. + - If your deployment has any other hops (an edge proxy, load balancer, etc) between the reverse proxy and the client, you'll need PROXY protocol support throughout that chain. For Traefik this additionally requires [enabling PROXY protocol on your entry points][traefik-docs::entrypoint::proxyprotocol]. -The following changes can be achieved completely by adding the content to the appropriate files by using the projects [function to overwrite config files][docs-optionalconfig]. +???+ example "Traefik labels for DMS" -Changes for `postfix` can be applied by adding the following content to `docker-data/dms/config/postfix-main.cf`: + ```yaml title="compose.yaml" + services: + dms: + image: ghcr.io/docker-mailserver/docker-mailserver:latest + hostname: mail.example.com + labels: + - traefik.enable=true -```cf -postscreen_upstream_proxy_protocol = haproxy -``` + # These are examples, configure the equivalent for any additional ports you proxy. + # Explicit TLS (STARTTLS): + - traefik.tcp.routers.mail-smtp.rule=HostSNI(`*`) + - traefik.tcp.routers.mail-smtp.entrypoints=smtp + - traefik.tcp.routers.mail-smtp.service=smtp + - traefik.tcp.services.mail-smtp.loadbalancer.server.port=25 + - traefik.tcp.services.mail-smtp.loadbalancer.proxyProtocol.version=2 -and to `docker-data/dms/config/postfix-master.cf`: + # Implicit TLS is no different, except for optional HostSNI support: + - traefik.tcp.routers.mail-submissions.rule=HostSNI(`*`) + - traefik.tcp.routers.mail-submissions.entrypoints=smtp-submissions + - traefik.tcp.routers.mail-submissions.service=smtp-submissions + - traefik.tcp.services.mail-submissions.loadbalancer.server.port=465 + - traefik.tcp.services.mail-submissions.loadbalancer.proxyProtocol.version=2 + # NOTE: Optionally match by SNI rule, this requires TLS passthrough (not compatible with STARTTLS): + #- traefik.tcp.routers.mail-submissions.rule=HostSNI(`mail.example.com`) + #- traefik.tcp.routers.mail-submissions.tls.passthrough=true + ``` -```cf -submission/inet/smtpd_upstream_proxy_protocol=haproxy -submissions/inet/smtpd_upstream_proxy_protocol=haproxy -``` + !!! note "PROXY protocol compatibility" -Changes for `dovecot` can be applied by adding the following content to `docker-data/dms/config/dovecot.cf`: + Only TCP routers support enabling PROXY Protocol (via [`proxyProtocol.version=2`][traefik-docs::service-tcp::proxyprotocol]) -```cf -haproxy_trusted_networks = , -haproxy_timeout = 3 secs -service imap-login { - inet_listener imaps { - haproxy = yes - ssl = yes - port = 10993 - } -} -``` + Postfix and Dovecot are both compatible with PROXY protocol v1 and v2. -!!! note - Port `10993` is used here to avoid conflicts with internal systems like `postscreen` and `amavis` as they will exchange messages on the default port and obviously have a different origin then compared to the proxy. +??? abstract "Technical Details - Ports (Traefik config)" + + !!! info "Explicit TLS (STARTTLS)" + + **Service Ports:** `mail-smtp` (25), `mail-submission` (587), `mail-imap` (143), `mail-pop3` (110), `mail-managesieve` (4190) + + --- + + - [Traefik expects the TCP router to not enable TLS][traefik-docs::router-tcp::server-first-protocols] (_see "Server First protocols"_) for these connections. They begin in plaintext and potentially upgrade the connection to TLS, Traefik has no involvement in STARTTLS. + - Without an initial TLS connection, the [`HostSNI` router rule is not usable][traefik-docs::router-tcp::host-sni] (_see "HostSNI & TLS"_). This limits routing flexibility for these ports (_eg: routing these ports by the FQDN to different DMS containers_). + + !!! info "Implicit TLS" + + **Service Ports:** `mail-submissions` (465), `mail-imaps` (993), `mail-pop3s` (995) + + --- + + The `HostSNI` router rule could specify the DMS FQDN instead of `*`: + + - This requires the router to have TLS enabled, so that Traefik can inspect the server name sent by the client. + - Traefik can only match the SNI to `*` when the client does not provide a server name. Some clients must explicitly opt-in, such as CLI clients `openssl` (`-servername`) and `swaks` (`--tls-sni`). + - Add [`tls.passthrough=true` to the router][traefik-docs::router-tcp::passthrough] (_this implicitly enables TLS_). + - Traefik should not terminate TLS, decryption should occur within DMS instead when proxying to the same implicit TLS ports. + - Passthrough ignores any certificates configured for Traefik; DMS must be configured with the certificates instead (_[DMS can use `acme.json` from Traefik][docs::tls::traefik]_). + + Unlike proxying HTTPS (port 443) to a container via HTTP (port 80), the equivalent for DMS service ports is not supported: + + - Port 25 must secure the connection via STARTTLS to be reached publicly. + - STARTTLS ports requiring authentication for Postfix (587) and Dovecot (110, 143, 4190) are configured to only permit authentication over an encrypted connection. + - Support would require routing the implicit TLS ports to their explicit TLS equivalent ports with auth restrictions removed. `tls.passthrough.true` would not be required, additionally port 25 would always be unencrypted (_if the proxy exclusively manages TLS/certs_), or unreachable by public MTAs attempting delivery if the proxy enables implicit TLS for this port. + +### DMS (Postfix + Dovecot) + +???+ example "Enable PROXY protocol on existing service ports" + + This can be handled via our config override support. + + --- + + Postfix via [`postfix-master.cf`][docs::overrides::postfix]: + + ```cf title="docker-data/dms/config/postfix-master.cf" + smtp/inet/postscreen_upstream_proxy_protocol=haproxy + submission/inet/smtpd_upstream_proxy_protocol=haproxy + submissions/inet/smtpd_upstream_proxy_protocol=haproxy + ``` + + [`postscreen_upstream_proxy_protocol`][postfix-docs::settings::postscreen_upstream_proxy_protocol] and [`smtpd_upstream_proxy_protocol`][postfix-docs::settings::smtpd_upstream_proxy_protocol] both specify the protocol type used by a proxy. `haproxy` represents the PROXY protocol. + + --- + + Dovecot via [`dovecot.cf`][docs::overrides::dovecot]: + + ```cf title="docker-data/dms/config/dovecot.cf" + haproxy_trusted_networks = 172.16.42.2 + + service imap-login { + inet_listener imap { + haproxy = yes + } + + inet_listener imaps { + haproxy = yes + } + } + + service pop3-login { + inet_listener pop3 { + haproxy = yes + } + + inet_listener pop3s { + haproxy = yes + } + } + + service managesieve-login { + inet_listener sieve { + haproxy = yes + } + } + ``` + + - [`haproxy_trusted_networks`][dovecot-docs::settings::haproxy-trusted-networks] must reference the reverse proxy IP, or a wider subnet using CIDR notation. + - [`haproxy = yes`][dovecot-docs::service-config::haproxy] for the TCP listeners of each login service. + +!!! warning "Internal traffic (_within the network or DMS itself_)" + + - Direct connections to DMS from other containers within the internal network will be rejected when they don't provide the required PROXY header. + - This can also affect services running within the DMS container itself if they attempt to make a connection and aren't PROXY protocol capable. + + --- + + A solution is to configure alternative service ports that offer PROXY protocol support (as shown next). + + Alternatively routing connections to DMS through the local reverse proxy via [DNS query rewriting][gh-dms::dns-rewrite-example] can work too. + +??? example "Configuring services with separate ports for PROXY protocol" + + In this example we'll take the original service ports and add `10000` for the new PROXY protocol service ports. + + Traefik labels will need to update their service ports accordingly (eg: `.loadbalancer.server.port=10465`). + + --- + + Postfix config now requires [our `user-patches.sh` support][docs::overrides::user-patches] to add new services in `/etc/postfix/master.cf`: + + ```bash title="docker-data/dms/config/user-patches.sh" + #!/bin/bash + + # Duplicate the config for the submission(s) service ports (587 / 465) with adjustments for the PROXY ports (10587 / 10465) and `syslog_name` setting: + postconf -Mf submission/inet | sed -e s/^submission/10587/ -e 's/submission/submission-proxyprotocol/' >> /etc/postfix/master.cf + postconf -Mf submissions/inet | sed -e s/^submissions/10465/ -e 's/submissions/submissions-proxyprotocol/' >> /etc/postfix/master.cf + # Enable PROXY Protocol support for these new service variants: + postconf -P 10587/inet/smtpd_upstream_proxy_protocol=haproxy + postconf -P 10465/inet/smtpd_upstream_proxy_protocol=haproxy + + # Create a variant for port 25 too (NOTE: Port 10025 is already assigned in DMS to Amavis): + postconf -Mf smtp/inet | sed -e s/^smtp/12525/ >> /etc/postfix/master.cf + # Enable PROXY Protocol support (different setting as port 25 is handled via postscreen), optionally configure a `syslog_name` to distinguish in logs: + postconf -P 12525/inet/postscreen_upstream_proxy_protocol=haproxy 12525/inet/syslog_name=smtp-proxyprotocol + ``` + + --- + + Dovecot is mostly the same as before: + + - A new service name instead of targeting one to modify. + - Add the new port assignment. + - Set [`ssl = yes`][dovecot-docs::service-config::ssl] when implicit TLS is needed. + + ```cf title="docker-data/dms/config/dovecot.cf" + haproxy_trusted_networks = 172.16.42.2 + + service imap-login { + inet_listener imap-proxied { + haproxy = yes + port = 10143 + } + + inet_listener imaps-proxied { + haproxy = yes + port = 10993 + ssl = yes + } + } + + service pop3-login { + inet_listener pop3-proxied { + haproxy = yes + port = 10110 + } + + inet_listener pop3s-proxied { + haproxy = yes + port = 10995 + ssl = yes + } + } + + service managesieve-login { + inet_listener sieve-proxied { + haproxy = yes + port = 14190 + } + } + ``` + +## Verification + +Send an email through the reverse proxy. If you do not use the DNS query rewriting approach, you'll need to do this from an external client. + +??? example "Sending a generic test mail through `swaks` CLI" + + Run a `swaks` command and then check your DMS logs for the expected client IP, it should no longer be using the reverse proxy IP. + + ```bash + # NOTE: It is common to find port 25 is blocked from outbound connections, you may only be able to test the submission(s) ports. + swaks --helo not-relevant.test --server mail.example.com --port 25 -tls --from hello@not-relevant.test --to user@example.com + ``` + + - You can specify the `--server` as the DMS FQDN or an IP address, where either should connect to the reverse proxy service. + - `not-relevant.test` technically may be subject to some tests, at least for port 25. With the submission(s) ports those should be exempt. + - `-tls` will use STARTTLS on port 25, you can exclude it to send unencrypted, but it would still go through the same port/route being tested. + - To test the submission ports use `--port 587 -tls` or `--port 465 -tlsc` with your credentials `--auth-user user@example.com --auth-password secret` + - Add `--tls-sni mail.example.com` if you have configured `HostSNI` in Traefik router rules (_SNI routing is only valid for implicit TLS ports_). + +??? warning "Do not rely on local testing alone" + + Testing from the Docker host technically works, however the IP is likely subject to more manipulation via `iptables` than an external client. + + The IP will likely appear as from the gateway IP of the Docker network associated to the reverse proxy, where that gateway IP then becomes the client IP when writing the PROXY protocol header. + +## Security concerns + +### Forgery + +Since the PROXY protocol sends a header with the client IP rewritten for software to use instead, this could be abused by bad actors. + +Software on the receiving end of the connection often supports configuring an IP or CIDR range of clients to trust receiving the PROXY protocol header from. + +??? warning "Risk exposure" + + If you trust more than the reverse proxy IP, you must consider the risk exposure: + + - Any container within the network that is compromised could impersonate another IP (_container or external client_) which may have been configured to have additional access/exceptions granted. + - If the reverse proxy is on a separate network/host than DMS, exposure of the PROXY protocol enabled ports outside the network increases the importance of narrowing trust. For example with the [known IPv6 to subnet Gateway IP routing gotcha][docs::ipv6::security-risks] in Docker, trusting the entire subnet DMS belongs to would wrongly trust external clients that have the subnet Gateway IP to impersonate any client IP. + - There is a [known risk with Layer 2 switching][docker::networking::l2-switch-gotcha] (_applicable to VPC networks, impact varies by cloud vendor_): + - Neighbouring hosts can indirectly route to ports published on the interfaces of a separate host system that shouldn't be reachable (_eg: localhost `127.0.0.1`, or a private subnet `172.16.0.0/12`_). + - The scope of this in Docker is limited to published ports only when Docker uses `iptables` with the kernel tunable `sysctl net.ipv4.ip_forward=1` (enabled implicitly). Port access is via `HOST:CONTAINER` ports published to their respective interface(s), that includes the container IP + port. + + While some concerns raised above are rather specific, these type of issues aren't exclusive to Docker and difficult to keep on top of as software is constantly changing. Limit the trusted networks where possible. + +??? warning "Postfix has no concept of trusted proxies" + + Postfix does not appear to have a way to configure trusted proxies like Dovecot does (`haproxy_trusted_networks`). + + [`postscreen_access_list`][postfix-docs::settings::postscreen_access_list] (_or [`smtpd_client_restrictions`][postfix-docs::settings::smtpd_client_restrictions] with [`check_client_access`][postfix-docs::settings::check_client_access] for ports 587/465_) can both restrict access by IP via a [CIDR lookup table][postfix-docs::config-table::cidr], however the client IP is already rewritten at this point via PROXY protocol. + + Thus those settings cannot be used for restricting access to only trusted proxies, only to the actual clients. + + A similar setting [`mynetworks`][postfix-docs::settings::mynetworks] / [`PERMIT_DOCKER`][docs::env::permit_docker] manages elevated trust for bypassing security restrictions. While it is intended for trusted clients, it has no relevance to trusting proxies for the same reasons. + + +### Monitoring + +While PROXY protocol works well with the reverse proxy, you may have some containers internally that interact with DMS on behalf of multiple clients. + +??? example "Roundcube + Fail2Ban" + + You may have other services with functionality like an API to send mail through DMS that likewise delegates credentials through DMS. + + Roundcube is an example of this where authentication is delegated to DMS, which introduces the same concern with loss of client IP. + + - While this service does implement some support for preserving the client IP, it is limited. + - This may be problematic when monitoring services like Fail2Ban are enabled that scan logs for multiple failed authentication attempts which triggers a ban on the shared IP address. + + You should adjust configuration of these monitoring services to monitor for auth failures from those services directly instead, adding an exclusion for that service IP from any DMS logs monitored (_but be mindful of PROXY header forgery risks_). + +[docs::overrides::dovecot]: ../../config/advanced/override-defaults/dovecot.md +[docs::overrides::postfix]: ../../config/advanced/override-defaults/postfix.md +[docs::overrides::user-patches]: ../../config/advanced/override-defaults/user-patches.md +[docs::ipv6::security-risks]: ../../config/advanced/ipv6.md#what-can-go-wrong +[docs::tls::traefik]: ../../config/security/ssl.md#traefik-v2 +[docs::env::permit_docker]: ../../config/environment.md#permit_docker +[gh-dms::dns-rewrite-example]: https://github.com/docker-mailserver/docker-mailserver/issues/3866#issuecomment-1928877236 + +[nginx-docs::proxyprotocol]: https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol +[caddy::plugin::l4]: https://github.com/mholt/caddy-l4 + +[traefik-web]: https://traefik.io +[traefik-docs::entrypoint::proxyprotocol]: https://doc.traefik.io/traefik/routing/entrypoints/#proxyprotocol +[traefik-docs::provider-docker::network]: https://doc.traefik.io/traefik/providers/docker/#network +[traefik-docs::router-tcp::server-first-protocols]: https://doc.traefik.io/traefik/routing/routers/#entrypoints_1 +[traefik-docs::router-tcp::host-sni]: https://doc.traefik.io/traefik/routing/routers/#rule_1 +[traefik-docs::router-tcp::passthrough]: https://doc.traefik.io/traefik/routing/routers/#passthrough +[traefik-docs::service-tcp::proxyprotocol]:https://doc.traefik.io/traefik/routing/services/#proxy-protocol + +[dovecot-docs::settings::haproxy-trusted-networks]: https://doc.dovecot.org/settings/core/#core_setting-haproxy_trusted_networks +[dovecot-docs::service-config::haproxy]: https://doc.dovecot.org/configuration_manual/service_configuration/#haproxy-v2-2-19 +[dovecot-docs::service-config::ssl]: https://doc.dovecot.org/configuration_manual/service_configuration/#ssl + +[postfix-docs::config-table::cidr]: https://www.postfix.org/cidr_table.5.html +[postfix-docs::settings::check_client_access]: https://www.postfix.org/postconf.5.html#check_client_access +[postfix-docs::settings::mynetworks]: https://www.postfix.org/postconf.5.html#mynetworks +[postfix-docs::settings::postscreen_access_list]: https://www.postfix.org/postconf.5.html#postscreen_access_list +[postfix-docs::settings::postscreen_upstream_proxy_protocol]: https://www.postfix.org/postconf.5.html#postscreen_upstream_proxy_protocol +[postfix-docs::settings::smtpd_client_restrictions]: https://www.postfix.org/postconf.5.html#smtpd_client_restrictions +[postfix-docs::settings::smtpd_upstream_proxy_protocol]: https://www.postfix.org/postconf.5.html#smtpd_upstream_proxy_protocol + +[docker::networking::l2-switch-gotcha]: https://github.com/moby/moby/issues/45610 +[networking::spec:proxy-protocol]: https://github.com/haproxy/haproxy/blob/master/doc/proxy-protocol.txt +[networking::http-header::x-forwarded-for]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For +[networking::http-header::forwarded]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded +[networking::osi-model]: https://www.cloudflare.com/learning/ddos/glossary/open-systems-interconnection-model-osi/ From 79a9656f4876e50264060eafbc066287a036a045 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 14 Feb 2024 09:48:35 +0000 Subject: [PATCH 036/267] docs: update `CONTRIBUTORS.md` (#3883) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- CONTRIBUTORS.md | 510 +++++++++++++++++++++++++----------------------- 1 file changed, 262 insertions(+), 248 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index a7c4e661..1e252291 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -628,6 +628,13 @@ Thanks goes to these wonderful people ✨ GoliathLabs + + + frugan-dev +
+ frugan-dev + + tbutter @@ -641,15 +648,15 @@ Thanks goes to these wonderful people ✨
yogo1212 - + + willtho89
willtho89 - - + mpanneck @@ -658,10 +665,17 @@ Thanks goes to these wonderful people ✨ - - andrewlow + + ubenmackin
- andrewlow + ubenmackin + + + + + craue +
+ craue @@ -671,6 +685,14 @@ Thanks goes to these wonderful people ✨ abh + + + andrewlow +
+ andrewlow + + + aminvakil @@ -685,21 +707,6 @@ Thanks goes to these wonderful people ✨ elbracht - - - ubenmackin -
- ubenmackin - - - - - - craue -
- craue - - danielpanteleit @@ -727,85 +734,14 @@ Thanks goes to these wonderful people ✨
DuncanvR - + + emazzotta
emazzotta - - - - - fl42 -
- fl42 - - - - - ipernet -
- ipernet - - - - - H4R0 -
- H4R0 - - - - - eltociear -
- eltociear - - - - - jamesfryer -
- jamesfryer - - - - - millaguie -
- millaguie - - - - - - artonge -
- artonge - - - - - jedateach -
- jedateach - - - - - spacecowboy -
- spacecowboy - - - - - martinwepner -
- martinwepner - @@ -815,39 +751,96 @@ Thanks goes to these wonderful people ✨ - - Thiritin + + martinwepner
- Thiritin + martinwepner + + + + + artonge +
+ artonge + + + + + spacecowboy +
+ spacecowboy + + + + + jedateach +
+ jedateach - - thomasschmit + + millaguie
- thomasschmit + millaguie - - TechnicLab + + eltociear
- TechnicLab + eltociear - - sylvaindumont + + H4R0
- sylvaindumont + H4R0 - - syl20bnr + + jamesfryer
- syl20bnr + jamesfryer + + + + + ipernet +
+ ipernet + + + + + fl42 +
+ fl42 + + + + + + simonsystem +
+ simonsystem + + + + + stephan-devop +
+ stephan-devop + + + + + stigok +
+ stigok @@ -858,25 +851,53 @@ Thanks goes to these wonderful people ✨ - - stigok + + syl20bnr
- stigok + syl20bnr + + + + + sylvaindumont +
+ sylvaindumont - - stephan-devop + + TechnicLab
- stephan-devop + TechnicLab - - radicand + + ShiriNmi1520
- radicand + ShiriNmi1520 + + + + + thomasschmit +
+ thomasschmit + + + + + Thiritin +
+ Thiritin + + + + + 42wim +
+ 42wim @@ -885,7 +906,8 @@ Thanks goes to these wonderful people ✨
tweibert - + + torus @@ -906,8 +928,7 @@ Thanks goes to these wonderful people ✨
Twist235 - - + k3it @@ -928,27 +949,6 @@ Thanks goes to these wonderful people ✨
vilisas - - - - 42wim -
- 42wim - - - - - ShiriNmi1520 -
- ShiriNmi1520 - - - - - Zepmann -
- Zepmann - @@ -1015,13 +1015,28 @@ Thanks goes to these wonderful people ✨ piwai + + + rahilarious +
+ rahilarious + + + + + 0xflotus +
+ 0xflotus + + remoe
remoe - + + romansey @@ -1035,8 +1050,7 @@ Thanks goes to these wonderful people ✨
norrs - - + MightySCollins @@ -1064,7 +1078,8 @@ Thanks goes to these wonderful people ✨
svdb0 - + + 3ap @@ -1078,8 +1093,7 @@ Thanks goes to these wonderful people ✨
shyim - - + sjmudd @@ -1088,17 +1102,10 @@ Thanks goes to these wonderful people ✨ - - simonsystem + + mchamplain
- simonsystem - - - - - allddd -
- allddd + mchamplain @@ -1114,15 +1121,15 @@ Thanks goes to these wonderful people ✨
mplx - + + odinis
odinis - - + okamidash @@ -1157,15 +1164,15 @@ Thanks goes to these wonderful people ✨
presocratics - + + rhyst
rhyst - - + rmlhuk @@ -1200,15 +1207,15 @@ Thanks goes to these wonderful people ✨
sportshead - + + squash
squash - - + strarsis @@ -1243,15 +1250,29 @@ Thanks goes to these wonderful people ✨
wolkenschieber - + + worldworm
worldworm - - + + + + Zepmann +
+ Zepmann + + + + + allddd +
+ allddd + + arcaine2 @@ -1272,7 +1293,8 @@ Thanks goes to these wonderful people ✨
brainkiller - + + cternes @@ -1293,8 +1315,7 @@ Thanks goes to these wonderful people ✨
dimalo - - + eleith @@ -1315,7 +1336,8 @@ Thanks goes to these wonderful people ✨
helmutundarnold - + + hnws @@ -1336,8 +1358,7 @@ Thanks goes to these wonderful people ✨
idaadi - - + ixeft @@ -1358,7 +1379,8 @@ Thanks goes to these wonderful people ✨
paralax - + + jpduyx @@ -1379,8 +1401,7 @@ Thanks goes to these wonderful people ✨
callmemagnus - - + marios88 @@ -1395,35 +1416,21 @@ Thanks goes to these wonderful people ✨ matrixes - - - mchamplain -
- mchamplain - - - - - crash7 -
- crash7 - - auchri
auchri - + + arkanovicz
arkanovicz - - + CBeerta @@ -1458,15 +1465,15 @@ Thanks goes to these wonderful people ✨
dbellavista - + + danielvandenberg95
danielvandenberg95 - - + denisix @@ -1501,15 +1508,15 @@ Thanks goes to these wonderful people ✨
vedtam - + + edvorg
edvorg - - + eliroca @@ -1544,20 +1551,20 @@ Thanks goes to these wonderful people ✨
felixn - + + flole
flole - - + - - 0xflotus + + froks
- 0xflotus + froks @@ -1587,15 +1594,15 @@ Thanks goes to these wonderful people ✨
askz - + + aspettl
aspettl - - + acch @@ -1630,15 +1637,15 @@ Thanks goes to these wonderful people ✨
ch3sh1r - + + eglia
eglia - - + groupmsl @@ -1673,15 +1680,15 @@ Thanks goes to these wonderful people ✨
arunvc - + + astrocket
astrocket - - + baxerus @@ -1704,10 +1711,10 @@ Thanks goes to these wonderful people ✨ - - akkumar + + crash7
- akkumar + crash7 @@ -1716,15 +1723,15 @@ Thanks goes to these wonderful people ✨
thechubbypanda - + + KCrawley
KCrawley - - + khuedoan @@ -1759,15 +1766,15 @@ Thanks goes to these wonderful people ✨
luke- - + + LucidityCrash
LucidityCrash - - + MadsRC @@ -1802,15 +1809,15 @@ Thanks goes to these wonderful people ✨
michaeljensen - + + exhuma
exhuma - - + milas @@ -1845,20 +1852,20 @@ Thanks goes to these wonderful people ✨
naveensrinivasan - + + neuralp
neuralp - - + - - froks + + radicand
- froks + radicand @@ -1868,13 +1875,6 @@ Thanks goes to these wonderful people ✨ fkefer - - - frugan-dev -
- frugan-dev - - Marsu31 @@ -1925,21 +1925,28 @@ Thanks goes to these wonderful people ✨ Influencer + + + JacksonZ03 +
+ JacksonZ03 + + JamBalaya56562
JamBalaya56562 - + + jcalfee
jcalfee - - + mivek @@ -1974,15 +1981,15 @@ Thanks goes to these wonderful people ✨
jirislav - + + jmccl
jmccl - - + jurekbarth @@ -2003,6 +2010,13 @@ Thanks goes to these wonderful people ✨
Kaan88 + + + + akkumar +
+ akkumar + From a815bf5ab44388baecb597a0b83f71872a2d34bc Mon Sep 17 00:00:00 2001 From: Robbert Klarenbeek Date: Fri, 16 Feb 2024 08:24:39 +0100 Subject: [PATCH 037/267] fix: Apply SELinux security context after moving to mail-state (#3890) * fix: Apply SELinux security context after moving to mail-state * fix: Ignore failing chcon on non-SELinux systems --- CHANGELOG.md | 1 + target/scripts/startup/setup.d/mail_state.sh | 6 ++++++ 2 files changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index c8030f43..2faf36f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -82,6 +82,7 @@ The most noteworthy change of this release is the update of the container's base - `RELAY_HOST` ENV no longer enforces configuring outbound SMTP to require credentials. Like `DEFAULT_RELAY_HOST` it can now configure a relay where credentials are optional. - Restarting DMS should not be required when configuring relay hosts without these ENV, but solely via `setup relay ...`, as change detection events now apply relevant Postfix setting changes for supporting credentials too. - Rspamd configuration: Add a missing comma in `local_networks` so that all internal IP addresses are actually considered as internal ([#3862](https://github.com/docker-mailserver/docker-mailserver/pull/3862)) +- Ensure correct SELinux security context labels for files and directories moved to the mail-state volume during setup ([#3890](https://github.com/docker-mailserver/docker-mailserver/pull/3890)) ## [v13.3.1](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v13.3.1) diff --git a/target/scripts/startup/setup.d/mail_state.sh b/target/scripts/startup/setup.d/mail_state.sh index 9c43fea4..5acf6762 100644 --- a/target/scripts/startup/setup.d/mail_state.sh +++ b/target/scripts/startup/setup.d/mail_state.sh @@ -48,6 +48,9 @@ function _setup_save_states() { _log 'trace' "Moving ${SERVICEFILE} to ${DEST}" # Empty volume was mounted, or new content from enabling a feature ENV: mv "${SERVICEFILE}" "${DEST}" + # Apply SELinux security context to match the state directory, so access + # is not restricted to the current running container: + chcon -R --reference="${STATEDIR}" "${DEST}" 2>/dev/null || true fi # Symlink the original file in the container ($SERVICEFILE) to be @@ -69,6 +72,9 @@ function _setup_save_states() { _log 'trace' "Moving contents of ${SERVICEDIR} to ${DEST}" # Empty volume was mounted, or new content from enabling a feature ENV: mv "${SERVICEDIR}" "${DEST}" + # Apply SELinux security context to match the state directory, so access + # is not restricted to the current running container: + chcon -R --reference="${STATEDIR}" "${DEST}" 2>/dev/null || true fi # Symlink the original path in the container ($SERVICEDIR) to be From d86c3cb1598d44afed1db60d3de8e3d1438dcd7c Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Tue, 20 Feb 2024 21:21:22 +1300 Subject: [PATCH 038/267] chore: `packages.sh` - Remove redundant comment (#3900) --- target/scripts/build/packages.sh | 1 - 1 file changed, 1 deletion(-) diff --git a/target/scripts/build/packages.sh b/target/scripts/build/packages.sh index 1f769739..5360ae95 100644 --- a/target/scripts/build/packages.sh +++ b/target/scripts/build/packages.sh @@ -147,7 +147,6 @@ function _install_dovecot() { _log 'trace' 'Using Dovecot community repository' curl -sSfL https://repo.dovecot.org/DOVECOT-REPO-GPG | gpg --import gpg --export ED409DA1 > /etc/apt/trusted.gpg.d/dovecot.gpg - # VERSION_CODENAME sourced from /etc/os-release echo "deb https://repo.dovecot.org/ce-2.3-latest/debian/${VERSION_CODENAME} ${VERSION_CODENAME} main" > /etc/apt/sources.list.d/dovecot.list _log 'trace' 'Updating Dovecot package signatures' From 67faa95b0b0e355afc84b86cf1b69ab89c674ab1 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Tue, 20 Feb 2024 21:33:04 +1300 Subject: [PATCH 039/267] fix(`setup`): `open-dkim` log for conflicting implementations (#3899) --- target/bin/open-dkim | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/target/bin/open-dkim b/target/bin/open-dkim index 808ef8cc..9410ecfa 100755 --- a/target/bin/open-dkim +++ b/target/bin/open-dkim @@ -5,7 +5,7 @@ source /usr/local/bin/helpers/index.sh if [[ -f /etc/dms-settings ]] && [[ $(_get_dms_env_value 'ENABLE_RSPAMD') -eq 1 ]]; then if [[ $(_get_dms_env_value 'ENABLE_OPENDKIM') -eq 1 ]]; then - log 'error' "You enabled Rspamd and OpenDKIM - OpenDKIM will be implicitly used for DKIM keys" + _log 'warn' "Conflicting DKIM support, both Rspamd and OpenDKIM enabled - OpenDKIM will manage DKIM keys" else /usr/local/bin/rspamd-dkim "${@}" exit From e232e43d32b4693a43a841250f2bd0c350f48c89 Mon Sep 17 00:00:00 2001 From: Casper Date: Wed, 21 Feb 2024 11:19:41 +0100 Subject: [PATCH 040/267] fix: fetchmail environment variables (#3901) --- CHANGELOG.md | 1 + target/supervisor/conf.d/supervisor-app.conf | 1 + 2 files changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2faf36f7..5b05b5d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -83,6 +83,7 @@ The most noteworthy change of this release is the update of the container's base - Restarting DMS should not be required when configuring relay hosts without these ENV, but solely via `setup relay ...`, as change detection events now apply relevant Postfix setting changes for supporting credentials too. - Rspamd configuration: Add a missing comma in `local_networks` so that all internal IP addresses are actually considered as internal ([#3862](https://github.com/docker-mailserver/docker-mailserver/pull/3862)) - Ensure correct SELinux security context labels for files and directories moved to the mail-state volume during setup ([#3890](https://github.com/docker-mailserver/docker-mailserver/pull/3890)) +- Use correct environment variable for fetchmail ([#3901](https://github.com/docker-mailserver/docker-mailserver/pull/3901)) ## [v13.3.1](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v13.3.1) diff --git a/target/supervisor/conf.d/supervisor-app.conf b/target/supervisor/conf.d/supervisor-app.conf index 50a7acc3..7f106456 100644 --- a/target/supervisor/conf.d/supervisor-app.conf +++ b/target/supervisor/conf.d/supervisor-app.conf @@ -122,6 +122,7 @@ autorestart=true stdout_logfile=/var/log/supervisor/%(program_name)s.log stderr_logfile=/var/log/supervisor/%(program_name)s.log user=fetchmail +environment=HOME="/var/lib/fetchmail",USER="fetchmail" command=/usr/bin/fetchmail -f /etc/fetchmailrc --nodetach --daemon "%(ENV_FETCHMAIL_POLL)s" -i /var/lib/fetchmail/.fetchmail-UIDL-cache --pidfile /var/run/fetchmail/fetchmail.pid [program:postfix] From 95dfc71b54a6445190555b2f192cc4e195fe38dc Mon Sep 17 00:00:00 2001 From: Jesse Portnoy Date: Sat, 24 Feb 2024 23:06:58 +0000 Subject: [PATCH 041/267] Fix typo and broken README link (#3906) --- docs/content/config/environment.md | 2 +- docs/content/usage.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/content/config/environment.md b/docs/content/config/environment.md index 323b6321..a0916aa3 100644 --- a/docs/content/config/environment.md +++ b/docs/content/config/environment.md @@ -212,7 +212,7 @@ Configures the handling of creating mails with forged sender addresses. ##### ENABLE_SRS -Enables the Sender Rewriting Scheme. SRS is needed if DMS acts as forwarder. See [postsrsd](https://github.com/roehling/postsrsd/blob/master/README.md#sender-rewriting-scheme-crash-course) for further explanation. +Enables the Sender Rewriting Scheme. SRS is needed if DMS acts as forwarder. See [postsrsd](https://github.com/roehling/postsrsd/blob/main/README.rst) for further explanation. - **0** => Disabled - 1 => Enabled diff --git a/docs/content/usage.md b/docs/content/usage.md index 30f6e822..76011fe7 100644 --- a/docs/content/usage.md +++ b/docs/content/usage.md @@ -2,7 +2,7 @@ title: Usage --- -This pages explains how to get started with DMS. The guide uses Docker Compose as a reference. In our examples, a volume mounts the host location [`docker-data/dms/config/`][docs::dms-volumes-config] to `/tmp/docker-mailserver/` inside the container. +This page explains how to get started with DMS. The guide uses Docker Compose as a reference. In our examples, a volume mounts the host location [`docker-data/dms/config/`][docs::dms-volumes-config] to `/tmp/docker-mailserver/` inside the container. [docs::dms-volumes-config]: ./config/advanced/optional-config.md#volumes-config From d3ccaddb708dcb427850e0dfe5bbbfee3047f8ec Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sun, 25 Feb 2024 12:54:49 +0100 Subject: [PATCH 042/267] docs: updated `CONTRIBUTORS.md` (#3909) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- CONTRIBUTORS.md | 239 +++++++++++++++++++++++++----------------------- 1 file changed, 123 insertions(+), 116 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 1e252291..c387a654 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -291,28 +291,21 @@ Thanks goes to these wonderful people ✨ pyy - - - dennis95stumm -
- dennis95stumm - - arneke
arneke - - + akmet
akmet - + + diiigle @@ -347,15 +340,15 @@ Thanks goes to these wonderful people ✨
lukecyca - - + jsonn
jsonn - + + jamebus @@ -363,13 +356,6 @@ Thanks goes to these wonderful people ✨ jamebus - - - dashohoxha -
- dashohoxha - - mathuin @@ -377,6 +363,13 @@ Thanks goes to these wonderful people ✨ mathuin + + + dashohoxha +
+ dashohoxha + + egavard @@ -390,15 +383,15 @@ Thanks goes to these wonderful people ✨
weo - - + Zehir
Zehir - + + guardiande @@ -433,15 +426,15 @@ Thanks goes to these wonderful people ✨
VanVan - - + mjung
mjung - + + m-schmoock @@ -476,15 +469,15 @@ Thanks goes to these wonderful people ✨
Starbix - - + citec
citec - + + yajo @@ -519,15 +512,15 @@ Thanks goes to these wonderful people ✨
MakerMatrix - - + pbek
pbek - + + keslerm @@ -562,15 +555,15 @@ Thanks goes to these wonderful people ✨
bobbravo2 - - + r-pufky
r-pufky - + + vincentDcmps @@ -605,15 +598,15 @@ Thanks goes to these wonderful people ✨
j-marz - - + lokipo
lokipo - + + msheakoski @@ -648,15 +641,15 @@ Thanks goes to these wonderful people ✨
yogo1212 - - + willtho89
willtho89 - + + mpanneck @@ -691,15 +684,15 @@ Thanks goes to these wonderful people ✨
andrewlow - - + aminvakil
aminvakil - + + elbracht @@ -734,15 +727,15 @@ Thanks goes to these wonderful people ✨
DuncanvR - - + emazzotta
emazzotta - + + nueaf @@ -777,34 +770,20 @@ Thanks goes to these wonderful people ✨
jedateach - - + millaguie
millaguie - + + - - eltociear + + fl42
- eltociear - - - - - H4R0 -
- H4R0 - - - - - jamesfryer -
- jamesfryer + fl42 @@ -815,10 +794,31 @@ Thanks goes to these wonderful people ✨ - - fl42 + + eltociear
- fl42 + eltociear + + + + + jamesfryer +
+ jamesfryer + + + + + H4R0 +
+ H4R0 + + + + + nilshoell +
+ nilshoell @@ -872,13 +872,6 @@ Thanks goes to these wonderful people ✨ TechnicLab - - - ShiriNmi1520 -
- ShiriNmi1520 - - thomasschmit @@ -893,21 +886,13 @@ Thanks goes to these wonderful people ✨ Thiritin - - - 42wim -
- 42wim - - tweibert
tweibert - - + torus @@ -921,7 +906,8 @@ Thanks goes to these wonderful people ✨
VictorKoenders - + + Twist235 @@ -949,15 +935,22 @@ Thanks goes to these wonderful people ✨
vilisas - - + - - nilshoell + + 42wim
- nilshoell + 42wim + + + ShiriNmi1520 +
+ ShiriNmi1520 + + + nknapp @@ -992,15 +985,15 @@ Thanks goes to these wonderful people ✨
mrPjer - - + p3dda
p3dda - + + peter-hartmann @@ -1035,6 +1028,13 @@ Thanks goes to these wonderful people ✨
remoe + + + + robbertkl +
+ robbertkl + @@ -1717,14 +1717,21 @@ Thanks goes to these wonderful people ✨ crash7 + + + fkefer +
+ fkefer + + + thechubbypanda
thechubbypanda - - + KCrawley @@ -1759,15 +1766,15 @@ Thanks goes to these wonderful people ✨
linhandev - + + luke-
luke- - - + LucidityCrash @@ -1802,15 +1809,15 @@ Thanks goes to these wonderful people ✨
dragetd - + + michaeljensen
michaeljensen - - + exhuma @@ -1845,15 +1852,15 @@ Thanks goes to these wonderful people ✨
mpldr - + + naveensrinivasan
naveensrinivasan - - + neuralp @@ -1868,13 +1875,6 @@ Thanks goes to these wonderful people ✨ radicand - - - fkefer -
- fkefer - - Marsu31 @@ -1968,21 +1968,28 @@ Thanks goes to these wonderful people ✨ Jeidnx + + + jessp01 +
+ jessp01 + + JiLleON
JiLleON - + + jirislav
jirislav - - + jmccl From 2c1faa72444a132785362f61a08b9cf659a5d5a9 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 26 Feb 2024 13:45:50 +0100 Subject: [PATCH 043/267] chore(deps): Bump myrotvorets/set-commit-status-action (#3911) --- .github/workflows/docs-preview-deploy.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/docs-preview-deploy.yml b/.github/workflows/docs-preview-deploy.yml index 3b6f3c67..dbe57752 100644 --- a/.github/workflows/docs-preview-deploy.yml +++ b/.github/workflows/docs-preview-deploy.yml @@ -45,7 +45,7 @@ jobs: # but presently does not work correctly via split workflow. It is useful in a split workflow as the 1st stage # no longer indicates if the entire workflow/deployment was successful. - name: 'Commit Status: Set Workflow Status as Pending' - uses: myrotvorets/set-commit-status-action@v2.0.0 + uses: myrotvorets/set-commit-status-action@v2.0.1 with: token: ${{ secrets.GITHUB_TOKEN }} status: pending @@ -105,7 +105,7 @@ jobs: Built with commit: ${{ env.PR_HEADSHA }} - name: 'Commit Status: Update deployment status' - uses: myrotvorets/set-commit-status-action@v2.0.0 + uses: myrotvorets/set-commit-status-action@v2.0.1 # Always run this step regardless of job failing early: if: ${{ always() }} env: From 512f39c7ebc6edce8b5ae5302a2777c4f9173bea Mon Sep 17 00:00:00 2001 From: Dominic Germain Date: Wed, 28 Feb 2024 09:34:30 -0500 Subject: [PATCH 044/267] feat: Configurable number of rotated log files (#3907) --- CHANGELOG.md | 5 +++++ docs/content/config/environment.md | 6 ++++++ mailserver.env | 3 +++ target/scripts/startup/setup.d/log.sh | 8 +++++++- target/scripts/startup/setup.d/security/rspamd.sh | 2 +- target/scripts/startup/variables-stack.sh | 1 + 6 files changed, 23 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b05b5d1..009e2355 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -60,6 +60,11 @@ The most noteworthy change of this release is the update of the container's base - `undef` was previously supported as an opt-out with `SA_SPAM_SUBJECT`. This is no longer valid, the equivalent opt-out value is now an empty value (_or rather the omission of this ENV being configured_). - The feature to include [`_SCORE_` tag](https://spamassassin.apache.org/full/4.0.x/doc/Mail_SpamAssassin_Conf.html#rewrite_header-subject-from-to-STRING) in your value to be replaced by the associated spam score is no longer available. +### Added + +- **Environment Variables:** + - `LOGROTATE_COUNT` defines the number of files kept by logrotate ([#3907](https://github.com/docker-mailserver/docker-mailserver/pull/3907)) + ### Updates - **Environment Variables:** diff --git a/docs/content/config/environment.md b/docs/content/config/environment.md index a0916aa3..83fcdfa6 100644 --- a/docs/content/config/environment.md +++ b/docs/content/config/environment.md @@ -545,6 +545,12 @@ Changes the interval in which log files are rotated. This variable can also determine the interval for Postfix's log summary reports, see [`PFLOGSUMM_TRIGGER`](#pflogsumm_trigger). +##### LOGROTATE_COUNT + +Defines how many files are kept by logrotate. + +- **4** => Number of files + #### SpamAssassin ##### ENABLE_SPAMASSASSIN diff --git a/mailserver.env b/mailserver.env index 44a9a484..0d00de0a 100644 --- a/mailserver.env +++ b/mailserver.env @@ -346,6 +346,9 @@ REPORT_SENDER= # Note: This variable can also determine the interval for Postfix's log summary reports, see [`PFLOGSUMM_TRIGGER`](#pflogsumm_trigger). LOGROTATE_INTERVAL=weekly +# Defines how many log files are kept by logrorate +LOGROTATE_COUNT=4 + # If enabled, employs `reject_unknown_client_hostname` to sender restrictions in Postfix's configuration. # diff --git a/target/scripts/startup/setup.d/log.sh b/target/scripts/startup/setup.d/log.sh index cf282966..06aa679d 100644 --- a/target/scripts/startup/setup.d/log.sh +++ b/target/scripts/startup/setup.d/log.sh @@ -19,13 +19,19 @@ function _setup_logrotate() { _dms_panic__invalid_value 'LOGROTATE_INTERVAL' 'Setup -> Logrotate' fi + if [[ ${LOGROTATE_COUNT} =~ ^[0-9]+$ ]]; then + _log 'trace' "Logrotate count set to ${LOGROTATE_COUNT}" + else + _dms_panic__invalid_value 'LOGROTATE_COUNT' 'Setup -> Logrotate' + fi + cat >/etc/logrotate.d/maillog << EOF /var/log/mail/mail.log { compress copytruncate delaycompress - rotate 4 + rotate ${LOGROTATE_COUNT} ${LOGROTATE_INTERVAL} } EOF diff --git a/target/scripts/startup/setup.d/security/rspamd.sh b/target/scripts/startup/setup.d/security/rspamd.sh index 98f83de6..6b54addd 100644 --- a/target/scripts/startup/setup.d/security/rspamd.sh +++ b/target/scripts/startup/setup.d/security/rspamd.sh @@ -109,7 +109,7 @@ function __rspamd__setup_logfile() { compress copytruncate delaycompress - rotate 4 + rotate ${LOGROTATE_COUNT} ${LOGROTATE_INTERVAL} } EOF diff --git a/target/scripts/startup/variables-stack.sh b/target/scripts/startup/variables-stack.sh index eb5bf149..a3be72b8 100644 --- a/target/scripts/startup/variables-stack.sh +++ b/target/scripts/startup/variables-stack.sh @@ -145,6 +145,7 @@ function __environment_variables_general_setup() { VARS[GETMAIL_POLL]="${GETMAIL_POLL:=5}" VARS[LOG_LEVEL]="${LOG_LEVEL:=info}" VARS[LOGROTATE_INTERVAL]="${LOGROTATE_INTERVAL:=weekly}" + VARS[LOGROTATE_COUNT]="${LOGROTATE_COUNT:=4}" VARS[LOGWATCH_INTERVAL]="${LOGWATCH_INTERVAL:=none}" VARS[LOGWATCH_RECIPIENT]="${LOGWATCH_RECIPIENT:=${REPORT_RECIPIENT}}" VARS[LOGWATCH_SENDER]="${LOGWATCH_SENDER:=${REPORT_SENDER}}" From aa9465773c99a25dea7ca9534f1928ed6301693a Mon Sep 17 00:00:00 2001 From: Casper Date: Wed, 28 Feb 2024 22:08:19 +0100 Subject: [PATCH 045/267] Rename supervisor-app.conf to dms-services.conf (#3908) * rename supervisor-app.conf to dms-services.conf * changelog added --- CHANGELOG.md | 2 ++ .../conf.d/{supervisor-app.conf => dms-services.conf} | 0 .../set3/container_configuration/process_check_restart.bats | 4 ++-- 3 files changed, 4 insertions(+), 2 deletions(-) rename target/supervisor/conf.d/{supervisor-app.conf => dms-services.conf} (100%) diff --git a/CHANGELOG.md b/CHANGELOG.md index 009e2355..62da354f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -59,6 +59,8 @@ The most noteworthy change of this release is the update of the container's base - The default has changed to not prepend any prefix to the subject unless configured to do so. If you relied on the implicit prefix, you will now need to provide one explicitly. - `undef` was previously supported as an opt-out with `SA_SPAM_SUBJECT`. This is no longer valid, the equivalent opt-out value is now an empty value (_or rather the omission of this ENV being configured_). - The feature to include [`_SCORE_` tag](https://spamassassin.apache.org/full/4.0.x/doc/Mail_SpamAssassin_Conf.html#rewrite_header-subject-from-to-STRING) in your value to be replaced by the associated spam score is no longer available. +- **Supervisord**: + - `supervisor-app.conf` renamed to `dms-services.conf` ### Added diff --git a/target/supervisor/conf.d/supervisor-app.conf b/target/supervisor/conf.d/dms-services.conf similarity index 100% rename from target/supervisor/conf.d/supervisor-app.conf rename to target/supervisor/conf.d/dms-services.conf diff --git a/test/tests/parallel/set3/container_configuration/process_check_restart.bats b/test/tests/parallel/set3/container_configuration/process_check_restart.bats index 4f0fd90f..78eabd7a 100644 --- a/test/tests/parallel/set3/container_configuration/process_check_restart.bats +++ b/test/tests/parallel/set3/container_configuration/process_check_restart.bats @@ -184,7 +184,7 @@ function _check_if_process_is_running() { # `--list-full` provides information for matching against (full process path) # `--full` allows matching the process against the full path (required if a process is not the exec command, such as started by python3 command without a shebang) - # `--oldest` should select the parent process when there are multiple results, typically the command defined in `supervisor-app.conf` + # `--oldest` should select the parent process when there are multiple results, typically the command defined in `dms-services.conf` local IS_RUNNING=$(_exec_in_container pgrep --full --list-full "${MIN_SECS_RUNNING[@]}" --oldest "${PROCESS}") # When no matches are found, nothing is returned. Provide something we can assert on (helpful for debugging): @@ -199,7 +199,7 @@ function _check_if_process_is_running() { # The process manager (supervisord) should perform a graceful shutdown: # NOTE: Time limit should never be below these configured values: -# - supervisor-app.conf:stopwaitsecs +# - dms-services.conf:stopwaitsecs # - compose.yaml:stop_grace_period function _should_stop_cleanly() { run docker stop -t 60 "${CONTAINER_NAME}" From 736f2e44bcddac31b0bed506e5e60756d8dd032a Mon Sep 17 00:00:00 2001 From: Casper Date: Fri, 1 Mar 2024 01:00:23 +0100 Subject: [PATCH 046/267] Fail2Ban: Align logrotate count & interval (#3915) --- CHANGELOG.md | 1 + target/scripts/startup/setup.d/security/misc.sh | 6 ++++++ 2 files changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 62da354f..6b663f46 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,7 @@ The most noteworthy change of this release is the update of the container's base - **Environment Variables:** - `LOGROTATE_COUNT` defines the number of files kept by logrotate ([#3907](https://github.com/docker-mailserver/docker-mailserver/pull/3907)) + - The fail2ban log file is now also taken into account by `LOGROTATE_COUNT` and `LOGROTATE_INTERVAL` ([#3915](https://github.com/docker-mailserver/docker-mailserver/pull/3915)) ### Updates diff --git a/target/scripts/startup/setup.d/security/misc.sh b/target/scripts/startup/setup.d/security/misc.sh index aefeba20..df810b2e 100644 --- a/target/scripts/startup/setup.d/security/misc.sh +++ b/target/scripts/startup/setup.d/security/misc.sh @@ -189,14 +189,17 @@ function __setup__security__fail2ban() { _log 'debug' 'Enabling and configuring Fail2Ban' if [[ -e /tmp/docker-mailserver/fail2ban-fail2ban.cf ]]; then + _log 'trace' 'Custom fail2ban-fail2ban.cf found' cp /tmp/docker-mailserver/fail2ban-fail2ban.cf /etc/fail2ban/fail2ban.local fi if [[ -e /tmp/docker-mailserver/fail2ban-jail.cf ]]; then + _log 'trace' 'Custom fail2ban-jail.cf found' cp /tmp/docker-mailserver/fail2ban-jail.cf /etc/fail2ban/jail.d/user-jail.local fi if [[ ${FAIL2BAN_BLOCKTYPE} != 'reject' ]]; then + _log 'trace' "Setting fail2ban blocktype to 'drop'" echo -e '[Init]\nblocktype = drop' >/etc/fail2ban/action.d/nftables-common.local fi @@ -205,6 +208,9 @@ function __setup__security__fail2ban() { _log 'debug' 'Fail2Ban is disabled' rm -f /etc/logrotate.d/fail2ban fi + _log 'trace' 'Configuring fail2ban logrotate rotate count and interval' + sedfile -i "s|rotate 4$|rotate ${LOGROTATE_COUNT}|" /etc/logrotate.d/fail2ban + sedfile -i "s|weekly$|${LOGROTATE_INTERVAL}|" /etc/logrotate.d/fail2ban } function __setup__security__amavis() { From 12f5101d84331e23dea2192b05c502de5e610116 Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Sat, 2 Mar 2024 02:42:47 +0100 Subject: [PATCH 047/267] Rspamd: improve SPF, DKIM and DMARC Symbol Weights (#3913) --- CHANGELOG.md | 5 +- docs/content/config/security/rspamd.md | 32 ++++---- target/rspamd/scores.d/policies_group.conf | 86 +++++++++++----------- 3 files changed, 66 insertions(+), 57 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b663f46..30c7dd0c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -78,8 +78,9 @@ The most noteworthy change of this release is the update of the container's base - **Tests:** - Refactored helper methods for sending e-mails with specific `Message-ID` headers and the helpers for retrieving + filtering logs, which together help isolate logs relevant to specific mail when multiple mails have been processed within a single test. ([#3786](https://github.com/docker-mailserver/docker-mailserver/pull/3786)) - **Rspamd**: - - The `rewrite_subject` action, is now disabled by default. It has been replaced with the new `SPAM_SUBJECT` environment variable, which implements the functionality via a Sieve script instead in favor of being anti-spam service agnostic ([3820](https://github.com/docker-mailserver/docker-mailserver/pull/3820)) - - `RSPAMD_NEURAL` was added and is disabled by default. If switched on it wil enable the experimental Rspamd Neural network module to add a layer of analysis to spam detection using neural network technology. ([3833](https://github.com/docker-mailserver/docker-mailserver/pull/3833)) + - The `rewrite_subject` action, is now disabled by default. It has been replaced with the new `SPAM_SUBJECT` environment variable, which implements the functionality via a Sieve script instead which is anti-spam service agnostic ([#3820](https://github.com/docker-mailserver/docker-mailserver/pull/3820)) + - `RSPAMD_NEURAL` was added and is disabled by default. If switched on it will enable the experimental Rspamd "Neural network" module to add a layer of analysis to spam detection ([#3833](https://github.com/docker-mailserver/docker-mailserver/pull/3833)) + - The symbol weights of SPF, DKIM and DMARC have been adjusted again. Fixes a bug and includes more appropriate combinations of symbols ([#3913](https://github.com/docker-mailserver/docker-mailserver/pull/3913)) ### Fixes diff --git a/docs/content/config/security/rspamd.md b/docs/content/config/security/rspamd.md index 6dc5f202..5cb901b7 100644 --- a/docs/content/config/security/rspamd.md +++ b/docs/content/config/security/rspamd.md @@ -4,9 +4,9 @@ title: 'Security | Rspamd' ## About -Rspamd is a ["fast, free and open-source spam filtering system"][www::rspamd-homepage]. DMS integrates Rspamd like any other service. We provide a basic but easy to maintain setup of Rspamd. +Rspamd is a ["fast, free and open-source spam filtering system"][rspamd-web]. DMS integrates Rspamd like any other service. We provide a basic but easy to maintain setup of Rspamd. -If you want to take a look at the default configuration files for Rspamd that DMS packs, navigate to [`target/rspamd/` inside the repository][repo::dms-default-configuration]. Please consult the [section "The Default Configuration"](#the-default-configuration) section down below for a written overview. +If you want to take a look at the default configuration files for Rspamd that DMS packs, navigate to [`target/rspamd/` inside the repository][dms-repo::default-rspamd-configuration]. Please consult the [section "The Default Configuration"](#the-default-configuration) section down below for a written overview. ## Related Environment Variables @@ -56,7 +56,7 @@ And then there is a corresponding `X-Rspamd-Action` header, which shows the over X-Rspamd-Action no action ``` -Since the score is `-2.80`, nothing will happen and the e-mail is not classified as spam. Our custom [`actions.conf`][www::rspamd-actions-config] defines what to do at certain scores: +Since the score is `-2.80`, nothing will happen and the e-mail is not classified as spam. Our custom [`actions.conf`][dms-repo::rspamd-actions-config] defines what to do at certain scores: 1. At a score of 4, the e-mail is to be _greylisted_; 2. At a score of 6, the e-mail is _marked with a header_ (`X-Spam: Yes`); @@ -64,13 +64,13 @@ Since the score is `-2.80`, nothing will happen and the e-mail is not classified --- -There is more to spam analysis than meets the eye: we have not covered the [Bayes training and filters][www::rspamd-docs-bayes] here, nor have we discussed [Sieve rules for e-mails that are marked as spam][docs::spam-to-junk]. +There is more to spam analysis than meets the eye: we have not covered the [Bayes training and filters][rspamd-docs::bayes] here, nor have we discussed [Sieve rules for e-mails that are marked as spam][docs::spam-to-junk]. Even the calculation of the score with the individual symbols has been presented to you in a simplified manner. But with the knowledge from above, you're equipped to read on and use Rspamd confidently. Keep on reading to understand the integration even better - you will want to know about your anti-spam software, not only to keep the bad e-mail out, but also to make sure the good e-mail arrive properly! ### Workers -The proxy worker operates in [self-scan mode][www::rspamd-docs-proxy-self-scan-mode]. This simplifies the setup as we do not require a normal worker. You can easily change this though by [overriding the configuration by DMS](#providing-custom-settings-overriding-settings). +The proxy worker operates in [self-scan mode][rspamd-docs::proxy-self-scan-mode]. This simplifies the setup as we do not require a normal worker. You can easily change this though by [overriding the configuration by DMS](#providing-custom-settings-overriding-settings). DMS does not set a default password for the controller worker. You may want to do that yourself. In setups where you already have an authentication provider in front of the Rspamd webpage, you may want to [set the `secure_ip ` option to `"0.0.0.0/0"` for the controller worker](#with-the-help-of-a-custom-file) to disable password authentication inside Rspamd completely. @@ -88,18 +88,24 @@ Redis uses `/etc/redis/redis.conf` for configuration: ### Web Interface -Rspamd provides a [web interface][www::rspamd-docs-web-interface], which contains statistics and data Rspamd collects. The interface is enabled by default and reachable on port 11334. +Rspamd provides a [web interface][rspamd-docs::web-interface], which contains statistics and data Rspamd collects. The interface is enabled by default and reachable on port 11334. ![Rspamd Web Interface](https://rspamd.com/img/webui.png) ### DNS -DMS does not supply custom values for DNS servers to Rspamd. If you need to use custom DNS servers, which could be required when using [DNS-based black/whitelists](#rbls-realtime-blacklists-dnsbls-dns-based-blacklists), you need to adjust [`options.inc`][www::rspamd-docs-basic-options] yourself. +DMS does not supply custom values for DNS servers to Rspamd. If you need to use custom DNS servers, which could be required when using [DNS-based black/whitelists](#rbls-realtime-blacklists-dnsbls-dns-based-blacklists), you need to adjust [`options.inc`][rspamd-docs::basic-options] yourself. !!! tip "Making DNS Servers Configurable" If you want to see an environment variable (like `RSPAMD_DNS_SERVERS`) to support custom DNS servers for Rspamd being added to DMS, please raise a feature request issue. +!!! warning + + Rspamd heavily relies on a properly working DNS server that it can use to resolve DNS queries. If your DNS server is misconfigured, you will encounter issues when Rspamd queries DNS to assess if mail is spam. Legitimate mail is then unintentionally marked as spam or worse, rejected entirely. + + When Rspamd is deciding if mail is spam, it will check DNS records for SPF, DKIM and DMARC. Each of those has an associated symbol for DNS temporary errors with a non-zero weight assigned. That weight contributes towards the spam score assessed by Rspamd which is normally desirable - provided your network DNS is functioning correctly, otherwise when DNS is broken all mail is biased towards spam due to these failed DNS lookups. + !!! danger While we do not provide values for custom DNS servers by default, we set `soft_reject_on_timeout = true;` by default. This setting will cause a soft reject if a task (presumably a DNS request) timeout takes place. @@ -112,7 +118,7 @@ You can find the Rspamd logs at `/var/log/mail/rspamd.log`, and the correspondin ### Modules -You can find a list of all Rspamd modules [on their website][www::rspamd-docs-modules]. +You can find a list of all Rspamd modules [on their website][rspamd-docs::modules]. #### Disabled By Default @@ -140,9 +146,9 @@ DMS brings sane default settings for Rspamd. They are located at `/etc/rspamd/lo !!! question "What is [`docker-data/dms/config/`][docs::dms-volumes-config]?" -If you want to overwrite the default settings or provide your settings, you can place files at `docker-data/dms/config/rspamd/override.d/`. Files from this directory are copied to `/etc/rspamd/override.d/` during startup. These files [forcibly override][www::rspamd-docs-override-dir] Rspamd and DMS default settings. +If you want to overwrite the default settings or provide your settings, you can place files at `docker-data/dms/config/rspamd/override.d/`. Files from this directory are copied to `/etc/rspamd/override.d/` during startup. These files [forcibly override][rspamd-docs::override-dir] Rspamd and DMS default settings. -!!! question "What is the [`local.d` directory and how does it compare to `override.d`][www::rspamd-docs-config-directories]?" +!!! question "What is the [`local.d` directory and how does it compare to `override.d`][rspamd-docs::config-directories]?" !!! warning "Clashing Overrides" @@ -163,7 +169,7 @@ where `COMMAND` can be: 3. `set-option-for-module`: sets the value for option `ARGUMENT2` to `ARGUMENT3` inside module `ARGUMENT1` 4. `set-option-for-controller`: set the value of option `ARGUMENT1` to `ARGUMENT2` for the controller worker 5. `set-option-for-proxy`: set the value of option `ARGUMENT1` to `ARGUMENT2` for the proxy worker -6. `set-common-option`: set the option `ARGUMENT1` that [defines basic Rspamd behavior][www::rspamd-docs-basic-options] to value `ARGUMENT2` +6. `set-common-option`: set the option `ARGUMENT1` that [defines basic Rspamd behavior][rspamd-docs::basic-options] to value `ARGUMENT2` 7. `add-line`: this will add the complete line after `ARGUMENT1` (with all characters) to the file `/etc/rspamd/override.d/` !!! example "An Example Is [Shown Down Below](#adjusting-and-extending-the-very-basic-configuration)" @@ -231,11 +237,11 @@ There is a dedicated [section for setting up DKIM with Rspamd in our documentati ### _Abusix_ Integration -This subsection provides information about the integration of [Abusix][www::abusix], "a set of blocklists that work as an additional email security layer for your existing mail environment". The setup is straight-forward and well documented: +This subsection provides information about the integration of [Abusix][abusix-web], "a set of blocklists that work as an additional email security layer for your existing mail environment". The setup is straight-forward and well documented: 1. [Create an account](https://app.abusix.com/signup) 2. Retrieve your API key -3. Navigate to the ["Getting Started" documentation for Rspamd][www::abusix-rspamd-integration] and follow the steps described there +3. Navigate to the ["Getting Started" documentation for Rspamd][abusix-docs::rspamd-integration] and follow the steps described there 4. Make sure to change `` to your private API key We recommend mounting the files directly into the container, as they are rather big and not manageable with our [`custom-command.conf` script](#with-the-help-of-a-custom-file). If mounted to the correct location, Rspamd will automatically pick them up. diff --git a/target/rspamd/scores.d/policies_group.conf b/target/rspamd/scores.d/policies_group.conf index 5f9426e9..f2048665 100644 --- a/target/rspamd/scores.d/policies_group.conf +++ b/target/rspamd/scores.d/policies_group.conf @@ -1,72 +1,75 @@ # Please refer to # https://github.com/docker-mailserver/docker-mailserver/issues/3690 # for understanding this file and its scores' values. +# +# This configuration is not 100% compliant with RFC7489. +# This is intentional! Rspamd has additional symbols than those defined in this file. +# 100% compliance is not desirable as those symbols will change the overall spam score. symbols = { # SPF - "R_SPF_ALLOW" { + "R_SPF_ALLOW" { # SPF check succeeded weight = -1; description = "SPF verification allows sending"; groups = ["spf"]; } - "R_SPF_NA" { + "R_SPF_NA" { # SPF is not available for this domain weight = 1.5; description = "Missing SPF record"; one_shot = true; groups = ["spf"]; } - "R_SPF_SOFTFAIL" { - weight = 2.5; - description = "SPF verification soft-failed"; - groups = ["spf"]; - } - "R_SPF_FAIL" { - weight = 4.5; - description = "SPF verification failed"; - groups = ["spf"]; - } - - "R_SPF_NEUTRAL" { # == R_SPF_NA + "R_SPF_NEUTRAL" { # same as R_SPF_NA weight = 1.5; description = "SPF policy is neutral"; groups = ["spf"]; } - "R_SPF_DNSFAIL" { # == R_SPF_SOFTFAIL + "R_SPF_SOFTFAIL" { # there was a temporary DNS issue and SPF could not be checked + weight = 2.5; + description = "SPF verification soft-failed"; + groups = ["spf"]; + } + "R_SPF_DNSFAIL" { # same as R_SPF_SOFTFAIL weight = 2.5; description = "SPF DNS failure"; groups = ["spf"]; } - "R_SPF_PERMFAIL" { # == R_SPF_FAIL + "R_SPF_FAIL" { # SPF check failed + weight = 4.5; + description = "SPF verification failed"; + groups = ["spf"]; + } + "R_SPF_PERMFAIL" { # same as R_SPF_FAIL weight = 4.5; description = "SPF record is malformed or persistent DNS error"; groups = ["spf"]; } # DKIM - "R_DKIM_ALLOW" { + "R_DKIM_ALLOW" { # DKIM check succeeded weight = -1; description = "DKIM verification succeed"; one_shot = true; groups = ["dkim"]; } - "R_DKIM_NA" { - weight = 0; + "R_DKIM_NA" { # DKIM is not available for this domain + weight = 1; description = "Missing DKIM signature"; one_shot = true; groups = ["dkim"]; } - "R_DKIM_TEMPFAIL" { + "R_DKIM_TEMPFAIL" { # there was a temporary DNS issue and DKIM could not be checked weight = 1.5; description = "DKIM verification soft-failed"; groups = ["dkim"]; } - "R_DKIM_PERMFAIL" { + "R_DKIM_PERMFAIL" { # DKIM check failed weight = 4.5; description = "DKIM verification hard-failed (invalid)"; groups = ["dkim"]; } - "R_DKIM_REJECT" { # == R_DKIM_PERMFAIL + "R_DKIM_REJECT" { # same as R_DKIM_PERMFAIL weight = 4.5; description = "DKIM verification failed"; one_shot = true; @@ -74,35 +77,34 @@ symbols = { } # DMARC - "DMARC_NA" { - weight = 1; - description = "No DMARC record"; - groups = ["dmarc"]; - } - "DMARC_POLICY_QUARANTINE" { - weight = 1.5; - description = "DMARC quarantine policy"; - groups = ["dmarc"]; - } - "DMARC_POLICY_REJECT" { - weight = 2; - description = "DMARC reject policy"; - groups = ["dmarc"]; - } - - "DMARC_POLICY_ALLOW" { # no equivalent + "DMARC_POLICY_ALLOW" { # DMARC check succeeded weight = -1; description = "DMARC permit policy"; groups = ["dmarc"]; } - "DMARC_POLICY_ALLOW_WITH_FAILURES" { # no equivalent - weight = -0.5; + "DMARC_POLICY_ALLOW_WITH_FAILURES" { # DMARC check succeeded but either SPF or DKIM was not successful + weight = 0; description = "DMARC permit policy with DKIM/SPF failure"; groups = ["dmarc"]; } - "DMARC_POLICY_SOFTFAIL" { # == DMARC_POLICY_QUARANTINE + "DMARC_NA" { # DMARC is not available for this domain + weight = 0.5; + description = "No DMARC record"; + groups = ["dmarc"]; + } + "DMARC_POLICY_SOFTFAIL" { # there was a temporary DNS issue and DMARC could not be checked weight = 1.5; description = "DMARC soft-failed"; groups = ["dmarc"]; } + "DMARC_POLICY_QUARANTINE" { # DMARC check failed and the policy is to quarantine + weight = 3; + description = "DMARC quarantine policy"; + groups = ["dmarc"]; + } + "DMARC_POLICY_REJECT" { # DMARC check failed and the policy is to reject + weight = 5.5; + description = "DMARC reject policy"; + groups = ["dmarc"]; + } } From 83a48e8958c0fcba9778117f235e6064b401f3a1 Mon Sep 17 00:00:00 2001 From: Casper Date: Sun, 3 Mar 2024 22:48:42 +0100 Subject: [PATCH 048/267] Fail2ban logrotate interval/count: substitute only when necessary (#3919) --- CHANGELOG.md | 2 +- target/scripts/startup/setup.d/security/misc.sh | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 30c7dd0c..b60a7afa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,7 +66,7 @@ The most noteworthy change of this release is the update of the container's base - **Environment Variables:** - `LOGROTATE_COUNT` defines the number of files kept by logrotate ([#3907](https://github.com/docker-mailserver/docker-mailserver/pull/3907)) - - The fail2ban log file is now also taken into account by `LOGROTATE_COUNT` and `LOGROTATE_INTERVAL` ([#3915](https://github.com/docker-mailserver/docker-mailserver/pull/3915)) + - The fail2ban log file is now also taken into account by `LOGROTATE_COUNT` and `LOGROTATE_INTERVAL` ([#3915](https://github.com/docker-mailserver/docker-mailserver/pull/3915), [#3919](https://github.com/docker-mailserver/docker-mailserver/pull/3919)) ### Updates diff --git a/target/scripts/startup/setup.d/security/misc.sh b/target/scripts/startup/setup.d/security/misc.sh index df810b2e..303ae62d 100644 --- a/target/scripts/startup/setup.d/security/misc.sh +++ b/target/scripts/startup/setup.d/security/misc.sh @@ -209,8 +209,8 @@ function __setup__security__fail2ban() { rm -f /etc/logrotate.d/fail2ban fi _log 'trace' 'Configuring fail2ban logrotate rotate count and interval' - sedfile -i "s|rotate 4$|rotate ${LOGROTATE_COUNT}|" /etc/logrotate.d/fail2ban - sedfile -i "s|weekly$|${LOGROTATE_INTERVAL}|" /etc/logrotate.d/fail2ban + [[ ${LOGROTATE_COUNT} -ne 4 ]] && sedfile -i "s|rotate 4$|rotate ${LOGROTATE_COUNT}|" /etc/logrotate.d/fail2ban + [[ ${LOGROTATE_INTERVAL} != "weekly" ]] && sedfile -i "s|weekly$|${LOGROTATE_INTERVAL}|" /etc/logrotate.d/fail2ban } function __setup__security__amavis() { From 0c8d8f26d9323789569cab9ceac33e0b541380e0 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sun, 3 Mar 2024 22:50:06 +0100 Subject: [PATCH 049/267] docs: updated `CONTRIBUTORS.md` (#3916) --- CONTRIBUTORS.md | 356 +++++++++++++++++++++++++----------------------- 1 file changed, 182 insertions(+), 174 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index c387a654..ffe5cecd 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -642,40 +642,19 @@ Thanks goes to these wonderful people ✨ yogo1212 - - - willtho89 -
- willtho89 - - - mpanneck
mpanneck - + + - - ubenmackin + + willtho89
- ubenmackin - - - - - craue -
- craue - - - - - abh -
- abh + willtho89 @@ -685,14 +664,20 @@ Thanks goes to these wonderful people ✨ andrewlow + + + abh +
+ abh + + aminvakil
aminvakil - - + elbracht @@ -700,6 +685,21 @@ Thanks goes to these wonderful people ✨ elbracht + + + craue +
+ craue + + + + + + ubenmackin +
+ ubenmackin + + danielpanteleit @@ -735,49 +735,6 @@ Thanks goes to these wonderful people ✨ emazzotta - - - - nueaf -
- nueaf - - - - - martinwepner -
- martinwepner - - - - - artonge -
- artonge - - - - - spacecowboy -
- spacecowboy - - - - - jedateach -
- jedateach - - - - - millaguie -
- millaguie - - @@ -793,6 +750,13 @@ Thanks goes to these wonderful people ✨ ipernet + + + H4R0 +
+ H4R0 + + eltociear @@ -808,60 +772,53 @@ Thanks goes to these wonderful people ✨ - - H4R0 + + millaguie
- H4R0 - - - - - nilshoell -
- nilshoell + millaguie - - simonsystem + + martinwepner
- simonsystem + martinwepner - - stephan-devop + + artonge
- stephan-devop + artonge - - stigok + + jedateach
- stigok + jedateach - - 5ven + + spacecowboy
- 5ven + spacecowboy - - syl20bnr + + nueaf
- syl20bnr + nueaf - - sylvaindumont + + thomasschmit
- sylvaindumont + thomasschmit @@ -873,10 +830,53 @@ Thanks goes to these wonderful people ✨ - - thomasschmit + + sylvaindumont
- thomasschmit + sylvaindumont + + + + + syl20bnr +
+ syl20bnr + + + + + 5ven +
+ 5ven + + + + + stigok +
+ stigok + + + + + stephan-devop +
+ stephan-devop + + + + + + simonsystem +
+ simonsystem + + + + + radicand +
+ radicand @@ -951,6 +951,13 @@ Thanks goes to these wonderful people ✨ + + + nilshoell +
+ nilshoell + + nknapp @@ -985,15 +992,15 @@ Thanks goes to these wonderful people ✨
mrPjer - + + p3dda
p3dda - - + peter-hartmann @@ -1015,13 +1022,6 @@ Thanks goes to these wonderful people ✨ rahilarious - - - 0xflotus -
- 0xflotus - - remoe @@ -1416,14 +1416,21 @@ Thanks goes to these wonderful people ✨ matrixes + + + 0xflotus +
+ 0xflotus + + + auchri
auchri - - + arkanovicz @@ -1458,15 +1465,15 @@ Thanks goes to these wonderful people ✨
dkarski - + + dbellavista
dbellavista - - + danielvandenberg95 @@ -1495,6 +1502,14 @@ Thanks goes to these wonderful people ✨ mazzz1y + + + doominator42 +
+ doominator42 + + + aydodo @@ -1508,8 +1523,7 @@ Thanks goes to these wonderful people ✨
vedtam - - + edvorg @@ -1537,7 +1551,8 @@ Thanks goes to these wonderful people ✨
ErikEngerd - + + huncode @@ -1551,8 +1566,7 @@ Thanks goes to these wonderful people ✨
felixn - - + flole @@ -1560,13 +1574,6 @@ Thanks goes to these wonderful people ✨ flole - - - froks -
- froks - - ifokeev @@ -1587,15 +1594,15 @@ Thanks goes to these wonderful people ✨
2b - + + askz
askz - - + aspettl @@ -1630,15 +1637,15 @@ Thanks goes to these wonderful people ✨
alexanderneu - + + ch3sh1r
ch3sh1r - - + eglia @@ -1673,15 +1680,15 @@ Thanks goes to these wonderful people ✨
MrFreezeex - + + arunvc
arunvc - - + astrocket @@ -1716,15 +1723,22 @@ Thanks goes to these wonderful people ✨
crash7 - - - - fkefer -
- fkefer - + + + froks +
+ froks + + + + + akkumar +
+ akkumar + + thechubbypanda @@ -1752,7 +1766,8 @@ Thanks goes to these wonderful people ✨
JustAnother1 - + + LeoWinterDE @@ -1766,8 +1781,7 @@ Thanks goes to these wonderful people ✨
linhandev - - + luke- @@ -1795,7 +1809,8 @@ Thanks goes to these wonderful people ✨
madmath03 - + + maxemann96 @@ -1809,8 +1824,7 @@ Thanks goes to these wonderful people ✨
dragetd - - + michaeljensen @@ -1838,7 +1852,8 @@ Thanks goes to these wonderful people ✨
mcchots - + + MohammedNoureldin @@ -1852,8 +1867,7 @@ Thanks goes to these wonderful people ✨
mpldr - - + naveensrinivasan @@ -1869,10 +1883,10 @@ Thanks goes to these wonderful people ✨ - - radicand + + fkefer
- radicand + fkefer @@ -1881,7 +1895,8 @@ Thanks goes to these wonderful people ✨
Marsu31 - + + glandais @@ -1895,8 +1910,7 @@ Thanks goes to these wonderful people ✨
GiovanH - - + harryyoud @@ -1924,7 +1938,8 @@ Thanks goes to these wonderful people ✨
Influencer - + + JacksonZ03 @@ -1938,8 +1953,7 @@ Thanks goes to these wonderful people ✨
JamBalaya56562 - - + jcalfee @@ -1967,7 +1981,8 @@ Thanks goes to these wonderful people ✨
Jeidnx - + + jessp01 @@ -1981,8 +1996,7 @@ Thanks goes to these wonderful people ✨
JiLleON - - + jirislav @@ -2010,20 +2024,14 @@ Thanks goes to these wonderful people ✨
JOduMonT - + + Kaan88
Kaan88 - - - - akkumar -
- akkumar - From 899b644a04cfbfed10d03d67a6864a2ef988ee32 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 4 Mar 2024 18:19:57 +0100 Subject: [PATCH 050/267] chore(deps): Bump docker/setup-buildx-action from 3.0.0 to 3.1.0 (#3924) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index ccef46f5..660b0a2b 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -79,7 +79,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.0.0 + uses: docker/setup-buildx-action@v3.1.0 # NOTE: AMD64 can build within 2 minutes - name: 'Build images' diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 83357062..bd776cfd 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -40,7 +40,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.0.0 + uses: docker/setup-buildx-action@v3.1.0 # Try get the cached build layers from a prior `generic_build.yml` job. # NOTE: Until adopting `type=gha` scoped cache exporter (in `docker/build-push-action`), diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 2c1d1045..bc26ef4f 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -38,7 +38,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.0.0 + uses: docker/setup-buildx-action@v3.1.0 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 78af5ded..f800cfb4 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -37,7 +37,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.0.0 + uses: docker/setup-buildx-action@v3.1.0 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. From d227d6dc7368274fa5c2b206e77b33274b4569bc Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Tue, 5 Mar 2024 20:33:04 +1300 Subject: [PATCH 051/267] docs: Reference systemd timer example (`cerbot renew`) (#3921) --- docs/content/config/security/ssl.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/content/config/security/ssl.md b/docs/content/config/security/ssl.md index 1b84c4df..a9174483 100644 --- a/docs/content/config/security/ssl.md +++ b/docs/content/config/security/ssl.md @@ -134,6 +134,8 @@ Certbot provisions certificates to `/etc/letsencrypt`. Add a volume to store the ``` This process can also be [automated via _cron_ or _systemd timers_][certbot::automated-renewal]. + + - [Example with a systemd timer][certbot::automated-renewal::example-systemd-timer] !!! note "Using a different ACME CA" @@ -903,6 +905,7 @@ Despite this, if you must use non-standard DH parameters or you would like to sw [certbot::standalone]: https://certbot.eff.org/docs/using.html#standalone [certbot::renew]: https://certbot.eff.org/docs/using.html#renewing-certificates [certbot::automated-renewal]: https://certbot.eff.org/docs/using.html#automated-renewals +[certbot::automated-renewal::example-systemd-timer]: https://github.com/orgs/docker-mailserver/discussions/3917#discussioncomment-8661690 [certbot::custom-ca]: https://certbot.eff.org/docs/using.htmlchanging-the-acme-server [certbot::webroot]: https://certbot.eff.org/docs/using.html#webroot From b5b193ca4ca114dd14e725ab4980cf011ed58633 Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Tue, 5 Mar 2024 10:48:49 +0100 Subject: [PATCH 052/267] Rspamd: minor tweaks and follow-up for SPF, DKIM and DMARC symbols (#3923) * move `policies_group.conf` to correct location I originally assumed the file had to be placed into `scores.d`, but I now know that `local.d` is actually correct. * add configuration for composite symbols See updates to #3690: Additional Rspamd Symbols Rspamd has so-called composite symbols that trigger when a condition is met. Especially AUTH_NA and AUTH_NA_OR_FAIL will adjust the scores of various lines in the table above. This needs to be taken into account. * update CHANGELOG --- CHANGELOG.md | 2 +- Dockerfile | 1 - target/rspamd/local.d/composites.conf | 18 ++++++++++++++++++ .../{scores.d => local.d}/policies_group.conf | 0 4 files changed, 19 insertions(+), 2 deletions(-) create mode 100644 target/rspamd/local.d/composites.conf rename target/rspamd/{scores.d => local.d}/policies_group.conf (100%) diff --git a/CHANGELOG.md b/CHANGELOG.md index b60a7afa..1ec71ded 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -80,7 +80,7 @@ The most noteworthy change of this release is the update of the container's base - **Rspamd**: - The `rewrite_subject` action, is now disabled by default. It has been replaced with the new `SPAM_SUBJECT` environment variable, which implements the functionality via a Sieve script instead which is anti-spam service agnostic ([#3820](https://github.com/docker-mailserver/docker-mailserver/pull/3820)) - `RSPAMD_NEURAL` was added and is disabled by default. If switched on it will enable the experimental Rspamd "Neural network" module to add a layer of analysis to spam detection ([#3833](https://github.com/docker-mailserver/docker-mailserver/pull/3833)) - - The symbol weights of SPF, DKIM and DMARC have been adjusted again. Fixes a bug and includes more appropriate combinations of symbols ([#3913](https://github.com/docker-mailserver/docker-mailserver/pull/3913)) + - The symbol weights of SPF, DKIM and DMARC have been adjusted again. Fixes a bug and includes more appropriate combinations of symbols ([#3913](https://github.com/docker-mailserver/docker-mailserver/pull/3913), [#3923](https://github.com/docker-mailserver/docker-mailserver/pull/3923)) ### Fixes diff --git a/Dockerfile b/Dockerfile index 8110cebf..8b2a502c 100644 --- a/Dockerfile +++ b/Dockerfile @@ -104,7 +104,6 @@ EOF # ----------------------------------------------- COPY target/rspamd/local.d/ /etc/rspamd/local.d/ -COPY target/rspamd/scores.d/* /etc/rspamd/scores.d/ # ----------------------------------------------- # --- OAUTH2 ------------------------------------ diff --git a/target/rspamd/local.d/composites.conf b/target/rspamd/local.d/composites.conf new file mode 100644 index 00000000..ab58f45d --- /dev/null +++ b/target/rspamd/local.d/composites.conf @@ -0,0 +1,18 @@ +# In addition to `policies_group.conf`, this file contains +# symbols that are applied when certain other symbols are +# applied (or not applied). +# +# We are especially interested in the `policy` field, because +# there are cases in which `remove_weight` is undesirable. + +# When neither SPF, DKIM, nor DMARC are available, we want +# to increase the base score so we apply at least greylisting. +AUTH_NA { + score = 2.5; + policy = "leave"; +} + +AUTH_NA_OR_FAIL { + score = 1; + policy = "leave"; +} diff --git a/target/rspamd/scores.d/policies_group.conf b/target/rspamd/local.d/policies_group.conf similarity index 100% rename from target/rspamd/scores.d/policies_group.conf rename to target/rspamd/local.d/policies_group.conf From e21e5e04902432079418e96fc5ccba3400dd5582 Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Wed, 6 Mar 2024 08:44:34 +0100 Subject: [PATCH 053/267] Rspamd: update history key in Redis configuration (#3927) --- CHANGELOG.md | 4 +++- target/scripts/startup/setup.d/security/rspamd.sh | 9 +++++++++ test/tests/parallel/set1/spam_virus/rspamd_full.bats | 12 ++++++++++++ 3 files changed, 24 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1ec71ded..e1ba19b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,7 +52,7 @@ The most noteworthy change of this release is the update of the container's base - `smtp_sasl_security_options = noanonymous` (_credentials are mandatory for outbound mail delivery_) - `smtp_tls_security_level = encrypt` (_the outbound MTA connection must always be secure due to credentials sent_) - **Environment Variables**: - - `SA_SPAM_SUBJECT` has been renamed into `SPAM_SUBJECT` to become anti-spam service agnostic. ([3820](https://github.com/docker-mailserver/docker-mailserver/pull/3820)) + - `SA_SPAM_SUBJECT` has been renamed into `SPAM_SUBJECT` to become anti-spam service agnostic. ([#3820](https://github.com/docker-mailserver/docker-mailserver/pull/3820)) - As this functionality is now handled in Dovecot via a Sieve script instead of the respective anti-spam service during Postfix processing, this feature will only apply to mail stored in Dovecot. If you have relied on this feature in a different context, it will no longer be available. - Rspamd previously handled this functionality via the `rewrite_subject` action which as now been disabled by default in favor of the new approach with `SPAM_SUBJECT`. - `SA_SPAM_SUBJECT` is now deprecated and will log a warning if used. The value is copied as a fallback to `SPAM_SUBJECT`. @@ -61,6 +61,8 @@ The most noteworthy change of this release is the update of the container's base - The feature to include [`_SCORE_` tag](https://spamassassin.apache.org/full/4.0.x/doc/Mail_SpamAssassin_Conf.html#rewrite_header-subject-from-to-STRING) in your value to be replaced by the associated spam score is no longer available. - **Supervisord**: - `supervisor-app.conf` renamed to `dms-services.conf` +- **Rspamd**: + - the Redis history key has been changed in order to not incorporate the hostname of the container (which is desirable in Kubernetes environments) ([#3927](https://github.com/docker-mailserver/docker-mailserver/pull/3927)) ### Added diff --git a/target/scripts/startup/setup.d/security/rspamd.sh b/target/scripts/startup/setup.d/security/rspamd.sh index 6b54addd..702a27ee 100644 --- a/target/scripts/startup/setup.d/security/rspamd.sh +++ b/target/scripts/startup/setup.d/security/rspamd.sh @@ -126,6 +126,15 @@ function __rspamd__setup_redis() { servers = "127.0.0.1:6379"; expand_keys = true; +EOF + + # We do not use `{{HOSTNAME}}` but only `{{COMPRES}}` to better support + # Kubernetes, see https://github.com/orgs/docker-mailserver/discussions/3922 + cat >"${RSPAMD_LOCAL_D}/history_redis.conf" << "EOF" +# documentation: https://rspamd.com/doc/modules/history_redis.html + +key_prefix = "rs_history{{COMPRESS}}"; + EOF # Here we adjust the Redis default configuration that we supply to Redis when starting it. diff --git a/test/tests/parallel/set1/spam_virus/rspamd_full.bats b/test/tests/parallel/set1/spam_virus/rspamd_full.bats index 9b14860b..32891a27 100644 --- a/test/tests/parallel/set1/spam_virus/rspamd_full.bats +++ b/test/tests/parallel/set1/spam_virus/rspamd_full.bats @@ -86,6 +86,18 @@ function teardown_file() { _default_teardown ; } refute_line --regexp 'rewrite_subject = [0-9]+;' } +@test 'Rspamd Redis configuration is correct' { + _run_in_container rspamadm configdump redis + assert_success + assert_line 'expand_keys = true;' + assert_line 'servers = "127.0.0.1:6379";' + + _run_in_container rspamadm configdump history_redis + assert_success + assert_line 'compress = true;' + assert_line 'key_prefix = "rs_history{{COMPRESS}}";' +} + @test "contents of '/etc/rspamd/override.d/' are copied" { local OVERRIDE_D='/etc/rspamd/override.d' _file_exists_in_container "${OVERRIDE_D}/testmodule_complicated.conf" From 364969919757216f97a205cf05e98dfae0c7c924 Mon Sep 17 00:00:00 2001 From: Kirill Kirilenko Date: Thu, 7 Mar 2024 01:13:22 +0300 Subject: [PATCH 054/267] fix: Move spam to mailbox associated to the `\Junk` special-use attribute (#3925) Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- CHANGELOG.md | 4 ++++ docs/content/config/environment.md | 8 ++++--- target/dovecot/90-sieve.conf | 2 +- .../scripts/startup/setup.d/security/misc.sh | 4 ++-- test/config/junk-mailbox/user-patches.sh | 21 +++++++++++++++++++ .../parallel/set1/spam_virus/rspamd_full.bats | 2 +- .../set1/spam_virus/spam_junk_folder.bats | 13 ++++++++---- 7 files changed, 43 insertions(+), 11 deletions(-) create mode 100755 test/config/junk-mailbox/user-patches.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index e1ba19b4..de5642bf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,6 +36,10 @@ The most noteworthy change of this release is the update of the container's base - DMS `main.cf` has renamed `postscreen_dnsbl_whitelist_threshold` to `postscreen_dnsbl_allowlist_threshold` as part of this change. - `smtpd_relay_restrictions` (relay policy) is now evaluated after `smtpd_recipient_restrictions` (spam policy). Previously it was evaluated before `smtpd_recipient_restrictions`. Mail to be relayed via DMS must now pass through the spam policy first. - The TLS fingerprint policy has changed the default from MD5 to SHA256 (_DMS does not modify this Postfix parameter, but may affect any user customizations that do_). +- **Dovecot** + - The "Junk" mailbox (folder) is now referenced by it's [special-use flag `\Junk`](https://docker-mailserver.github.io/docker-mailserver/v13.3/examples/use-cases/imap-folders/) instead of an explicit mailbox. ([#3925](https://github.com/docker-mailserver/docker-mailserver/pull/3925)) + - This provides compatibility for the Junk mailbox when it's folder name differs (_eg: Renamed to "Spam"_). + - Potential breakage if your deployment modifies our `spam_to_junk.sieve` sieve script (_which is created during container startup when ENV `MOVE_SPAM_TO_JUNK=1`_) that handles storing spam mail into a users "Junk" mailbox folder. - **rsyslog:** - Debian 12 adjusted the `rsyslog` configuration for the default file template from `RSYSLOG_TraditionalFileFormat` to `RSYSLOG_FileFormat` (_upstream default since 2012_). This change may affect you if you have any monitoring / analysis of log output (_eg: `mail.log` / `docker logs`_). - The two formats are roughly equivalent to [RFC 3164](https://www.rfc-editor.org/rfc/rfc3164)) and [RFC 5424](https://datatracker.ietf.org/doc/html/rfc5424#section-1) respectively. diff --git a/docs/content/config/environment.md b/docs/content/config/environment.md index 83fcdfa6..9adfd5d9 100644 --- a/docs/content/config/environment.md +++ b/docs/content/config/environment.md @@ -328,10 +328,12 @@ Note: More information at ##### MOVE_SPAM_TO_JUNK -- 0 => Spam messages will be delivered in the mailbox. -- **1** => Spam messages will be delivered in the `Junk` folder. +- 0 => Spam messages will be delivered to the inbox. +- **1** => Spam messages will be delivered to the Junk mailbox. -Routes mail identified as spam into the recipient(s) Junk folder (_via a Dovecot Sieve script_). +Routes mail identified as spam into the recipient(s) Junk mailbox (_a specialized folder for junk associated to the [special-use flag `\Junk`][docs::dovecot::special-use-flag], handled via a Dovecot sieve script internally_). + +[docs::dovecot::special-use-flag]: ../examples/use-cases/imap-folders.md !!! info diff --git a/target/dovecot/90-sieve.conf b/target/dovecot/90-sieve.conf index 8b559be2..f388009a 100644 --- a/target/dovecot/90-sieve.conf +++ b/target/dovecot/90-sieve.conf @@ -51,7 +51,7 @@ plugin { # deprecated imapflags extension in addition to all extensions were already # enabled by default. #sieve_extensions = +notify +imapflags - sieve_extensions = +notify +imapflags +vnd.dovecot.pipe +vnd.dovecot.filter + sieve_extensions = +notify +imapflags +special-use +vnd.dovecot.pipe +vnd.dovecot.filter # Which Sieve language extensions are ONLY available in global scripts. This # can be used to restrict the use of certain Sieve extensions to administrator diff --git a/target/scripts/startup/setup.d/security/misc.sh b/target/scripts/startup/setup.d/security/misc.sh index 303ae62d..bb460716 100644 --- a/target/scripts/startup/setup.d/security/misc.sh +++ b/target/scripts/startup/setup.d/security/misc.sh @@ -304,11 +304,11 @@ function _setup_spam_to_junk() { _log 'debug' 'Spam emails will be moved to the Junk folder' mkdir -p /usr/lib/dovecot/sieve-global/after/ cat >/usr/lib/dovecot/sieve-global/after/spam_to_junk.sieve << EOF -require ["fileinto","mailbox"]; +require ["fileinto","special-use"]; if anyof (header :contains "X-Spam-Flag" "YES", header :contains "X-Spam" "Yes") { - fileinto "Junk"; + fileinto :specialuse "\\\\Junk" "Junk"; } EOF sievec /usr/lib/dovecot/sieve-global/after/spam_to_junk.sieve diff --git a/test/config/junk-mailbox/user-patches.sh b/test/config/junk-mailbox/user-patches.sh new file mode 100755 index 00000000..0a16a21f --- /dev/null +++ b/test/config/junk-mailbox/user-patches.sh @@ -0,0 +1,21 @@ +#!/bin/bash +## +# This user script will be executed between configuration and starting daemons +# To enable it you must save it in your config directory as "user-patches.sh" +## + +echo "[user-patches.sh] Adjusting 'Junk' mailbox name to verify delivery to Junk mailbox based on special-use flag instead of mailbox's name" + +sed -i -e 's/mailbox Junk/mailbox Spam/' /etc/dovecot/conf.d/15-mailboxes.conf + +### Before / After ### + +# mailbox Junk { +# auto = subscribe +# special_use = \Junk +# } + +# mailbox Spam { +# auto = subscribe +# special_use = \Junk +# } diff --git a/test/tests/parallel/set1/spam_virus/rspamd_full.bats b/test/tests/parallel/set1/spam_virus/rspamd_full.bats index 32891a27..3bc9fb10 100644 --- a/test/tests/parallel/set1/spam_virus/rspamd_full.bats +++ b/test/tests/parallel/set1/spam_virus/rspamd_full.bats @@ -244,7 +244,7 @@ function teardown_file() { _default_teardown ; } _service_log_should_contain_string 'rspamd' 'add header "Gtube pattern"' _print_mail_log_for_msgid 'rspamd-test-email-header' - assert_output --partial "fileinto action: stored mail into mailbox 'Junk'" + assert_output --partial "fileinto action: stored mail into mailbox [SPECIAL-USE \\Junk]" _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 2 _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/.Junk/new/ 1 diff --git a/test/tests/parallel/set1/spam_virus/spam_junk_folder.bats b/test/tests/parallel/set1/spam_virus/spam_junk_folder.bats index 0c873382..5590d6f7 100644 --- a/test/tests/parallel/set1/spam_virus/spam_junk_folder.bats +++ b/test/tests/parallel/set1/spam_virus/spam_junk_folder.bats @@ -49,9 +49,11 @@ function teardown() { _default_teardown ; } _should_receive_spam_at '/var/mail/localhost.localdomain/user1/new/' } -@test "(enabled + MOVE_SPAM_TO_JUNK=1) should deliver spam message into Junk folder" { +@test "(enabled + MOVE_SPAM_TO_JUNK=1) should deliver spam message into Junk mailbox" { export CONTAINER_NAME=${CONTAINER3_NAME} + _init_with_defaults + local CUSTOM_SETUP_ARGUMENTS=( --env ENABLE_AMAVIS=1 --env ENABLE_SPAMASSASSIN=1 @@ -60,14 +62,17 @@ function teardown() { _default_teardown ; } --env MOVE_SPAM_TO_JUNK=1 --env PERMIT_DOCKER=container ) - _init_with_defaults + + # Adjust 'Junk' mailbox name to verify delivery to Junk mailbox based on special-use flag instead of mailbox's name + mv "${TEST_TMP_CONFIG}/junk-mailbox/user-patches.sh" "${TEST_TMP_CONFIG}/" + _common_container_setup 'CUSTOM_SETUP_ARGUMENTS' _should_send_spam_message _should_be_received_by_amavis 'Passed SPAM {RelayedTaggedInbound,Quarantined}' - # Should move delivered spam to Junk folder - _should_receive_spam_at '/var/mail/localhost.localdomain/user1/.Junk/new/' + # Should move delivered spam to the Junk mailbox (adjusted to be located at '.Spam/') + _should_receive_spam_at '/var/mail/localhost.localdomain/user1/.Spam/new/' } # NOTE: Same as test for `CONTAINER3_NAME`, only differing by ENV `MARK_SPAM_AS_READ=1` + `_should_receive_spam_at` location From 267fc552d220414be9f8ba7dbb2f0ed2431f3228 Mon Sep 17 00:00:00 2001 From: Casper Date: Sat, 9 Mar 2024 14:21:02 +0100 Subject: [PATCH 055/267] getmail: remove temp file usage (#3920) --- target/scripts/startup/setup.d/getmail.sh | 8 ++++---- target/scripts/startup/setup.d/security/rspamd.sh | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/target/scripts/startup/setup.d/getmail.sh b/target/scripts/startup/setup.d/getmail.sh index a6c304c6..6f280118 100644 --- a/target/scripts/startup/setup.d/getmail.sh +++ b/target/scripts/startup/setup.d/getmail.sh @@ -18,10 +18,10 @@ function _setup_getmail() { CONFIGS=1 ID=$(cut -d '-' -f 3 <<< "${FILE}" | cut -d '.' -f 1) local GETMAIL_CONFIG="${GETMAILRC}/getmailrc-${ID}" - cat /etc/getmailrc_general >"${GETMAIL_CONFIG}.tmp" - echo -e "message_log = /var/log/mail/getmail-${ID}.log\n" >>"${GETMAIL_CONFIG}.tmp" - cat "${GETMAIL_CONFIG}.tmp" "${FILE}" >"${GETMAIL_CONFIG}" - rm "${GETMAIL_CONFIG}.tmp" + + cat /etc/getmailrc_general >"${GETMAIL_CONFIG}" + echo -e "message_log = /var/log/mail/getmail-${ID}.log\n" >>"${GETMAIL_CONFIG}" + cat "${FILE}" >>"${GETMAIL_CONFIG}" fi done diff --git a/target/scripts/startup/setup.d/security/rspamd.sh b/target/scripts/startup/setup.d/security/rspamd.sh index 702a27ee..37916282 100644 --- a/target/scripts/startup/setup.d/security/rspamd.sh +++ b/target/scripts/startup/setup.d/security/rspamd.sh @@ -128,7 +128,7 @@ expand_keys = true; EOF - # We do not use `{{HOSTNAME}}` but only `{{COMPRES}}` to better support + # We do not use `{{HOSTNAME}}` but only `{{COMPRESS}}` to better support # Kubernetes, see https://github.com/orgs/docker-mailserver/discussions/3922 cat >"${RSPAMD_LOCAL_D}/history_redis.conf" << "EOF" # documentation: https://rspamd.com/doc/modules/history_redis.html From 8bdda5f43325c826f5e0bd42bf74d19bb83e0a8e Mon Sep 17 00:00:00 2001 From: Ikko Eltociear Ashimine Date: Mon, 11 Mar 2024 20:02:22 +0900 Subject: [PATCH 056/267] Update user-patches.sh (#3932) --- test/config/rspamd_full/user-patches.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/config/rspamd_full/user-patches.sh b/test/config/rspamd_full/user-patches.sh index 9f4dc9fb..f731ff36 100644 --- a/test/config/rspamd_full/user-patches.sh +++ b/test/config/rspamd_full/user-patches.sh @@ -8,7 +8,7 @@ echo "enable_test_patterns = true;" >>/etc/rspamd/local.d/options.inc # We want Dovecot to be very detailed about what it is doing, -# specificially for Sieve because we need to check whether the +# specifically for Sieve because we need to check whether the # Sieve scripts are executed so Rspamd is trained when using # `RSPAMD_LEARN=1`. echo 'mail_debug = yes' >>/etc/dovecot/dovecot.conf From 9bc8869715f1d431274067c8b830d72cf861924e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 11 Mar 2024 19:40:48 +0100 Subject: [PATCH 057/267] chore(deps): Bump docker/build-push-action from 5.1.0 to 5.2.0 (#3934) --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 660b0a2b..aa3051c8 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -83,7 +83,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v5.1.0 + uses: docker/build-push-action@v5.2.0 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index bd776cfd..a729e5b8 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v5.1.0 + uses: docker/build-push-action@v5.2.0 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index bc26ef4f..73db6f6c 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v5.1.0 + uses: docker/build-push-action@v5.2.0 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index f800cfb4..8ca0b8b2 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v5.1.0 + uses: docker/build-push-action@v5.2.0 with: context: . tags: mailserver-testing:ci From a04b53f4f8ba39217739043059a5272e2ba84fd6 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 12 Mar 2024 11:46:34 +1300 Subject: [PATCH 058/267] chore(deps): Bump nwtgck/actions-netlify from 2.1 to 3.0 (#3933) Bumps [nwtgck/actions-netlify](https://github.com/nwtgck/actions-netlify) from 2.1 to 3.0. - [Release notes](https://github.com/nwtgck/actions-netlify/releases) - [Changelog](https://github.com/nwtgck/actions-netlify/blob/develop/CHANGELOG.md) - [Commits](https://github.com/nwtgck/actions-netlify/compare/v2.1...v3.0) --- updated-dependencies: - dependency-name: nwtgck/actions-netlify dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/docs-preview-deploy.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/docs-preview-deploy.yml b/.github/workflows/docs-preview-deploy.yml index dbe57752..7c924f86 100644 --- a/.github/workflows/docs-preview-deploy.yml +++ b/.github/workflows/docs-preview-deploy.yml @@ -55,7 +55,7 @@ jobs: context: 'Deploy Preview (pull_request => workflow_run)' - name: 'Send preview build to Netlify' - uses: nwtgck/actions-netlify@v2.1 + uses: nwtgck/actions-netlify@v3.0 id: preview timeout-minutes: 1 env: From 2133b51e782ae207475788a9b168b0008a5b450b Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Tue, 12 Mar 2024 09:31:44 +0100 Subject: [PATCH 059/267] docs: rewrite Kubernetes page (#3928) --- docs/content/assets/css/customizations.css | 34 + docs/content/config/advanced/kubernetes.md | 1192 ++++++++++------- .../tutorials/mailserver-behind-proxy.md | 7 +- docs/mkdocs.yml | 5 + 4 files changed, 778 insertions(+), 460 deletions(-) diff --git a/docs/content/assets/css/customizations.css b/docs/content/assets/css/customizations.css index 25cb0274..7255a837 100644 --- a/docs/content/assets/css/customizations.css +++ b/docs/content/assets/css/customizations.css @@ -107,3 +107,37 @@ div.md-content article.md-content__inner a.toclink code { .md-nav__item--nested > .md-nav__link { font-weight: 700; } + +/* ============================================================================================================= */ + +/* + TaskList style for a pro/con list. Presently only used for this type of list in the kubernetes docs. + Uses a custom icon for the unchecked (con) state: :octicons-x-circle-fill-24: + https://github.com/squidfunk/mkdocs-material/discussions/6811#discussioncomment-8700795 + + TODO: Can better scope the style under a class name when migrating to block extension syntax: + https://github.com/facelessuser/pymdown-extensions/discussions/1973 +*/ + +:root { + --md-tasklist-icon--failed: url('data:image/svg+xml;charset=utf-8,'); +} + +.md-typeset [type="checkbox"] + .task-list-indicator::before { + background-color: rgb(216, 87, 48); + -webkit-mask-image: var(--md-tasklist-icon--failed); + mask-image: var(--md-tasklist-icon--failed); +} + +/* More suitable shade of green */ +.md-typeset [type=checkbox]:checked+.task-list-indicator:before { + background-color: rgb(97, 216, 42); +} + +/* Tiny layout shift */ +[dir=ltr] .md-typeset .task-list-indicator:before { + left: -1.6em; + top: 1px; +} + +/* ============================================================================================================= */ diff --git a/docs/content/config/advanced/kubernetes.md b/docs/content/config/advanced/kubernetes.md index c277852a..97a7e414 100644 --- a/docs/content/config/advanced/kubernetes.md +++ b/docs/content/config/advanced/kubernetes.md @@ -4,518 +4,794 @@ title: 'Advanced | Kubernetes' ## Introduction -This article describes how to deploy DMS to Kubernetes. Please note that there is also a [Helm chart] available. +This article describes how to deploy DMS to Kubernetes. We highly recommend everyone to use our community [DMS Helm chart][github-web::docker-mailserver-helm]. -!!! attention "Requirements" +!!! note "Requirements" - We assume basic knowledge about Kubernetes from the reader. Moreover, we assume the reader to have a basic understanding of mail servers. Ideally, the reader has deployed DMS before in an easier setup with Docker (Compose). + 1. Basic knowledge about Kubernetes from the reader. + 2. A basic understanding of mail servers. + 3. Ideally, the reader has already deployed DMS before with a simpler setup (_`docker run` or Docker Compose_). -!!! warning "About Support for Kubernetes" +!!! warning "Limited Support" - Please note that Kubernetes **is not** officially supported and we do not build images specifically designed for it. When opening an issue, please remember that only Docker & Docker Compose are officially supported. + DMS **does not officially support Kubernetes**. This content is entirely community-supported. If you find errors, please open an issue and raise a PR. - This content is entirely community-supported. If you find errors, please open an issue and provide a PR. +## Manually Writing Manifests -## Manifests +If using our Helm chart is not viable for you, here is some guidance to start with your own manifests. -### Configuration + +!!! quote "" -We want to provide the basic configuration in the form of environment variables with a `ConfigMap`. Note that this is just an example configuration; tune the `ConfigMap` to your needs. + === "`ConfigMap`" -```yaml ---- -apiVersion: v1 -kind: ConfigMap + Provide the basic configuration via environment variables with a `ConfigMap`. + + !!! example -metadata: - name: mailserver.environment + Below is only an example configuration, adjust the `ConfigMap` to your own needs. -immutable: false + ```yaml + --- + apiVersion: v1 + kind: ConfigMap -data: - TLS_LEVEL: modern - POSTSCREEN_ACTION: drop - OVERRIDE_HOSTNAME: mail.example.com - FAIL2BAN_BLOCKTYPE: drop - POSTMASTER_ADDRESS: postmaster@example.com - UPDATE_CHECK_INTERVAL: 10d - POSTFIX_INET_PROTOCOLS: ipv4 - ENABLE_CLAMAV: '1' - ENABLE_POSTGREY: '0' - ENABLE_FAIL2BAN: '1' - AMAVIS_LOGLEVEL: '-1' - SPOOF_PROTECTION: '1' - MOVE_SPAM_TO_JUNK: '1' - ENABLE_UPDATE_CHECK: '1' - ENABLE_SPAMASSASSIN: '1' - SUPERVISOR_LOGLEVEL: warn - SPAMASSASSIN_SPAM_TO_INBOX: '1' + metadata: + name: mailserver.environment - # here, we provide an example for the SSL configuration - SSL_TYPE: manual - SSL_CERT_PATH: /secrets/ssl/rsa/tls.crt - SSL_KEY_PATH: /secrets/ssl/rsa/tls.key -``` + immutable: false -We can also make use of user-provided configuration files, e.g. `user-patches.sh`, `postfix-accounts.cf` and more, to adjust DMS to our likings. We encourage you to have a look at [Kustomize][kustomize] for creating `ConfigMap`s from multiple files, but for now, we will provide a simple, hand-written example. This example is absolutely minimal and only goes to show what can be done. + data: + TLS_LEVEL: modern + POSTSCREEN_ACTION: drop + OVERRIDE_HOSTNAME: mail.example.com + FAIL2BAN_BLOCKTYPE: drop + POSTMASTER_ADDRESS: postmaster@example.com + UPDATE_CHECK_INTERVAL: 10d + POSTFIX_INET_PROTOCOLS: ipv4 + ENABLE_CLAMAV: '1' + ENABLE_POSTGREY: '0' + ENABLE_FAIL2BAN: '1' + AMAVIS_LOGLEVEL: '-1' + SPOOF_PROTECTION: '1' + MOVE_SPAM_TO_JUNK: '1' + ENABLE_UPDATE_CHECK: '1' + ENABLE_SPAMASSASSIN: '1' + SUPERVISOR_LOGLEVEL: warn + SPAMASSASSIN_SPAM_TO_INBOX: '1' -```yaml ---- -apiVersion: v1 -kind: ConfigMap - -metadata: - name: mailserver.files + # here, we provide an example for the SSL configuration + SSL_TYPE: manual + SSL_CERT_PATH: /secrets/ssl/rsa/tls.crt + SSL_KEY_PATH: /secrets/ssl/rsa/tls.key + ``` -data: - postfix-accounts.cf: | - test@example.com|{SHA512-CRYPT}$6$someHashValueHere - other@example.com|{SHA512-CRYPT}$6$someOtherHashValueHere -``` - -!!! attention "Static Configuration" - - With the configuration shown above, you can **not** dynamically add accounts as the configuration file mounted into the mail server can not be written to. - - Use persistent volumes for production deployments. - -### Persistence - -Thereafter, we need persistence for our data. Make sure you have a storage provisioner and that you choose the correct `storageClassName`. - -```yaml ---- -apiVersion: v1 -kind: PersistentVolumeClaim - -metadata: - name: data - -spec: - storageClassName: local-path - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 25Gi -``` - -### Service - -A `Service` is required for getting the traffic to the pod itself. The service is somewhat crucial. Its configuration determines whether the original IP from the sender will be kept. [More about this further down below](#exposing-your-mail-server-to-the-outside-world). - -The configuration you're seeing does keep the original IP, but you will not be able to scale this way. We have chosen to go this route in this case because we think most Kubernetes users will only want to have one instance. - -```yaml ---- -apiVersion: v1 -kind: Service - -metadata: - name: mailserver - labels: - app: mailserver - -spec: - type: LoadBalancer - - selector: - app: mailserver - - ports: - # Transfer - - name: transfer - port: 25 - targetPort: transfer - protocol: TCP - # ESMTP with implicit TLS - - name: esmtp-implicit - port: 465 - targetPort: esmtp-implicit - protocol: TCP - # ESMTP with explicit TLS (STARTTLS) - - name: esmtp-explicit - port: 587 - targetPort: esmtp-explicit - protocol: TCP - # IMAPS with implicit TLS - - name: imap-implicit - port: 993 - targetPort: imap-implicit - protocol: TCP - -``` - -### Deployments - -Last but not least, the `Deployment` becomes the most complex component. It instructs Kubernetes how to run the DMS container and how to apply your `ConfigMaps`, persisted storage, etc. Additionally, we can set options to enforce runtime security here. - -```yaml ---- -apiVersion: apps/v1 -kind: Deployment - -metadata: - name: mailserver - - annotations: - ignore-check.kube-linter.io/run-as-non-root: >- - 'mailserver' needs to run as root - ignore-check.kube-linter.io/privileged-ports: >- - 'mailserver' needs privileged ports - ignore-check.kube-linter.io/no-read-only-root-fs: >- - There are too many files written to make The - root FS read-only - -spec: - replicas: 1 - selector: - matchLabels: - app: mailserver - - template: - metadata: - labels: - app: mailserver - - annotations: - container.apparmor.security.beta.kubernetes.io/mailserver: runtime/default - - spec: - hostname: mail - containers: - - name: mailserver - image: ghcr.io/docker-mailserver/docker-mailserver:latest - imagePullPolicy: IfNotPresent - - securityContext: - # Required to support SGID via `postdrop` executable - # in `/var/mail-state` for Postfix (maildrop + public dirs): - # https://github.com/docker-mailserver/docker-mailserver/pull/3625 - allowPrivilegeEscalation: true - readOnlyRootFilesystem: false - runAsUser: 0 - runAsGroup: 0 - runAsNonRoot: false - privileged: false - capabilities: - add: - # file permission capabilities - - CHOWN - - FOWNER - - MKNOD - - SETGID - - SETUID - - DAC_OVERRIDE - # network capabilities - - NET_ADMIN # needed for F2B - - NET_RAW # needed for F2B - - NET_BIND_SERVICE - # miscellaneous capabilities - - SYS_CHROOT - - KILL - drop: [ALL] - seccompProfile: - type: RuntimeDefault - - # You want to tune this to your needs. If you disable ClamAV, - # you can use less RAM and CPU. This becomes important in - # case you're low on resources and Kubernetes refuses to - # schedule new pods. - resources: - limits: - memory: 4Gi - cpu: 1500m - requests: - memory: 2Gi - cpu: 600m - - volumeMounts: - - name: files - subPath: postfix-accounts.cf - mountPath: /tmp/docker-mailserver/postfix-accounts.cf - readOnly: true - - # PVCs - - name: data - mountPath: /var/mail - subPath: data - readOnly: false - - name: data - mountPath: /var/mail-state - subPath: state - readOnly: false - - name: data - mountPath: /var/log/mail - subPath: log - readOnly: false - - # certificates - - name: certificates-rsa - mountPath: /secrets/ssl/rsa/ - readOnly: true - - # other - - name: tmp-files - mountPath: /tmp - readOnly: false - - ports: - - name: transfer - containerPort: 25 - protocol: TCP - - name: esmtp-implicit - containerPort: 465 - protocol: TCP - - name: esmtp-explicit - containerPort: 587 - - name: imap-implicit - containerPort: 993 - protocol: TCP - - envFrom: - - configMapRef: - name: mailserver.environment - - restartPolicy: Always - - volumes: - # configuration files - - name: files - configMap: - name: mailserver.files - - # PVCs - - name: data - persistentVolumeClaim: - claimName: data - - # certificates - - name: certificates-rsa - secret: - secretName: mail-tls-certificate-rsa - items: - - key: tls.key - path: tls.key - - key: tls.crt - path: tls.crt - - # other - - name: tmp-files - emptyDir: {} -``` - -### Certificates - An Example - -In this example, we use [`cert-manager`][cert-manager] to supply RSA certificates. You can also supply RSA certificates as fallback certificates, which DMS supports out of the box with `SSL_ALT_CERT_PATH` and `SSL_ALT_KEY_PATH`, and provide ECDSA as the proper certificates. - -```yaml ---- -apiVersion: cert-manager.io/v1 -kind: Certificate - -metadata: - name: mail-tls-certificate-rsa - -spec: - secretName: mail-tls-certificate-rsa - isCA: false - privateKey: - algorithm: RSA - encoding: PKCS1 - size: 2048 - dnsNames: [mail.example.com] - issuerRef: - name: mail-issuer - kind: Issuer -``` - -!!! attention - - You will need to have [`cert-manager`][cert-manager] configured. Especially the issue will need to be configured. Since we do not know how you want or need your certificates to be supplied, we do not provide more configuration here. The documentation for [`cert-manager`][cert-manager] is excellent. - -### Sensitive Data - -!!! attention "Sensitive Data" - - For storing OpenDKIM keys, TLS certificates or any sort of sensitive data, you should be using `Secret`s. You can mount secrets like `ConfigMap`s and use them the same way. - -The [TLS docs page][docs-tls] provides guidance when it comes to certificates and transport layer security. Always provide sensitive information vai `Secrets`. + You can also make use of user-provided configuration files (_e.g. `user-patches.sh`, `postfix-accounts.cf`, etc_), to customize DMS to your needs. + + ??? example "Providing config files" + + Here is a minimal example that supplies a `postfix-accounts.cf` file inline with two users: + + ```yaml + --- + apiVersion: v1 + kind: ConfigMap + + metadata: + name: mailserver.files + + data: + postfix-accounts.cf: | + test@example.com|{SHA512-CRYPT}$6$someHashValueHere + other@example.com|{SHA512-CRYPT}$6$someOtherHashValueHere + ``` + + !!! warning "Static Configuration" + + The inline `postfix-accounts.cf` config example above provides file content that is static. It is mounted as read-only at runtime, thus cannot support modifications. + + For production deployments, use persistent volumes instead (via `PersistentVolumeClaim`). That will enable files like `postfix-account.cf` to add and remove accounts, while also persisting those changes externally from the container. + + !!! tip "Modularize your `ConfigMap`" + + [Kustomize][kustomize] can be a useful tool as it supports creating a `ConfigMap` from multiple files. + + === "`PersistentVolumeClaim`" + + To persist data externally from the DMS container, configure a `PersistentVolumeClaim` (PVC). + + Make sure you have a storage system (like Longhorn, Rook, etc.) and that you choose the correct `storageClassName` (according to your storage system). + + !!! example + + ```yaml + --- + apiVersion: v1 + kind: PersistentVolumeClaim + + metadata: + name: data + + spec: + storageClassName: local-path + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 25Gi + ``` + + === "`Service`" + + A [`Service`][k8s-docs::config::service] is required for getting the traffic to the pod itself. It configures a load balancer with the ports you'll need. + + The configuration for a `Service` affects if the original IP from a connecting client is preserved (_this is important_). [More about this further down below](#exposing-your-mail-server-to-the-outside-world). + + !!! example + + ```yaml + --- + apiVersion: v1 + kind: Service + + metadata: + name: mailserver + labels: + app: mailserver + + spec: + type: LoadBalancer + + selector: + app: mailserver + + ports: + # smtp + - name: smtp + port: 25 + targetPort: smtp + protocol: TCP + # submissions (ESMTP with implicit TLS) + - name: submission + port: 465 + targetPort: submissions + protocol: TCP + # submission (ESMTP with explicit TLS) + - name: submission + port: 587 + targetPort: submission + protocol: TCP + # imaps (implicit TLS) + - name: imaps + port: 993 + targetPort: imaps + protocol: TCP + ``` + + === "`Certificate`" + + !!! example "Using [`cert-manager`][cert-manager] to supply TLS certificates" + + ```yaml + --- + apiVersion: cert-manager.io/v1 + kind: Certificate + + metadata: + name: mail-tls-certificate-rsa + + spec: + secretName: mail-tls-certificate-rsa + isCA: false + privateKey: + algorithm: RSA + encoding: PKCS1 + size: 2048 + dnsNames: [mail.example.com] + issuerRef: + name: mail-issuer + kind: Issuer + ``` + + The [TLS docs page][docs-tls] provides guidance when it comes to certificates and transport layer security. + + !!! tip "ECDSA + RSA (fallback)" + + You could supply RSA certificates as fallback certificates instead, with ECDSA as the primary. DMS supports dual certificates via the ENV `SSL_ALT_CERT_PATH` and `SSL_ALT_KEY_PATH`. + + !!! warning "Always provide sensitive information via a `Secret`" + + For storing OpenDKIM keys, TLS certificates, or any sort of sensitive data - you should be using `Secret`s. + + A `Secret` is similar to `ConfigMap`, it can be used and mounted as a volume as demonstrated in the [`Deployment` manifest][docs::k8s::config-deployment] tab. + + === "`Deployment`" + + The [`Deployment`][k8s-docs::config::deployment] config is the most complex component. + + - It instructs Kubernetes how to run the DMS container and how to apply your `ConfigMap`s, persisted storage, etc. + - Additional options can be set to enforce runtime security. + + ???+ example + + ```yaml + --- + apiVersion: apps/v1 + kind: Deployment + + metadata: + name: mailserver + + annotations: + ignore-check.kube-linter.io/run-as-non-root: >- + 'mailserver' needs to run as root + ignore-check.kube-linter.io/privileged-ports: >- + 'mailserver' needs privileged ports + ignore-check.kube-linter.io/no-read-only-root-fs: >- + There are too many files written to make the root FS read-only + + spec: + replicas: 1 + selector: + matchLabels: + app: mailserver + + template: + metadata: + labels: + app: mailserver + + annotations: + container.apparmor.security.beta.kubernetes.io/mailserver: runtime/default + + spec: + hostname: mail + containers: + - name: mailserver + image: ghcr.io/docker-mailserver/docker-mailserver:latest + imagePullPolicy: IfNotPresent + + securityContext: + # `allowPrivilegeEscalation: true` is required to support SGID via the `postdrop` + # executable in `/var/mail-state` for Postfix (maildrop + public dirs): + # https://github.com/docker-mailserver/docker-mailserver/pull/3625 + allowPrivilegeEscalation: true + readOnlyRootFilesystem: false + runAsUser: 0 + runAsGroup: 0 + runAsNonRoot: false + privileged: false + capabilities: + add: + # file permission capabilities + - CHOWN + - FOWNER + - MKNOD + - SETGID + - SETUID + - DAC_OVERRIDE + # network capabilities + - NET_ADMIN # needed for F2B + - NET_RAW # needed for F2B + - NET_BIND_SERVICE + # miscellaneous capabilities + - SYS_CHROOT + - KILL + drop: [ALL] + seccompProfile: + type: RuntimeDefault + + # Tune this to your needs. + # If you disable ClamAV, you can use less RAM and CPU. + # This becomes important in case you're low on resources + # and Kubernetes refuses to schedule new pods. + resources: + limits: + memory: 4Gi + cpu: 1500m + requests: + memory: 2Gi + cpu: 600m + + volumeMounts: + - name: files + subPath: postfix-accounts.cf + mountPath: /tmp/docker-mailserver/postfix-accounts.cf + readOnly: true + + # PVCs + - name: data + mountPath: /var/mail + subPath: data + readOnly: false + - name: data + mountPath: /var/mail-state + subPath: state + readOnly: false + - name: data + mountPath: /var/log/mail + subPath: log + readOnly: false + + # certificates + - name: certificates-rsa + mountPath: /secrets/ssl/rsa/ + readOnly: true + + ports: + - name: smtp + containerPort: 25 + protocol: TCP + - name: submissions + containerPort: 465 + protocol: TCP + - name: submission + containerPort: 587 + - name: imaps + containerPort: 993 + protocol: TCP + + envFrom: + - configMapRef: + name: mailserver.environment + + restartPolicy: Always + + volumes: + # configuration files + - name: files + configMap: + name: mailserver.files + + # PVCs + - name: data + persistentVolumeClaim: + claimName: data + + # certificates + - name: certificates-rsa + secret: + secretName: mail-tls-certificate-rsa + items: + - key: tls.key + path: tls.key + - key: tls.crt + path: tls.crt + ``` ## Exposing your Mail Server to the Outside World -The more difficult part with Kubernetes is to expose a deployed DMS to the outside world. Kubernetes provides multiple ways for doing that; each has downsides and complexity. The major problem with exposing DMS to outside world in Kubernetes is to [preserve the real client IP][Kubernetes-service-source-ip]. The real client IP is required by DMS for performing IP-based SPF checks and spam checks. If you do not require SPF checks for incoming mails, you may disable them in your [Postfix configuration][docs-postfix] by dropping the line that states: `check_policy_service unix:private/policyd-spf`. +The more difficult part with Kubernetes is to expose a deployed DMS instance to the outside world. -The easiest approach was covered above, using `#!yaml externalTrafficPolicy: Local`, which disables the service proxy, but makes the service local as well (which does not scale). This approach only works when you are given the correct (that is, a public and routable) IP address by a load balancer (like MetalLB). In this sense, the approach above is similar to the next example below. We want to provide you with a few alternatives too. **But** we also want to communicate the idea of another simple method: you could use a load-balancer without an external IP and DNAT the network traffic to the mail server. After all, this does not interfere with SPF checks because it keeps the origin IP address. If no dedicated external IP address is available, you could try the latter approach, if one is available, use the former. +The major problem with exposing DMS to the outside world in Kubernetes is to [preserve the real client IP][k8s-docs::service-source-ip]. The real client IP is required by DMS for performing IP-based DNS and spam checks. -### External IPs Service +Kubernetes provides multiple ways to address this; each has its upsides and downsides. -The simplest way is to expose DMS as a [Service][Kubernetes-network-service] with [external IPs][Kubernetes-network-external-ip]. This is very similar to the approach taken above. Here, an external IP is given to the service directly by you. With the approach above, you tell your load-balancer to do this. + +!!! quote "" -```yaml ---- -apiVersion: v1 -kind: Service + === "Configure IP Manually" -metadata: - name: mailserver - labels: - app: mailserver + ???+ abstract "Advantages / Disadvantages" -spec: - selector: - app: mailserver - ports: - - name: smtp - port: 25 - targetPort: smtp - # ... + - [x] Simple + - [ ] Requires the node to have a dedicated, publicly routable IP address + - [ ] Limited to a single node (_associated to the dedicated IP address_) + - [ ] Your deployment requires an explicit IP in your configuration (_or an entire Load Balancer_). - externalIPs: - - 80.11.12.10 -``` + !!! info "Requirements" -This approach + 1. You can dedicate a **publicly routable IP** address for the DMS configured `Service`. + 2. A dedicated IP is required to allow your mail server to have matching `A` and `PTR` records (_which other mail servers will use to verify trust when they receive mail sent from your DMS instance_). -- does not preserve the real client IP, so SPF check of incoming mail will fail. -- requires you to specify the exposed IPs explicitly. + !!! example -### Proxy port to Service + Assign the DMS `Service` an external IP directly, or delegate an LB to assign the IP on your behalf. -The [proxy pod][Kubernetes-proxy-service] helps to avoid the necessity of specifying external IPs explicitly. This comes at the cost of complexity; you must deploy a proxy pod on each [Node][Kubernetes-nodes] you want to expose DMS on. + === "External-IP Service" -This approach + The DMS `Service` is configured with an "[external IP][k8s-docs::network-external-ip]" manually. Append your externally reachable IP address to `spec.externalIPs`. -- does not preserve the real client IP, so SPF check of incoming mail will fail. + ```yaml + --- + apiVersion: v1 + kind: Service -### Bind to concrete Node and use host network + metadata: + name: mailserver + labels: + app: mailserver -One way to preserve the real client IP is to use `hostPort` and `hostNetwork: true`. This comes at the cost of availability; you can reach DMS from the outside world only via IPs of [Node][Kubernetes-nodes] where DMS is deployed. + spec: + selector: + app: mailserver + ports: + - name: smtp + port: 25 + targetPort: smtp + # ... -```yaml ---- -apiVersion: extensions/v1beta1 -kind: Deployment + externalIPs: + - 10.20.30.40 + ``` -metadata: - name: mailserver + === "Load-Balancer" -# ... - spec: - hostNetwork: true + The config differs depending on your choice of load balancer. This example uses [MetalLB][metallb-web]. - # ... - containers: - # ... - ports: - - name: smtp - containerPort: 25 - hostPort: 25 - - name: smtp-auth - containerPort: 587 - hostPort: 587 - - name: imap-secure - containerPort: 993 - hostPort: 993 - # ... -``` + ```yaml + --- + apiVersion: v1 + kind: Service -With this approach, + metadata: + name: mailserver + labels: + app: mailserver + annotations: + metallb.universe.tf/address-pool: mailserver -- it is not possible to access DMS via other cluster Nodes, only via the Node DMS was deployed at. -- every Port within the Container is exposed on the Host side. + # ... -### Proxy Port to Service via PROXY Protocol + --- + apiVersion: metallb.io/v1beta1 + kind: IPAddressPool -This way is ideologically the same as [using a proxy pod](#proxy-port-to-service), but instead of a separate proxy pod, you configure your ingress to proxy TCP traffic to the DMS pod using the PROXY protocol, which preserves the real client IP. + metadata: + name: mail + namespace: metallb-system -#### Configure your Ingress + spec: + addresses: [ ] + autoAssign: true -With an [NGINX ingress controller][Kubernetes-nginx], set `externalTrafficPolicy: Local` for its service, and add the following to the TCP services config map (as described [here][Kubernetes-nginx-expose]): + --- + apiVersion: metallb.io/v1beta1 + kind: L2Advertisement -```yaml -25: "mailserver/mailserver:25::PROXY" -465: "mailserver/mailserver:465::PROXY" -587: "mailserver/mailserver:587::PROXY" -993: "mailserver/mailserver:993::PROXY" -``` + metadata: + name: mail + namespace: metallb-system -!!! help "HAProxy" - With [HAProxy][dockerhub-haproxy], the configuration should look similar to the above. If you know what it actually looks like, add an example here. :smiley: + spec: + ipAddressPools: [ mailserver ] + ``` -#### Configure the Mailserver + === "Host network" -Then, configure both [Postfix][docs-postfix] and [Dovecot][docs-dovecot] to expect the PROXY protocol: + ???+ abstract "Advantages / Disadvantages" -??? example "HAProxy Example" + - [x] Simple + - [ ] Requires the node to have a dedicated, publicly routable IP address + - [ ] Limited to a single node (_associated to the dedicated IP address_) + - [ ] It is not possible to access DMS via other cluster nodes, only via the node that DMS was deployed on + - [ ] Every port within the container is exposed on the host side - ```yaml - kind: ConfigMap - apiVersion: v1 - metadata: - name: mailserver.config - labels: - app: mailserver - data: - postfix-main.cf: | - postscreen_upstream_proxy_protocol = haproxy - postfix-master.cf: | - smtp/inet/postscreen_upstream_proxy_protocol=haproxy - submission/inet/smtpd_upstream_proxy_protocol=haproxy - submissions/inet/smtpd_upstream_proxy_protocol=haproxy - dovecot.cf: | - # Assuming your ingress controller is bound to 10.0.0.0/8 - haproxy_trusted_networks = 10.0.0.0/8, 127.0.0.0/8 - service imap-login { - inet_listener imap { - haproxy = yes - } - inet_listener imaps { - haproxy = yes - } - } - # ... - --- + !!! example - kind: Deployment - apiVersion: extensions/v1beta1 - metadata: - name: mailserver - spec: - template: - spec: - containers: - - name: docker-mailserver - volumeMounts: - - name: config - subPath: postfix-main.cf - mountPath: /tmp/docker-mailserver/postfix-main.cf - readOnly: true - - name: config - subPath: postfix-master.cf - mountPath: /tmp/docker-mailserver/postfix-master.cf - readOnly: true - - name: config - subPath: dovecot.cf - mountPath: /tmp/docker-mailserver/dovecot.cf - readOnly: true - ``` + Using `hostPort` and `hostNetwork: true` is a similar approach to [`network_mode: host` with Docker Compose][docker-docs::compose::network_mode]. -With this approach, + ```yaml + --- + apiVersion: apps/v1 + kind: Deployment -- it is not possible to access DMS via cluster-DNS, as the PROXY protocol is required for incoming connections. + metadata: + name: mailserver -[Helm chart]: https://github.com/docker-mailserver/docker-mailserver-helm -[kustomize]: https://kustomize.io/ -[cert-manager]: https://cert-manager.io/docs/ + # ... + spec: + hostNetwork: true + # ... + containers: + # ... + ports: + - name: smtp + containerPort: 25 + hostPort: 25 + - name: submissions + containerPort: 465 + hostPort: 465 + - name: submission + containerPort: 587 + hostPort: 587 + - name: imaps + containerPort: 993 + hostPort: 993 + ``` + + === "Using the PROXY Protocol" + + ???+ abstract "Advantages / Disadvantages" + + - [x] Preserves the origin IP address of clients (_which is crucial for DNS related checks_) + - [x] Aligns with a best practice for Kubernetes by using a dedicated ingress, routing external traffic to the k8s cluster (_with the benefits of flexible routing rules_) + - [x] Avoids the restraint of a single [node][k8s-docs::nodes] (_as a workaround to preserve the original client IP_) + - [ ] Introduces complexity by requiring: + - A reverse-proxy / ingress controller (_potentially extra setup_) + - Kubernetes manifest changes for the DMS configured `Service` + - DMS configuration changes for Postfix and Dovecot + - [ ] To keep support for direct connections to DMS services internally within cluster, service ports must be "duplicated" to offer an alternative port for connections using PROXY protocol + + ??? question "What is the PROXY protocol?" + + PROXY protocol is a network protocol for preserving a client’s IP address when the client’s TCP connection passes through a proxy. + + It is a common feature supported among reverse-proxy services (_NGINX, HAProxy, Traefik_), which you may already have handling ingress traffic for your cluster. + + ```mermaid + flowchart LR + A(External Mail Server) -->|Incoming connection| B + subgraph cluster + B("Ingress Acting as a Proxy") -->|PROXY protocol connection| C(DMS) + end + ``` + + For more information on the PROXY protocol, refer to [our dedicated docs page][docs-mailserver-behind-proxy] on the topic. + + ???+ example "Configure the Ingress Controller" + + === "Traefik" + + On Traefik's side, the configuration is very simple. + + - Create an entrypoint for each port that you want to expose (_probably 25, 465, 587 and 993_). + - Each entrypoint should configure an [`IngressRouteTCP`][traefik-docs::k8s::ingress-route-tcp] that routes to the equivalent internal DMS `Service` port which supports PROXY protocol connections. + + The below snippet demonstrates an example for two entrypoints, `submissions` (port 465) and `imaps` (port 993). + + ```yaml + --- + apiVersion: v1 + kind: Service + + metadata: + name: mailserver + + spec: + # This an optimization to get rid of additional routing steps. + # Previously "type: LoadBalancer" + type: ClusterIP + + --- + apiVersion: traefik.io/v1alpha1 + kind: IngressRouteTCP + + metadata: + name: smtp + + spec: + entryPoints: [ submissions ] + routes: + - match: HostSNI(`*`) + services: + - name: mailserver + namespace: mail + port: subs-proxy # note the 15 character limit here + proxyProtocol: + version: 2 + + --- + apiVersion: traefik.io/v1alpha1 + kind: IngressRouteTCP + + metadata: + name: imaps + + spec: + entryPoints: [ imaps ] + routes: + - match: HostSNI(`*`) + services: + - name: mailserver + namespace: mail + port: imaps-proxy + proxyProtocol: + version: 2 + ``` + + !!! info "`*-proxy` port name suffix" + + The `IngressRouteTCP` example configs above reference ports with a `*-proxy` suffix. + + - These port variants will be defined in the [`Deployment` manifest][docs::k8s::config-deployment], and are scoped to the `mailserver` service (via `spec.routes.services.name`). + - The suffix is used to distinguish that these ports are only compatible with connections using the PROXY protocol, which is what your ingress controller should be managing for you by adding the correct PROXY protocol headers to TCP connections it routes to DMS. + + === "NGINX" + + With an [NGINX ingress controller][k8s-docs::nginx], add the following to the TCP services config map (_as described [here][k8s-docs::nginx-expose]_): + + ```yaml + 25: "mailserver/mailserver:25::PROXY" + 465: "mailserver/mailserver:465::PROXY" + 587: "mailserver/mailserver:587::PROXY" + 993: "mailserver/mailserver:993::PROXY" + ``` + + ???+ example "Adjust DMS config for Dovecot + Postfix" + + ??? warning "Only ingress should connect to DMS with PROXY protocol" + + While Dovecot will restrict connections via PROXY protocol to only clients trusted configured via `haproxy_trusted_networks`, Postfix does not have an equivalent setting. Public clients should always route through ingress to establish a PROXY protocol connection. + + You are responsible for properly managing traffic inside your cluster and to **ensure that only trustworthy entities** can connect to the designated PROXY protocol ports. + + With Kubernetes, this is usually the task of the CNI (_container network interface_). + + !!! tip "Advised approach" + + The _"Separate PROXY protocol ports"_ tab below introduces a little more complexity, but provides better compatibility for internal connections to DMS. + + === "Only accept connections with PROXY protocol" + + !!! warning "Connections to DMS within the internal cluster will be rejected" + + The services for these ports can only enable PROXY protocol support by mandating the protocol on all connections for these ports. + + This can be problematic when you also need to support internal cluster traffic directly to DMS (_instead of routing indirectly through the ingress controller_). + + Here is an example configuration for [Postfix][docs-postfix], [Dovecot][docs-dovecot], and the required adjustments for the [`Deployment` manifest][docs::k8s::config-deployment]. The port names are adjusted here only to convey the additional context described earlier. + + ```yaml + kind: ConfigMap + apiVersion: v1 + metadata: + name: mailserver-extra-config + labels: + app: mailserver + data: + postfix-main.cf: | + postscreen_upstream_proxy_protocol = haproxy + postfix-master.cf: | + smtp/inet/postscreen_upstream_proxy_protocol=haproxy + submission/inet/smtpd_upstream_proxy_protocol=haproxy + submissions/inet/smtpd_upstream_proxy_protocol=haproxy + dovecot.cf: | + haproxy_trusted_networks = + service imap-login { + inet_listener imap { + haproxy = yes + } + inet_listener imaps { + haproxy = yes + } + } + # ... + + --- + kind: Deployment + apiVersion: apps/v1 + metadata: + name: mailserver + spec: + template: + spec: + containers: + - name: docker-mailserver + # ... + ports: + - name: smtp-proxy + containerPort: 25 + protocol: TCP + - name: imap-proxy + containerPort: 143 + protocol: TCP + - name: subs-proxy + containerPort: 465 + protocol: TCP + - name: sub-proxy + containerPort: 587 + protocol: TCP + - name: imaps-proxy + containerPort: 993 + protocol: TCP + # ... + volumeMounts: + - name: config + subPath: postfix-main.cf + mountPath: /tmp/docker-mailserver/postfix-main.cf + readOnly: true + - name: config + subPath: postfix-master.cf + mountPath: /tmp/docker-mailserver/postfix-master.cf + readOnly: true + - name: config + subPath: dovecot.cf + mountPath: /tmp/docker-mailserver/dovecot.cf + readOnly: true + ``` + + === "Separate PROXY protocol ports for ingress" + + !!! info + + Supporting internal cluster connections to DMS without using PROXY protocol requires both Postfix and Dovecot to be configured with alternative ports for each service port (_which only differ by enforcing PROXY protocol connections_). + + - The ingress controller will route public connections to the internal alternative ports for DMS (`*-proxy` variants). + - Internal cluster connections will instead use the original ports configured for the DMS container directly (_which are private to the cluster network_). + + In this example we'll create a copy of the original service ports with PROXY protocol enabled, and increment the port number assigned by `10000`. + + Create a `user-patches.sh` file to apply these config changes during container startup: + + ```bash + #!/bin/bash + + # Duplicate the config for the submission(s) service ports (587 / 465) with adjustments for the PROXY ports (10587 / 10465) and `syslog_name` setting: + postconf -Mf submission/inet | sed -e s/^submission/10587/ -e 's/submission/submission-proxyprotocol/' >> /etc/postfix/master.cf + postconf -Mf submissions/inet | sed -e s/^submissions/10465/ -e 's/submissions/submissions-proxyprotocol/' >> /etc/postfix/master.cf + # Enable PROXY Protocol support for these new service variants: + postconf -P 10587/inet/smtpd_upstream_proxy_protocol=haproxy + postconf -P 10465/inet/smtpd_upstream_proxy_protocol=haproxy + + # Create a variant for port 25 too (NOTE: Port 10025 is already assigned in DMS to Amavis): + postconf -Mf smtp/inet | sed -e s/^smtp/12525/ >> /etc/postfix/master.cf + # Enable PROXY Protocol support (different setting as port 25 is handled via postscreen), optionally configure a `syslog_name` to distinguish in logs: + postconf -P 12525/inet/postscreen_upstream_proxy_protocol=haproxy 12525/inet/syslog_name=smtp-proxyprotocol + ``` + + For Dovecot, you can configure [`dovecot.cf`][docs-dovecot] to look like this: + + ```cf + haproxy_trusted_networks = + + service imap-login { + inet_listener imap-proxied { + haproxy = yes + port = 10143 + } + + inet_listener imaps-proxied { + haproxy = yes + port = 10993 + ssl = yes + } + } + ``` + + Update the [`Deployment` manifest][docs::k8s::config-deployment] `ports` section by appending these new ports: + + ```yaml + - name: smtp-proxy + # not 10025 in this example due to a possible clash with Amavis + containerPort: 12525 + protocol: TCP + - name: imap-proxy + containerPort: 10143 + protocol: TCP + - name: subs-proxy + containerPort: 10465 + protocol: TCP + - name: sub-proxy + containerPort: 10587 + protocol: TCP + - name: imaps-proxy + containerPort: 10993 + protocol: TCP + ``` + + !!! note + + If you use other Dovecot ports (110, 995, 4190), you may want to configure those similar to above. The `dovecot.cf` config for these ports is [documented here][docs-mailserver-behind-proxy] (_in the equivalent section of that page_). + +[docs::k8s::config-deployment]: #deployment [docs-tls]: ../security/ssl.md [docs-dovecot]: ./override-defaults/dovecot.md [docs-postfix]: ./override-defaults/postfix.md -[dockerhub-haproxy]: https://hub.docker.com/_/haproxy -[Kubernetes-nginx]: https://kubernetes.github.io/ingress-nginx -[Kubernetes-nginx-expose]: https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services -[Kubernetes-network-service]: https://kubernetes.io/docs/concepts/services-networking/service -[Kubernetes-network-external-ip]: https://kubernetes.io/docs/concepts/services-networking/service/#external-ips -[Kubernetes-nodes]: https://kubernetes.io/docs/concepts/architecture/nodes -[Kubernetes-proxy-service]: https://github.com/kubernetes/contrib/tree/master/for-demos/proxy-to-service -[Kubernetes-service-source-ip]: https://kubernetes.io/docs/tutorials/services/source-ip +[docs-mailserver-behind-proxy]: ../../examples/tutorials/mailserver-behind-proxy.md + +[github-web::docker-mailserver-helm]: https://github.com/docker-mailserver/docker-mailserver-helm +[docker-docs::compose::network_mode]: https://docs.docker.com/compose/compose-file/compose-file-v3/#network_mode +[kustomize]: https://kustomize.io/ +[cert-manager]: https://cert-manager.io/docs/ +[metallb-web]: https://metallb.universe.tf/ + +[k8s-docs::config::service]: https://kubernetes.io/docs/concepts/services-networking/service +[k8s-docs::config::deployment]: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#creating-a-deployment +[k8s-docs::nodes]: https://kubernetes.io/docs/concepts/architecture/nodes +[k8s-docs::nginx]: https://kubernetes.github.io/ingress-nginx +[k8s-docs::nginx-expose]: https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services +[k8s-docs::service-source-ip]: https://kubernetes.io/docs/tutorials/services/source-ip +[k8s-docs::network-external-ip]: https://kubernetes.io/docs/concepts/services-networking/service/#external-ips + +[traefik-docs::k8s::ingress-route-tcp]: https://doc.traefik.io/traefik/routing/providers/kubernetes-crd/#kind-ingressroutetcp diff --git a/docs/content/examples/tutorials/mailserver-behind-proxy.md b/docs/content/examples/tutorials/mailserver-behind-proxy.md index 47116fde..60e4c539 100644 --- a/docs/content/examples/tutorials/mailserver-behind-proxy.md +++ b/docs/content/examples/tutorials/mailserver-behind-proxy.md @@ -14,6 +14,8 @@ This reduces many of the benefits for why you might use a reverse proxy, but the Some deployments may require a service to route traffic (kubernetes) when deploying, in which case the below advice is important to understand well. +The guide here has also been adapted for [our Kubernetes docs][docs::kubernetes]. + ## What can go wrong? Without a reverse proxy involved, a service is typically aware of the client IP for a connection. @@ -354,9 +356,8 @@ Software on the receiving end of the connection often supports configuring an IP [`postscreen_access_list`][postfix-docs::settings::postscreen_access_list] (_or [`smtpd_client_restrictions`][postfix-docs::settings::smtpd_client_restrictions] with [`check_client_access`][postfix-docs::settings::check_client_access] for ports 587/465_) can both restrict access by IP via a [CIDR lookup table][postfix-docs::config-table::cidr], however the client IP is already rewritten at this point via PROXY protocol. Thus those settings cannot be used for restricting access to only trusted proxies, only to the actual clients. - - A similar setting [`mynetworks`][postfix-docs::settings::mynetworks] / [`PERMIT_DOCKER`][docs::env::permit_docker] manages elevated trust for bypassing security restrictions. While it is intended for trusted clients, it has no relevance to trusting proxies for the same reasons. + A similar setting [`mynetworks`][postfix-docs::settings::mynetworks] / [`PERMIT_DOCKER`][docs::env::permit_docker] manages elevated trust for bypassing security restrictions. While it is intended for trusted clients, it has no relevance to trusting proxies for the same reasons. ### Monitoring @@ -373,6 +374,8 @@ While PROXY protocol works well with the reverse proxy, you may have some contai You should adjust configuration of these monitoring services to monitor for auth failures from those services directly instead, adding an exclusion for that service IP from any DMS logs monitored (_but be mindful of PROXY header forgery risks_). +[docs::kubernetes]: ../../config/advanced/kubernetes.md#using-the-proxy-protocol + [docs::overrides::dovecot]: ../../config/advanced/override-defaults/dovecot.md [docs::overrides::postfix]: ../../config/advanced/override-defaults/postfix.md [docs::overrides::user-patches]: ../../config/advanced/override-defaults/user-patches.md diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 84366423..e0ff50c9 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -82,6 +82,11 @@ markdown_extensions: format: !!python/name:pymdownx.superfences.fence_code_format - pymdownx.tabbed: alternate_style: true + slugify: !!python/object/apply:pymdownx.slugs.slugify + kwds: + case: lower + - pymdownx.tasklist: + custom_checkbox: true - pymdownx.magiclink - pymdownx.inlinehilite - pymdownx.tilde From ede95e6f7fca0b02b756bec62d9f6cf81b64dfb1 Mon Sep 17 00:00:00 2001 From: Rahil Bhimjiani Date: Thu, 14 Mar 2024 02:44:14 +0530 Subject: [PATCH 060/267] docs: Update links for account management in `README.md` (#3937) --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index cfd453ff..e34aeb12 100644 --- a/README.md +++ b/README.md @@ -38,8 +38,8 @@ A production-ready fullstack but simple containerized mail server (SMTP, IMAP, L ## :package: Included Services -- [Postfix](http://www.postfix.org) with SMTP or LDAP authentication and support for [extension delimiters](https://docker-mailserver.github.io/docker-mailserver/latest/config/user-management/aliases/#address-tags-extension-delimiters-an-alternative-to-aliases) -- [Dovecot](https://www.dovecot.org) with SASL, IMAP, POP3, LDAP, [basic Sieve support](https://docker-mailserver.github.io/docker-mailserver/latest/config/advanced/mail-sieve) and [quotas](https://docker-mailserver.github.io/docker-mailserver/latest/config/user-management/accounts#notes) +- [Postfix](http://www.postfix.org) with SMTP or LDAP authentication and support for [extension delimiters](https://docker-mailserver.github.io/docker-mailserver/v13.3/config/user-management/#address-tags-extension-delimiters-as-an-alternative-to-aliases) +- [Dovecot](https://www.dovecot.org) with SASL, IMAP, POP3, LDAP, [basic Sieve support](https://docker-mailserver.github.io/docker-mailserver/latest/config/advanced/mail-sieve) and [quotas](https://docker-mailserver.github.io/docker-mailserver/v13.3/config/user-management/#quotas) - [Rspamd](https://rspamd.com/) - [Amavis](https://www.amavis.org/) - [SpamAssassin](http://spamassassin.apache.org/) supporting custom rules From cdcd86420e83984228dbb027d5f148f3cea9e0c6 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Thu, 14 Mar 2024 15:24:33 +1300 Subject: [PATCH 061/267] docs: Add IPv6 troubleshooting tip (#3938) Sometimes a user may have a configuration error and get halfway there. This should help point them in the right direction. --- docs/content/config/advanced/ipv6.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/content/config/advanced/ipv6.md b/docs/content/config/advanced/ipv6.md index 77dfb0c9..7e9c904c 100644 --- a/docs/content/config/advanced/ipv6.md +++ b/docs/content/config/advanced/ipv6.md @@ -177,6 +177,12 @@ docker run --rm -d --network dms-ipv6 -p 80:80 traefik/whoami curl --max-time 5 http://[2001:db8::1]:80 ``` +!!! warning "IPv6 gateway IP" + + If instead of the remote IPv6 address, you may notice the gateway IP for the IPv6 subnet your DMS container belongs to. + + This will happen when DMS has an IPv6 IP address assigned, for the same reason as with IPv4, `userland-proxy: true`. It indicates that your `daemon.json` has not been configured correctly or had the updated config applied for `ip6tables :true` + `experimental: true`. Make sure you used `systemctl restart docker` after updating `daemon.json`. + !!! info "IPv6 ULA address priority" DNS lookups that have records for both IPv4 and IPv6 addresses (_eg: `localhost`_) may prefer IPv4 over IPv6 (ULA) for private addresses, whereas for public addresses IPv6 has priority. This shouldn't be anything to worry about, but can come across as a surprise when testing your IPv6 setup on the same host instead of from a remote client. From 910667d5866cfdb2e04e893960bb009b626882fd Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 14 Mar 2024 13:22:43 +0100 Subject: [PATCH 062/267] docs: updated `CONTRIBUTORS.md` (#3930) --- CONTRIBUTORS.md | 419 ++++++++++++++++++++++++------------------------ 1 file changed, 213 insertions(+), 206 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index ffe5cecd..9b8e4043 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -658,10 +658,17 @@ Thanks goes to these wonderful people ✨ - - andrewlow + + ubenmackin
- andrewlow + ubenmackin + + + + + craue +
+ craue @@ -671,13 +678,21 @@ Thanks goes to these wonderful people ✨ abh + + + andrewlow +
+ andrewlow + + aminvakil
aminvakil - + + elbracht @@ -685,21 +700,6 @@ Thanks goes to these wonderful people ✨ elbracht - - - craue -
- craue - - - - - - ubenmackin -
- ubenmackin - - danielpanteleit @@ -737,48 +737,12 @@ Thanks goes to these wonderful people ✨ - - fl42 + + nueaf
- fl42 + nueaf - - - ipernet -
- ipernet - - - - - H4R0 -
- H4R0 - - - - - eltociear -
- eltociear - - - - - jamesfryer -
- jamesfryer - - - - - millaguie -
- millaguie - - - martinwepner @@ -793,13 +757,6 @@ Thanks goes to these wonderful people ✨ artonge - - - jedateach -
- jedateach - - spacecowboy @@ -808,46 +765,82 @@ Thanks goes to these wonderful people ✨ - - nueaf + + jedateach
- nueaf + jedateach - - thomasschmit + + millaguie
- thomasschmit + millaguie - - TechnicLab + + eltociear
- TechnicLab + eltociear - - sylvaindumont + + H4R0
- sylvaindumont + H4R0 - - syl20bnr + + jamesfryer
- syl20bnr + jamesfryer - - 5ven + + ipernet
- 5ven + ipernet + + + + + fl42 +
+ fl42 + + + + + radicand +
+ radicand + + + + + + sjmudd +
+ sjmudd + + + + + simonsystem +
+ simonsystem + + + + + stephan-devop +
+ stephan-devop @@ -858,25 +851,53 @@ Thanks goes to these wonderful people ✨ - - stephan-devop + + 5ven
- stephan-devop + 5ven + + + + + syl20bnr +
+ syl20bnr - - simonsystem + + sylvaindumont
- simonsystem + sylvaindumont - - radicand + + TechnicLab
- radicand + TechnicLab + + + + + 42wim +
+ 42wim + + + + + vilisas +
+ vilisas + + + + + thomasschmit +
+ thomasschmit @@ -885,7 +906,8 @@ Thanks goes to these wonderful people ✨
Thiritin - + + tweibert @@ -906,8 +928,7 @@ Thanks goes to these wonderful people ✨
VictorKoenders - - + Twist235 @@ -928,27 +949,6 @@ Thanks goes to these wonderful people ✨
Drakulix - - - - vilisas -
- vilisas - - - - - 42wim -
- 42wim - - - - - ShiriNmi1520 -
- ShiriNmi1520 - @@ -1015,6 +1015,13 @@ Thanks goes to these wonderful people ✨ piwai + + + auchri +
+ auchri + + rahilarious @@ -1028,15 +1035,15 @@ Thanks goes to these wonderful people ✨
remoe - + + robbertkl
robbertkl - - + romansey @@ -1071,15 +1078,15 @@ Thanks goes to these wonderful people ✨
klamann - + + svdb0
svdb0 - - + 3ap @@ -1095,10 +1102,10 @@ Thanks goes to these wonderful people ✨ - - sjmudd + + ShiriNmi1520
- sjmudd + ShiriNmi1520 @@ -1114,15 +1121,15 @@ Thanks goes to these wonderful people ✨
millerjason - + + mplx
mplx - - + odinis @@ -1157,15 +1164,15 @@ Thanks goes to these wonderful people ✨
pravynandas - + + presocratics
presocratics - - + rhyst @@ -1200,15 +1207,15 @@ Thanks goes to these wonderful people ✨
smargold476 - + + sportshead
sportshead - - + squash @@ -1243,15 +1250,15 @@ Thanks goes to these wonderful people ✨
wligtenberg - + + wolkenschieber
wolkenschieber - - + worldworm @@ -1286,15 +1293,15 @@ Thanks goes to these wonderful people ✨
awb99 - + + brainkiller
brainkiller - - + cternes @@ -1329,15 +1336,15 @@ Thanks goes to these wonderful people ✨
ghnp5 - + + helmutundarnold
helmutundarnold - - + hnws @@ -1372,15 +1379,15 @@ Thanks goes to these wonderful people ✨
jjtt - + + paralax
paralax - - + jpduyx @@ -1415,22 +1422,8 @@ Thanks goes to these wonderful people ✨
matrixes - - - - 0xflotus -
- 0xflotus - - - - auchri -
- auchri - - arkanovicz @@ -1465,15 +1458,15 @@ Thanks goes to these wonderful people ✨
dkarski - - + dbellavista
dbellavista - + + danielvandenberg95 @@ -1508,15 +1501,15 @@ Thanks goes to these wonderful people ✨
doominator42 - - + aydodo
aydodo - + + vedtam @@ -1551,15 +1544,15 @@ Thanks goes to these wonderful people ✨
ErikEngerd - - + huncode
huncode - + + felixn @@ -1574,6 +1567,20 @@ Thanks goes to these wonderful people ✨ flole + + + akkumar +
+ akkumar + + + + + 0xflotus +
+ 0xflotus + + ifokeev @@ -1587,15 +1594,15 @@ Thanks goes to these wonderful people ✨
20th - + + 2b
2b - - + askz @@ -1630,15 +1637,15 @@ Thanks goes to these wonderful people ✨
kachkaev - + + alexanderneu
alexanderneu - - + ch3sh1r @@ -1673,15 +1680,15 @@ Thanks goes to these wonderful people ✨
iRhonin - + + MrFreezeex
MrFreezeex - - + arunvc @@ -1716,28 +1723,14 @@ Thanks goes to these wonderful people ✨
erdos4d - + + crash7
crash7 - - - - - froks -
- froks - - - - - akkumar -
- akkumar - @@ -1760,6 +1753,13 @@ Thanks goes to these wonderful people ✨ khuedoan + + + UltraCoderRU +
+ UltraCoderRU + + JustAnother1 @@ -1882,21 +1882,28 @@ Thanks goes to these wonderful people ✨ neuralp + + + froks +
+ froks + + fkefer
fkefer - + + Marsu31
Marsu31 - - + glandais @@ -1931,15 +1938,15 @@ Thanks goes to these wonderful people ✨
sirgantrithon - + + Influencer
Influencer - - + JacksonZ03 @@ -1974,15 +1981,15 @@ Thanks goes to these wonderful people ✨
init-js - + + Jeidnx
Jeidnx - - + jessp01 @@ -2017,15 +2024,15 @@ Thanks goes to these wonderful people ✨
jurekbarth - + + JOduMonT
JOduMonT - - + Kaan88 From 066773e79fe8e49d7f9b4ae4ff85f1da75abab96 Mon Sep 17 00:00:00 2001 From: Casper Date: Sun, 17 Mar 2024 16:31:55 +0100 Subject: [PATCH 063/267] Better support regular container restarts (#3929) Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> --- CHANGELOG.md | 3 +++ target/scripts/start-mailserver.sh | 36 +++++++++++++++++---------- target/scripts/startup/check-stack.sh | 9 ------- target/scripts/startup/setup-stack.sh | 2 +- 4 files changed, 27 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index de5642bf..31f906d5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -74,6 +74,9 @@ The most noteworthy change of this release is the update of the container's base - `LOGROTATE_COUNT` defines the number of files kept by logrotate ([#3907](https://github.com/docker-mailserver/docker-mailserver/pull/3907)) - The fail2ban log file is now also taken into account by `LOGROTATE_COUNT` and `LOGROTATE_INTERVAL` ([#3915](https://github.com/docker-mailserver/docker-mailserver/pull/3915), [#3919](https://github.com/docker-mailserver/docker-mailserver/pull/3919)) +- **Internal:** + - Regular container restarts are now better supported. Setup scripts that ran previously will now be skipped ([#3929](https://github.com/docker-mailserver/docker-mailserver/pull/3929)) + ### Updates - **Environment Variables:** diff --git a/target/scripts/start-mailserver.sh b/target/scripts/start-mailserver.sh index f18abdf6..cbe38da9 100755 --- a/target/scripts/start-mailserver.sh +++ b/target/scripts/start-mailserver.sh @@ -33,7 +33,6 @@ function _register_functions() { # ? >> Checks - _register_check_function '_check_improper_restart' _register_check_function '_check_hostname' _register_check_function '_check_log_level' _register_check_function '_check_spam_prefix' @@ -170,24 +169,35 @@ function _register_functions() { # ? >> Executing all stacks / actual start of DMS # ------------------------------------------------------------ -_early_supervisor_setup -_early_variables_setup +# Ensure DMS only adjusts config files for a new container. +# Container restarts should skip as they retain the modified config. +if [[ ! -f /CONTAINER_START ]]; then + _early_supervisor_setup + _early_variables_setup -_log 'info' "Welcome to docker-mailserver ${DMS_RELEASE}" + _log 'info' "Welcome to docker-mailserver ${DMS_RELEASE}" -_register_functions -_check -_setup -[[ ${LOG_LEVEL} =~ (debug|trace) ]] && print-environment -_run_user_patches -_start_daemons + _register_functions + _check + _setup + _run_user_patches +else + # container was restarted + _early_variables_setup + + _log 'info' 'Container was restarted. Skipping setup routines.' + _log 'info' "Welcome to docker-mailserver ${DMS_RELEASE}" + + _register_functions +fi # marker to check if container was restarted date >/CONTAINER_START +[[ ${LOG_LEVEL} =~ (debug|trace) ]] && print-environment +_start_daemons + _log 'info' "${HOSTNAME} is up and running" touch /var/log/mail/mail.log -tail -Fn 0 /var/log/mail/mail.log - -exit 0 +exec tail -Fn 0 /var/log/mail/mail.log diff --git a/target/scripts/startup/check-stack.sh b/target/scripts/startup/check-stack.sh index cc98cfe2..766fcccf 100644 --- a/target/scripts/startup/check-stack.sh +++ b/target/scripts/startup/check-stack.sh @@ -14,15 +14,6 @@ function _check() { done } -function _check_improper_restart() { - _log 'debug' 'Checking for improper restart' - - if [[ -f /CONTAINER_START ]]; then - _log 'warn' 'This container was (likely) improperly restarted which can result in undefined behavior' - _log 'warn' "Please use 'docker compose up --force-recreate' or equivalent (view our troubleshooting docs)" - fi -} - function _check_hostname() { _log 'debug' 'Checking that hostname/domainname is provided or overridden' diff --git a/target/scripts/startup/setup-stack.sh b/target/scripts/startup/setup-stack.sh index 81e1a98e..8c8e6461 100644 --- a/target/scripts/startup/setup-stack.sh +++ b/target/scripts/startup/setup-stack.sh @@ -40,7 +40,7 @@ function _early_supervisor_setup() { if ! grep -q "loglevel = ${SUPERVISOR_LOGLEVEL}" /etc/supervisor/supervisord.conf; then case "${SUPERVISOR_LOGLEVEL}" in ( 'critical' | 'error' | 'info' | 'debug' ) - sed -i -E \ + sedfile -i -E \ "s|(loglevel).*|\1 = ${SUPERVISOR_LOGLEVEL}|g" \ /etc/supervisor/supervisord.conf From 7017f4c0817efbfe810b8f61e343d026adff7d26 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 20 Mar 2024 09:46:14 +1300 Subject: [PATCH 064/267] chore(deps): Bump docker/build-push-action from 5.2.0 to 5.3.0 (#3947) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 5.2.0 to 5.3.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/v5.2.0...v5.3.0) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index aa3051c8..6a29126d 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -83,7 +83,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v5.2.0 + uses: docker/build-push-action@v5.3.0 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index a729e5b8..7d9b6166 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v5.2.0 + uses: docker/build-push-action@v5.3.0 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 73db6f6c..7b60f319 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v5.2.0 + uses: docker/build-push-action@v5.3.0 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 8ca0b8b2..d9d5639b 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v5.2.0 + uses: docker/build-push-action@v5.3.0 with: context: . tags: mailserver-testing:ci From 849293f88c8a5faebbe2c704badae92b923b2ac5 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 19 Mar 2024 20:48:35 +0000 Subject: [PATCH 065/267] chore(deps): Bump docker/setup-buildx-action from 3.1.0 to 3.2.0 (#3946) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3.1.0 to 3.2.0. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](https://github.com/docker/setup-buildx-action/compare/v3.1.0...v3.2.0) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 6a29126d..f0bd2111 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -79,7 +79,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.1.0 + uses: docker/setup-buildx-action@v3.2.0 # NOTE: AMD64 can build within 2 minutes - name: 'Build images' diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 7d9b6166..266090b6 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -40,7 +40,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.1.0 + uses: docker/setup-buildx-action@v3.2.0 # Try get the cached build layers from a prior `generic_build.yml` job. # NOTE: Until adopting `type=gha` scoped cache exporter (in `docker/build-push-action`), diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 7b60f319..4c49ca3e 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -38,7 +38,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.1.0 + uses: docker/setup-buildx-action@v3.2.0 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index d9d5639b..23544820 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -37,7 +37,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.1.0 + uses: docker/setup-buildx-action@v3.2.0 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. From 3125cad45a7895e93e4779aed1a7c86ad8cf3ee8 Mon Sep 17 00:00:00 2001 From: Casper Date: Thu, 21 Mar 2024 00:53:04 +0100 Subject: [PATCH 066/267] Enable spamassassin only, when amavis is enabled too. (#3943) --- CHANGELOG.md | 1 + target/scripts/startup/setup.d/security/misc.sh | 9 +++++---- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 31f906d5..89efb030 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -84,6 +84,7 @@ The most noteworthy change of this release is the update of the container's base - It's only functionality remaining was to opt-out of run-time state consolidation with `ONE_DIR=0` (_when a volume was already mounted to `/var/mail-state`_). - **Internal:** - Changed the Postgrey whitelist retrieved during build to [source directly from Github](https://github.com/schweikert/postgrey/blob/master/postgrey_whitelist_clients) as the list is updated more frequently than the [author publishes at their website](https://postgrey.schweikert.ch) ([#3879](https://github.com/docker-mailserver/docker-mailserver/pull/3879)) + - Enable spamassassin only, when amavis is enabled too. ([#3943](https://github.com/docker-mailserver/docker-mailserver/pull/3943)) - **Tests:** - Refactored helper methods for sending e-mails with specific `Message-ID` headers and the helpers for retrieving + filtering logs, which together help isolate logs relevant to specific mail when multiple mails have been processed within a single test. ([#3786](https://github.com/docker-mailserver/docker-mailserver/pull/3786)) - **Rspamd**: diff --git a/target/scripts/startup/setup.d/security/misc.sh b/target/scripts/startup/setup.d/security/misc.sh index bb460716..444589df 100644 --- a/target/scripts/startup/setup.d/security/misc.sh +++ b/target/scripts/startup/setup.d/security/misc.sh @@ -67,6 +67,11 @@ function __setup__security__postscreen() { } function __setup__security__spamassassin() { + if [[ ${ENABLE_AMAVIS} -ne 1 && ${ENABLE_SPAMASSASSIN} -eq 1 ]]; then + _log 'warn' 'Spamassassin does not work when Amavis is disabled. Enable Amavis to fix it.' + ENABLE_SPAMASSASSIN=0 + fi + if [[ ${ENABLE_SPAMASSASSIN} -eq 1 ]]; then _log 'debug' 'Enabling and configuring SpamAssassin' @@ -240,10 +245,6 @@ function __setup__security__amavis() { if [[ ${ENABLE_CLAMAV} -eq 1 ]] && [[ ${ENABLE_RSPAMD} -eq 0 ]]; then _log 'warn' 'ClamAV will not work when Amavis & rspamd are disabled. Enable either Amavis or rspamd to fix it.' fi - - if [[ ${ENABLE_SPAMASSASSIN} -eq 1 ]]; then - _log 'warn' 'Spamassassin will not work when Amavis is disabled. Enable Amavis to fix it.' - fi fi } From 0dad7c49a4f4e6880c091ef0cdbed828cfd86068 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 21 Mar 2024 20:53:00 +0100 Subject: [PATCH 067/267] docs: updated `CONTRIBUTORS.md` (#3944) --- CONTRIBUTORS.md | 472 ++++++++++++++++++++++++------------------------ 1 file changed, 236 insertions(+), 236 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 9b8e4043..7c81c968 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -320,49 +320,6 @@ Thanks goes to these wonderful people ✨ kiliant - - - robertdolca -
- robertdolca - - - - - okainov -
- okainov - - - - - lukecyca -
- lukecyca - - - - - jsonn -
- jsonn - - - - - - jamebus -
- jamebus - - - - - mathuin -
- mathuin - - dashohoxha @@ -378,53 +335,53 @@ Thanks goes to these wonderful people ✨ - - weo + + mathuin
- weo + mathuin - - Zehir + + jamebus
- Zehir + jamebus - - guardiande + + robertdolca
- guardiande + robertdolca - - kamuri + + okainov
- kamuri + okainov - - davidszp + + jsonn
- davidszp + jsonn - - andreasgerstmayr + + lukecyca
- andreasgerstmayr + lukecyca - - VanVan + + m-schmoock
- VanVan + m-schmoock @@ -436,10 +393,17 @@ Thanks goes to these wonderful people ✨ - - m-schmoock + + eltociear
- m-schmoock + eltociear + + + + + VanVan +
+ VanVan @@ -449,6 +413,42 @@ Thanks goes to these wonderful people ✨ voordev + + + andreasgerstmayr +
+ andreasgerstmayr + + + + + davidszp +
+ davidszp + + + + + kamuri +
+ kamuri + + + + + + guardiande +
+ guardiande + + + + + Zehir +
+ Zehir + + Birkenstab @@ -485,6 +485,20 @@ Thanks goes to these wonderful people ✨ yajo + + + MakerMatrix +
+ MakerMatrix + + + + + weo +
+ weo + + analogue @@ -505,27 +519,20 @@ Thanks goes to these wonderful people ✨
reneploetz - - - - MakerMatrix -
- MakerMatrix - - + + pbek
pbek - - + - - keslerm + + yogo1212
- keslerm + yogo1212 @@ -542,6 +549,21 @@ Thanks goes to these wonderful people ✨ p-fruck + + + tbutter +
+ tbutter + + + + + rahilarious +
+ rahilarious + + + Rillke @@ -562,8 +584,7 @@ Thanks goes to these wonderful people ✨
r-pufky - - + vincentDcmps @@ -571,6 +592,21 @@ Thanks goes to these wonderful people ✨ vincentDcmps + + + frugan-dev +
+ frugan-dev + + + + + mpanneck +
+ mpanneck + + + andymel123 @@ -605,49 +641,13 @@ Thanks goes to these wonderful people ✨
lokipo - - + msheakoski
msheakoski - - - - GoliathLabs -
- GoliathLabs - - - - - frugan-dev -
- frugan-dev - - - - - tbutter -
- tbutter - - - - - yogo1212 -
- yogo1212 - - - - - mpanneck -
- mpanneck - @@ -657,6 +657,13 @@ Thanks goes to these wonderful people ✨ willtho89 + + + GoliathLabs +
+ GoliathLabs + + ubenmackin @@ -684,15 +691,15 @@ Thanks goes to these wonderful people ✨
andrewlow - + + aminvakil
aminvakil - - + elbracht @@ -727,21 +734,14 @@ Thanks goes to these wonderful people ✨
DuncanvR - + + emazzotta
emazzotta - - - - - nueaf -
- nueaf - @@ -772,25 +772,25 @@ Thanks goes to these wonderful people ✨ - - millaguie + + nueaf
- millaguie + nueaf - - eltociear + + keslerm
- eltociear + keslerm - - H4R0 + + millaguie
- H4R0 + millaguie @@ -800,13 +800,6 @@ Thanks goes to these wonderful people ✨ jamesfryer - - - ipernet -
- ipernet - - fl42 @@ -815,18 +808,25 @@ Thanks goes to these wonderful people ✨ - - radicand + + H4R0
- radicand + H4R0 + + + + + ipernet +
+ ipernet - - sjmudd + + neuralp
- sjmudd + neuralp @@ -879,20 +879,6 @@ Thanks goes to these wonderful people ✨ TechnicLab - - - 42wim -
- 42wim - - - - - vilisas -
- vilisas - - thomasschmit @@ -906,8 +892,7 @@ Thanks goes to these wonderful people ✨
Thiritin - - + tweibert @@ -921,7 +906,8 @@ Thanks goes to these wonderful people ✨
torus - + + VictorKoenders @@ -949,8 +935,36 @@ Thanks goes to these wonderful people ✨
Drakulix + + + + vilisas +
+ vilisas + + + + + 42wim +
+ 42wim + + + + ShiriNmi1520 +
+ ShiriNmi1520 + + + + + radicand +
+ radicand + + nilshoell @@ -978,7 +992,8 @@ Thanks goes to these wonderful people ✨
OrvilleQ - + + ovidiucp @@ -992,8 +1007,7 @@ Thanks goes to these wonderful people ✨
mrPjer - - + p3dda @@ -1015,20 +1029,6 @@ Thanks goes to these wonderful people ✨ piwai - - - auchri -
- auchri - - - - - rahilarious -
- rahilarious - - remoe @@ -1102,10 +1102,10 @@ Thanks goes to these wonderful people ✨ - - ShiriNmi1520 + + sjmudd
- ShiriNmi1520 + sjmudd @@ -1424,6 +1424,20 @@ Thanks goes to these wonderful people ✨ + + + 0xflotus +
+ 0xflotus + + + + + auchri +
+ auchri + + arkanovicz @@ -1451,7 +1465,8 @@ Thanks goes to these wonderful people ✨
espitall - + + dkarski @@ -1465,8 +1480,7 @@ Thanks goes to these wonderful people ✨
dbellavista - - + danielvandenberg95 @@ -1494,7 +1508,8 @@ Thanks goes to these wonderful people ✨
mazzz1y - + + doominator42 @@ -1508,8 +1523,7 @@ Thanks goes to these wonderful people ✨
aydodo - - + vedtam @@ -1537,7 +1551,8 @@ Thanks goes to these wonderful people ✨
ekkis - + + ErikEngerd @@ -1551,8 +1566,7 @@ Thanks goes to these wonderful people ✨
huncode - - + felixn @@ -1567,20 +1581,6 @@ Thanks goes to these wonderful people ✨ flole - - - akkumar -
- akkumar - - - - - 0xflotus -
- 0xflotus - - ifokeev @@ -1732,6 +1732,20 @@ Thanks goes to these wonderful people ✨ crash7 + + + froks +
+ froks + + + + + akkumar +
+ akkumar + + thechubbypanda @@ -1752,7 +1766,8 @@ Thanks goes to these wonderful people ✨
khuedoan - + + UltraCoderRU @@ -1766,8 +1781,7 @@ Thanks goes to these wonderful people ✨
JustAnother1 - - + LeoWinterDE @@ -1795,7 +1809,8 @@ Thanks goes to these wonderful people ✨
LucidityCrash - + + MadsRC @@ -1809,8 +1824,7 @@ Thanks goes to these wonderful people ✨
madmath03 - - + maxemann96 @@ -1838,7 +1852,8 @@ Thanks goes to these wonderful people ✨
exhuma - + + milas @@ -1852,8 +1867,7 @@ Thanks goes to these wonderful people ✨
mcchots - - + MohammedNoureldin @@ -1875,20 +1889,6 @@ Thanks goes to these wonderful people ✨ naveensrinivasan - - - neuralp -
- neuralp - - - - - froks -
- froks - - fkefer From 082e076377b48378761890a6a39acfda2a0db68d Mon Sep 17 00:00:00 2001 From: Inseo Song Date: Thu, 28 Mar 2024 13:02:11 +0900 Subject: [PATCH 068/267] docs: Add relay host config guide for Gmail (#3958) Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- .../advanced/mail-forwarding/gmail-smtp.md | 50 +++++++++++++++++++ docs/mkdocs.yml | 1 + 2 files changed, 51 insertions(+) create mode 100644 docs/content/config/advanced/mail-forwarding/gmail-smtp.md diff --git a/docs/content/config/advanced/mail-forwarding/gmail-smtp.md b/docs/content/config/advanced/mail-forwarding/gmail-smtp.md new file mode 100644 index 00000000..a581acd1 --- /dev/null +++ b/docs/content/config/advanced/mail-forwarding/gmail-smtp.md @@ -0,0 +1,50 @@ +--- +title: 'Mail Forwarding | Configure Gmail as a relay host' +--- + +This page provides a guide for configuring DMS to use [GMAIL as an SMTP relay host][gmail-smtp]. + +!!! example "Configuration via ENV" + + [Configure a relay host in DMS][docs::relay]. This example shows how the related ENV settings map to the Gmail service config: + + - `RELAY_HOST` should be configured as [advised by Gmail][gmail-smtp::relay-host], there are two SMTP endpoints to choose: + - `smtp.gmail.com` (_for a personal Gmail account_) + - `smtp-relay.gmail.com` (_when using Google Workspace_) + - `RELAY_PORT` should be set to [one of the supported Gmail SMTP ports][gmail-smtp::relay-port] (_eg: 587 for STARTTLS_). + - `RELAY_USER` should be your gmail address (`user@gmail.com`). + - `RELAY_PASSWORD` should be your [App Password][gmail-smtp::app-password], **not** your personal gmail account password. + + ```env + RELAY_HOST=smtp.gmail.com + RELAY_PORT=587 + # Alternative to RELAY_HOST + RELAY_PORT which is compatible with LDAP: + DEFAULT_RELAY_HOST=[smtp.gmail.com]:587 + + RELAY_USER=username@gmail.com + RELAY_PASSWORD=secret + ``` + +!!! tip + + - As per our main [relay host docs page][docs::relay], you may prefer to configure your credentials via `setup relay add-auth` instead of the `RELAY_USER` + `RELAY_PASSWORD` ENV. + - If you configure for `smtp-relay.gmail.com`, the `DEFAULT_RELAY_HOST` ENV should be all you need as shown in the above example. Credentials can be optional when using Google Workspace (`smtp-relay.gmail.com`), which supports restricting connections to trusted IP addresses. + +!!! note "Verify the relay host is configured correctly" + + To verify proper operation, send an email to an external account of yours and inspect the mail headers. + + You will also see the connection to the Gmail relay host (`smtp.gmail.com`) in the mail logs: + + ```log + postfix/smtp[910]: Trusted TLS connection established to smtp.gmail.com[64.233.188.109]:587: + TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) + postfix/smtp[910]: 4BCB547D9D: to=, relay=smtp.gmail.com[64.233.188.109]:587, + delay=2.9, delays=0.01/0.02/1.7/1.2, dsn=2.0.0, status=sent (250 2.0.0 OK 17... - gsmtp) + ``` + +[docs::relay]: ./relay-hosts.md +[gmail-smtp]: https://support.google.com/a/answer/2956491 +[gmail-smtp::relay-host]: https://support.google.com/a/answer/176600 +[gmail-smtp::relay-port]: https://support.google.com/a/answer/2956491 +[gmail-smtp::app-password]: https://support.google.com/accounts/answer/185833 diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index e0ff50c9..9d7fc81d 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -161,6 +161,7 @@ nav: - 'Email Forwarding': - 'Relay Hosts': config/advanced/mail-forwarding/relay-hosts.md - 'AWS SES': config/advanced/mail-forwarding/aws-ses.md + - 'Configure Gmail as a relay host': config/advanced/mail-forwarding/gmail-smtp.md - 'Full-Text Search': config/advanced/full-text-search.md - 'Kubernetes': config/advanced/kubernetes.md - 'IPv6': config/advanced/ipv6.md From 4f10089c90625eb4ccb8bff1cf6e40630e0d8b6c Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Fri, 29 Mar 2024 02:07:13 +0100 Subject: [PATCH 069/267] docs: add note about custom F2B setup with PROXY protocol (#3964) --- docs/content/config/advanced/kubernetes.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/content/config/advanced/kubernetes.md b/docs/content/config/advanced/kubernetes.md index 97a7e414..8f5ac901 100644 --- a/docs/content/config/advanced/kubernetes.md +++ b/docs/content/config/advanced/kubernetes.md @@ -26,7 +26,7 @@ If using our Helm chart is not viable for you, here is some guidance to start wi === "`ConfigMap`" Provide the basic configuration via environment variables with a `ConfigMap`. - + !!! example Below is only an example configuration, adjust the `ConfigMap` to your own needs. @@ -512,6 +512,7 @@ Kubernetes provides multiple ways to address this; each has its upsides and down - Kubernetes manifest changes for the DMS configured `Service` - DMS configuration changes for Postfix and Dovecot - [ ] To keep support for direct connections to DMS services internally within cluster, service ports must be "duplicated" to offer an alternative port for connections using PROXY protocol + - [ ] Custom Fail2Ban required: Because the traffic to DMS is now coming from the proxy, banning the origin IP address will have no effect; you'll need to implement a [custom solution for your setup][github-web::docker-mailserver::proxy-protocol-fail2ban]. ??? question "What is the PROXY protocol?" @@ -795,3 +796,4 @@ Kubernetes provides multiple ways to address this; each has its upsides and down [k8s-docs::network-external-ip]: https://kubernetes.io/docs/concepts/services-networking/service/#external-ips [traefik-docs::k8s::ingress-route-tcp]: https://doc.traefik.io/traefik/routing/providers/kubernetes-crd/#kind-ingressroutetcp +[github-web::docker-mailserver::proxy-protocol-fail2ban]: https://github.com/docker-mailserver/docker-mailserver/issues/1761#issuecomment-2016879319 From 6733a172d77480e1cbc919125c7d9b7b7fd467ac Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Sun, 31 Mar 2024 04:14:02 +0200 Subject: [PATCH 070/267] docs: add FAQ entry about DNS servers and drop feature request on custom DNS servers for Rspamd (#3966) * add FAQ entry about DNS servers I also opted for including a quote from @polarthene which illustrates how DNS servers are a difficult topic and should not be DMS' responsibility. * link to DNS FAQ from Rspamd page & drop feature request The feature request annotation has been removed because we decided it's not DMS responsibility to ensure correctly working DNS servers. --- docs/content/config/security/rspamd.md | 8 +++----- docs/content/faq.md | 13 ++++++++++++- 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/docs/content/config/security/rspamd.md b/docs/content/config/security/rspamd.md index 5cb901b7..fd0fe25e 100644 --- a/docs/content/config/security/rspamd.md +++ b/docs/content/config/security/rspamd.md @@ -94,11 +94,7 @@ Rspamd provides a [web interface][rspamd-docs::web-interface], which contains st ### DNS -DMS does not supply custom values for DNS servers to Rspamd. If you need to use custom DNS servers, which could be required when using [DNS-based black/whitelists](#rbls-realtime-blacklists-dnsbls-dns-based-blacklists), you need to adjust [`options.inc`][rspamd-docs::basic-options] yourself. - -!!! tip "Making DNS Servers Configurable" - - If you want to see an environment variable (like `RSPAMD_DNS_SERVERS`) to support custom DNS servers for Rspamd being added to DMS, please raise a feature request issue. +DMS does not supply custom values for DNS servers (to Rspamd). If you need to use custom DNS servers, which could be required when using [DNS-based deny/allowlists](#rbls-real-time-blacklists-dnsbls-dns-based-blacklists), you need to adjust [`options.inc`][rspamd-docs::basic-options] yourself. Make sure to also read our [FAQ page on DNS servers][docs::faq::dns-servers]. !!! warning @@ -270,3 +266,5 @@ While _Abusix_ can be integrated into Postfix, Postscreen and a multitude of oth [docs::dms-volumes-config]: ../advanced/optional-config.md#volumes-config [docs::dms-volumes-state]: ../advanced/optional-config.md#volumes-state + +[docs::faq::dns-servers]: ../../faq.md#what-about-dns-servers diff --git a/docs/content/faq.md b/docs/content/faq.md index 6b1782e3..4add0589 100644 --- a/docs/content/faq.md +++ b/docs/content/faq.md @@ -79,6 +79,14 @@ volumes: Optionally, you can set the `TZ` ENV variable; e.g. `TZ=Europe/Berlin`. Check [this list](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for which values are allowed. +### What About DNS Servers? + +Properly working DNS servers are crucial for differentiating spam from legitimate e-mails. Records like `SPF`, `DKIM` and `DMARC` records, as well as working name (resolving `A` records) and reverse name (resolving `PTR` records) resolution ensures legitimate e-mails arrive while e-mails that are more likely phishing and spam do not. + +Anti-spam measures (like SpamAssassin or Rspamd) make use of DNS block lists. To learn more check out our [Rspamd documentation on this topic][docs::rspamd-rbl-dnsbl]. In case you want to utilize RBL/DNSBLs, you need a recursive DNS resolver (_not big custom resolvers like Cloudflare, Quad9, Google, etc._). + +DMS does not integrate support for an internal DNS service as this is a [responsibility that is sensitive to the host environment][gh-discussion::dms-avoid-maintaining-internal-dns]. You can configure internal services within DMS to use your own managed DNS server, or configure for such at the host or container level (_such as with [`compose.yaml`][docker-compose::docs::config-dns]_). + ### What is the file format? All files are using the Unix format with `LF` line endings. Please do not use `CRLF`. @@ -376,7 +384,7 @@ The default setup `@local_domains_acl = ( ".$mydomain" );` does not match subdom Put received spams in `.Junk/` imap folder using `SPAMASSASSIN_SPAM_TO_INBOX=1` and `MOVE_SPAM_TO_JUNK=1` and add a _user_ cron like the following: -!!! example +!!! example **NOTE:** This example assumes you have a [`/var/mail-state` volume][docs::dms-volumes-state] mounted. @@ -482,6 +490,7 @@ $spam_quarantine_to = "quarantine\@example.com"; [fail2ban-customize]: ./config/security/fail2ban.md [docs::dms-volumes-state]: ./config/advanced/optional-config.md#volumes-state +[docs::rspamd-rbl-dnsbl]: ./config/security/rspamd.md#rbls-real-time-blacklists-dnsbls-dns-based-blacklists [docs-maintenance]: ./config/advanced/maintenance/update-and-cleanup.md [docs-override-postfix]: ./config/advanced/override-defaults/postfix.md [docs-userpatches]: ./config/advanced/override-defaults/user-patches.md @@ -495,4 +504,6 @@ $spam_quarantine_to = "quarantine\@example.com"; [github-issue-1405-comment]: https://github.com/docker-mailserver/docker-mailserver/issues/1405#issuecomment-590106498 [github-issue-1639]: https://github.com/docker-mailserver/docker-mailserver/issues/1639 [github-issue-1792]: https://github.com/docker-mailserver/docker-mailserver/pull/1792 +[gh-discussion::dms-avoid-maintaining-internal-dns]: https://github.com/orgs/docker-mailserver/discussions/3959#discussioncomment-8956322 +[docker-compose::docs::config-dns]: https://docs.docker.com/compose/compose-file/compose-file-v3/#dns [hanscees-userpatches]: https://github.com/hanscees/dockerscripts/blob/master/scripts/tomav-user-patches.sh From d502dae0687b25c81aa3f010d88ad5552bc9ab41 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Mon, 1 Apr 2024 17:20:35 +0200 Subject: [PATCH 071/267] docs: updated `CONTRIBUTORS.md` (#3967) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- CONTRIBUTORS.md | 329 ++++++++++++++++++++++++------------------------ 1 file changed, 168 insertions(+), 161 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 7c81c968..ea523c94 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -19,6 +19,13 @@ Thanks goes to these wonderful people ✨ fbartels + + + polarathene +
+ polarathene + + NorseGaud @@ -39,15 +46,15 @@ Thanks goes to these wonderful people ✨
wernerfred - + + georglauterbach
georglauterbach - - + tomav @@ -55,13 +62,6 @@ Thanks goes to these wonderful people ✨ tomav - - - polarathene -
- polarathene - - erik-wramner @@ -743,6 +743,20 @@ Thanks goes to these wonderful people ✨ emazzotta + + + keslerm +
+ keslerm + + + + + nueaf +
+ nueaf + + martinwepner @@ -763,7 +777,8 @@ Thanks goes to these wonderful people ✨
spacecowboy - + + jedateach @@ -771,28 +786,6 @@ Thanks goes to these wonderful people ✨ jedateach - - - nueaf -
- nueaf - - - - - - keslerm -
- keslerm - - - - - millaguie -
- millaguie - - jamesfryer @@ -800,13 +793,6 @@ Thanks goes to these wonderful people ✨ jamesfryer - - - fl42 -
- fl42 - - H4R0 @@ -814,12 +800,26 @@ Thanks goes to these wonderful people ✨ H4R0 + + + millaguie +
+ millaguie + + ipernet
ipernet + + + + fl42 +
+ fl42 + @@ -829,6 +829,13 @@ Thanks goes to these wonderful people ✨ neuralp + + + sjmudd +
+ sjmudd + + simonsystem @@ -856,15 +863,15 @@ Thanks goes to these wonderful people ✨
5ven - + + syl20bnr
syl20bnr - - + sylvaindumont @@ -879,13 +886,28 @@ Thanks goes to these wonderful people ✨ TechnicLab + + + 42wim +
+ 42wim + + + + + vilisas +
+ vilisas + + thomasschmit
thomasschmit - + + Thiritin @@ -906,8 +928,7 @@ Thanks goes to these wonderful people ✨
torus - - + VictorKoenders @@ -928,7 +949,8 @@ Thanks goes to these wonderful people ✨
k3it - + + Drakulix @@ -936,28 +958,6 @@ Thanks goes to these wonderful people ✨ Drakulix - - - vilisas -
- vilisas - - - - - 42wim -
- 42wim - - - - - - ShiriNmi1520 -
- ShiriNmi1520 - - radicand @@ -1015,6 +1015,13 @@ Thanks goes to these wonderful people ✨ p3dda + + + auchri +
+ auchri + + peter-hartmann @@ -1028,15 +1035,15 @@ Thanks goes to these wonderful people ✨
piwai - + + remoe
remoe - - + robbertkl @@ -1071,15 +1078,15 @@ Thanks goes to these wonderful people ✨
501st-alpha1 - + + klamann
klamann - - + svdb0 @@ -1102,10 +1109,10 @@ Thanks goes to these wonderful people ✨ - - sjmudd + + ShiriNmi1520
- sjmudd + ShiriNmi1520 @@ -1114,15 +1121,15 @@ Thanks goes to these wonderful people ✨
mchamplain - + + millerjason
millerjason - - + mplx @@ -1157,15 +1164,15 @@ Thanks goes to these wonderful people ✨
ontheair81 - + + pravynandas
pravynandas - - + presocratics @@ -1200,15 +1207,15 @@ Thanks goes to these wonderful people ✨
schnippl0r - + + smargold476
smargold476 - - + sportshead @@ -1243,15 +1250,15 @@ Thanks goes to these wonderful people ✨
vivacarvajalito - + + wligtenberg
wligtenberg - - + wolkenschieber @@ -1286,15 +1293,15 @@ Thanks goes to these wonderful people ✨
arcaine2 - + + awb99
awb99 - - + brainkiller @@ -1329,15 +1336,15 @@ Thanks goes to these wonderful people ✨
eleith - + + ghnp5
ghnp5 - - + helmutundarnold @@ -1372,15 +1379,15 @@ Thanks goes to these wonderful people ✨
ixeft - + + jjtt
jjtt - - + paralax @@ -1415,28 +1422,14 @@ Thanks goes to these wonderful people ✨
marios88 - + + matrixes
matrixes - - - - - 0xflotus -
- 0xflotus - - - - - auchri -
- auchri - @@ -1465,15 +1458,15 @@ Thanks goes to these wonderful people ✨
espitall - - + dkarski
dkarski - + + dbellavista @@ -1508,15 +1501,15 @@ Thanks goes to these wonderful people ✨
mazzz1y - - + doominator42
doominator42 - + + aydodo @@ -1551,15 +1544,15 @@ Thanks goes to these wonderful people ✨
ekkis - - + ErikEngerd
ErikEngerd - + + huncode @@ -1581,21 +1574,35 @@ Thanks goes to these wonderful people ✨ flole + + + Kaan88 +
+ Kaan88 + + + + + 0xflotus +
+ 0xflotus + + ifokeev
ifokeev - + + 20th
20th - - + 2b @@ -1630,15 +1637,15 @@ Thanks goes to these wonderful people ✨
vifino - + + kachkaev
kachkaev - - + alexanderneu @@ -1673,15 +1680,15 @@ Thanks goes to these wonderful people ✨
green-anger - + + iRhonin
iRhonin - - + MrFreezeex @@ -1716,15 +1723,15 @@ Thanks goes to these wonderful people ✨
spock - + + erdos4d
erdos4d - - + crash7 @@ -1732,13 +1739,6 @@ Thanks goes to these wonderful people ✨ crash7 - - - froks -
- froks - - akkumar @@ -1889,14 +1889,21 @@ Thanks goes to these wonderful people ✨ naveensrinivasan + + + froks +
+ froks + + + fkefer
fkefer - - + Marsu31 @@ -1931,15 +1938,15 @@ Thanks goes to these wonderful people ✨
HeySora - + + sirgantrithon
sirgantrithon - - + Influencer @@ -1947,6 +1954,13 @@ Thanks goes to these wonderful people ✨ Influencer + + + in-seo +
+ in-seo + + JacksonZ03 @@ -1967,7 +1981,8 @@ Thanks goes to these wonderful people ✨
jcalfee - + + mivek @@ -1981,8 +1996,7 @@ Thanks goes to these wonderful people ✨
init-js - - + Jeidnx @@ -2010,7 +2024,8 @@ Thanks goes to these wonderful people ✨
jirislav - + + jmccl @@ -2024,21 +2039,13 @@ Thanks goes to these wonderful people ✨
jurekbarth - - + JOduMonT
JOduMonT - - - - Kaan88 -
- Kaan88 - From 8c5cf03203ff164a89f45848e542256f2725bc21 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 8 Apr 2024 17:56:12 +0200 Subject: [PATCH 072/267] chore(deps): Bump docker/setup-buildx-action from 3.2.0 to 3.3.0 (#3972) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index f0bd2111..da5d51c1 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -79,7 +79,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.2.0 + uses: docker/setup-buildx-action@v3.3.0 # NOTE: AMD64 can build within 2 minutes - name: 'Build images' diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 266090b6..ead38a3f 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -40,7 +40,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.2.0 + uses: docker/setup-buildx-action@v3.3.0 # Try get the cached build layers from a prior `generic_build.yml` job. # NOTE: Until adopting `type=gha` scoped cache exporter (in `docker/build-push-action`), diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 4c49ca3e..4e648e32 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -38,7 +38,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.2.0 + uses: docker/setup-buildx-action@v3.3.0 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 23544820..2f1a3803 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -37,7 +37,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.2.0 + uses: docker/setup-buildx-action@v3.3.0 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. From ad5d1011f82401415421b0738209d855bcb12443 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 11 Apr 2024 10:18:17 +0200 Subject: [PATCH 073/267] docs: updated `CONTRIBUTORS.md` (#3971) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- CONTRIBUTORS.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index ea523c94..01c7897f 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -1783,10 +1783,10 @@ Thanks goes to these wonderful people ✨ - - LeoWinterDE + + leowinterde
- LeoWinterDE + leowinterde From f2314259829d170984eacd5790406a6f745ae032 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 15 Apr 2024 18:05:07 +0200 Subject: [PATCH 074/267] chore(deps): Bump peaceiris/actions-gh-pages from 3.9.3 to 4.0.0 (#3978) --- .github/workflows/docs-production-deploy.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/docs-production-deploy.yml b/.github/workflows/docs-production-deploy.yml index cb8bdbed..2c6c1e2c 100644 --- a/.github/workflows/docs-production-deploy.yml +++ b/.github/workflows/docs-production-deploy.yml @@ -59,7 +59,7 @@ jobs: {} + - name: 'Deploy to Github Pages' - uses: peaceiris/actions-gh-pages@v3.9.3 + uses: peaceiris/actions-gh-pages@v4.0.0 with: github_token: ${{ secrets.GITHUB_TOKEN }} # Build directory contents to publish to the `gh-pages` branch: From dc5185003058dbdb1d18947a870a09c43a275d85 Mon Sep 17 00:00:00 2001 From: fanqiaojun <166898955+fanqiaojun@users.noreply.github.com> Date: Tue, 16 Apr 2024 03:48:55 +0800 Subject: [PATCH 075/267] chore: remove repetitive words (#3977) --- docs/content/config/advanced/auth-ldap.md | 2 +- docs/content/examples/use-cases/auth-lua.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/content/config/advanced/auth-ldap.md b/docs/content/config/advanced/auth-ldap.md index ea24526d..397e42eb 100644 --- a/docs/content/config/advanced/auth-ldap.md +++ b/docs/content/config/advanced/auth-ldap.md @@ -26,7 +26,7 @@ Those variables contain the LDAP lookup filters for postfix, using `%s` as the p - Technically, there is no difference between `ALIAS` and `GROUP`, but ideally you should use `ALIAS` for personal aliases for a singular person (like `ceo@example.org`) and `GROUP` for multiple people (like `hr@example.org`). - ...for outgoing email, the sender address is put through the `SENDERS` filter, and only if the authenticated user is one of the returned entries, the email can be sent. - This only applies if `SPOOF_PROTECTION=1`. - - If the `SENDERS` filter is missing, the `USER`, `ALIAS` and `GROUP` filters will be used in in a disjunction (OR). + - If the `SENDERS` filter is missing, the `USER`, `ALIAS` and `GROUP` filters will be used in a disjunction (OR). - To for example allow users from the `admin` group to spoof any sender email address, and to force everyone else to only use their personal mailbox address for outgoing email, you can use something like this: `(|(memberOf=cn=admin,*)(mail=%s))` ???+ example diff --git a/docs/content/examples/use-cases/auth-lua.md b/docs/content/examples/use-cases/auth-lua.md index 82586885..e34ed3b0 100644 --- a/docs/content/examples/use-cases/auth-lua.md +++ b/docs/content/examples/use-cases/auth-lua.md @@ -34,7 +34,7 @@ A drawback of this method is that any (compromised) Nextcloud application passwo To answer the questions asked earlier for this specific scenario: -1. Do I want to use Lua to identify mailboxes and verify that users are are authorized to use mail services? **No. Provisioning is done through LDAP.** +1. Do I want to use Lua to identify mailboxes and verify that users are authorized to use mail services? **No. Provisioning is done through LDAP.** 1. Do I want to use Lua to verify passwords that users authenticate with for IMAP/POP3/SMTP in their mail clients? **Yes. Password authentication is done through Lua against Nextcloud.** 1. If the answer is 'yes' to question 1 or 2: are there other methods that better facilitate my use case instead of custom scripts which rely on me being a developer and not just a user? **No. Only HTTP can be used to authenticate against Nextcloud, which is not supported natively by Dovecot or DMS.** From d87e4d3bfd7780def3c187b0f31c9034954cdfe7 Mon Sep 17 00:00:00 2001 From: Iztok Fister Jr Date: Tue, 16 Apr 2024 22:25:45 +0200 Subject: [PATCH 076/267] docs: Fix typos (#3979) --- docs/content/config/environment.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/content/config/environment.md b/docs/content/config/environment.md index 9adfd5d9..867c7459 100644 --- a/docs/content/config/environment.md +++ b/docs/content/config/environment.md @@ -456,8 +456,8 @@ Default: 6 (which corresponds to the `add_header` action) ##### RSPAMD_NEURAL -Can be used to enable or disable the [Neural network module][rspamd-docs-neural-network]. This is an experimental anti-spam weigh method using three neuaral networks in the configuration added here. As far as we can tell it trains itsself by using other modules to find out what spam is. It will take a while (a week or more) to train its first neural network. The config trains new networks all the time and discards of old networks. -Since it is experimental it is switched of by default. +Can be used to enable or disable the [Neural network module][rspamd-docs-neural-network]. This is an experimental anti-spam weigh method using three neural networks in the configuration added here. As far as we can tell it trains itself by using other modules to find out what spam is. It will take a while (a week or more) to train its first neural network. The config trains new networks all the time and discards old networks. +Since it is experimental, it is switched off by default. - **0** => Disabled - 1 => Enabled From 942920615c0ab3d9946d6ff1db6a83f9034351cc Mon Sep 17 00:00:00 2001 From: Tobia Bocchi <29007647+tobiabocchi@users.noreply.github.com> Date: Thu, 18 Apr 2024 03:08:26 +0200 Subject: [PATCH 077/267] docs: Fix typo on usage page (#3980) --- docs/content/usage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/usage.md b/docs/content/usage.md index 76011fe7..087547ea 100644 --- a/docs/content/usage.md +++ b/docs/content/usage.md @@ -11,7 +11,7 @@ This page explains how to get started with DMS. The guide uses Docker Compose as Before you can get started with deploying your own mail server, there are some requirements to be met: 1. You need to have a host that you can manage. -2. You need to own a domain, and you need to able to manage DNS for this domain. +2. You need to own a domain, and you need to be able to manage DNS for this domain. ### Host Setup From ac22caf74eff031ad8086ac147949c0b15c24398 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Sat, 20 Apr 2024 11:25:02 +1200 Subject: [PATCH 078/267] docs: Updates to TLS page (Caddy, testing, etc) (#3981) --- docs/content/config/security/ssl.md | 227 +++++++++++++++------------- 1 file changed, 124 insertions(+), 103 deletions(-) diff --git a/docs/content/config/security/ssl.md b/docs/content/config/security/ssl.md index a9174483..a5ba005a 100644 --- a/docs/content/config/security/ssl.md +++ b/docs/content/config/security/ssl.md @@ -481,113 +481,108 @@ DSM-generated letsencrypt certificates get auto-renewed every three months. ### Caddy -For Caddy v2 you can specify the `key_type` in your server's global settings, which would end up looking something like this if you're using a `Caddyfile`: +[Caddy][web::caddy] is an open-source web server with built-in TLS certificate generation. You can use the [official Docker image][dockerhub::caddy] and write your own `Caddyfile`. -```caddyfile -{ - debug - admin localhost:2019 - http_port 80 - https_port 443 - default_sni example.com - key_type rsa2048 -} -``` +!!! example -If you are instead using a json config for Caddy v2, you can set it in your site's TLS automation policies: + ```yaml title="compose.yaml" + services: + # Basic Caddy service to provision certs: + reverse-proxy: + image: caddy:2.7 + ports: + - 80:80 + - 443:443 + volumes: + - ./Caddyfile:/etc/caddy/Caddyfile:ro + - ${CADDY_DATA_DIR}:/data -??? example "Caddy v2 JSON example snippet" + # Share the Caddy data volume for certs and configure SSL_TYPE to `letsencrypt` + mailserver: + image: ghcr.io/docker-mailserver/docker-mailserver:latest + hostname: mail.example.com + environment: + SSL_TYPE: letsencrypt + # While you could use a named data volume instead of a bind mount volume, it would require the long-syntax to rename cert files: + # https://docs.docker.com/compose/compose-file/05-services/#volumes + volumes: + - ${CADDY_DATA_DIR}/certificates/acme-v02.api.letsencrypt.org-directory/mail.example.com/mail.example.com.crt:/etc/letsencrypt/live/mail.example.com/fullchain.pem + - ${CADDY_DATA_DIR}/certificates/acme-v02.api.letsencrypt.org-directory/mail.example.com/mail.example.com.key:/etc/letsencrypt/live/mail.example.com/privkey.pem + ``` - ```json - { - "apps": { - "http": { - "servers": { - "srv0": { - "listen": [ - ":443" - ], - "routes": [ - { - "match": [ - { - "host": [ - "mail.example.com", - ] - } - ], - "handle": [ - { - "handler": "subroute", - "routes": [ - { - "handle": [ - { - "body": "", - "handler": "static_response" - } - ] - } - ] - } - ], - "terminal": true - }, - ] - } - } - }, - "tls": { - "automation": { - "policies": [ - { - "subjects": [ - "mail.example.com", - ], - "key_type": "rsa2048", - "issuer": { - "email": "admin@example.com", - "module": "acme" - } - }, - { - "issuer": { - "email": "admin@example.com", - "module": "acme" - } - } - ] - } - } + ```caddyfile title="Caddyfile" + mail.example.com { + tls internal { + key_type rsa2048 } + + # Optional, can be useful for troubleshooting + # connection to Caddy with correct certificate: + respond "Hello DMS" } ``` -The generated certificates can then be mounted: + While DMS does not need a webserver to work, this workaround will provision a TLS certificate for DMS to use. -```yaml -volumes: - - ${CADDY_DATA_DIR}/certificates/acme-v02.api.letsencrypt.org-directory/mail.example.com/mail.example.com.crt:/etc/letsencrypt/live/mail.example.com/fullchain.pem - - ${CADDY_DATA_DIR}/certificates/acme-v02.api.letsencrypt.org-directory/mail.example.com/mail.example.com.key:/etc/letsencrypt/live/mail.example.com/privkey.pem -``` + - [`tls internal`][caddy-docs::tls-internal] will create a local self-signed cert for testing. This targets only the site-address, unlike the global `local_certs` option. + - [`key_type`][caddy-docs::key-type] can be used in the `tls` block if you need to enforce RSA as the key type for certificates provisioned. The default is currently ECDSA (P-256). -### Traefik v2 +??? example "With `caddy-docker-proxy`" -[Traefik][traefik::github] is an open-source application proxy using the [ACME protocol][ietf::rfc::acme]. [Traefik][traefik::github] can request certificates for domains and subdomains, and it will take care of renewals, challenge negotiations, etc. We strongly recommend to use [Traefik][traefik::github]'s major version 2. + Using [`lucaslorentz/caddy-docker-proxy`][github::caddy-docker-proxy] allows you to generate a `Caddyfile` by adding labels to your services in `compose.yaml`: -[Traefik][traefik::github]'s storage format is natively supported if the `acme.json` store is mounted into the container at `/etc/letsencrypt/acme.json`. The file is also monitored for changes and will trigger a reload of the mail services (Postfix and Dovecot). + ```yaml title="compose.yaml" + services: + reverse-proxy: + image: lucaslorentz/caddy-docker-proxy:2.8 + ports: + - 80:80 + - 443:443 + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - ${CADDY_DATA_DIR}:/data + labels: + # Set global config here, this option has an empty value to enable self-signed certs for local testing: + # NOTE: Remove this label when going to production. + caddy.local_certs: "" -Wildcard certificates are supported. If your FQDN is `mail.example.com` and your wildcard certificate is `*.example.com`, add the ENV: `#!bash SSL_DOMAIN=example.com`. + # Use labels to configure Caddy to provision DMS certs + mailserver: + image: ghcr.io/docker-mailserver/docker-mailserver:latest + hostname: mail.example.com + environment: + SSL_TYPE: letsencrypt + volumes: + - ${CADDY_DATA_DIR}/certificates/acme-v02.api.letsencrypt.org-directory/mail.example.com/mail.example.com.crt:/etc/letsencrypt/live/mail.example.com/fullchain.pem + - ${CADDY_DATA_DIR}/certificates/acme-v02.api.letsencrypt.org-directory/mail.example.com/mail.example.com.key:/etc/letsencrypt/live/mail.example.com/privkey.pem + labels: + # Set your DMS FQDN here to add the site-address into the generated Caddyfile: + caddy_0: mail.example.com + # Add a dummy directive is required: + caddy_0.respond: "Hello DMS" + # Uncomment to make a proxy for Rspamd + # caddy_1: rspamd.example.com + # caddy_1.reverse_proxy: "{{upstreams 11334}}" + ``` -DMS will select it's certificate from `acme.json` checking these ENV for a matching FQDN (_in order of priority_): +!!! warning "Caddy certificate location varies" -1. `#!bash ${SSL_DOMAIN}` -2. `#!bash ${HOSTNAME}` -3. `#!bash ${DOMAINNAME}` + The path contains the certificate provisioner used. This path may be different from the example above for you and may change over time when multiple provisioner services are used][dms-pr-feedback::caddy-provisioning-gotcha]. -This setup only comes with one caveat: The domain has to be configured on another service for [Traefik][traefik::github] to actually request it from _Let's Encrypt_, i.e. [Traefik][traefik::github] will not issue a certificate without a service / router demanding it. + This can make the volume mounting for DMS to find the certificates non-deterministic, but you can [restrict provisioning to single service via the `acme_ca` setting][caddy::restrict-acme-provisioner]. + +### Traefik + +[Traefik][traefik::github] is an open-source application proxy using the [ACME protocol][ietf::rfc::acme]. Traefik can request certificates for domains and subdomains, and it will take care of renewals, challenge negotiations, etc. + +Traefik's storage format is natively supported if the `acme.json` store is mounted into the container at `/etc/letsencrypt/acme.json`. The file is also monitored for changes and will trigger a reload of the mail services (Postfix and Dovecot). + +DMS will select it's certificate from `acme.json` prioritizing a match for the DMS FQDN (hostname), while also checking one DNS level up (_eg: `mail.example.com` => `example.com`_). Wildcard certificates are supported. + +This setup only comes with one caveat - The domain has to be configured on another service for Traefik to actually request it from _Let's Encrypt_ (_i.e. Traefik will not issue a certificate without a service / router demanding it_). ???+ example "Example Code" + Here is an example setup for [`docker-compose`](https://docs.docker.com/compose/): ```yaml @@ -716,7 +711,7 @@ The local and internal paths may be whatever you prefer, so long as both `SSL_CE ## Testing a Certificate is Valid -- From your host: +!!! example "Connect to DMS on port 25" ```sh docker exec mailserver openssl s_client \ @@ -725,26 +720,42 @@ The local and internal paths may be whatever you prefer, so long as both `SSL_CE -CApath /etc/ssl/certs/ ``` -- Or: + The response should show the certificate chain with a line further down: `Verify return code: 0 (ok)` + + --- + + This example runs within the DMS container itself to verify the cert is working locally. + + - Adjust the `-connect` IP if testing externally from another system. Additionally testing for port 143 (Dovecot IMAP) is encouraged (_change the protocol for `-starttls` from `smtp` to `imap`_). + - `-CApath` will help verify the certificate chain, provided the location contains the root CA that signed your TLS cert for DMS. + +??? example "Verify certificate dates" ```sh docker exec mailserver openssl s_client \ - -connect 0.0.0.0:143 \ - -starttls imap \ - -CApath /etc/ssl/certs/ + -connect 0.0.0.0:25 \ + -starttls smtp \ + -CApath /etc/ssl/certs/ \ + 2>/dev/null | openssl x509 -noout -dates ``` -And you should see the certificate chain, the server certificate and: `Verify return code: 0 (ok)` +!!! tip "Testing and troubleshooting" -In addition, to verify certificate dates: + If you need to test a connection without resolving DNS, `curl` can connect with `--resolve` option to map an FQDN + Port to an IP address, instead of the request address provided. -```sh -docker exec mailserver openssl s_client \ - -connect 0.0.0.0:25 \ - -starttls smtp \ - -CApath /etc/ssl/certs/ \ - 2>/dev/null | openssl x509 -noout -dates -``` + ```bash + # NOTE: You may want to use `--insecure` if the cert was provisioned with a private CA not present on the curl client: + # Use `--verbose` for additional insights on the connection. + curl --resolve mail.example.com:443:127.0.0.1 https://mail.example.com + ``` + + Similarly with `openssl` you can connect to an IP as shown previously, but provide an explicit SNI if necessary with `-servername mail.example.com`. + + --- + + Both `curl` and `openssl` also support `-4` and `-6` for enforcing IPv4 or IPv6 lookup. + + This can be useful, such as when [DNS resolves the IP to different servers leading to different certificates returned][dms-discussion::gotcha-fqdn-bad-dns]. As shown in that link, `step certificate inspect` is also handy for viewing details of the cert returned or on disk. ## Plain-Text Access @@ -919,3 +930,13 @@ Despite this, if you must use non-standard DH parameters or you would like to sw [acme-companion::standalone]: https://github.com/nginx-proxy/acme-companion/blob/main/docs/Standalone-certificates.md [acme-companion::standalone-changes]: https://github.com/nginx-proxy/acme-companion/blob/main/docs/Standalone-certificates.md#picking-up-changes-to-letsencrypt_user_data [acme-companion::service-loop]: https://github.com/nginx-proxy/acme-companion/blob/main/docs/Container-utilities.md + +[web::caddy]: https://caddyserver.com +[dockerhub::caddy]: https://hub.docker.com/_/caddy +[github::caddy-docker-proxy]: https://github.com/lucaslorentz/caddy-docker-proxy +[dms-pr-feedback::caddy-provisioning-gotcha]: https://github.com/docker-mailserver/docker-mailserver/pull/3485/files#r1297512818 +[caddy-docs::tls-internal]: https://caddyserver.com/docs/caddyfile/directives/tls#syntax +[caddy-docs::key-type]: https://caddyserver.com/docs/caddyfile/options#key-type +[caddy::restrict-acme-provisioner]: https://caddy.community/t/is-there-a-way-on-a-caddyfile-to-force-a-specific-acme-ca/14506 + +[dms-discussion::gotcha-fqdn-bad-dns]: https://github.com/docker-mailserver/docker-mailserver/issues/3955#issuecomment-2027882633 From d739fe3785ec16dcda1dd2213f64298e2269ce72 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Mon, 22 Apr 2024 11:28:11 +1200 Subject: [PATCH 079/267] chore: Remove base-60 port quote warning from example `compose.yaml` (#3982) This should not be relevant to users of `docker compose` which is the primary demographic. --- compose.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/compose.yaml b/compose.yaml index e9c49596..8f5bfdb2 100644 --- a/compose.yaml +++ b/compose.yaml @@ -7,7 +7,6 @@ services: env_file: mailserver.env # More information about the mail-server ports: # https://docker-mailserver.github.io/docker-mailserver/latest/config/security/understanding-the-ports/ - # To avoid conflicts with yaml base-60 float, DO NOT remove the quotation marks. ports: - "25:25" # SMTP (explicit TLS => STARTTLS, Authentication is DISABLED => use port 465/587 instead) - "143:143" # IMAP4 (explicit TLS => STARTTLS) From df360516ff221e6c5c6380d62000100178038be8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=BCrst?= <7149167+furstblumier@users.noreply.github.com> Date: Mon, 22 Apr 2024 01:50:02 +0200 Subject: [PATCH 080/267] docs: Add config guide for relaying to and from a private DMS instance (#3973) Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Co-authored-by: Casper --- CHANGELOG.md | 2 + .../external-relay-only-mailserver.md | 233 ++++++++++++++++++ docs/mkdocs.yml | 1 + 3 files changed, 236 insertions(+) create mode 100644 docs/content/examples/use-cases/external-relay-only-mailserver.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 89efb030..10d30016 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -70,6 +70,8 @@ The most noteworthy change of this release is the update of the container's base ### Added +- **Docs:** + - A guide for configuring a public server to relay inbound and outbound mail from DMS on a private server ([#3973](https://github.com/docker-mailserver/docker-mailserver/pull/3973)) - **Environment Variables:** - `LOGROTATE_COUNT` defines the number of files kept by logrotate ([#3907](https://github.com/docker-mailserver/docker-mailserver/pull/3907)) - The fail2ban log file is now also taken into account by `LOGROTATE_COUNT` and `LOGROTATE_INTERVAL` ([#3915](https://github.com/docker-mailserver/docker-mailserver/pull/3915), [#3919](https://github.com/docker-mailserver/docker-mailserver/pull/3919)) diff --git a/docs/content/examples/use-cases/external-relay-only-mailserver.md b/docs/content/examples/use-cases/external-relay-only-mailserver.md new file mode 100644 index 00000000..44e83c91 --- /dev/null +++ b/docs/content/examples/use-cases/external-relay-only-mailserver.md @@ -0,0 +1,233 @@ +--- +title: 'Use Cases | Relay inbound and outbound mail for an internal DMS' +hide: + - toc +--- + +## Introduction + +!!! info "Community contributed guide" + + Adapted into a guide from [this discussion](https://github.com/orgs/docker-mailserver/discussions/3965). + + **Requirements:** + + - A _public server_ with a static IP, like many VPS providers offer. It will only relay mail to DMS, no mail is stored on this system. + - A _private server_ (e.g.: a local system at home) that will run DMS. + - Both servers are connected to the same network via a VPN (_optional convenience for trust via the `mynetworks` setting_). + + --- + + The guide below will assume the VPN is setup on `192.168.2.0/24` with: + + - The _public server_ is using `192.168.2.2` + - The _private server_ is using `192.168.2.3` + +The goal of this guide is to configure a _public server_ that can receive inbound mail and relay that over to DMS on a _private server_, which can likewise submit mail outbound through a _public server_ or service. + +The primary motivation is to keep your mail storage private instead of storing it to disk unencrypted on a VPS host. + +## DNS setup + +Follow our [standard guidance][docs::usage-dns-setup] for DNS setup. + +Set your A, MX and PTR records for the _public server_ as if it were running DMS. + +!!! example "DNS Zone file example" + + For this guide, we assume DNS is configured with: + + - A public reachable IP address of `11.22.33.44` + - Mail for `@example.com` addresses must have an MX record pointing to `mail.example.com`. + - An A record for `mail.example.com` pointing to the IP address of your _public server_. + + ```txt + $ORIGIN example.com + @ IN A 11.22.33.44 + mail IN A 11.22.33.44 + + ; mail server for example.com + @ IN MX 10 mail.example.com. + ``` + + SPF records should also be set up as you normally would for `mail.example.com`. + +## Public Server (Basic Postfix setup) + +You will need to install Postfix on your _public server_. The functionality that is needed for this setup is not yet implemented in DMS, so a vanilla Postfix will probably be easier to work with, especially since this server will only be used as an inbound and outbound relay. + +It's necessary to adjust some settings afterwards. + + +!!! quote "" + + === "Postfix main config" + + ??? example "Create or replace `/etc/postfix/main.cf`" + + ```cf + # See /usr/share/postfix/main.cf.dist for a commented, more complete version + + smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU) + biff = no + + # appending .domain is the MUA's job. + append_dot_mydomain = no + + # Uncomment the next line to generate "delayed mail" warnings + #delay_warning_time = 4h + + # See http://www.postfix.org/COMPATIBILITY_README.html -- default to 3.6 on + # fresh installs. + compatibility_level = 3.6 + + # TLS parameters + smtpd_tls_cert_file=/etc/postfix/certificates/mail.example.com.crt + smtpd_tls_key_file=/etc/postfix/certificates/mail.example.com.key + smtpd_tls_security_level=may + smtp_tls_CApath=/etc/ssl/certs + smtp_tls_security_level=may + smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache + + alias_database = hash:/etc/aliases + alias_maps = hash:/etc/aliases + maillog_file = /var/log/postfix.log + mailbox_size_limit = 0 + inet_interfaces = all + inet_protocols = ipv4 + readme_directory = no + recipient_delimiter = + + + # Customizations relevant to this guide: + myhostname = mail.example.com + myorigin = example.com + mydestination = localhost + mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 192.168.2.0/24 + smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination + transport_maps = hash:/etc/postfix/transport + relay_domains = $mydestination, hash:/etc/postfix/relay + + # Disable local system accounts and delivery: + local_recipient_maps = + local_transport = error:local mail delivery is disabled + ``` + + Let's highlight some of the important parts: + + - Avoid including `mail.example.com` in `mydestination`, in fact you can just set `localhost` or nothing at all here as we want all mail to be relayed to our _private server_ (DMS). + - `mynetworks` should contain your VPN network (_eg: `192.168.2.0/24` subnet_). + - Important are `transport_maps = hash:/etc/postfix/transport` and `relay_domains = $mydestination, hash:/etc/postfix/relay`, with their file contents covered below. + - For good measure, also disable `local_recipient_maps`. + - You should have a valid certificate configured for `mail.example.com`. + + !!! warning "Open relay" + + Please be aware that setting `mynetworks` to a public CIDR will leave you with an open relay. **Only** set it to the CIDR of your VPN beyond the localhost ranges. + + === "Route outbound mail through a separate transport" + + When mail arrives to the _public server_ for an `@example.com` address, we want to send it via the `relay` transport to our _private server_ over port 25 for delivery to DMS. + + [`transport_maps`][postfix-docs::transport_maps] is configured with a [`transport` table][postfix-docs::transport_table] file that matches recipient addresses and assigns a non-default transport. This setting has priority over [`relay_transport`][postfix-docs::relay_transport]. + + !!! example "Create `/etc/postfix/transport`" + + ```txt + example.com relay:[192.168.2.3]:25 + ``` + + **Other considerations:** + + - If you have multiple domains, you can add them here too (on separate lines). + - If you use a smarthost add `* relay:[X.X.X.X]:port` to the bottom (eg: `* relay:[relay1.org]:587`), which will relay everything outbound via this relay host. + + !!! tip + + Instead of a file, you could alternatively configure `main.cf` with `transport_maps = inline:{ example.com=relay:[192.168.2.3]:25 }` + + === "Configure recipient domains to relay mail" + + We want `example.com` to be relayed inbound and everything else relayed outbound. + + [`relay_domains`][postfix-docs::relay_domains] is configured with a file with a list of domains that should be relayed (one per line), the 2nd value is required but can be anything. + + !!! example "Create `/etc/postfix/relay`" + + ```txt + example.com OK + ``` + + !!! tip + + Instead of a file, you could alternatively configure `main.cf` with `relay_domains = example.com`. + +!!! note "Files configured with `hash:` table type must run `postmap` to apply changes" + + Run `postmap /etc/postfix/transport` and `postmap /etc/postfix/relay` after creating or updating either of these files, this processes them into a separate file for Postfix to use. + +## Private Server (Running DMS) + +You can set up your DMS instance as you normally would. + +- Be careful not to give it a hostname of `mail.example.com`. Instead, use `internal-mail.example.com` or something similar. +- DKIM can be setup as usual since it considers checks whether the message body has been tampered with, which our public relay doesn't do. Set DKIM up for `mail.example.com`. + +Next, we need to configure our _private server_ to relay all outbound mail through the _public server_ (or a separate smarthost service). The setup is [similar to the default relay setup][docs::relay-host-details]. + + +!!! quote "" + + === "Configure the relay host" + + !!! example "Create `postfix-relaymap.cf`" + + ```txt + @example.com [192.168.2.2]:25 + ``` + + Meaning all mail sent outbound from `@example.com` addresses will be relayed through the _public server_ at that VPN IP. + + The _public server_ `mynetworks` setting from earlier trusts any mail received on port 25 from the VPN network, which is what allows the mail to be sent outbound when it'd otherwise be denied. + + === "Trust the _public server_" + + !!! example "Create `postfix-main.cf`" + + ```txt + mynetworks = 192.168.2.0/24 + ``` + + This will trust any connection from the VPN network to DMS, such as from the _public server_ when relaying mail over to DMS at the _private server_. + + This step is necessary to skip some security measures that DMS normally checks for, like verifying DNS records like SPF are valid. As the mail is being relayed, those checks would fail otherwise as the IP of your _public server_ would not be authorized to send mail on behalf of the sender address in mail being relayed. + + ??? tip "Alternative to `mynetworks` setting" + + Instead of trusting connections by their IP with the `mynetworks` setting, those same security measures can be skipped for any authenticated deliveries to DMS over port 587 instead. + + This is a bit more work. `mynetworks` on the _public server_ `main.cf` Postfix config is for trusting DMS when it sends mail from the _private server_, thus you'll need to have that public Postfix service configured with a login account that DMS can use. + + On the _private server_, DMS needs to know the credentials for that login account, that is handled with `postfix-sasl-password.cf`: + + ```txt + @example.com user:secret + ``` + + You could also relay mail through SendGrid, AWS SES or similar instead of the _public server_ you're running to receive mail from. Login credentials for those relay services are provided via the same `postfix-sasl-password.cf` file. + + --- + + Likewise for the _public server_ to send mail to DMS, it would need to be configured to relay mail with credentials too, removing the need for `mynetworks` on the DMS `postfix-main.cf` config. + + The extra effort to require authentication instead of blind trust of your private subnet can be beneficial at reducing the impact of a compromised system or service on that network that wasn't expected to be permitted to send mail. + +## IMAP / POP3 + +IMAP and POP3 need to point towards your _private server_, since that is where the mailboxes are located, which means you need to have a way for your MUA to connect to it. + +[docs::usage-dns-setup]: ../../usage.md#minimal-dns-setup +[docs::relay-host-details]: ../../config/advanced/mail-forwarding/relay-hosts.md#technical-details +[postfix-docs::relay_domains]: https://www.postfix.org/postconf.5.html#relay_domains +[postfix-docs::relay_transport]: https://www.postfix.org/postconf.5.html#relay_transport +[postfix-docs::transport_maps]: https://www.postfix.org/postconf.5.html#transport_maps +[postfix-docs::transport_table]: https://www.postfix.org/transport.5.html diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 9d7fc81d..59263443 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -180,6 +180,7 @@ nav: - 'iOS Mail Push Support': examples/use-cases/ios-mail-push-support.md - 'Lua Authentication': examples/use-cases/auth-lua.md - 'Bind outbound SMTP to a specific network': examples/use-cases/bind-smtp-network-interface.md + - 'Relay inbound and outbound mail for an internal DMS': examples/use-cases/external-relay-only-mailserver.md - 'FAQ' : faq.md - 'Contributing': - 'General Information': contributing/general.md From 1051a5d921a07bae4cfe2f43e981d058baa05fb4 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 22 Apr 2024 15:46:34 +0200 Subject: [PATCH 081/267] chore(deps): Bump akhilmhdh/contributors-readme-action (#3987) Bumps [akhilmhdh/contributors-readme-action](https://github.com/akhilmhdh/contributors-readme-action) from 2.3.6 to 2.3.8. - [Release notes](https://github.com/akhilmhdh/contributors-readme-action/releases) - [Commits](https://github.com/akhilmhdh/contributors-readme-action/compare/v2.3.6...v2.3.8) --- updated-dependencies: - dependency-name: akhilmhdh/contributors-readme-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/contributors.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/contributors.yml b/.github/workflows/contributors.yml index 6f8476f2..9a71b648 100644 --- a/.github/workflows/contributors.yml +++ b/.github/workflows/contributors.yml @@ -19,7 +19,7 @@ jobs: uses: actions/checkout@v4 - name: 'Update CONTRIBUTORS.md' - uses: akhilmhdh/contributors-readme-action@v2.3.6 + uses: akhilmhdh/contributors-readme-action@v2.3.8 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} with: From 162e66276a4d4c836d6eb4434fdfb3cfb35e2fad Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 25 Apr 2024 08:38:38 +0200 Subject: [PATCH 082/267] docs: updated `CONTRIBUTORS.md` (#3984) --- CONTRIBUTORS.md | 287 ++++++++++++++++++++++++++---------------------- 1 file changed, 154 insertions(+), 133 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 01c7897f..e1027231 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -787,10 +787,17 @@ Thanks goes to these wonderful people ✨ - - jamesfryer + + fl42
- jamesfryer + fl42 + + + + + ipernet +
+ ipernet @@ -808,34 +815,13 @@ Thanks goes to these wonderful people ✨ - - ipernet + + jamesfryer
- ipernet - - - - - fl42 -
- fl42 + jamesfryer - - - neuralp -
- neuralp - - - - - sjmudd -
- sjmudd - - simonsystem @@ -863,8 +849,7 @@ Thanks goes to these wonderful people ✨
5ven - - + syl20bnr @@ -878,7 +863,8 @@ Thanks goes to these wonderful people ✨
sylvaindumont - + + TechnicLab @@ -886,28 +872,13 @@ Thanks goes to these wonderful people ✨ TechnicLab - - - 42wim -
- 42wim - - - - - vilisas -
- vilisas - - thomasschmit
thomasschmit - - + Thiritin @@ -915,6 +886,13 @@ Thanks goes to these wonderful people ✨ Thiritin + + + tobiabocchi +
+ tobiabocchi + + tweibert @@ -928,7 +906,8 @@ Thanks goes to these wonderful people ✨
torus - + + VictorKoenders @@ -949,8 +928,7 @@ Thanks goes to these wonderful people ✨
k3it - - + Drakulix @@ -958,6 +936,28 @@ Thanks goes to these wonderful people ✨ Drakulix + + + vilisas +
+ vilisas + + + + + 42wim +
+ 42wim + + + + + + ShiriNmi1520 +
+ ShiriNmi1520 + + radicand @@ -1015,13 +1015,6 @@ Thanks goes to these wonderful people ✨ p3dda - - - auchri -
- auchri - - peter-hartmann @@ -1029,6 +1022,13 @@ Thanks goes to these wonderful people ✨ peter-hartmann + + + neuralp +
+ neuralp + + piwai @@ -1109,20 +1109,27 @@ Thanks goes to these wonderful people ✨ - - ShiriNmi1520 + + sjmudd
- ShiriNmi1520 + sjmudd + + + Zepmann +
+ Zepmann + + + mchamplain
mchamplain - - + millerjason @@ -1157,15 +1164,15 @@ Thanks goes to these wonderful people ✨
olaf-mandel - + + ontheair81
ontheair81 - - + pravynandas @@ -1200,15 +1207,15 @@ Thanks goes to these wonderful people ✨
rriski - + + schnippl0r
schnippl0r - - + smargold476 @@ -1243,15 +1250,15 @@ Thanks goes to these wonderful people ✨
tamueller - + + vivacarvajalito
vivacarvajalito - - + wligtenberg @@ -1273,13 +1280,6 @@ Thanks goes to these wonderful people ✨ worldworm - - - Zepmann -
- Zepmann - - allddd @@ -1338,6 +1338,13 @@ Thanks goes to these wonderful people ✨ + + + fanqiaojun +
+ fanqiaojun + + ghnp5 @@ -1372,15 +1379,15 @@ Thanks goes to these wonderful people ✨
idaadi - + + ixeft
ixeft - - + jjtt @@ -1415,15 +1422,15 @@ Thanks goes to these wonderful people ✨
callmemagnus - + + marios88
marios88 - - + matrixes @@ -1431,6 +1438,13 @@ Thanks goes to these wonderful people ✨ matrixes + + + auchri +
+ auchri + + arkanovicz @@ -1451,7 +1465,8 @@ Thanks goes to these wonderful people ✨
damianmoore - + + espitall @@ -1465,8 +1480,7 @@ Thanks goes to these wonderful people ✨
dkarski - - + dbellavista @@ -1494,7 +1508,8 @@ Thanks goes to these wonderful people ✨
mlatorre31 - + + mazzz1y @@ -1508,8 +1523,7 @@ Thanks goes to these wonderful people ✨
doominator42 - - + aydodo @@ -1537,7 +1551,8 @@ Thanks goes to these wonderful people ✨
eliroca - + + ekkis @@ -1551,8 +1566,7 @@ Thanks goes to these wonderful people ✨
ErikEngerd - - + huncode @@ -1575,12 +1589,13 @@ Thanks goes to these wonderful people ✨ - - Kaan88 + + froks
- Kaan88 + froks - + + 0xflotus @@ -1594,8 +1609,7 @@ Thanks goes to these wonderful people ✨
ifokeev - - + 20th @@ -1623,7 +1637,8 @@ Thanks goes to these wonderful people ✨
aspettl - + + acch @@ -1637,8 +1652,7 @@ Thanks goes to these wonderful people ✨
vifino - - + kachkaev @@ -1666,7 +1680,8 @@ Thanks goes to these wonderful people ✨
eglia - + + groupmsl @@ -1680,8 +1695,7 @@ Thanks goes to these wonderful people ✨
green-anger - - + iRhonin @@ -1709,7 +1723,8 @@ Thanks goes to these wonderful people ✨
astrocket - + + baxerus @@ -1723,8 +1738,7 @@ Thanks goes to these wonderful people ✨
spock - - + erdos4d @@ -1739,13 +1753,21 @@ Thanks goes to these wonderful people ✨ crash7 + + + Kaan88 +
+ Kaan88 + + akkumar
akkumar - + + thechubbypanda @@ -1766,8 +1788,7 @@ Thanks goes to these wonderful people ✨
khuedoan - - + UltraCoderRU @@ -1788,7 +1809,8 @@ Thanks goes to these wonderful people ✨
leowinterde - + + linhandev @@ -1809,8 +1831,7 @@ Thanks goes to these wonderful people ✨
LucidityCrash - - + MadsRC @@ -1831,7 +1852,8 @@ Thanks goes to these wonderful people ✨
maxemann96 - + + dragetd @@ -1852,8 +1874,7 @@ Thanks goes to these wonderful people ✨
exhuma - - + milas @@ -1874,7 +1895,8 @@ Thanks goes to these wonderful people ✨
MohammedNoureldin - + + mpldr @@ -1889,14 +1911,6 @@ Thanks goes to these wonderful people ✨ naveensrinivasan - - - froks -
- froks - - - fkefer @@ -1924,7 +1938,8 @@ Thanks goes to these wonderful people ✨
GiovanH - + + harryyoud @@ -1938,8 +1953,7 @@ Thanks goes to these wonderful people ✨
HeySora - - + sirgantrithon @@ -1961,6 +1975,14 @@ Thanks goes to these wonderful people ✨ in-seo + + + firefly-cpp +
+ firefly-cpp + + + JacksonZ03 @@ -1981,8 +2003,7 @@ Thanks goes to these wonderful people ✨
jcalfee - - + mivek @@ -2003,7 +2024,8 @@ Thanks goes to these wonderful people ✨
Jeidnx - + + jessp01 @@ -2024,8 +2046,7 @@ Thanks goes to these wonderful people ✨
jirislav - - + jmccl From be8615f129b4616ea1d4f318da878e7fc01673e2 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sun, 28 Apr 2024 19:06:50 +0200 Subject: [PATCH 083/267] docs: updated `CONTRIBUTORS.md` (#3992) --- CONTRIBUTORS.md | 268 +++++++++++++++++++++++++----------------------- 1 file changed, 138 insertions(+), 130 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index e1027231..a96b2659 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -665,17 +665,17 @@ Thanks goes to these wonderful people ✨ - - ubenmackin + + andrewlow
- ubenmackin + andrewlow - - craue + + aminvakil
- craue + aminvakil @@ -685,27 +685,27 @@ Thanks goes to these wonderful people ✨ abh - - - andrewlow -
- andrewlow - - - - - - aminvakil -
- aminvakil - - elbracht
elbracht + + + + + craue +
+ craue + + + + + ubenmackin +
+ ubenmackin + @@ -786,27 +786,6 @@ Thanks goes to these wonderful people ✨ jedateach - - - fl42 -
- fl42 - - - - - ipernet -
- ipernet - - - - - H4R0 -
- H4R0 - - millaguie @@ -820,8 +799,36 @@ Thanks goes to these wonderful people ✨
jamesfryer + + + + H4R0 +
+ H4R0 + + + + + ipernet +
+ ipernet + + + + + fl42 +
+ fl42 + + + + 0xflotus +
+ 0xflotus + + simonsystem @@ -856,15 +863,15 @@ Thanks goes to these wonderful people ✨
syl20bnr - + + sylvaindumont
sylvaindumont - - + TechnicLab @@ -899,15 +906,15 @@ Thanks goes to these wonderful people ✨
tweibert - + + torus
torus - - + VictorKoenders @@ -942,20 +949,27 @@ Thanks goes to these wonderful people ✨
vilisas - + + 42wim
42wim - - + - - ShiriNmi1520 + + matrixes
- ShiriNmi1520 + matrixes + + + + + neuralp +
+ neuralp @@ -978,7 +992,8 @@ Thanks goes to these wonderful people ✨
nknapp - + + pcqnt @@ -992,8 +1007,7 @@ Thanks goes to these wonderful people ✨
OrvilleQ - - + ovidiucp @@ -1021,22 +1035,15 @@ Thanks goes to these wonderful people ✨
peter-hartmann - - - - neuralp -
- neuralp - - + + piwai
piwai - - + remoe @@ -1071,15 +1078,15 @@ Thanks goes to these wonderful people ✨
MightySCollins - + + 501st-alpha1
501st-alpha1 - - + klamann @@ -1114,13 +1121,6 @@ Thanks goes to these wonderful people ✨
sjmudd - - - - Zepmann -
- Zepmann - @@ -1280,6 +1280,21 @@ Thanks goes to these wonderful people ✨ worldworm + + + ShiriNmi1520 +
+ ShiriNmi1520 + + + + + Zepmann +
+ Zepmann + + + allddd @@ -1293,8 +1308,7 @@ Thanks goes to these wonderful people ✨
arcaine2 - - + awb99 @@ -1322,7 +1336,8 @@ Thanks goes to these wonderful people ✨
dborowy - + + dimalo @@ -1336,8 +1351,7 @@ Thanks goes to these wonderful people ✨
eleith - - + fanqiaojun @@ -1365,7 +1379,8 @@ Thanks goes to these wonderful people ✨
hnws - + + i-C-o-d-e-r @@ -1379,8 +1394,7 @@ Thanks goes to these wonderful people ✨
idaadi - - + ixeft @@ -1408,7 +1422,8 @@ Thanks goes to these wonderful people ✨
jpduyx - + + landergate @@ -1422,8 +1437,7 @@ Thanks goes to these wonderful people ✨
callmemagnus - - + marios88 @@ -1431,20 +1445,6 @@ Thanks goes to these wonderful people ✨ marios88 - - - matrixes -
- matrixes - - - - - auchri -
- auchri - - arkanovicz @@ -1597,10 +1597,10 @@ Thanks goes to these wonderful people ✨ - - 0xflotus + + fkefer
- 0xflotus + fkefer @@ -1753,21 +1753,28 @@ Thanks goes to these wonderful people ✨ crash7 + + + auchri +
+ auchri + + Kaan88
Kaan88 - + + akkumar
akkumar - - + thechubbypanda @@ -1802,15 +1809,15 @@ Thanks goes to these wonderful people ✨
JustAnother1 - + + leowinterde
leowinterde - - + linhandev @@ -1845,15 +1852,15 @@ Thanks goes to these wonderful people ✨
madmath03 - + + maxemann96
maxemann96 - - + dragetd @@ -1888,15 +1895,15 @@ Thanks goes to these wonderful people ✨
mcchots - + + MohammedNoureldin
MohammedNoureldin - - + mpldr @@ -1912,10 +1919,10 @@ Thanks goes to these wonderful people ✨ - - fkefer + + furstblumier
- fkefer + furstblumier @@ -1931,15 +1938,15 @@ Thanks goes to these wonderful people ✨
glandais - + + GiovanH
GiovanH - - + harryyoud @@ -1974,15 +1981,15 @@ Thanks goes to these wonderful people ✨
in-seo - + + firefly-cpp
firefly-cpp - - + JacksonZ03 @@ -2017,15 +2024,15 @@ Thanks goes to these wonderful people ✨
init-js - + + Jeidnx
Jeidnx - - + jessp01 @@ -2060,7 +2067,8 @@ Thanks goes to these wonderful people ✨
jurekbarth - + + JOduMonT From 83da191f3a4cbe1d8ee458860de6e93ea152b08f Mon Sep 17 00:00:00 2001 From: Wael <8544289+forzagreen@users.noreply.github.com> Date: Thu, 2 May 2024 02:08:29 +0200 Subject: [PATCH 084/267] docs: Fix link for `getmail6` (#3996) --- docs/content/config/advanced/mail-getmail.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/config/advanced/mail-getmail.md b/docs/content/config/advanced/mail-getmail.md index d08f5d07..f62388b0 100644 --- a/docs/content/config/advanced/mail-getmail.md +++ b/docs/content/config/advanced/mail-getmail.md @@ -95,7 +95,7 @@ environment: It is possible to utilize the `getmail-gmail-xoauth-tokens` helper to provide authentication using `xoauth2` for [gmail (example 12)][getmail-docs-xoauth-12] or [Microsoft Office 365 (example 13)][getmail-docs-xoauth-13] -[getmail-website]: https://www.getmail.org +[getmail-website]: https://www.getmail6.org [getmail-docs]: https://getmail6.org/configuration.html [getmail-docs-xoauth-12]: https://github.com/getmail6/getmail6/blob/1f95606156231f1e074ba62a9baa64f892a92ef8/docs/getmailrc-examples#L286 [getmail-docs-xoauth-13]: https://github.com/getmail6/getmail6/blob/1f95606156231f1e074ba62a9baa64f892a92ef8/docs/getmailrc-examples#L351 From 7dcbbd7173a8fd1713a77d55ed224d14c8970aa5 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Thu, 2 May 2024 19:41:25 +1200 Subject: [PATCH 085/267] fix(`accounts.sh`): Sync user home location for alias workaround (#3997) --- CHANGELOG.md | 1 + target/scripts/helpers/accounts.sh | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 10d30016..20ce0ed4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -105,6 +105,7 @@ The most noteworthy change of this release is the update of the container's base - Rspamd configuration: Add a missing comma in `local_networks` so that all internal IP addresses are actually considered as internal ([#3862](https://github.com/docker-mailserver/docker-mailserver/pull/3862)) - Ensure correct SELinux security context labels for files and directories moved to the mail-state volume during setup ([#3890](https://github.com/docker-mailserver/docker-mailserver/pull/3890)) - Use correct environment variable for fetchmail ([#3901](https://github.com/docker-mailserver/docker-mailserver/pull/3901)) +- Dovecot dummy accounts (_virtual alias workaround for dovecot feature `ENABLE_QUOTAS=1`_) now correctly matches the home location of the user for that alias ([#3997](https://github.com/docker-mailserver/docker-mailserver/pull/3997)) ## [v13.3.1](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v13.3.1) diff --git a/target/scripts/helpers/accounts.sh b/target/scripts/helpers/accounts.sh index 31ded04a..78464b88 100644 --- a/target/scripts/helpers/accounts.sh +++ b/target/scripts/helpers/accounts.sh @@ -134,7 +134,7 @@ function _create_dovecot_alias_dummy_accounts() { fi fi - DOVECOT_USERDB_LINE="${ALIAS}:${REAL_ACC[1]}:${DMS_VMAIL_UID}:${DMS_VMAIL_GID}::/var/mail/${REAL_DOMAINNAME}/${REAL_USERNAME}::${REAL_ACC[2]:-}" + DOVECOT_USERDB_LINE="${ALIAS}:${REAL_ACC[1]}:${DMS_VMAIL_UID}:${DMS_VMAIL_GID}::/var/mail/${REAL_DOMAINNAME}/${REAL_USERNAME}/home::${REAL_ACC[2]:-}" if grep -qi "^${ALIAS}:" "${DOVECOT_USERDB_FILE}"; then _log 'warn' "Alias '${ALIAS}' will not be added to '${DOVECOT_USERDB_FILE}' twice" else From d00edd7209d80f8f9b45bf9274fcb6cf858e6867 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Thu, 2 May 2024 19:44:54 +1200 Subject: [PATCH 086/267] docs: Revise fetchmail page (#3998) --- .../content/config/advanced/mail-fetchmail.md | 191 ++++++++++-------- 1 file changed, 109 insertions(+), 82 deletions(-) diff --git a/docs/content/config/advanced/mail-fetchmail.md b/docs/content/config/advanced/mail-fetchmail.md index 254196a1..936d5188 100644 --- a/docs/content/config/advanced/mail-fetchmail.md +++ b/docs/content/config/advanced/mail-fetchmail.md @@ -2,7 +2,7 @@ title: 'Advanced | Email Gathering with Fetchmail' --- -To enable the [fetchmail][fetchmail-website] service to retrieve e-mails set the environment variable `ENABLE_FETCHMAIL` to `1`. Your `compose.yaml` file should look like following snippet: +To enable the [fetchmail][fetchmail-website] service to retrieve e-mails, set the environment variable `ENABLE_FETCHMAIL` to `1`. Your `compose.yaml` file should look like following snippet: ```yaml environment: @@ -18,108 +18,135 @@ Generate a file called `fetchmail.cf` and place it in the `docker-data/dms/confi │   ├── fetchmail.cf │   ├── postfix-accounts.cf │   └── postfix-virtual.cf -├── compose.yaml -└── README.md +└── compose.yaml ``` ## Configuration -A detailed description of the configuration options can be found in the [online version of the manual page][fetchmail-docs]. +Configuration options for `fetchmail.cf` are covered at the [official fetchmail docs][fetchmail-docs-config] (_see the section "The run control file" and the table with "keyword" column for all settings_). -### IMAP Configuration +!!! example "Basic `fetchmail.cf` configuration" -!!! example + Retrieve mail from `remote-user@somewhere.com` and deliver it to `dms-user@example.com`: ```fetchmailrc - poll 'imap.gmail.com' proto imap - user 'username' - pass 'secret' - is 'user1@example.com' - ssl + poll 'mail.somewhere.com' + proto imap + user 'remote-user' + pass 'secret' + is 'dms-user@example.com' ``` -### POP3 Configuration + - `poll` sets the remote mail server to connect to retrieve mail from. + - `proto` lets you connect via IMAP or POP3. + - `user` and `pass` provide the login credentials for the remote mail service account to access. + - `is` configures where the fetched mail will be sent to (_eg: your local DMS account in `docker-data/dms/config/postfix-accounts.cf`_). -!!! example + --- - ```fetchmailrc - poll 'pop3.gmail.com' proto pop3 - user 'username' - pass 'secret' - is 'user2@example.com' - ssl + ??? warning "`proto imap` will still delete remote mail once fetched" + + This is due to a separate default setting `no keep`. Adding the setting `keep` to your config on a new line will prevent deleting the remote copy. + +??? example "Multiple users or remote servers" + + The official docs [config examples][fetchmail-config-examples] show a common convention to indent settings on subsequent lines for visually grouping per server. + + === "Minimal syntax" + + ```fetchmailrc + poll 'mail.somewhere.com' proto imap + user 'john.doe' pass 'secret' is 'johnny@example.com' + user 'jane.doe' pass 'secret' is 'jane@example.com' + + poll 'mail.somewhere-else.com' proto pop3 + user 'john.doe@somewhere-else.com' pass 'secret' is 'johnny@example.com' + ``` + + === "With optional syntax" + + - `#` for adding comments. + - The config file may include "noise" keywords to improve readability. + + ```fetchmailrc + # Retrieve mail for users `john.doe` and `jane.doe` via IMAP at this remote mail server: + poll 'mail.somewhere.com' with proto imap wants: + user 'john.doe' with pass 'secret', is 'johnny@example.com' here + user 'jane.doe' with pass 'secret', is 'jane@example.com' here + + # Also retrieve mail from this mail server (but via POP3). + # NOTE: This could also be all on a single line, or with each key + value as a separate line. + # Notice how the remote username includes a full email address, + # Some mail servers like DMS use the full email address as the username: + poll 'mail.somewhere-else.com' with proto pop3 wants: + user 'john.doe@somewhere-else.com' with pass 'secret', is 'johnny@example.com' here + ``` + +!!! tip "`FETCHMAIL_POLL` ENV: Override default polling interval" + + By default the fetchmail service will check every 5 minutes for new mail at the configured mail accounts. + + ```yaml + environment: + # The fetchmail polling interval in seconds: + FETCHMAIL_POLL: 60 ``` -!!! caution - - Don’t forget the last line! (_eg: `is 'user1@example.com'`_). After `is`, you have to specify an email address from the configuration file: `docker-data/dms/config/postfix-accounts.cf`. - -More details how to configure fetchmail can be found in the [fetchmail man page in the chapter “The run control file”][fetchmail-docs-run]. - -### Polling Interval - -By default the fetchmail service searches every 5 minutes for new mails on your external mail accounts. You can override this default value by changing the ENV variable `FETCHMAIL_POLL`: - -```yaml -environment: - - FETCHMAIL_POLL=60 -``` - -You must specify a numeric argument which is a polling interval in seconds. The example above polls every minute for new mails. - ## Debugging -To debug your `fetchmail.cf` configuration run this command: +To debug your `fetchmail.cf` configuration run this `setup debug` command: ```sh -./setup.sh debug fetchmail +docker exec -it dms-container-name setup debug fetchmail ``` -For more information about the configuration script `setup.sh` [read the corresponding docs][docs-setup]. +??? example "Sample output of `setup debug fetchmail`" -Here a sample output of `./setup.sh debug fetchmail`: + ```log + fetchmail: 6.3.26 querying outlook.office365.com (protocol POP3) at Mon Aug 29 22:11:09 2016: poll started + Trying to connect to 132.245.48.18/995...connected. + fetchmail: Server certificate: + fetchmail: Issuer Organization: Microsoft Corporation + fetchmail: Issuer CommonName: Microsoft IT SSL SHA2 + fetchmail: Subject CommonName: outlook.com + fetchmail: Subject Alternative Name: outlook.com + fetchmail: Subject Alternative Name: *.outlook.com + fetchmail: Subject Alternative Name: office365.com + fetchmail: Subject Alternative Name: *.office365.com + fetchmail: Subject Alternative Name: *.live.com + fetchmail: Subject Alternative Name: *.internal.outlook.com + fetchmail: Subject Alternative Name: *.outlook.office365.com + fetchmail: Subject Alternative Name: outlook.office.com + fetchmail: Subject Alternative Name: attachment.outlook.office.net + fetchmail: Subject Alternative Name: attachment.outlook.officeppe.net + fetchmail: Subject Alternative Name: *.office.com + fetchmail: outlook.office365.com key fingerprint: 3A:A4:58:42:56:CD:BD:11:19:5B:CF:1E:85:16:8E:4D + fetchmail: POP3< +OK The Microsoft Exchange POP3 service is ready. [SABFADEAUABSADAAMQBDAEEAMAAwADAANwAuAGUAdQByAHAAcgBkADAAMQAuAHAAcgBvAGQALgBlAHgAYwBoAGEAbgBnAGUAbABhAGIAcwAuAGMAbwBtAA==] + fetchmail: POP3> CAPA + fetchmail: POP3< +OK + fetchmail: POP3< TOP + fetchmail: POP3< UIDL + fetchmail: POP3< SASL PLAIN + fetchmail: POP3< USER + fetchmail: POP3< . + fetchmail: POP3> USER user1@outlook.com + fetchmail: POP3< +OK + fetchmail: POP3> PASS * + fetchmail: POP3< +OK User successfully logged on. + fetchmail: POP3> STAT + fetchmail: POP3< +OK 0 0 + fetchmail: No mail for user1@outlook.com at outlook.office365.com + fetchmail: POP3> QUIT + fetchmail: POP3< +OK Microsoft Exchange Server 2016 POP3 server signing off. + fetchmail: 6.3.26 querying outlook.office365.com (protocol POP3) at Mon Aug 29 22:11:11 2016: poll completed + fetchmail: normal termination, status 1 + ``` -```log -fetchmail: 6.3.26 querying outlook.office365.com (protocol POP3) at Mon Aug 29 22:11:09 2016: poll started -Trying to connect to 132.245.48.18/995...connected. -fetchmail: Server certificate: -fetchmail: Issuer Organization: Microsoft Corporation -fetchmail: Issuer CommonName: Microsoft IT SSL SHA2 -fetchmail: Subject CommonName: outlook.com -fetchmail: Subject Alternative Name: outlook.com -fetchmail: Subject Alternative Name: *.outlook.com -fetchmail: Subject Alternative Name: office365.com -fetchmail: Subject Alternative Name: *.office365.com -fetchmail: Subject Alternative Name: *.live.com -fetchmail: Subject Alternative Name: *.internal.outlook.com -fetchmail: Subject Alternative Name: *.outlook.office365.com -fetchmail: Subject Alternative Name: outlook.office.com -fetchmail: Subject Alternative Name: attachment.outlook.office.net -fetchmail: Subject Alternative Name: attachment.outlook.officeppe.net -fetchmail: Subject Alternative Name: *.office.com -fetchmail: outlook.office365.com key fingerprint: 3A:A4:58:42:56:CD:BD:11:19:5B:CF:1E:85:16:8E:4D -fetchmail: POP3< +OK The Microsoft Exchange POP3 service is ready. [SABFADEAUABSADAAMQBDAEEAMAAwADAANwAuAGUAdQByAHAAcgBkADAAMQAuAHAAcgBvAGQALgBlAHgAYwBoAGEAbgBnAGUAbABhAGIAcwAuAGMAbwBtAA==] -fetchmail: POP3> CAPA -fetchmail: POP3< +OK -fetchmail: POP3< TOP -fetchmail: POP3< UIDL -fetchmail: POP3< SASL PLAIN -fetchmail: POP3< USER -fetchmail: POP3< . -fetchmail: POP3> USER user1@outlook.com -fetchmail: POP3< +OK -fetchmail: POP3> PASS * -fetchmail: POP3< +OK User successfully logged on. -fetchmail: POP3> STAT -fetchmail: POP3< +OK 0 0 -fetchmail: No mail for user1@outlook.com at outlook.office365.com -fetchmail: POP3> QUIT -fetchmail: POP3< +OK Microsoft Exchange Server 2016 POP3 server signing off. -fetchmail: 6.3.26 querying outlook.office365.com (protocol POP3) at Mon Aug 29 22:11:11 2016: poll completed -fetchmail: normal termination, status 1 -``` +!!! tip "Troubleshoot with this reference `compose.yaml`" + + [A minimal `compose.yaml` example][fetchmail-compose-example] demonstrates how to run two instances of DMS locally, with one instance configured with `fetchmail.cf` and the other to simulate a remote mail server to fetch from. -[docs-setup]: ../../config/setup.sh.md [fetchmail-website]: https://www.fetchmail.info -[fetchmail-docs]: https://www.fetchmail.info/fetchmail-man.html -[fetchmail-docs-run]: https://www.fetchmail.info/fetchmail-man.html#31 +[fetchmail-docs-config]: https://www.fetchmail.info/fetchmail-man.html#the-run-control-file +[fetchmail-config-examples]: https://www.fetchmail.info/fetchmail-man.html#configuration-examples +[fetchmail-compose-example]: https://github.com/orgs/docker-mailserver/discussions/3994#discussioncomment-9290570 From 7822a9743034c486cf2ceeebde64eed6c3ace8c2 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Thu, 2 May 2024 19:48:05 +1200 Subject: [PATCH 087/267] docs(FAQ): Add advice for restricting login by IP (#3999) --- docs/content/faq.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/content/faq.md b/docs/content/faq.md index 4add0589..a350f937 100644 --- a/docs/content/faq.md +++ b/docs/content/faq.md @@ -291,6 +291,12 @@ mydestination = localhost.$mydomain, localhost proxy_interfaces = X.X.X.X (your public IP) ``` +For reverse proxy support you will want to view [our dedicated guide][docs::examples::reverse-proxy]. + +### How to restrict login by IP? + +There are a few ways you could approach this, see [this discussion answer][gh-discussion::restrict-login-by-ip] for advice. + ### How to adjust settings with the `user-patches.sh` script Suppose you want to change a number of settings that are not listed as variables or add things to the server that are not included? @@ -496,6 +502,7 @@ $spam_quarantine_to = "quarantine\@example.com"; [docs-userpatches]: ./config/advanced/override-defaults/user-patches.md [docs::env::sa_env]: ./config/environment.md#spamassassin [docs::env::sa_kill]: ./config/environment.md#sa_kill +[docs::examples::reverse-proxy]: ./examples/tutorials/mailserver-behind-proxy.md [github-comment-baredomain]: https://github.com/docker-mailserver/docker-mailserver/issues/3048#issuecomment-1432358353 [github-comment-override-hostname]: https://github.com/docker-mailserver/docker-mailserver/issues/1731#issuecomment-753968425 [github-issue-95]: https://github.com/docker-mailserver/docker-mailserver/issues/95 @@ -505,5 +512,6 @@ $spam_quarantine_to = "quarantine\@example.com"; [github-issue-1639]: https://github.com/docker-mailserver/docker-mailserver/issues/1639 [github-issue-1792]: https://github.com/docker-mailserver/docker-mailserver/pull/1792 [gh-discussion::dms-avoid-maintaining-internal-dns]: https://github.com/orgs/docker-mailserver/discussions/3959#discussioncomment-8956322 +[gh-discussion::restrict-login-by-ip]: https://github.com/orgs/docker-mailserver/discussions/3847 [docker-compose::docs::config-dns]: https://docs.docker.com/compose/compose-file/compose-file-v3/#dns [hanscees-userpatches]: https://github.com/hanscees/dockerscripts/blob/master/scripts/tomav-user-patches.sh From e2c2a22dcf2d8adc12f303b84b694945e8dc072e Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Fri, 3 May 2024 11:12:34 +1200 Subject: [PATCH 088/267] fix: `postfix-main.cf` may depend upon `postfix-master.cf` (#3880) Custom parameters must be referenced to be retained when `postconf -n` is run. If those parameters are referenced by `postfix-master.cf` this needs to update `master.cf` before updating `main.cf`. --- CHANGELOG.md | 4 ++- target/scripts/startup/setup.d/postfix.sh | 36 ++++++++++--------- test/config/override-configs/postfix-main.cf | 3 ++ .../config/override-configs/postfix-master.cf | 4 +++ .../tests/parallel/set1/config_overrides.bats | 21 +++++++++++ 5 files changed, 51 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 20ce0ed4..d651b0b5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -96,7 +96,9 @@ The most noteworthy change of this release is the update of the container's base ### Fixes -- DMS config files that are parsed line by line are now more robust to parse by detecting and fixing line-endings ([#3819](https://github.com/docker-mailserver/docker-mailserver/pull/3819)) +- DMS config: + - Files that are parsed line by line are now more robust to parse by detecting and fixing line-endings ([#3819](https://github.com/docker-mailserver/docker-mailserver/pull/3819)) + - The override config `postfix-main.cf` now retains custom parameters intended for use with `postfix-master.cf` ([#3880](https://github.com/docker-mailserver/docker-mailserver/pull/3880)) - Variables related to Rspamd are declared as `readonly`, which would cause warnings in the log when being re-declared; we now guard against this issue ([#3837](https://github.com/docker-mailserver/docker-mailserver/pull/3837)) - Relay host feature refactored ([#3845](https://github.com/docker-mailserver/docker-mailserver/pull/3845)) - `DEFAULT_RELAY_HOST` ENV can now also use the `RELAY_USER` + `RELAY_PASSWORD` ENV for supplying credentials. diff --git a/target/scripts/startup/setup.d/postfix.sh b/target/scripts/startup/setup.d/postfix.sh index 05052faa..3ebdeb9d 100644 --- a/target/scripts/startup/setup.d/postfix.sh +++ b/target/scripts/startup/setup.d/postfix.sh @@ -109,21 +109,6 @@ function _setup_postfix_late() { function __postfix__setup_override_configuration() { __postfix__log 'debug' 'Overriding / adjusting configuration with user-supplied values' - local OVERRIDE_CONFIG_POSTFIX_MAIN='/tmp/docker-mailserver/postfix-main.cf' - if [[ -f ${OVERRIDE_CONFIG_POSTFIX_MAIN} ]]; then - cat "${OVERRIDE_CONFIG_POSTFIX_MAIN}" >>/etc/postfix/main.cf - _adjust_mtime_for_postfix_maincf - - # do not directly output to 'main.cf' as this causes a read-write-conflict - postconf -n >/tmp/postfix-main-new.cf 2>/dev/null - - mv /tmp/postfix-main-new.cf /etc/postfix/main.cf - _adjust_mtime_for_postfix_maincf - __postfix__log 'trace' "Adjusted '/etc/postfix/main.cf' according to '${OVERRIDE_CONFIG_POSTFIX_MAIN}'" - else - __postfix__log 'trace' "No extra Postfix settings loaded because optional '${OVERRIDE_CONFIG_POSTFIX_MAIN}' was not provided" - fi - local OVERRIDE_CONFIG_POSTFIX_MASTER='/tmp/docker-mailserver/postfix-master.cf' if [[ -f ${OVERRIDE_CONFIG_POSTFIX_MASTER} ]]; then while read -r LINE; do @@ -131,7 +116,26 @@ function __postfix__setup_override_configuration() { done < <(_get_valid_lines_from_file "${OVERRIDE_CONFIG_POSTFIX_MASTER}") __postfix__log 'trace' "Adjusted '/etc/postfix/master.cf' according to '${OVERRIDE_CONFIG_POSTFIX_MASTER}'" else - __postfix__log 'trace' "No extra Postfix settings loaded because optional '${OVERRIDE_CONFIG_POSTFIX_MASTER}' was not provided" + __postfix__log 'trace' "No extra Postfix master settings loaded because optional '${OVERRIDE_CONFIG_POSTFIX_MASTER}' was not provided" + fi + + # NOTE: `postfix-main.cf` should be handled after `postfix-master.cf` as custom parameters require an existing reference + # in either `main.cf` or `master.cf` prior to `postconf` reading `main.cf`, otherwise it is discarded from output. + local OVERRIDE_CONFIG_POSTFIX_MAIN='/tmp/docker-mailserver/postfix-main.cf' + if [[ -f ${OVERRIDE_CONFIG_POSTFIX_MAIN} ]]; then + cat "${OVERRIDE_CONFIG_POSTFIX_MAIN}" >>/etc/postfix/main.cf + _adjust_mtime_for_postfix_maincf + + # Do not directly output to 'main.cf' as this causes a read-write-conflict. + # `postconf` output is filtered to skip expected warnings regarding overrides: + # https://github.com/docker-mailserver/docker-mailserver/pull/3880#discussion_r1510414576 + postconf -n >/tmp/postfix-main-new.cf 2> >(grep -v 'overriding earlier entry') + + mv /tmp/postfix-main-new.cf /etc/postfix/main.cf + _adjust_mtime_for_postfix_maincf + __postfix__log 'trace' "Adjusted '/etc/postfix/main.cf' according to '${OVERRIDE_CONFIG_POSTFIX_MAIN}'" + else + __postfix__log 'trace' "No extra Postfix main settings loaded because optional '${OVERRIDE_CONFIG_POSTFIX_MAIN}' was not provided" fi } diff --git a/test/config/override-configs/postfix-main.cf b/test/config/override-configs/postfix-main.cf index ce07bd56..fb29f3cc 100644 --- a/test/config/override-configs/postfix-main.cf +++ b/test/config/override-configs/postfix-main.cf @@ -2,3 +2,6 @@ max_idle = 600s # this is a comment # this is also a comment readme_directory = /tmp + +# This parameter is referenced by an override in `postfix-master.cf`: +custom_parameter = cidr:{{!172.16.0.42 REJECT}}, permit_sasl_authenticated, reject diff --git a/test/config/override-configs/postfix-master.cf b/test/config/override-configs/postfix-master.cf index 516fea81..f54d37d3 100644 --- a/test/config/override-configs/postfix-master.cf +++ b/test/config/override-configs/postfix-master.cf @@ -1,3 +1,7 @@ submission/inet/smtpd_sasl_security_options=noanonymous # this is a test comment, please don't delete me :'( # this is also a test comment, :O + +# DMS must first process `postfix-master.cf` to `master.cf` when any custom parameters are from `postfix-main.cf` +# before an updated `main.cf` is read via `postconf`, otherwise without an existing reference the parameter is excluded. +submission/inet/smtpd_client_restrictions=$custom_parameter diff --git a/test/tests/parallel/set1/config_overrides.bats b/test/tests/parallel/set1/config_overrides.bats index 0a7a5e7f..0281d197 100644 --- a/test/tests/parallel/set1/config_overrides.bats +++ b/test/tests/parallel/set1/config_overrides.bats @@ -15,6 +15,9 @@ function setup_file() { function teardown_file() { _default_teardown ; } +# The `postconf` command can query both `main.cf` and `master.cf` at `/etc/postfix/`. +# Reference: http://www.postfix.org/postconf.1.html + @test "Postfix - 'postfix-main.cf' overrides applied to '/etc/postfix/main.cf'" { _run_in_container grep -q 'max_idle = 600s' /tmp/docker-mailserver/postfix-main.cf assert_success @@ -37,6 +40,24 @@ function teardown_file() { _default_teardown ; } assert_output --partial '-o smtpd_sasl_security_options=noanonymous' } +# Custom parameter support works correctly: +# NOTE: This would only fail on a fresh container state, any restart would pass successfully: +# https://github.com/docker-mailserver/docker-mailserver/pull/3880 +@test "Postfix - 'postfix-master.cf' should apply before 'postfix-main.cf'" { + # Retrieve the value for this setting, `postfix-master.cf` should have the override set: + _run_in_container postconf -Ph 'submission/inet/smtpd_client_restrictions' + assert_success + refute_output --partial 'postconf: warning: /etc/postfix/master.cf: undefined parameter: custom_parameter' + #shellcheck disable=SC2016 + assert_output '$custom_parameter' + + # As it's a custom parameter (`$` prefix), ensure the parameter value expands correctly: + _run_in_container postconf -Phx 'submission/inet/smtpd_client_restrictions' + assert_success + refute_output --partial 'postconf: warning: /etc/postfix/master.cf: undefined parameter: custom_parameter' + assert_output 'cidr:{{!172.16.0.42 REJECT}}, permit_sasl_authenticated, reject' +} + @test "Dovecot - 'dovecot.cf' overrides applied to '/etc/dovecot/local.conf'" { _run_in_container grep -q 'mail_max_userip_connections = 69' /tmp/docker-mailserver/dovecot.cf assert_success From d52b813cd90833f5a0f0df01128c5bbd362c7870 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sun, 5 May 2024 11:42:59 +0200 Subject: [PATCH 089/267] docs: updated `CONTRIBUTORS.md` (#4002) --- CONTRIBUTORS.md | 175 +++++++++++++++++++++++++----------------------- 1 file changed, 91 insertions(+), 84 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index a96b2659..3ecae553 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -678,19 +678,19 @@ Thanks goes to these wonderful people ✨ aminvakil - - - abh -
- abh - - elbracht
elbracht + + + + abh +
+ abh + @@ -823,17 +823,10 @@ Thanks goes to these wonderful people ✨ - - 0xflotus + + neuralp
- 0xflotus - - - - - simonsystem -
- simonsystem + neuralp @@ -863,15 +856,15 @@ Thanks goes to these wonderful people ✨
syl20bnr - - + sylvaindumont
sylvaindumont - + + TechnicLab @@ -906,15 +899,15 @@ Thanks goes to these wonderful people ✨
tweibert - - + torus
torus - + + VictorKoenders @@ -949,6 +942,13 @@ Thanks goes to these wonderful people ✨
vilisas + + + + forzagreen +
+ forzagreen + @@ -965,13 +965,6 @@ Thanks goes to these wonderful people ✨ matrixes - - - neuralp -
- neuralp - - radicand @@ -992,15 +985,15 @@ Thanks goes to these wonderful people ✨
nknapp - - + pcqnt
pcqnt - + + OrvilleQ @@ -1035,15 +1028,15 @@ Thanks goes to these wonderful people ✨
peter-hartmann - - + piwai
piwai - + + remoe @@ -1078,15 +1071,15 @@ Thanks goes to these wonderful people ✨
MightySCollins - - + 501st-alpha1
501st-alpha1 - + + klamann @@ -1121,6 +1114,13 @@ Thanks goes to these wonderful people ✨
sjmudd + + + + simonsystem +
+ simonsystem + @@ -1445,6 +1445,13 @@ Thanks goes to these wonderful people ✨ marios88 + + + 0xflotus +
+ 0xflotus + + arkanovicz @@ -1458,15 +1465,15 @@ Thanks goes to these wonderful people ✨
CBeerta - + + damianmoore
damianmoore - - + espitall @@ -1501,15 +1508,15 @@ Thanks goes to these wonderful people ✨
denisix - + + mlatorre31
mlatorre31 - - + mazzz1y @@ -1544,15 +1551,15 @@ Thanks goes to these wonderful people ✨
edvorg - + + eliroca
eliroca - - + ekkis @@ -1587,15 +1594,15 @@ Thanks goes to these wonderful people ✨
flole - + + froks
froks - - + fkefer @@ -1630,15 +1637,15 @@ Thanks goes to these wonderful people ✨
askz - + + aspettl
aspettl - - + acch @@ -1673,15 +1680,15 @@ Thanks goes to these wonderful people ✨
ch3sh1r - + + eglia
eglia - - + groupmsl @@ -1716,15 +1723,15 @@ Thanks goes to these wonderful people ✨
arunvc - + + astrocket
astrocket - - + baxerus @@ -1759,15 +1766,15 @@ Thanks goes to these wonderful people ✨
auchri - + + Kaan88
Kaan88 - - + akkumar @@ -1802,15 +1809,15 @@ Thanks goes to these wonderful people ✨
UltraCoderRU - + + JustAnother1
JustAnother1 - - + leowinterde @@ -1845,15 +1852,15 @@ Thanks goes to these wonderful people ✨
MadsRC - + + madmath03
madmath03 - - + maxemann96 @@ -1888,15 +1895,15 @@ Thanks goes to these wonderful people ✨
milas - + + mcchots
mcchots - - + MohammedNoureldin @@ -1931,15 +1938,15 @@ Thanks goes to these wonderful people ✨
Marsu31 - + + glandais
glandais - - + GiovanH @@ -1974,15 +1981,15 @@ Thanks goes to these wonderful people ✨
Influencer - + + in-seo
in-seo - - + firefly-cpp @@ -2017,15 +2024,15 @@ Thanks goes to these wonderful people ✨
mivek - + + init-js
init-js - - + Jeidnx @@ -2060,15 +2067,15 @@ Thanks goes to these wonderful people ✨
jmccl - + + jurekbarth
jurekbarth - - + JOduMonT From 10f72224ca9ee3d2f82b187270d7099cf46537e1 Mon Sep 17 00:00:00 2001 From: "F. Eber" <138527669+gitfeber@users.noreply.github.com> Date: Sun, 5 May 2024 19:26:58 +0200 Subject: [PATCH 090/267] Update typo in kubernetes.md (#4003) Added the missing "s" on "submissions", otherwise this error comes up: The Service "mailserver" is invalid: spec.ports[2].name: Duplicate value: "submission" --- docs/content/config/advanced/kubernetes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/config/advanced/kubernetes.md b/docs/content/config/advanced/kubernetes.md index 8f5ac901..268094a6 100644 --- a/docs/content/config/advanced/kubernetes.md +++ b/docs/content/config/advanced/kubernetes.md @@ -152,7 +152,7 @@ If using our Helm chart is not viable for you, here is some guidance to start wi targetPort: smtp protocol: TCP # submissions (ESMTP with implicit TLS) - - name: submission + - name: submissions port: 465 targetPort: submissions protocol: TCP From 016d6b5255be6945b08616a79f92f496a583b775 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 6 May 2024 19:02:41 +0200 Subject: [PATCH 091/267] chore(deps): Bump akhilmhdh/contributors-readme-action (#4005) --- .github/workflows/contributors.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/contributors.yml b/.github/workflows/contributors.yml index 9a71b648..6ca6027b 100644 --- a/.github/workflows/contributors.yml +++ b/.github/workflows/contributors.yml @@ -19,7 +19,7 @@ jobs: uses: actions/checkout@v4 - name: 'Update CONTRIBUTORS.md' - uses: akhilmhdh/contributors-readme-action@v2.3.8 + uses: akhilmhdh/contributors-readme-action@v2.3.10 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} with: From dab3d9fe71091e6f8cc4c20ae49116de76056935 Mon Sep 17 00:00:00 2001 From: mmehnert Date: Sun, 12 May 2024 09:59:22 +0200 Subject: [PATCH 092/267] chore(logwatch): Add `ignore.conf` to ignore logs from Dovecot `index-worker` (#4012) --- CHANGELOG.md | 2 ++ Dockerfile | 1 + target/logwatch/ignore.conf | 2 ++ 3 files changed, 5 insertions(+) create mode 100644 target/logwatch/ignore.conf diff --git a/CHANGELOG.md b/CHANGELOG.md index d651b0b5..0ead51f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -93,6 +93,8 @@ The most noteworthy change of this release is the update of the container's base - The `rewrite_subject` action, is now disabled by default. It has been replaced with the new `SPAM_SUBJECT` environment variable, which implements the functionality via a Sieve script instead which is anti-spam service agnostic ([#3820](https://github.com/docker-mailserver/docker-mailserver/pull/3820)) - `RSPAMD_NEURAL` was added and is disabled by default. If switched on it will enable the experimental Rspamd "Neural network" module to add a layer of analysis to spam detection ([#3833](https://github.com/docker-mailserver/docker-mailserver/pull/3833)) - The symbol weights of SPF, DKIM and DMARC have been adjusted again. Fixes a bug and includes more appropriate combinations of symbols ([#3913](https://github.com/docker-mailserver/docker-mailserver/pull/3913), [#3923](https://github.com/docker-mailserver/docker-mailserver/pull/3923)) +- **Dovecot:** + - `logwatch` now filters out non-error logs related to the status of the `index-worker` process for FTS indexing. ([#4012](https://github.com/docker-mailserver/docker-mailserver/pull/4012)) ### Fixes diff --git a/Dockerfile b/Dockerfile index 8b2a502c..bf688ec1 100644 --- a/Dockerfile +++ b/Dockerfile @@ -273,6 +273,7 @@ EOF # ----------------------------------------------- COPY target/logwatch/maillog.conf /etc/logwatch/conf/logfiles/maillog.conf +COPY target/logwatch/ignore.conf /etc/logwatch/conf/ignore.conf # ----------------------------------------------- # --- Supervisord & Start ----------------------- diff --git a/target/logwatch/ignore.conf b/target/logwatch/ignore.conf new file mode 100644 index 00000000..5b7e9312 --- /dev/null +++ b/target/logwatch/ignore.conf @@ -0,0 +1,2 @@ +# ignore output from dovecot-fts-xapian about successfully indexed messages +dovecot: indexer-worker\([^\)]+\).*Indexed From 5bd8df68eb06b0f8879f7a953a6da607b77a4730 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sun, 12 May 2024 11:55:22 +0200 Subject: [PATCH 093/267] docs: updated `CONTRIBUTORS.md` (#4014) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- CONTRIBUTORS.md | 4216 ++++++++++++++++++++++++----------------------- 1 file changed, 2137 insertions(+), 2079 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 3ecae553..4275d39d 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -4,2084 +4,2142 @@ Thanks goes to these wonderful people ✨ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
- - casperklein -
- casperklein - - 		- - fbartels -
- fbartels - - 		- - polarathene -
- polarathene - - 		- - NorseGaud -
- NorseGaud - - 		- - williamdes -
- williamdes - - 		- - wernerfred -
- wernerfred - -
- - georglauterbach -
- georglauterbach - - 		- - tomav -
- tomav - - 		- - erik-wramner -
- erik-wramner - - 		- - chikamichi -
- chikamichi - - 		- - martin-schulze-vireso -
- martin-schulze-vireso - - 		- - Josef-Friedrich -
- Josef-Friedrich - -
- - johansmitsnl -
- johansmitsnl - - 		- - youtous -
- youtous - - 		- - 17Halbe -
- 17Halbe - - 		- - tve -
- tve - - 		- - gmasse -
- gmasse - - 		- - ap-wtioit -
- ap-wtioit - -
- - 00angus -
- 00angus - - 		- - alinmear -
- alinmear - - 		- - dominikwinter -
- dominikwinter - - 		- - crazystick -
- crazystick - - 		- - svenyonson -
- svenyonson - - 		- - swiesend -
- swiesend - -
- - stonemaster -
- stonemaster - - 		- - omarc1492 -
- omarc1492 - - 		- - phish108 -
- phish108 - - 		- - mwlczk -
- mwlczk - - 		- - tyranron -
- tyranron - - 		- - KyleOndy -
- KyleOndy - -
- - mindrunner -
- mindrunner - - 		- - MichaelSp -
- MichaelSp - - 		- - m-a-v -
- m-a-v - - 		- - bilak -
- bilak - - 		- - vortex852456 -
- vortex852456 - - 		- - hanscees -
- hanscees - -
- - chris54721 -
- chris54721 - - 		- - jrpear -
- jrpear - - 		- - moqmar -
- moqmar - - 		- - pyy -
- pyy - - 		- - arneke -
- arneke - - 		- - akmet -
- akmet - -
- - diiigle -
- diiigle - - 		- - kiliant -
- kiliant - - 		- - dashohoxha -
- dashohoxha - - 		- - egavard -
- egavard - - 		- - mathuin -
- mathuin - - 		- - jamebus -
- jamebus - -
- - robertdolca -
- robertdolca - - 		- - okainov -
- okainov - - 		- - jsonn -
- jsonn - - 		- - lukecyca -
- lukecyca - - 		- - m-schmoock -
- m-schmoock - - 		- - mjung -
- mjung - -
- - eltociear -
- eltociear - - 		- - VanVan -
- VanVan - - 		- - voordev -
- voordev - - 		- - andreasgerstmayr -
- andreasgerstmayr - - 		- - davidszp -
- davidszp - - 		- - kamuri -
- kamuri - -
- - guardiande -
- guardiande - - 		- - Zehir -
- Zehir - - 		- - Birkenstab -
- Birkenstab - - 		- - BrandonSchmitt -
- BrandonSchmitt - - 		- - Starbix -
- Starbix - - 		- - citec -
- citec - -
- - yajo -
- yajo - - 		- - MakerMatrix -
- MakerMatrix - - 		- - weo -
- weo - - 		- - analogue -
- analogue - - 		- - Rubytastic2 -
- Rubytastic2 - - 		- - reneploetz -
- reneploetz - -
- - pbek -
- pbek - - 		- - yogo1212 -
- yogo1212 - - 		- - castorinop -
- castorinop - - 		- - p-fruck -
- p-fruck - - 		- - tbutter -
- tbutter - - 		- - rahilarious -
- rahilarious - -
- - Rillke -
- Rillke - - 		- - bobbravo2 -
- bobbravo2 - - 		- - r-pufky -
- r-pufky - - 		- - vincentDcmps -
- vincentDcmps - - 		- - frugan-dev -
- frugan-dev - - 		- - mpanneck -
- mpanneck - -
- - andymel123 -
- andymel123 - - 		- - bigpigeon -
- bigpigeon - - 		- - engelant -
- engelant - - 		- - j-marz -
- j-marz - - 		- - lokipo -
- lokipo - - 		- - msheakoski -
- msheakoski - -
- - willtho89 -
- willtho89 - - 		- - GoliathLabs -
- GoliathLabs - - 		- - andrewlow -
- andrewlow - - 		- - aminvakil -
- aminvakil - - 		- - elbracht -
- elbracht - - 		- - abh -
- abh - -
- - craue -
- craue - - 		- - ubenmackin -
- ubenmackin - - 		- - danielpanteleit -
- danielpanteleit - - 		- - dmcgrandle -
- dmcgrandle - - 		- - theomega -
- theomega - - 		- - DuncanvR -
- DuncanvR - -
- - emazzotta -
- emazzotta - - 		- - keslerm -
- keslerm - - 		- - nueaf -
- nueaf - - 		- - martinwepner -
- martinwepner - - 		- - artonge -
- artonge - - 		- - spacecowboy -
- spacecowboy - -
- - jedateach -
- jedateach - - 		- - millaguie -
- millaguie - - 		- - jamesfryer -
- jamesfryer - - 		- - H4R0 -
- H4R0 - - 		- - ipernet -
- ipernet - - 		- - fl42 -
- fl42 - -
- - neuralp -
- neuralp - - 		- - stephan-devop -
- stephan-devop - - 		- - stigok -
- stigok - - 		- - 5ven -
- 5ven - - 		- - syl20bnr -
- syl20bnr - - 		- - sylvaindumont -
- sylvaindumont - -
- - TechnicLab -
- TechnicLab - - 		- - thomasschmit -
- thomasschmit - - 		- - Thiritin -
- Thiritin - - 		- - tobiabocchi -
- tobiabocchi - - 		- - tweibert -
- tweibert - - 		- - torus -
- torus - -
- - VictorKoenders -
- VictorKoenders - - 		- - Twist235 -
- Twist235 - - 		- - k3it -
- k3it - - 		- - Drakulix -
- Drakulix - - 		- - vilisas -
- vilisas - - 		- - forzagreen -
- forzagreen - -
- - 42wim -
- 42wim - - 		- - matrixes -
- matrixes - - 		- - radicand -
- radicand - - 		- - nilshoell -
- nilshoell - - 		- - nknapp -
- nknapp - - 		- - pcqnt -
- pcqnt - -
- - OrvilleQ -
- OrvilleQ - - 		- - ovidiucp -
- ovidiucp - - 		- - mrPjer -
- mrPjer - - 		- - p3dda -
- p3dda - - 		- - peter-hartmann -
- peter-hartmann - - 		- - piwai -
- piwai - -
- - remoe -
- remoe - - 		- - robbertkl -
- robbertkl - - 		- - romansey -
- romansey - - 		- - norrs -
- norrs - - 		- - MightySCollins -
- MightySCollins - - 		- - 501st-alpha1 -
- 501st-alpha1 - -
- - klamann -
- klamann - - 		- - svdb0 -
- svdb0 - - 		- - 3ap -
- 3ap - - 		- - shyim -
- shyim - - 		- - sjmudd -
- sjmudd - - 		- - simonsystem -
- simonsystem - -
- - mchamplain -
- mchamplain - - 		- - millerjason -
- millerjason - - 		- - mplx -
- mplx - - 		- - odinis -
- odinis - - 		- - okamidash -
- okamidash - - 		- - olaf-mandel -
- olaf-mandel - -
- - ontheair81 -
- ontheair81 - - 		- - pravynandas -
- pravynandas - - 		- - presocratics -
- presocratics - - 		- - rhyst -
- rhyst - - 		- - rmlhuk -
- rmlhuk - - 		- - rriski -
- rriski - -
- - schnippl0r -
- schnippl0r - - 		- - smargold476 -
- smargold476 - - 		- - sportshead -
- sportshead - - 		- - squash -
- squash - - 		- - strarsis -
- strarsis - - 		- - tamueller -
- tamueller - -
- - vivacarvajalito -
- vivacarvajalito - - 		- - wligtenberg -
- wligtenberg - - 		- - wolkenschieber -
- wolkenschieber - - 		- - worldworm -
- worldworm - - 		- - ShiriNmi1520 -
- ShiriNmi1520 - - 		- - Zepmann -
- Zepmann - -
- - allddd -
- allddd - - 		- - arcaine2 -
- arcaine2 - - 		- - awb99 -
- awb99 - - 		- - brainkiller -
- brainkiller - - 		- - cternes -
- cternes - - 		- - dborowy -
- dborowy - -
- - dimalo -
- dimalo - - 		- - eleith -
- eleith - - 		- - fanqiaojun -
- fanqiaojun - - 		- - ghnp5 -
- ghnp5 - - 		- - helmutundarnold -
- helmutundarnold - - 		- - hnws -
- hnws - -
- - i-C-o-d-e-r -
- i-C-o-d-e-r - - 		- - idaadi -
- idaadi - - 		- - ixeft -
- ixeft - - 		- - jjtt -
- jjtt - - 		- - paralax -
- paralax - - 		- - jpduyx -
- jpduyx - -
- - landergate -
- landergate - - 		- - callmemagnus -
- callmemagnus - - 		- - marios88 -
- marios88 - - 		- - 0xflotus -
- 0xflotus - - 		- - arkanovicz -
- arkanovicz - - 		- - CBeerta -
- CBeerta - -
- - damianmoore -
- damianmoore - - 		- - espitall -
- espitall - - 		- - dkarski -
- dkarski - - 		- - dbellavista -
- dbellavista - - 		- - danielvandenberg95 -
- danielvandenberg95 - - 		- - denisix -
- denisix - -
- - mlatorre31 -
- mlatorre31 - - 		- - mazzz1y -
- mazzz1y - - 		- - doominator42 -
- doominator42 - - 		- - aydodo -
- aydodo - - 		- - vedtam -
- vedtam - - 		- - edvorg -
- edvorg - -
- - eliroca -
- eliroca - - 		- - ekkis -
- ekkis - - 		- - ErikEngerd -
- ErikEngerd - - 		- - huncode -
- huncode - - 		- - felixn -
- felixn - - 		- - flole -
- flole - -
- - froks -
- froks - - 		- - fkefer -
- fkefer - - 		- - ifokeev -
- ifokeev - - 		- - 20th -
- 20th - - 		- - 2b -
- 2b - - 		- - askz -
- askz - -
- - aspettl -
- aspettl - - 		- - acch -
- acch - - 		- - vifino -
- vifino - - 		- - kachkaev -
- kachkaev - - 		- - alexanderneu -
- alexanderneu - - 		- - ch3sh1r -
- ch3sh1r - -
- - eglia -
- eglia - - 		- - groupmsl -
- groupmsl - - 		- - green-anger -
- green-anger - - 		- - iRhonin -
- iRhonin - - 		- - MrFreezeex -
- MrFreezeex - - 		- - arunvc -
- arunvc - -
- - astrocket -
- astrocket - - 		- - baxerus -
- baxerus - - 		- - spock -
- spock - - 		- - erdos4d -
- erdos4d - - 		- - crash7 -
- crash7 - - 		- - auchri -
- auchri - -
- - Kaan88 -
- Kaan88 - - 		- - akkumar -
- akkumar - - 		- - thechubbypanda -
- thechubbypanda - - 		- - KCrawley -
- KCrawley - - 		- - khuedoan -
- khuedoan - - 		- - UltraCoderRU -
- UltraCoderRU - -
- - JustAnother1 -
- JustAnother1 - - 		- - leowinterde -
- leowinterde - - 		- - linhandev -
- linhandev - - 		- - luke- -
- luke- - - 		- - LucidityCrash -
- LucidityCrash - - 		- - MadsRC -
- MadsRC - -
- - madmath03 -
- madmath03 - - 		- - maxemann96 -
- maxemann96 - - 		- - dragetd -
- dragetd - - 		- - michaeljensen -
- michaeljensen - - 		- - exhuma -
- exhuma - - 		- - milas -
- milas - -
- - mcchots -
- mcchots - - 		- - MohammedNoureldin -
- MohammedNoureldin - - 		- - mpldr -
- mpldr - - 		- - naveensrinivasan -
- naveensrinivasan - - 		- - furstblumier -
- furstblumier - - 		- - Marsu31 -
- Marsu31 - -
- - glandais -
- glandais - - 		- - GiovanH -
- GiovanH - - 		- - harryyoud -
- harryyoud - - 		- - HeySora -
- HeySora - - 		- - sirgantrithon -
- sirgantrithon - - 		- - Influencer -
- Influencer - -
- - in-seo -
- in-seo - - 		- - firefly-cpp -
- firefly-cpp - - 		- - JacksonZ03 -
- JacksonZ03 - - 		- - JamBalaya56562 -
- JamBalaya56562 - - 		- - jcalfee -
- jcalfee - - 		- - mivek -
- mivek - -
- - init-js -
- init-js - - 		- - Jeidnx -
- Jeidnx - - 		- - jessp01 -
- jessp01 - - 		- - JiLleON -
- JiLleON - - 		- - jirislav -
- jirislav - - 		- - jmccl -
- jmccl - -
- - jurekbarth -
- jurekbarth - - 		- - JOduMonT -
- JOduMonT - -
+ + casperklein +
+ casperklein + + 		+ + fbartels +
+ fbartels + + 		+ + polarathene +
+ polarathene + + 		+ + NorseGaud +
+ NorseGaud + + 		+ + williamdes +
+ williamdes + + 		+ + wernerfred +
+ wernerfred + +
+ + georglauterbach +
+ georglauterbach + + 		+ + tomav +
+ tomav + + 		+ + erik-wramner +
+ erik-wramner + + 		+ + chikamichi +
+ chikamichi + + 		+ + martin-schulze-vireso +
+ martin-schulze-vireso + + 		+ + Josef-Friedrich +
+ Josef-Friedrich + +
+ + johansmitsnl +
+ johansmitsnl + + 		+ + youtous +
+ youtous + + 		+ + 17Halbe +
+ 17Halbe + + 		+ + tve +
+ tve + + 		+ + gmasse +
+ gmasse + + 		+ + ap-wtioit +
+ ap-wtioit + +
+ + 00angus +
+ 00angus + + 		+ + alinmear +
+ alinmear + + 		+ + dominikwinter +
+ dominikwinter + + 		+ + crazystick +
+ crazystick + + 		+ + svenyonson +
+ svenyonson + + 		+ + swiesend +
+ swiesend + +
+ + stonemaster +
+ stonemaster + + 		+ + omarc1492 +
+ omarc1492 + + 		+ + phish108 +
+ phish108 + + 		+ + mwlczk +
+ mwlczk + + 		+ + tyranron +
+ tyranron + + 		+ + KyleOndy +
+ KyleOndy + +
+ + mindrunner +
+ mindrunner + + 		+ + MichaelSp +
+ MichaelSp + + 		+ + m-a-v +
+ m-a-v + + 		+ + bilak +
+ bilak + + 		+ + vortex852456 +
+ vortex852456 + + 		+ + hanscees +
+ hanscees + +
+ + chris54721 +
+ chris54721 + + 		+ + jrpear +
+ jrpear + + 		+ + moqmar +
+ moqmar + + 		+ + pyy +
+ pyy + + 		+ + arneke +
+ arneke + + 		+ + akmet +
+ akmet + +
+ + diiigle +
+ diiigle + + 		+ + kiliant +
+ kiliant + + 		+ + dashohoxha +
+ dashohoxha + + 		+ + egavard +
+ egavard + + 		+ + mathuin +
+ mathuin + + 		+ + jamebus +
+ jamebus + +
+ + robertdolca +
+ robertdolca + + 		+ + okainov +
+ okainov + + 		+ + jsonn +
+ jsonn + + 		+ + lukecyca +
+ lukecyca + + 		+ + m-schmoock +
+ m-schmoock + + 		+ + mjung +
+ mjung + +
+ + eltociear +
+ eltociear + + 		+ + VanVan +
+ VanVan + + 		+ + voordev +
+ voordev + + 		+ + andreasgerstmayr +
+ andreasgerstmayr + + 		+ + davidszp +
+ davidszp + + 		+ + kamuri +
+ kamuri + +
+ + guardiande +
+ guardiande + + 		+ + Zehir +
+ Zehir + + 		+ + Birkenstab +
+ Birkenstab + + 		+ + BrandonSchmitt +
+ BrandonSchmitt + + 		+ + Starbix +
+ Starbix + + 		+ + citec +
+ citec + +
+ + yajo +
+ yajo + + 		+ + MakerMatrix +
+ MakerMatrix + + 		+ + weo +
+ weo + + 		+ + analogue +
+ analogue + + 		+ + Rubytastic2 +
+ Rubytastic2 + + 		+ + reneploetz +
+ reneploetz + +
+ + pbek +
+ pbek + + 		+ + yogo1212 +
+ yogo1212 + + 		+ + castorinop +
+ castorinop + + 		+ + p-fruck +
+ p-fruck + + 		+ + tbutter +
+ tbutter + + 		+ + rahilarious +
+ rahilarious + +
+ + Rillke +
+ Rillke + + 		+ + bobbravo2 +
+ bobbravo2 + + 		+ + r-pufky +
+ r-pufky + + 		+ + vincentDcmps +
+ vincentDcmps + + 		+ + frugan-dev +
+ frugan-dev + + 		+ + mpanneck +
+ mpanneck + +
+ + andymel123 +
+ andymel123 + + 		+ + bigpigeon +
+ bigpigeon + + 		+ + engelant +
+ engelant + + 		+ + j-marz +
+ j-marz + + 		+ + lokipo +
+ lokipo + + 		+ + msheakoski +
+ msheakoski + +
+ + willtho89 +
+ willtho89 + + 		+ + GoliathLabs +
+ GoliathLabs + + 		+ + ubenmackin +
+ ubenmackin + + 		+ + craue +
+ craue + + 		+ + abh +
+ abh + + 		+ + andrewlow +
+ andrewlow + +
+ + aminvakil +
+ aminvakil + + 		+ + elbracht +
+ elbracht + + 		+ + danielpanteleit +
+ danielpanteleit + + 		+ + dmcgrandle +
+ dmcgrandle + + 		+ + theomega +
+ theomega + + 		+ + DuncanvR +
+ DuncanvR + +
+ + emazzotta +
+ emazzotta + + 		+ + keslerm +
+ keslerm + + 		+ + nueaf +
+ nueaf + + 		+ + martinwepner +
+ martinwepner + + 		+ + artonge +
+ artonge + + 		+ + spacecowboy +
+ spacecowboy + +
+ + jedateach +
+ jedateach + + 		+ + H4R0 +
+ H4R0 + + 		+ + jamesfryer +
+ jamesfryer + + 		+ + millaguie +
+ millaguie + + 		+ + ipernet +
+ ipernet + + 		+ + fl42 +
+ fl42 + +
+ + neuralp +
+ neuralp + + 		+ + simonsystem +
+ simonsystem + + 		+ + stephan-devop +
+ stephan-devop + + 		+ + stigok +
+ stigok + + 		+ + 5ven +
+ 5ven + + 		+ + syl20bnr +
+ syl20bnr + +
+ + sylvaindumont +
+ sylvaindumont + + 		+ + TechnicLab +
+ TechnicLab + + 		+ + thomasschmit +
+ thomasschmit + + 		+ + 42wim +
+ 42wim + + 		+ + Thiritin +
+ Thiritin + + 		+ + forzagreen +
+ forzagreen + +
+ + tobiabocchi +
+ tobiabocchi + + 		+ + tweibert +
+ tweibert + + 		+ + torus +
+ torus + + 		+ + VictorKoenders +
+ VictorKoenders + + 		+ + Twist235 +
+ Twist235 + + 		+ + k3it +
+ k3it + +
+ + vilisas +
+ vilisas + + 		+ + Drakulix +
+ Drakulix + + 		+ + radicand +
+ radicand + + 		+ + nilshoell +
+ nilshoell + + 		+ + nknapp +
+ nknapp + + 		+ + pcqnt +
+ pcqnt + +
+ + OrvilleQ +
+ OrvilleQ + + 		+ + ovidiucp +
+ ovidiucp + + 		+ + mrPjer +
+ mrPjer + + 		+ + p3dda +
+ p3dda + + 		+ + 0xflotus +
+ 0xflotus + + 		+ + peter-hartmann +
+ peter-hartmann + +
+ + piwai +
+ piwai + + 		+ + remoe +
+ remoe + + 		+ + robbertkl +
+ robbertkl + + 		+ + romansey +
+ romansey + + 		+ + norrs +
+ norrs + + 		+ + MightySCollins +
+ MightySCollins + +
+ + 501st-alpha1 +
+ 501st-alpha1 + + 		+ + klamann +
+ klamann + + 		+ + svdb0 +
+ svdb0 + + 		+ + 3ap +
+ 3ap + + 		+ + shyim +
+ shyim + + 		+ + sjmudd +
+ sjmudd + +
+ + matrixes +
+ matrixes + + 		+ + mchamplain +
+ mchamplain + + 		+ + millerjason +
+ millerjason + + 		+ + mplx +
+ mplx + + 		+ + odinis +
+ odinis + + 		+ + okamidash +
+ okamidash + +
+ + olaf-mandel +
+ olaf-mandel + + 		+ + ontheair81 +
+ ontheair81 + + 		+ + pravynandas +
+ pravynandas + + 		+ + presocratics +
+ presocratics + + 		+ + rhyst +
+ rhyst + + 		+ + rmlhuk +
+ rmlhuk + +
+ + rriski +
+ rriski + + 		+ + schnippl0r +
+ schnippl0r + + 		+ + smargold476 +
+ smargold476 + + 		+ + sportshead +
+ sportshead + + 		+ + squash +
+ squash + + 		+ + strarsis +
+ strarsis + +
+ + tamueller +
+ tamueller + + 		+ + vivacarvajalito +
+ vivacarvajalito + + 		+ + wligtenberg +
+ wligtenberg + + 		+ + wolkenschieber +
+ wolkenschieber + + 		+ + worldworm +
+ worldworm + + 		+ + ShiriNmi1520 +
+ ShiriNmi1520 + +
+ + Zepmann +
+ Zepmann + + 		+ + allddd +
+ allddd + + 		+ + arcaine2 +
+ arcaine2 + + 		+ + awb99 +
+ awb99 + + 		+ + brainkiller +
+ brainkiller + + 		+ + cternes +
+ cternes + +
+ + dborowy +
+ dborowy + + 		+ + dimalo +
+ dimalo + + 		+ + eleith +
+ eleith + + 		+ + fanqiaojun +
+ fanqiaojun + + 		+ + ghnp5 +
+ ghnp5 + + 		+ + helmutundarnold +
+ helmutundarnold + +
+ + hnws +
+ hnws + + 		+ + i-C-o-d-e-r +
+ i-C-o-d-e-r + + 		+ + idaadi +
+ idaadi + + 		+ + ixeft +
+ ixeft + + 		+ + jjtt +
+ jjtt + + 		+ + paralax +
+ paralax + +
+ + jpduyx +
+ jpduyx + + 		+ + landergate +
+ landergate + + 		+ + callmemagnus +
+ callmemagnus + + 		+ + marios88 +
+ marios88 + + 		+ + arkanovicz +
+ arkanovicz + + 		+ + CBeerta +
+ CBeerta + +
+ + damianmoore +
+ damianmoore + + 		+ + espitall +
+ espitall + + 		+ + dkarski +
+ dkarski + + 		+ + dbellavista +
+ dbellavista + + 		+ + danielvandenberg95 +
+ danielvandenberg95 + + 		+ + denisix +
+ denisix + +
+ + mlatorre31 +
+ mlatorre31 + + 		+ + mazzz1y +
+ mazzz1y + + 		+ + doominator42 +
+ doominator42 + + 		+ + aydodo +
+ aydodo + + 		+ + vedtam +
+ vedtam + + 		+ + edvorg +
+ edvorg + +
+ + eliroca +
+ eliroca + + 		+ + ekkis +
+ ekkis + + 		+ + ErikEngerd +
+ ErikEngerd + + 		+ + huncode +
+ huncode + + 		+ + gitfeber +
+ gitfeber + + 		+ + felixn +
+ felixn + +
+ + flole +
+ flole + + 		+ + froks +
+ froks + + 		+ + ifokeev +
+ ifokeev + + 		+ + 20th +
+ 20th + + 		+ + 2b +
+ 2b + + 		+ + askz +
+ askz + +
+ + aspettl +
+ aspettl + + 		+ + acch +
+ acch + + 		+ + vifino +
+ vifino + + 		+ + kachkaev +
+ kachkaev + + 		+ + alexanderneu +
+ alexanderneu + + 		+ + ch3sh1r +
+ ch3sh1r + +
+ + eglia +
+ eglia + + 		+ + groupmsl +
+ groupmsl + + 		+ + green-anger +
+ green-anger + + 		+ + iRhonin +
+ iRhonin + + 		+ + MrFreezeex +
+ MrFreezeex + + 		+ + arunvc +
+ arunvc + +
+ + astrocket +
+ astrocket + + 		+ + baxerus +
+ baxerus + + 		+ + spock +
+ spock + + 		+ + erdos4d +
+ erdos4d + + 		+ + crash7 +
+ crash7 + + 		+ + auchri +
+ auchri + +
+ + fkefer +
+ fkefer + + 		+ + Kaan88 +
+ Kaan88 + + 		+ + akkumar +
+ akkumar + + 		+ + thechubbypanda +
+ thechubbypanda + + 		+ + KCrawley +
+ KCrawley + + 		+ + khuedoan +
+ khuedoan + +
+ + UltraCoderRU +
+ UltraCoderRU + + 		+ + JustAnother1 +
+ JustAnother1 + + 		+ + leowinterde +
+ leowinterde + + 		+ + linhandev +
+ linhandev + + 		+ + luke- +
+ luke- + + 		+ + LucidityCrash +
+ LucidityCrash + +
+ + MadsRC +
+ MadsRC + + 		+ + madmath03 +
+ madmath03 + + 		+ + maxemann96 +
+ maxemann96 + + 		+ + dragetd +
+ dragetd + + 		+ + michaeljensen +
+ michaeljensen + + 		+ + exhuma +
+ exhuma + +
+ + milas +
+ milas + + 		+ + mcchots +
+ mcchots + + 		+ + MohammedNoureldin +
+ MohammedNoureldin + + 		+ + mpldr +
+ mpldr + + 		+ + naveensrinivasan +
+ naveensrinivasan + + 		+ + furstblumier +
+ furstblumier + +
+ + Marsu31 +
+ Marsu31 + + 		+ + glandais +
+ glandais + + 		+ + GiovanH +
+ GiovanH + + 		+ + harryyoud +
+ harryyoud + + 		+ + HeySora +
+ HeySora + + 		+ + sirgantrithon +
+ sirgantrithon + +
+ + Influencer +
+ Influencer + + 		+ + in-seo +
+ in-seo + + 		+ + firefly-cpp +
+ firefly-cpp + + 		+ + JacksonZ03 +
+ JacksonZ03 + + 		+ + JamBalaya56562 +
+ JamBalaya56562 + + 		+ + jcalfee +
+ jcalfee + +
+ + mivek +
+ mivek + + 		+ + init-js +
+ init-js + + 		+ + Jeidnx +
+ Jeidnx + + 		+ + jessp01 +
+ jessp01 + + 		+ + JiLleON +
+ JiLleON + + 		+ + jirislav +
+ jirislav + +
+ + jmccl +
+ jmccl + + 		+ + jurekbarth +
+ jurekbarth + + 		+ + JOduMonT +
+ JOduMonT + +
From 006f442cd0bdedb386acf328146b630887a88f84 Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Fri, 17 May 2024 09:57:07 +0200 Subject: [PATCH 094/267] Update `.ecrc.json` to exclude `CONTRIBUTORS.md` (#4020) --- test/linting/.ecrc.json | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/test/linting/.ecrc.json b/test/linting/.ecrc.json index 374615fe..44d501ba 100644 --- a/test/linting/.ecrc.json +++ b/test/linting/.ecrc.json @@ -3,6 +3,7 @@ "Exclude": [ "^test/bats/", "^test/test_helper/bats-(assert|support)", - "\\.git/" + "\\.git/", + "CONTRIBUTORS\\.md" ] } From a780fb331169dbd6a2d521b5d81900a4ef8636e7 Mon Sep 17 00:00:00 2001 From: pyy <4225935+pyy@users.noreply.github.com> Date: Fri, 17 May 2024 11:38:02 +0200 Subject: [PATCH 095/267] docs: Add tip for disabling the default SPF service (#4019) Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- docs/content/config/best-practices/dkim_dmarc_spf.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/content/config/best-practices/dkim_dmarc_spf.md b/docs/content/config/best-practices/dkim_dmarc_spf.md index 52d01dea..d035556f 100644 --- a/docs/content/config/best-practices/dkim_dmarc_spf.md +++ b/docs/content/config/best-practices/dkim_dmarc_spf.md @@ -302,9 +302,9 @@ The DMARC status may not be displayed instantly due to delays in DNS (caches). D [Source][wikipedia-spf] -!!! note "Disabling `policyd-spf`?" +!!! tip "Disabling the default SPF service `policy-spf`" - As of now, `policyd-spf` cannot be disabled. This is WIP. + Set [`ENABLE_POLICYD_SPF=0`][docs-env-spf-policyd] to opt-out of the default SPF service. Advised when Rspamd is configured to handle SPF instead. ### Adding an SPF Record @@ -348,6 +348,7 @@ volumes: [docs-volumes-config]: ../advanced/optional-config.md#volumes-config [docs-env-opendkim]: ../environment.md#enable_opendkim [docs-env-rspamd]: ../environment.md#enable_rspamd +[docs-env-spf-policyd]: ../environment.md#enable_policyd_spf [docs-rspamd-config-dropin]: ../security/rspamd.md#manually [cloudflare-dkim-dmarc-spf]: https://www.cloudflare.com/learning/email-security/dmarc-dkim-spf/ [rfc-8301]: https://datatracker.ietf.org/doc/html/rfc8301#section-3.2 From ed669bd3148d1565635a02cebcf4cf5f0cf9b780 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Sun, 19 May 2024 22:32:53 +1200 Subject: [PATCH 096/267] fix: `/var/mail-state` should not symlink non-existing directories (#4018) Fixes an issue with the Getmail service, view PR thread for additional details. - Log an error when the expected service state directory doesn't exist. - The location `/var/lib/getmail/` doesn't seem like it should have been introduced. Drop it in favor of `/tmp/docker-mailserver/getmail`. It appears to be for storing remote mail that was retrieved if not configured to send to Dovecot like our docs advise. This location was never valid anyway (_as referenced issue covers_). --- CHANGELOG.md | 1 + target/bin/debug-fetchmail | 6 ++++++ target/bin/debug-getmail | 8 +------- target/bin/getmail-cron | 3 ++- target/scripts/startup/setup.d/getmail.sh | 5 +++++ target/scripts/startup/setup.d/mail_state.sh | 5 +++-- 6 files changed, 18 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ead51f5..5564d8aa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -109,6 +109,7 @@ The most noteworthy change of this release is the update of the container's base - Rspamd configuration: Add a missing comma in `local_networks` so that all internal IP addresses are actually considered as internal ([#3862](https://github.com/docker-mailserver/docker-mailserver/pull/3862)) - Ensure correct SELinux security context labels for files and directories moved to the mail-state volume during setup ([#3890](https://github.com/docker-mailserver/docker-mailserver/pull/3890)) - Use correct environment variable for fetchmail ([#3901](https://github.com/docker-mailserver/docker-mailserver/pull/3901)) +- When using `ENABLE_GETMAIL=1` the undocumented internal location `/var/lib/getmail/` usage has been removed. Only the config volume `/tmp/docker-mailserver/getmail/` location is supported when Getmail has not been configured to deliver mail to Dovecot as advised in the DMS docs ([#4018](https://github.com/docker-mailserver/docker-mailserver/pull/4018)) - Dovecot dummy accounts (_virtual alias workaround for dovecot feature `ENABLE_QUOTAS=1`_) now correctly matches the home location of the user for that alias ([#3997](https://github.com/docker-mailserver/docker-mailserver/pull/3997)) ## [v13.3.1](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v13.3.1) diff --git a/target/bin/debug-fetchmail b/target/bin/debug-fetchmail index 7e267184..87d30e2b 100755 --- a/target/bin/debug-fetchmail +++ b/target/bin/debug-fetchmail @@ -5,6 +5,12 @@ source /usr/local/bin/helpers/log.sh # shellcheck source=../scripts/startup/setup.d/fetchmail.sh source /usr/local/bin/setup.d/fetchmail.sh +# TODO: This should probably not implicitly enable the feature. +# The setup method will feature gate and output a debug log if +# the feature is not enabled. +# +# Dropping the ENV here will require updating legacy test: +# test/tests/parallel/set3/scripts/setup_cli.bats ENABLE_FETCHMAIL=1 _setup_fetchmail su -s /bin/sh -c "/usr/bin/fetchmail \ diff --git a/target/bin/debug-getmail b/target/bin/debug-getmail index 27834475..60270215 100644 --- a/target/bin/debug-getmail +++ b/target/bin/debug-getmail @@ -7,13 +7,7 @@ source /usr/local/bin/setup.d/getmail.sh _setup_getmail -if [[ -d /var/lib/getmail ]]; then - GETMAILDIR=/var/lib/getmail -else - mkdir -p /tmp/docker-mailserver/getmail - GETMAILDIR=/tmp/docker-mailserver/getmail -fi - +GETMAILDIR=/tmp/docker-mailserver/getmail for FILE in /etc/getmailrc.d/getmailrc*; do getmail --getmaildir "${GETMAILDIR}" --rcfile "${FILE}" --dump | tail -n +6 done diff --git a/target/bin/getmail-cron b/target/bin/getmail-cron index 8e1e4749..fec03906 100644 --- a/target/bin/getmail-cron +++ b/target/bin/getmail-cron @@ -1,7 +1,8 @@ #! /bin/bash +GETMAILDIR=/tmp/docker-mailserver/getmail for FILE in /etc/getmailrc.d/getmailrc*; do if ! pgrep -f "${FILE}$" &>/dev/null; then - getmail --getmaildir /var/lib/getmail --rcfile "${FILE}" + getmail --getmaildir "${GETMAILDIR}" --rcfile "${FILE}" fi done diff --git a/target/scripts/startup/setup.d/getmail.sh b/target/scripts/startup/setup.d/getmail.sh index 6f280118..c1345063 100644 --- a/target/scripts/startup/setup.d/getmail.sh +++ b/target/scripts/startup/setup.d/getmail.sh @@ -31,6 +31,11 @@ function _setup_getmail() { EOF chmod -R 600 "${GETMAILRC}" fi + + # Both the debug command and cron job (that runs getmail) for getmail + # expect this location to exist. + GETMAILDIR=/tmp/docker-mailserver/getmail + mkdir -p "${GETMAILDIR}" else _log 'debug' 'Getmail is disabled' fi diff --git a/target/scripts/startup/setup.d/mail_state.sh b/target/scripts/startup/setup.d/mail_state.sh index 5acf6762..de15ee3b 100644 --- a/target/scripts/startup/setup.d/mail_state.sh +++ b/target/scripts/startup/setup.d/mail_state.sh @@ -23,7 +23,6 @@ function _setup_save_states() { [[ ${ENABLE_CLAMAV} -eq 1 ]] && SERVICEDIRS+=('lib/clamav') [[ ${ENABLE_FAIL2BAN} -eq 1 ]] && SERVICEDIRS+=('lib/fail2ban') [[ ${ENABLE_FETCHMAIL} -eq 1 ]] && SERVICEDIRS+=('lib/fetchmail') - [[ ${ENABLE_GETMAIL} -eq 1 ]] && SERVICEDIRS+=('lib/getmail') [[ ${ENABLE_MTA_STS} -eq 1 ]] && SERVICEDIRS+=('lib/mta-sts') [[ ${ENABLE_POSTGREY} -eq 1 ]] && SERVICEDIRS+=('lib/postgrey') [[ ${ENABLE_RSPAMD} -eq 1 ]] && SERVICEDIRS+=('lib/rspamd') @@ -70,11 +69,13 @@ function _setup_save_states() { rm -rf "${SERVICEDIR}" elif [[ -d ${SERVICEDIR} ]]; then _log 'trace' "Moving contents of ${SERVICEDIR} to ${DEST}" - # Empty volume was mounted, or new content from enabling a feature ENV: + # An empty volume was mounted, or new content dir now exists from enabling a feature ENV: mv "${SERVICEDIR}" "${DEST}" # Apply SELinux security context to match the state directory, so access # is not restricted to the current running container: chcon -R --reference="${STATEDIR}" "${DEST}" 2>/dev/null || true + else + _log 'error' "${SERVICEDIR} should exist but is missing" fi # Symlink the original path in the container ($SERVICEDIR) to be From 03c905e6f11fa8c14416acd1d73fa6d75020e820 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sun, 19 May 2024 14:06:59 +0200 Subject: [PATCH 097/267] docs: updated `CONTRIBUTORS.md` (#4021) --- CONTRIBUTORS.md | 453 ++++++++++++++++++++++++------------------------ 1 file changed, 230 insertions(+), 223 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 4275d39d..1600440c 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -284,13 +284,6 @@ Thanks goes to these wonderful people ✨ jrpear - - - moqmar -
- moqmar - - pyy @@ -298,6 +291,13 @@ Thanks goes to these wonderful people ✨ pyy + + + moqmar +
+ moqmar + + arneke @@ -328,13 +328,6 @@ Thanks goes to these wonderful people ✨ kiliant - - - dashohoxha -
- dashohoxha - - egavard @@ -342,6 +335,13 @@ Thanks goes to these wonderful people ✨ egavard + + + dashohoxha +
+ dashohoxha + + mathuin @@ -358,20 +358,6 @@ Thanks goes to these wonderful people ✨ - - - robertdolca -
- robertdolca - - - - - okainov -
- okainov - - jsonn @@ -387,26 +373,17 @@ Thanks goes to these wonderful people ✨ - - m-schmoock + + okainov
- m-schmoock + okainov - - mjung + + robertdolca
- mjung - - - - - - - eltociear -
- eltociear + robertdolca @@ -417,10 +394,19 @@ Thanks goes to these wonderful people ✨ - - voordev + + eltociear
- voordev + eltociear + + + + + + + mjung +
+ mjung @@ -430,6 +416,13 @@ Thanks goes to these wonderful people ✨ andreasgerstmayr + + + m-schmoock +
+ m-schmoock + + davidszp @@ -444,8 +437,6 @@ Thanks goes to these wonderful people ✨ kamuri - - guardiande @@ -453,6 +444,8 @@ Thanks goes to these wonderful people ✨ guardiande + + Zehir @@ -460,6 +453,13 @@ Thanks goes to these wonderful people ✨ Zehir + + + voordev +
+ voordev + + Birkenstab @@ -505,17 +505,17 @@ Thanks goes to these wonderful people ✨ - - weo + + pbek
- weo + pbek - - analogue + + reneploetz
- analogue + reneploetz @@ -526,26 +526,26 @@ Thanks goes to these wonderful people ✨ - - reneploetz + + analogue
- reneploetz + analogue - - pbek + + weo
- pbek + weo - - yogo1212 + + ubenmackin
- yogo1212 + ubenmackin @@ -562,13 +562,6 @@ Thanks goes to these wonderful people ✨ p-fruck - - - tbutter -
- tbutter - - rahilarious @@ -576,8 +569,6 @@ Thanks goes to these wonderful people ✨ rahilarious - - Rillke @@ -585,6 +576,8 @@ Thanks goes to these wonderful people ✨ Rillke + + bobbravo2 @@ -606,22 +599,6 @@ Thanks goes to these wonderful people ✨ vincentDcmps - - - frugan-dev -
- frugan-dev - - - - - mpanneck -
- mpanneck - - - - andymel123 @@ -643,6 +620,8 @@ Thanks goes to these wonderful people ✨ engelant + + j-marz @@ -663,15 +642,6 @@ Thanks goes to these wonderful people ✨
msheakoski - - - - - - willtho89 -
- willtho89 - @@ -681,10 +651,40 @@ Thanks goes to these wonderful people ✨ - - ubenmackin + + frugan-dev
- ubenmackin + frugan-dev + + + + + tbutter +
+ tbutter + + + + + + + yogo1212 +
+ yogo1212 + + + + + willtho89 +
+ willtho89 + + + + + mpanneck +
+ mpanneck @@ -806,24 +806,10 @@ Thanks goes to these wonderful people ✨ - - H4R0 + + fl42
- H4R0 - - - - - jamesfryer -
- jamesfryer - - - - - millaguie -
- millaguie + fl42 @@ -834,26 +820,33 @@ Thanks goes to these wonderful people ✨ - - fl42 + + H4R0
- fl42 + H4R0 + + + + + millaguie +
+ millaguie + + + + + jamesfryer +
+ jamesfryer - - neuralp + + radicand
- neuralp - - - - - simonsystem -
- simonsystem + radicand @@ -884,8 +877,6 @@ Thanks goes to these wonderful people ✨ syl20bnr - - sylvaindumont @@ -893,6 +884,8 @@ Thanks goes to these wonderful people ✨ sylvaindumont + + TechnicLab @@ -907,13 +900,6 @@ Thanks goes to these wonderful people ✨ thomasschmit - - - 42wim -
- 42wim - - Thiritin @@ -921,15 +907,6 @@ Thanks goes to these wonderful people ✨ Thiritin - - - forzagreen -
- forzagreen - - - - tobiabocchi @@ -951,6 +928,8 @@ Thanks goes to these wonderful people ✨ torus + + VictorKoenders @@ -971,15 +950,6 @@ Thanks goes to these wonderful people ✨
k3it - - - - - - vilisas -
- vilisas - @@ -989,10 +959,33 @@ Thanks goes to these wonderful people ✨ - - radicand + + vilisas
- radicand + vilisas + + + + + forzagreen +
+ forzagreen + + + + + + + 42wim +
+ 42wim + + + + + ShiriNmi1520 +
+ ShiriNmi1520 @@ -1016,8 +1009,6 @@ Thanks goes to these wonderful people ✨ pcqnt - - OrvilleQ @@ -1025,6 +1016,8 @@ Thanks goes to these wonderful people ✨ OrvilleQ + + ovidiucp @@ -1046,13 +1039,6 @@ Thanks goes to these wonderful people ✨ p3dda - - - 0xflotus -
- 0xflotus - - peter-hartmann @@ -1060,8 +1046,13 @@ Thanks goes to these wonderful people ✨ peter-hartmann - - + + + neuralp +
+ neuralp + + piwai @@ -1069,6 +1060,8 @@ Thanks goes to these wonderful people ✨ piwai + + remoe @@ -1104,8 +1097,6 @@ Thanks goes to these wonderful people ✨ MightySCollins - - 501st-alpha1 @@ -1113,6 +1104,8 @@ Thanks goes to these wonderful people ✨ 501st-alpha1 + + klamann @@ -1148,15 +1141,15 @@ Thanks goes to these wonderful people ✨ sjmudd - - - - matrixes + + simonsystem
- matrixes + simonsystem + + mchamplain @@ -1171,6 +1164,13 @@ Thanks goes to these wonderful people ✨ millerjason + + + mmehnert +
+ mmehnert + + mplx @@ -1317,15 +1317,6 @@ Thanks goes to these wonderful people ✨ worldworm - - - ShiriNmi1520 -
- ShiriNmi1520 - - - - Zepmann @@ -1333,6 +1324,8 @@ Thanks goes to these wonderful people ✨ Zepmann + + allddd @@ -1368,8 +1361,6 @@ Thanks goes to these wonderful people ✨ cternes - - dborowy @@ -1377,6 +1368,8 @@ Thanks goes to these wonderful people ✨ dborowy + + dimalo @@ -1412,8 +1405,6 @@ Thanks goes to these wonderful people ✨ helmutundarnold - - hnws @@ -1421,6 +1412,8 @@ Thanks goes to these wonderful people ✨ hnws + + i-C-o-d-e-r @@ -1456,8 +1449,6 @@ Thanks goes to these wonderful people ✨ paralax - - jpduyx @@ -1465,6 +1456,8 @@ Thanks goes to these wonderful people ✨ jpduyx + + landergate @@ -1486,6 +1479,20 @@ Thanks goes to these wonderful people ✨ marios88 + + + matrixes +
+ matrixes + + + + + 0xflotus +
+ 0xflotus + + arkanovicz @@ -1493,6 +1500,8 @@ Thanks goes to these wonderful people ✨ arkanovicz + + CBeerta @@ -1500,8 +1509,6 @@ Thanks goes to these wonderful people ✨ CBeerta - - damianmoore @@ -1537,6 +1544,8 @@ Thanks goes to these wonderful people ✨ danielvandenberg95 + + denisix @@ -1544,8 +1553,6 @@ Thanks goes to these wonderful people ✨ denisix - - mlatorre31 @@ -1581,6 +1588,8 @@ Thanks goes to these wonderful people ✨ vedtam + + edvorg @@ -1588,8 +1597,6 @@ Thanks goes to these wonderful people ✨ edvorg - - eliroca @@ -1625,6 +1632,8 @@ Thanks goes to these wonderful people ✨ gitfeber + + felixn @@ -1632,8 +1641,6 @@ Thanks goes to these wonderful people ✨ felixn - - flole @@ -1669,6 +1676,8 @@ Thanks goes to these wonderful people ✨ 2b + + askz @@ -1676,8 +1685,6 @@ Thanks goes to these wonderful people ✨ askz - - aspettl @@ -1713,6 +1720,8 @@ Thanks goes to these wonderful people ✨ alexanderneu + + ch3sh1r @@ -1720,8 +1729,6 @@ Thanks goes to these wonderful people ✨ ch3sh1r - - eglia @@ -1757,6 +1764,8 @@ Thanks goes to these wonderful people ✨ MrFreezeex + + arunvc @@ -1764,8 +1773,6 @@ Thanks goes to these wonderful people ✨ arunvc - - astrocket @@ -1801,6 +1808,8 @@ Thanks goes to these wonderful people ✨ crash7 + + auchri @@ -1808,8 +1817,6 @@ Thanks goes to these wonderful people ✨ auchri - - fkefer @@ -1845,6 +1852,8 @@ Thanks goes to these wonderful people ✨ KCrawley + + khuedoan @@ -1852,8 +1861,6 @@ Thanks goes to these wonderful people ✨ khuedoan - - UltraCoderRU @@ -1889,6 +1896,8 @@ Thanks goes to these wonderful people ✨ luke- + + LucidityCrash @@ -1896,8 +1905,6 @@ Thanks goes to these wonderful people ✨ LucidityCrash - - MadsRC @@ -1933,6 +1940,8 @@ Thanks goes to these wonderful people ✨ michaeljensen + + exhuma @@ -1940,8 +1949,6 @@ Thanks goes to these wonderful people ✨ exhuma - - milas @@ -1977,6 +1984,8 @@ Thanks goes to these wonderful people ✨ naveensrinivasan + + furstblumier @@ -1984,8 +1993,6 @@ Thanks goes to these wonderful people ✨ furstblumier - - Marsu31 @@ -2021,6 +2028,8 @@ Thanks goes to these wonderful people ✨ HeySora + + sirgantrithon @@ -2028,8 +2037,6 @@ Thanks goes to these wonderful people ✨ sirgantrithon - - Influencer @@ -2065,6 +2072,8 @@ Thanks goes to these wonderful people ✨ JamBalaya56562 + + jcalfee @@ -2072,8 +2081,6 @@ Thanks goes to these wonderful people ✨ jcalfee - - mivek @@ -2109,6 +2116,8 @@ Thanks goes to these wonderful people ✨ JiLleON + + jirislav @@ -2116,8 +2125,6 @@ Thanks goes to these wonderful people ✨ jirislav - - jmccl From 92b06c40204d97d03a1d3172bb1964a069690759 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ji=C5=99=C3=AD=20=C5=A0tefka?= Date: Mon, 20 May 2024 15:07:42 +0200 Subject: [PATCH 098/267] docs(rspamd): Add guidance for setting WebUI password (#4023) * fix(docs/rspamd): Add section on how to setup WebUI password * Apply review suggestion * Apply suggestions from code review * Update rspamd.md --------- Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- docs/content/config/security/rspamd.md | 30 +++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/docs/content/config/security/rspamd.md b/docs/content/config/security/rspamd.md index fd0fe25e..b0f65990 100644 --- a/docs/content/config/security/rspamd.md +++ b/docs/content/config/security/rspamd.md @@ -84,14 +84,30 @@ When Rspamd is enabled, we implicitly also start an instance of Redis in the con Redis uses `/etc/redis/redis.conf` for configuration: - We adjust this file when enabling the internal Redis service. -- If you have an external instance of Redis to use, the internal Redis service can be opt-out via setting the ENV [`ENABLE_RSPAMD_REDIS=0`](../environment.md#enable_rspamd_redis) (_link also details required changes to the DMS Rspamd config_). +- If you have an external instance of Redis to use, the internal Redis service can be opt-out via setting the ENV [`ENABLE_RSPAMD_REDIS=0`][docs::env::enable-redis] (_link also details required changes to the DMS Rspamd config_). ### Web Interface -Rspamd provides a [web interface][rspamd-docs::web-interface], which contains statistics and data Rspamd collects. The interface is enabled by default and reachable on port 11334. +Rspamd provides a [web interface][rspamd-docs::web-ui], which contains statistics and data Rspamd collects. The interface is enabled by default and reachable on port 11334. ![Rspamd Web Interface](https://rspamd.com/img/webui.png) +To use the web interface you will need to configure a password, [otherwise you won't be able to log in][rspamd-docs::web-ui::password]. + +??? example "Set a custom password" + + Add this line to [your rspamd `custom-commands.conf` config](#with-the-help-of-a-custom-file) which sets the `password` option of the _controller worker_: + + ``` + set-option-for-controller password "your hashed password here" + ``` + + The password hash can be generated via the `rspamadm pw` command: + + ```bash + docker exec -it rspamadm pw + ``` + ### DNS DMS does not supply custom values for DNS servers (to Rspamd). If you need to use custom DNS servers, which could be required when using [DNS-based deny/allowlists](#rbls-real-time-blacklists-dnsbls-dns-based-blacklists), you need to adjust [`options.inc`][rspamd-docs::basic-options] yourself. Make sure to also read our [FAQ page on DNS servers][docs::faq::dns-servers]. @@ -126,13 +142,13 @@ You can choose to enable ClamAV, and Rspamd will then use it to check for viruse #### RBLs (Real-time Blacklists) / DNSBLs (DNS-based Blacklists) -The [RBL module](https://rspamd.com/doc/modules/rbl.html) is enabled by default. As a consequence, Rspamd will perform DNS lookups to various blacklists. Whether an RBL or a DNSBL is queried depends on where the domain name was obtained: RBL servers are queried with IP addresses extracted from message headers, DNSBL server are queried with domains and IP addresses extracted from the message body \[[source][www::rbl-vs-dnsbl]\]. +The [RBL module][rspamd-docs::modules::rbl] is enabled by default. As a consequence, Rspamd will perform DNS lookups to various blacklists. Whether an RBL or a DNSBL is queried depends on where the domain name was obtained: RBL servers are queried with IP addresses extracted from message headers, DNSBL server are queried with domains and IP addresses extracted from the message body \[[source][www::rbl-vs-dnsbl]\]. !!! danger "Rspamd and DNS Block Lists" When the RBL module is enabled, Rspamd will do a variety of DNS requests to (amongst other things) DNSBLs. There are a variety of issues involved when using DNSBLs. Rspamd will try to mitigate some of them by properly evaluating all return codes. This evaluation is a best effort though, so if the DNSBL operators change or add return codes, it may take a while for Rspamd to adjust as well. - If you want to use DNSBLs, **try to use your own DNS resolver** and make sure it is set up correctly, i.e. it should be a non-public & **recursive** resolver. Otherwise, you might not be able ([see this Spamhaus post](https://www.spamhaus.org/faq/section/DNSBL%20Usage#365)) to make use of the block lists. + If you want to use DNSBLs, **try to use your own DNS resolver** and make sure it is set up correctly, i.e. it should be a non-public & **recursive** resolver. Otherwise, you might not be able ([see this Spamhaus post][spamhaus::faq::dnsbl-usage]) to make use of the block lists. ## Providing Custom Settings & Overriding Settings @@ -247,8 +263,10 @@ While _Abusix_ can be integrated into Postfix, Postscreen and a multitude of oth [rspamd-web]: https://rspamd.com/ [rspamd-docs::bayes]: https://rspamd.com/doc/configuration/statistic.html [rspamd-docs::proxy-self-scan-mode]: https://rspamd.com/doc/workers/rspamd_proxy.html#self-scan-mode -[rspamd-docs::web-interface]: https://rspamd.com/webui/ +[rspamd-docs::web-ui]: https://rspamd.com/webui/ +[rspamd-docs::web-ui::password]: https://www.rspamd.com/doc/tutorials/quickstart.html#setting-the-controller-password [rspamd-docs::modules]: https://rspamd.com/doc/modules/ +[rspamd-docs::modules::rbl]: https://rspamd.com/doc/modules/rbl.html [rspamd-docs::override-dir]: https://www.rspamd.com/doc/faq.html#what-are-the-locald-and-overrided-directories [rspamd-docs::config-directories]: https://rspamd.com/doc/faq.html#what-are-the-locald-and-overrided-directories [rspamd-docs::basic-options]: https://rspamd.com/doc/configuration/options.html @@ -256,10 +274,12 @@ While _Abusix_ can be integrated into Postfix, Postscreen and a multitude of oth [www::rbl-vs-dnsbl]: https://forum.eset.com/topic/25277-dnsbl-vs-rbl-mail-security/?do=findComment&comment=119818 [abusix-web]: https://abusix.com/ [abusix-docs::rspamd-integration]: https://docs.abusix.com/abusix-mail-intelligence/gbG8EcJ3x3fSUv8cMZLiwA/getting-started/dmw9dcwSGSNQiLTssFAnBW#rspamd +[spamhaus::faq::dnsbl-usage]: https://www.spamhaus.org/faq/section/DNSBL%20Usage#365 [dms-repo::rspamd-actions-config]: https://github.com/docker-mailserver/docker-mailserver/blob/v14.0.0/target/rspamd/local.d/actions.conf [dms-repo::default-rspamd-configuration]: https://github.com/docker-mailserver/docker-mailserver/tree/v14.0.0/target/rspamd +[docs::env::enable-redis]: ../environment.md#enable_rspamd_redis [docs::spam-to-junk]: ../environment.md#move_spam_to_junk [docs::dkim-dmarc-spf]: ../best-practices/dkim_dmarc_spf.md [docs::dkim-with-rspamd]: ../best-practices/dkim_dmarc_spf.md#dkim From 993c7b044f126aa941ea30d4ac86dd38c5dc5fe4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ji=C5=99=C3=AD=20=C5=A0tefka?= Date: Tue, 21 May 2024 16:49:39 +0200 Subject: [PATCH 099/267] breaking: Drop Dovecot support for Solr (#4025) --- CHANGELOG.md | 3 ++ .../config/advanced/full-text-search.md | 49 +------------------ target/scripts/build/packages.sh | 2 +- 3 files changed, 5 insertions(+), 49 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5564d8aa..04f8adb9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,6 +40,9 @@ The most noteworthy change of this release is the update of the container's base - The "Junk" mailbox (folder) is now referenced by it's [special-use flag `\Junk`](https://docker-mailserver.github.io/docker-mailserver/v13.3/examples/use-cases/imap-folders/) instead of an explicit mailbox. ([#3925](https://github.com/docker-mailserver/docker-mailserver/pull/3925)) - This provides compatibility for the Junk mailbox when it's folder name differs (_eg: Renamed to "Spam"_). - Potential breakage if your deployment modifies our `spam_to_junk.sieve` sieve script (_which is created during container startup when ENV `MOVE_SPAM_TO_JUNK=1`_) that handles storing spam mail into a users "Junk" mailbox folder. + - **Removed support for Solr integration:** ([#4025](https://github.com/docker-mailserver/docker-mailserver/pull/4025)) + - This was a community contributed feature for FTS (Full Text Search), the docs advise using an image that has not been maintained for over 2 years and lacks ARM64 support. Based on user engagement over the years this feature has very niche value to continue to support, thus is being removed. + - If you use Solr, support can be restored if you're willing to contribute docs for the feature that resolves the concerns raised - **rsyslog:** - Debian 12 adjusted the `rsyslog` configuration for the default file template from `RSYSLOG_TraditionalFileFormat` to `RSYSLOG_FileFormat` (_upstream default since 2012_). This change may affect you if you have any monitoring / analysis of log output (_eg: `mail.log` / `docker logs`_). - The two formats are roughly equivalent to [RFC 3164](https://www.rfc-editor.org/rfc/rfc3164)) and [RFC 5424](https://datatracker.ietf.org/doc/html/rfc5424#section-1) respectively. diff --git a/docs/content/config/advanced/full-text-search.md b/docs/content/config/advanced/full-text-search.md index 2e29e496..45e7ffc7 100644 --- a/docs/content/config/advanced/full-text-search.md +++ b/docs/content/config/advanced/full-text-search.md @@ -16,7 +16,7 @@ The [dovecot-fts-xapian](https://github.com/grosjo/fts-xapian) plugin makes use The indexes will be stored as a subfolder named `xapian-indexes` inside your local `mail-data` folder (_`/var/mail` internally_). With the default settings, 10GB of email data may generate around 4GB of indexed data. -While indexing is memory intensive, you can configure the plugin to limit the amount of memory consumed by the index workers. With Xapian being small and fast, this plugin is a good choice for low memory environments (2GB) as compared to Solr. +While indexing is memory intensive, you can configure the plugin to limit the amount of memory consumed by the index workers. With Xapian being small and fast, this plugin is a good choice for low memory environments (2GB). #### Setup @@ -137,53 +137,6 @@ While indexing is memory intensive, you can configure the plugin to limit the am - ./docker-data/dms/cron/fts_xapian:/etc/cron.d/fts_xapian ``` - -### Solr - -The [dovecot-solr Plugin](https://wiki2.dovecot.org/Plugins/FTS/Solr) is used in conjunction with [Apache Solr](https://lucene.apache.org/solr/) running in a separate container. This is quite straightforward to setup using the following instructions. - -Solr is a mature and fast indexing backend that runs on the JVM. The indexes are relatively compact compared to the size of your total email. - -However, Solr also requires a fair bit of RAM. While Solr is [highly tuneable](https://solr.apache.org/guide/7_0/query-settings-in-solrconfig.html), it may require a bit of testing to get it right. - -#### Setup - -1. `compose.yaml`: - - ```yaml - solr: - image: lmmdock/dovecot-solr:latest - volumes: - - ./docker-data/dms/config/dovecot/solr-dovecot:/opt/solr/server/solr/dovecot - restart: always - - mailserver: - depends_on: - - solr - image: ghcr.io/docker-mailserver/docker-mailserver:latest - ... - volumes: - ... - - ./docker-data/dms/config/dovecot/10-plugin.conf:/etc/dovecot/conf.d/10-plugin.conf:ro - ... - ``` - -2. `./docker-data/dms/config/dovecot/10-plugin.conf`: - - ```conf - mail_plugins = $mail_plugins fts fts_solr - - plugin { - fts = solr - fts_autoindex = yes - fts_solr = url=http://solr:8983/solr/dovecot/ - } - ``` - -3. Recreate containers: `docker compose down ; docker compose up -d` - -4. Flag all user mailbox FTS indexes as invalid, so they are rescanned on demand when they are next searched: `docker compose exec mailserver doveadm fts rescan -A` - #### Further Discussion See [#905](https://github.com/docker-mailserver/docker-mailserver/issues/905) diff --git a/target/scripts/build/packages.sh b/target/scripts/build/packages.sh index 5360ae95..6b21cc72 100644 --- a/target/scripts/build/packages.sh +++ b/target/scripts/build/packages.sh @@ -135,7 +135,7 @@ function _install_dovecot() { local DOVECOT_PACKAGES=( dovecot-core dovecot-imapd dovecot-ldap dovecot-lmtpd dovecot-managesieved - dovecot-pop3d dovecot-sieve dovecot-solr + dovecot-pop3d dovecot-sieve ) # Dovecot packages for community supported features. From 411984928415e5cbfdd3c473f171dfd3c3383c96 Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Sat, 25 May 2024 19:49:45 +0200 Subject: [PATCH 100/267] update: Dovecot FTS Xapian from 1.5.5 to 1.7.12 (#4034) * update `compile.sh` and Dovecot FTS Xapian to 1.7.12 - I updated from 1.5.5. Moreover, I adjusted the script to have what I consider better style. * update Dockerfile to use recent updates * update CHANGELOG --- CHANGELOG.md | 1 + Dockerfile | 4 ++-- target/scripts/build/compile.sh | 22 +++++++++++++++------- 3 files changed, 18 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 04f8adb9..c791ac5a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -98,6 +98,7 @@ The most noteworthy change of this release is the update of the container's base - The symbol weights of SPF, DKIM and DMARC have been adjusted again. Fixes a bug and includes more appropriate combinations of symbols ([#3913](https://github.com/docker-mailserver/docker-mailserver/pull/3913), [#3923](https://github.com/docker-mailserver/docker-mailserver/pull/3923)) - **Dovecot:** - `logwatch` now filters out non-error logs related to the status of the `index-worker` process for FTS indexing. ([#4012](https://github.com/docker-mailserver/docker-mailserver/pull/4012)) + - updated FTS Xapian from version 1.5.5 to 1.7.12 ### Fixes diff --git a/Dockerfile b/Dockerfile index bf688ec1..854c2332 100644 --- a/Dockerfile +++ b/Dockerfile @@ -82,8 +82,8 @@ EOF # install fts_xapian plugin -COPY --from=stage-compile dovecot-fts-xapian-1.5.5_1.5.5_*.deb / -RUN dpkg -i /dovecot-fts-xapian-1.5.5_1.5.5_*.deb && rm /dovecot-fts-xapian-1.5.5_1.5.5_*.deb +COPY --from=stage-compile dovecot-fts-xapian-1.7.12_1.7.12_*.deb / +RUN dpkg -i /dovecot-fts-xapian-1.7.12_1.7.12_*.deb && rm /dovecot-fts-xapian-1.7.12_1.7.12_*.deb COPY target/dovecot/*.inc target/dovecot/*.conf /etc/dovecot/conf.d/ COPY target/dovecot/dovecot-purge.cron /etc/cron.d/dovecot-purge.disabled diff --git a/target/scripts/build/compile.sh b/target/scripts/build/compile.sh index 56feea72..4c6fba82 100644 --- a/target/scripts/build/compile.sh +++ b/target/scripts/build/compile.sh @@ -8,21 +8,29 @@ set -eE -u -o pipefail # shellcheck source=../helpers/log.sh source /usr/local/bin/helpers/log.sh +# shellcheck disable=SC2310 _log_level_is 'trace' && QUIET='-y' || QUIET='-qq' function _compile_dovecot_fts_xapian() { apt-get "${QUIET}" update - apt-get "${QUIET}" --no-install-recommends install automake libtool pkg-config libicu-dev libsqlite3-dev libxapian-dev make build-essential dh-make devscripts dovecot-dev - curl -Lso dovecot-fts-xapian.tar.gz https://github.com/grosjo/fts-xapian/releases/download/1.5.5/dovecot-fts-xapian-1.5.5.tar.gz - tar xzvf dovecot-fts-xapian.tar.gz - cd fts-xapian-1.5.5 - USER=root dh_make -p dovecot-fts-xapian-1.5.5 --single --native --copyright gpl2 -y + apt-get "${QUIET}" --no-install-recommends install \ + automake libtool pkg-config libicu-dev libsqlite3-dev libxapian-dev make build-essential dh-make devscripts dovecot-dev + + local XAPIAN_VERSION='1.7.12' + curl -sSfL -o dovecot-fts-xapian.tar.gz \ + "https://github.com/grosjo/fts-xapian/releases/download/${XAPIAN_VERSION}/dovecot-fts-xapian-${XAPIAN_VERSION}.tar.gz" + tar xf dovecot-fts-xapian.tar.gz + + cd "fts-xapian-${XAPIAN_VERSION}" + USER=root dh_make -p "dovecot-fts-xapian-${XAPIAN_VERSION}" --single --native --copyright gpl2 -y + rm debian/*.ex cp PACKAGES/DEB/control debian/ cp PACKAGES/DEB/changelog debian/ cp PACKAGES/DEB/compat debian/ - sed -i 's/1\.4\.11-6/1.5.5/g' debian/control - sed -i 's/1\.4\.11-6/1.5.5/g' debian/changelog + + sed -i -E "s|(dovecot-fts-xapian)-[1-9\.-]+|\1-${XAPIAN_VERSION}|g" debian/control + sed -i -E "s|(dovecot-fts-xapian)-[1-9\.-]+ \(.*\)(.*)|\1-${XAPIAN_VERSION} (${XAPIAN_VERSION})\2|g" debian/changelog debuild -us -uc -B | tee /tmp/debuild.log 2>&1 } From b222035112fa61fa6a9b6fe16a04b8005cc4e0a1 Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Sat, 25 May 2024 19:56:19 +0200 Subject: [PATCH 101/267] scripts: perform additional checks when updating/adding/deletting accounts (#4033) * normalize accounts to lowercase * update CHANGELOG * add test to verify bug fix works correctly --- CHANGELOG.md | 2 ++ .../helpers/database/manage/postfix-accounts.sh | 10 ++++++++++ .../tests/parallel/set3/mta/account_management.bats | 13 +++++++++++++ 3 files changed, 25 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index c791ac5a..e2285f6d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -70,6 +70,8 @@ The most noteworthy change of this release is the update of the container's base - `supervisor-app.conf` renamed to `dms-services.conf` - **Rspamd**: - the Redis history key has been changed in order to not incorporate the hostname of the container (which is desirable in Kubernetes environments) ([#3927](https://github.com/docker-mailserver/docker-mailserver/pull/3927)) +- **Account Management** + - addresses (accounts) are now normalized to lowercase automatically and a warning is logged in case uppercase letters are supplied ### Added diff --git a/target/scripts/helpers/database/manage/postfix-accounts.sh b/target/scripts/helpers/database/manage/postfix-accounts.sh index 8a2a6133..987254fc 100644 --- a/target/scripts/helpers/database/manage/postfix-accounts.sh +++ b/target/scripts/helpers/database/manage/postfix-accounts.sh @@ -13,6 +13,7 @@ function _manage_accounts() { local PASSWD=${4} _arg_expect_mail_account + _arg_check_mail_account case "${ACTION}" in ( 'create' | 'update' ) @@ -69,6 +70,15 @@ function _arg_expect_mail_account() { [[ ${MAIL_ACCOUNT} =~ .*\@.* ]] || { __usage ; _exit_with_error "'${MAIL_ACCOUNT}' should include the domain (eg: user@example.com)" ; } } +# Checks the mail account string, e.g. on uppercase letters. +function _arg_check_mail_account() { + if grep -q -E '[[:upper:]]+' <<< "${MAIL_ACCOUNT}"; then + local MAIL_ACCOUNT_NORMALIZED=${MAIL_ACCOUNT,,} + _log 'warn' "Mail account '${MAIL_ACCOUNT}' has uppercase letters and will be normalized to '${MAIL_ACCOUNT_NORMALIZED}'" + MAIL_ACCOUNT=${MAIL_ACCOUNT_NORMALIZED} + fi +} + function _account_should_not_exist_yet() { __account_already_exists && _exit_with_error "'${MAIL_ACCOUNT}' already exists" if [[ -f ${DATABASE_VIRTUAL} ]] && grep -q "^${MAIL_ACCOUNT}" "${DATABASE_VIRTUAL}"; then diff --git a/test/tests/parallel/set3/mta/account_management.bats b/test/tests/parallel/set3/mta/account_management.bats index bc97d710..f8d5f9de 100644 --- a/test/tests/parallel/set3/mta/account_management.bats +++ b/test/tests/parallel/set3/mta/account_management.bats @@ -74,6 +74,19 @@ function teardown_file() { _default_teardown ; } __should_add_new_user 'user3@domain.tld' } +@test "should add new user 'USeRx@domain.tld' as 'userx@domain.tld' into 'postfix-accounts.cf' and log a warning" { + local MAIL_ACCOUNT='USeRx@domain.tld' + local NORMALIZED_MAIL_ACCOUNT='userx@domain.tld' + + _run_in_container setup email add "${MAIL_ACCOUNT}" mypassword + assert_success + assert_output --partial "'USeRx@domain.tld' has uppercase letters and will be normalized to 'userx@domain.tld'" + + __check_mail_account_exists "${NORMALIZED_MAIL_ACCOUNT}" + assert_success + assert_output "${NORMALIZED_MAIL_ACCOUNT}" +} + # To catch mistakes from substring matching: @test "should add new user 'auser3@domain.tld' into 'postfix-accounts.cf'" { __should_add_new_user 'auser3@domain.tld' From 94bde85ac6273caf186b9efc94df414ea0a6319c Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sun, 26 May 2024 10:41:01 +0200 Subject: [PATCH 102/267] docs: updated `CONTRIBUTORS.md` (#4036) --- CONTRIBUTORS.md | 427 ++++++++++++++++++++++++------------------------ 1 file changed, 217 insertions(+), 210 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 1600440c..c627f750 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -329,47 +329,10 @@ Thanks goes to these wonderful people ✨ - - egavard + + robertdolca
- egavard - - - - - dashohoxha -
- dashohoxha - - - - - mathuin -
- mathuin - - - - - jamebus -
- jamebus - - - - - - - jsonn -
- jsonn - - - - - lukecyca -
- lukecyca + robertdolca @@ -380,10 +343,91 @@ Thanks goes to these wonderful people ✨ - - robertdolca + + lukecyca
- robertdolca + lukecyca + + + + + jsonn +
+ jsonn + + + + + + + jamebus +
+ jamebus + + + + + mathuin +
+ mathuin + + + + + dashohoxha +
+ dashohoxha + + + + + egavard +
+ egavard + + + + + voordev +
+ voordev + + + + + Zehir +
+ Zehir + + + + + + + guardiande +
+ guardiande + + + + + kamuri +
+ kamuri + + + + + davidszp +
+ davidszp + + + + + andreasgerstmayr +
+ andreasgerstmayr @@ -409,13 +453,6 @@ Thanks goes to these wonderful people ✨ mjung - - - andreasgerstmayr -
- andreasgerstmayr - - m-schmoock @@ -423,43 +460,6 @@ Thanks goes to these wonderful people ✨ m-schmoock - - - davidszp -
- davidszp - - - - - kamuri -
- kamuri - - - - - guardiande -
- guardiande - - - - - - - Zehir -
- Zehir - - - - - voordev -
- voordev - - Birkenstab @@ -498,24 +498,17 @@ Thanks goes to these wonderful people ✨ - - MakerMatrix + + weo
- MakerMatrix + weo - - pbek + + analogue
- pbek - - - - - reneploetz -
- reneploetz + analogue @@ -526,26 +519,47 @@ Thanks goes to these wonderful people ✨ - - analogue + + reneploetz
- analogue + reneploetz + + + + + pbek +
+ pbek - - weo + + MakerMatrix
- weo + MakerMatrix - - ubenmackin + + mpanneck
- ubenmackin + mpanneck + + + + + keslerm +
+ keslerm + + + + + willtho89 +
+ willtho89 @@ -562,6 +576,8 @@ Thanks goes to these wonderful people ✨ p-fruck + + rahilarious @@ -576,8 +592,6 @@ Thanks goes to these wonderful people ✨ Rillke - - bobbravo2 @@ -592,12 +606,28 @@ Thanks goes to these wonderful people ✨ r-pufky + + + yogo1212 +
+ yogo1212 + + vincentDcmps
vincentDcmps + + + + + + tbutter +
+ tbutter + @@ -620,8 +650,6 @@ Thanks goes to these wonderful people ✨ engelant - - j-marz @@ -636,6 +664,8 @@ Thanks goes to these wonderful people ✨ lokipo + + msheakoski @@ -658,33 +688,10 @@ Thanks goes to these wonderful people ✨ - - tbutter + + ubenmackin
- tbutter - - - - - - - yogo1212 -
- yogo1212 - - - - - willtho89 -
- willtho89 - - - - - mpanneck -
- mpanneck + ubenmackin @@ -701,6 +708,8 @@ Thanks goes to these wonderful people ✨ abh + + andrewlow @@ -708,8 +717,6 @@ Thanks goes to these wonderful people ✨ andrewlow - - aminvakil @@ -745,6 +752,8 @@ Thanks goes to these wonderful people ✨ theomega + + DuncanvR @@ -752,8 +761,6 @@ Thanks goes to these wonderful people ✨ DuncanvR - - emazzotta @@ -761,13 +768,6 @@ Thanks goes to these wonderful people ✨ emazzotta - - - keslerm -
- keslerm - - nueaf @@ -798,6 +798,13 @@ Thanks goes to these wonderful people ✨ + + + jiriks74 +
+ jiriks74 + + jedateach @@ -833,6 +840,8 @@ Thanks goes to these wonderful people ✨ millaguie + + jamesfryer @@ -840,8 +849,6 @@ Thanks goes to these wonderful people ✨ jamesfryer - - radicand @@ -877,6 +884,8 @@ Thanks goes to these wonderful people ✨ syl20bnr + + sylvaindumont @@ -884,8 +893,6 @@ Thanks goes to these wonderful people ✨ sylvaindumont - - TechnicLab @@ -921,6 +928,8 @@ Thanks goes to these wonderful people ✨ tweibert + + torus @@ -928,8 +937,6 @@ Thanks goes to these wonderful people ✨ torus - - VictorKoenders @@ -965,6 +972,8 @@ Thanks goes to these wonderful people ✨ vilisas + + forzagreen @@ -972,8 +981,6 @@ Thanks goes to these wonderful people ✨ forzagreen - - 42wim @@ -1009,6 +1016,8 @@ Thanks goes to these wonderful people ✨ pcqnt + + OrvilleQ @@ -1016,8 +1025,6 @@ Thanks goes to these wonderful people ✨ OrvilleQ - - ovidiucp @@ -1053,6 +1060,8 @@ Thanks goes to these wonderful people ✨ neuralp + + piwai @@ -1060,8 +1069,6 @@ Thanks goes to these wonderful people ✨ piwai - - remoe @@ -1097,6 +1104,8 @@ Thanks goes to these wonderful people ✨ MightySCollins + + 501st-alpha1 @@ -1104,8 +1113,6 @@ Thanks goes to these wonderful people ✨ 501st-alpha1 - - klamann @@ -1141,6 +1148,8 @@ Thanks goes to these wonderful people ✨ sjmudd + + simonsystem @@ -1148,8 +1157,6 @@ Thanks goes to these wonderful people ✨ simonsystem - - mchamplain @@ -1185,6 +1192,8 @@ Thanks goes to these wonderful people ✨ odinis + + okamidash @@ -1192,8 +1201,6 @@ Thanks goes to these wonderful people ✨ okamidash - - olaf-mandel @@ -1229,6 +1236,8 @@ Thanks goes to these wonderful people ✨ rhyst + + rmlhuk @@ -1236,8 +1245,6 @@ Thanks goes to these wonderful people ✨ rmlhuk - - rriski @@ -1273,6 +1280,8 @@ Thanks goes to these wonderful people ✨ squash + + strarsis @@ -1280,8 +1289,6 @@ Thanks goes to these wonderful people ✨ strarsis - - tamueller @@ -1317,6 +1324,8 @@ Thanks goes to these wonderful people ✨ worldworm + + Zepmann @@ -1324,8 +1333,6 @@ Thanks goes to these wonderful people ✨ Zepmann - - allddd @@ -1361,6 +1368,8 @@ Thanks goes to these wonderful people ✨ cternes + + dborowy @@ -1368,8 +1377,6 @@ Thanks goes to these wonderful people ✨ dborowy - - dimalo @@ -1405,6 +1412,8 @@ Thanks goes to these wonderful people ✨ helmutundarnold + + hnws @@ -1412,8 +1421,6 @@ Thanks goes to these wonderful people ✨ hnws - - i-C-o-d-e-r @@ -1449,6 +1456,8 @@ Thanks goes to these wonderful people ✨ paralax + + jpduyx @@ -1456,8 +1465,6 @@ Thanks goes to these wonderful people ✨ jpduyx - - landergate @@ -1493,6 +1500,8 @@ Thanks goes to these wonderful people ✨ 0xflotus + + arkanovicz @@ -1500,8 +1509,6 @@ Thanks goes to these wonderful people ✨ arkanovicz - - CBeerta @@ -1537,6 +1544,8 @@ Thanks goes to these wonderful people ✨ dbellavista + + danielvandenberg95 @@ -1544,8 +1553,6 @@ Thanks goes to these wonderful people ✨ danielvandenberg95 - - denisix @@ -1581,6 +1588,8 @@ Thanks goes to these wonderful people ✨ aydodo + + vedtam @@ -1588,8 +1597,6 @@ Thanks goes to these wonderful people ✨ vedtam - - edvorg @@ -1625,6 +1632,8 @@ Thanks goes to these wonderful people ✨ huncode + + gitfeber @@ -1632,8 +1641,6 @@ Thanks goes to these wonderful people ✨ gitfeber - - felixn @@ -1669,6 +1676,8 @@ Thanks goes to these wonderful people ✨ 20th + + 2b @@ -1676,8 +1685,6 @@ Thanks goes to these wonderful people ✨ 2b - - askz @@ -1713,6 +1720,8 @@ Thanks goes to these wonderful people ✨ kachkaev + + alexanderneu @@ -1720,8 +1729,6 @@ Thanks goes to these wonderful people ✨ alexanderneu - - ch3sh1r @@ -1757,6 +1764,8 @@ Thanks goes to these wonderful people ✨ iRhonin + + MrFreezeex @@ -1764,8 +1773,6 @@ Thanks goes to these wonderful people ✨ MrFreezeex - - arunvc @@ -1801,6 +1808,8 @@ Thanks goes to these wonderful people ✨ erdos4d + + crash7 @@ -1808,8 +1817,6 @@ Thanks goes to these wonderful people ✨ crash7 - - auchri @@ -1845,6 +1852,8 @@ Thanks goes to these wonderful people ✨ thechubbypanda + + KCrawley @@ -1852,8 +1861,6 @@ Thanks goes to these wonderful people ✨ KCrawley - - khuedoan @@ -1889,6 +1896,8 @@ Thanks goes to these wonderful people ✨ linhandev + + luke- @@ -1896,8 +1905,6 @@ Thanks goes to these wonderful people ✨ luke- - - LucidityCrash @@ -1933,6 +1940,8 @@ Thanks goes to these wonderful people ✨ dragetd + + michaeljensen @@ -1940,8 +1949,6 @@ Thanks goes to these wonderful people ✨ michaeljensen - - exhuma @@ -1977,6 +1984,8 @@ Thanks goes to these wonderful people ✨ mpldr + + naveensrinivasan @@ -1984,8 +1993,6 @@ Thanks goes to these wonderful people ✨ naveensrinivasan - - furstblumier @@ -2021,6 +2028,8 @@ Thanks goes to these wonderful people ✨ harryyoud + + HeySora @@ -2028,8 +2037,6 @@ Thanks goes to these wonderful people ✨ HeySora - - sirgantrithon @@ -2065,6 +2072,8 @@ Thanks goes to these wonderful people ✨ JacksonZ03 + + JamBalaya56562 @@ -2072,8 +2081,6 @@ Thanks goes to these wonderful people ✨ JamBalaya56562 - - jcalfee @@ -2109,6 +2116,8 @@ Thanks goes to these wonderful people ✨ jessp01 + + JiLleON @@ -2116,8 +2125,6 @@ Thanks goes to these wonderful people ✨ JiLleON - - jirislav From a96a4e2768216167b7c80f28a3d4323ca1db0244 Mon Sep 17 00:00:00 2001 From: Tobias Knecht Date: Tue, 28 May 2024 13:20:15 +0200 Subject: [PATCH 103/267] Abusix docs links update. (#4038) --- docs/content/config/security/rspamd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/config/security/rspamd.md b/docs/content/config/security/rspamd.md index b0f65990..a98d3123 100644 --- a/docs/content/config/security/rspamd.md +++ b/docs/content/config/security/rspamd.md @@ -273,7 +273,7 @@ While _Abusix_ can be integrated into Postfix, Postscreen and a multitude of oth [www::rbl-vs-dnsbl]: https://forum.eset.com/topic/25277-dnsbl-vs-rbl-mail-security/?do=findComment&comment=119818 [abusix-web]: https://abusix.com/ -[abusix-docs::rspamd-integration]: https://docs.abusix.com/abusix-mail-intelligence/gbG8EcJ3x3fSUv8cMZLiwA/getting-started/dmw9dcwSGSNQiLTssFAnBW#rspamd +[abusix-docs::rspamd-integration]: https://abusix.com/docs/rspamd/ [spamhaus::faq::dnsbl-usage]: https://www.spamhaus.org/faq/section/DNSBL%20Usage#365 [dms-repo::rspamd-actions-config]: https://github.com/docker-mailserver/docker-mailserver/blob/v14.0.0/target/rspamd/local.d/actions.conf From 95d965fb768d828667ff1de53cc3b7ba8260299e Mon Sep 17 00:00:00 2001 From: Guillaume VARA Date: Tue, 28 May 2024 17:52:30 +0200 Subject: [PATCH 104/267] docs(k8s): Advise `externalTrafficPolicy: Local` if no PROXY protocol configured (#4039) Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- docs/content/config/advanced/kubernetes.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/content/config/advanced/kubernetes.md b/docs/content/config/advanced/kubernetes.md index 268094a6..337f1e36 100644 --- a/docs/content/config/advanced/kubernetes.md +++ b/docs/content/config/advanced/kubernetes.md @@ -140,6 +140,9 @@ If using our Helm chart is not viable for you, here is some guidance to start wi app: mailserver spec: + # `Local` is most likely required, otherwise every incoming request would be identified by the external IP, + # which will get banned by Fail2Ban when monitored services are not configured for PROXY protocol + externalTrafficPolicy: Local type: LoadBalancer selector: From f8b3f402763a8ac1112971ec53a2041398e7ce4c Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Wed, 29 May 2024 02:28:51 +0200 Subject: [PATCH 105/267] scripts: update log format (#4035) Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- CHANGELOG.md | 2 ++ target/scripts/check-for-changes.sh | 30 +++++++++---------- target/scripts/helpers/log.sh | 28 +++++++++-------- target/scripts/update-check.sh | 12 ++++---- .../set1/spam_virus/rspamd_partly.bats | 2 +- test/tests/parallel/set1/tls/dhparams.bats | 2 +- 6 files changed, 40 insertions(+), 36 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e2285f6d..30511983 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -43,6 +43,8 @@ The most noteworthy change of this release is the update of the container's base - **Removed support for Solr integration:** ([#4025](https://github.com/docker-mailserver/docker-mailserver/pull/4025)) - This was a community contributed feature for FTS (Full Text Search), the docs advise using an image that has not been maintained for over 2 years and lacks ARM64 support. Based on user engagement over the years this feature has very niche value to continue to support, thus is being removed. - If you use Solr, support can be restored if you're willing to contribute docs for the feature that resolves the concerns raised +- **Log**: + - The format of DMS specific logs (_from our scripts, not running services_) has been changed. The new format is ` : ` ([#4035](https://github.com/docker-mailserver/docker-mailserver/pull/4035)) - **rsyslog:** - Debian 12 adjusted the `rsyslog` configuration for the default file template from `RSYSLOG_TraditionalFileFormat` to `RSYSLOG_FileFormat` (_upstream default since 2012_). This change may affect you if you have any monitoring / analysis of log output (_eg: `mail.log` / `docker logs`_). - The two formats are roughly equivalent to [RFC 3164](https://www.rfc-editor.org/rfc/rfc3164)) and [RFC 5424](https://datatracker.ietf.org/doc/html/rfc5424#section-1) respectively. diff --git a/target/scripts/check-for-changes.sh b/target/scripts/check-for-changes.sh index 3f268710..bf5cd90b 100755 --- a/target/scripts/check-for-changes.sh +++ b/target/scripts/check-for-changes.sh @@ -7,7 +7,7 @@ # shellcheck source=./helpers/index.sh source /usr/local/bin/helpers/index.sh -_log_with_date 'debug' 'Starting changedetector' +_log 'debug' 'Starting changedetector' # ATTENTION: Do not remove! # This script requires some environment variables to be properly set. @@ -30,9 +30,9 @@ if [[ ! -f ${CHKSUM_FILE} ]]; then _exit_with_error "'${CHKSUM_FILE}' is missing" 0 fi -_log_with_date 'trace' "Using postmaster address '${POSTMASTER_ADDRESS}'" +_log 'trace' "Using postmaster address '${POSTMASTER_ADDRESS}'" -_log_with_date 'debug' "Changedetector is ready" +_log 'debug' "Changedetector is ready" function _check_for_changes() { # get chksum and check it, no need to lock config yet @@ -44,7 +44,7 @@ function _check_for_changes() { # 1 – files differ # 2 – inaccessible or missing argument if [[ ${?} -eq 1 ]]; then - _log_with_date 'info' 'Change detected' + _log 'info' 'Change detected' _create_lock # Shared config safety lock local CHANGED @@ -52,7 +52,7 @@ function _check_for_changes() { _handle_changes _remove_lock - _log_with_date 'debug' 'Completed handling of detected change' + _log 'debug' 'Completed handling of detected change' # mark changes as applied mv "${CHKSUM_FILE}.new" "${CHKSUM_FILE}" @@ -64,7 +64,7 @@ function _handle_changes() { local VHOST_UPDATED=0 # These two configs are the source for /etc/postfix/vhost (managed mail domains) if [[ ${CHANGED} =~ ${DMS_DIR}/postfix-(accounts|virtual).cf ]]; then - _log_with_date 'trace' 'Regenerating vhosts (Postfix)' + _log 'trace' 'Regenerating vhosts (Postfix)' # Regenerate via `helpers/postfix.sh`: _create_postfix_vhost @@ -75,7 +75,7 @@ function _handle_changes() { _postfix_dovecot_changes _rspamd_changes - _log_with_date 'debug' 'Reloading services due to detected changes' + _log 'debug' 'Reloading services due to detected changes' [[ ${ENABLE_AMAVIS} -eq 1 ]] && _reload_amavis _reload_postfix @@ -119,7 +119,7 @@ function _postfix_dovecot_changes() { || [[ ${CHANGED} =~ ${DMS_DIR}/dovecot-quotas.cf ]] \ || [[ ${CHANGED} =~ ${DMS_DIR}/dovecot-masters.cf ]] then - _log_with_date 'trace' 'Regenerating accounts (Dovecot + Postfix)' + _log 'trace' 'Regenerating accounts (Dovecot + Postfix)' [[ ${SMTP_ONLY} -ne 1 ]] && _create_accounts fi @@ -131,7 +131,7 @@ function _postfix_dovecot_changes() { || [[ ${CHANGED} =~ ${DMS_DIR}/postfix-relaymap.cf ]] \ || [[ ${CHANGED} =~ ${DMS_DIR}/postfix-sasl-password.cf ]] then - _log_with_date 'trace' 'Regenerating relay config (Postfix)' + _log 'trace' 'Regenerating relay config (Postfix)' _process_relayhost_configs fi @@ -159,14 +159,14 @@ function _ssl_changes() { || [[ ${CHANGED} =~ ${SSL_ALT_CERT_PATH:-${REGEX_NEVER_MATCH}} ]] \ || [[ ${CHANGED} =~ ${SSL_ALT_KEY_PATH:-${REGEX_NEVER_MATCH}} ]] then - _log_with_date 'debug' 'Manual certificates have changed - extracting certificates' + _log 'debug' 'Manual certificates have changed - extracting certificates' _setup_ssl fi # `acme.json` is only relevant to Traefik, and is where it stores the certificates it manages. # When a change is detected it's assumed to be a possible cert renewal that needs to be # extracted for `docker-mailserver` services to adjust to. elif [[ ${CHANGED} =~ /etc/letsencrypt/acme.json ]]; then - _log_with_date 'debug' "'/etc/letsencrypt/acme.json' has changed - extracting certificates" + _log 'debug' "'/etc/letsencrypt/acme.json' has changed - extracting certificates" _setup_ssl # Prevent an unnecessary change detection from the newly extracted cert files by updating their hashes in advance: @@ -188,23 +188,23 @@ function _rspamd_changes() { # "${RSPAMD_DMS_D}/override.d" if [[ ${CHANGED} =~ ${RSPAMD_DMS_OVERRIDE_D}/.* ]]; then - _log_with_date 'trace' 'Rspamd - Copying configuration overrides' + _log 'trace' 'Rspamd - Copying configuration overrides' rm "${RSPAMD_OVERRIDE_D}"/* cp "${RSPAMD_DMS_OVERRIDE_D}"/* "${RSPAMD_OVERRIDE_D}" fi # "${RSPAMD_DMS_D}/custom-commands.conf" if [[ ${CHANGED} =~ ${RSPAMD_DMS_CUSTOM_COMMANDS_F} ]]; then - _log_with_date 'trace' 'Rspamd - Generating new configuration from custom commands' + _log 'trace' 'Rspamd - Generating new configuration from custom commands' _rspamd_handle_user_modules_adjustments fi # "${RSPAMD_DMS_D}/dkim" if [[ ${CHANGED} =~ ${RSPAMD_DMS_DKIM_D} ]]; then - _log_with_date 'trace' 'Rspamd - DKIM files updated' + _log 'trace' 'Rspamd - DKIM files updated' fi - _log_with_date 'debug' 'Rspamd configuration has changed - restarting service' + _log 'debug' 'Rspamd configuration has changed - restarting service' supervisorctl restart rspamd fi } diff --git a/target/scripts/helpers/log.sh b/target/scripts/helpers/log.sh index d771a3f5..d98f96a8 100644 --- a/target/scripts/helpers/log.sh +++ b/target/scripts/helpers/log.sh @@ -53,8 +53,7 @@ function _log() { return 1 fi - local LEVEL_AS_INT - local MESSAGE="${RESET}[" + local LEVEL_AS_INT LOG_COLOR LOG_LEVEL_NAME MESSAGE case "$(_get_log_level_or_default)" in ( 'trace' ) LEVEL_AS_INT=5 ;; @@ -67,27 +66,35 @@ function _log() { case "${1}" in ( 'trace' ) [[ ${LEVEL_AS_INT} -ge 5 ]] || return 0 - MESSAGE+=" ${CYAN}TRACE " + LOG_COLOR='CYAN' + LOG_LEVEL_NAME='TRACE' ;; ( 'debug' ) [[ ${LEVEL_AS_INT} -ge 4 ]] || return 0 - MESSAGE+=" ${PURPLE}DEBUG " + LOG_COLOR='PURPLE' + LOG_LEVEL_NAME='DEBUG' ;; ( 'info' ) [[ ${LEVEL_AS_INT} -ge 3 ]] || return 0 - MESSAGE+=" ${BLUE}INF " + LOG_COLOR='BLUE' + # the whitespace is intentional (for alignment purposes) + LOG_LEVEL_NAME='INFO ' ;; ( 'warn' ) [[ ${LEVEL_AS_INT} -ge 2 ]] || return 0 - MESSAGE+=" ${LYELLOW}WARNING " + LOG_COLOR='LYELLOW' + # the whitespace is intentional (for alignment purposes) + LOG_LEVEL_NAME='WARN ' ;; ( 'error' ) [[ ${LEVEL_AS_INT} -ge 1 ]] || return 0 - MESSAGE+=" ${LRED}ERROR " ;; + LOG_COLOR='LRED' + LOG_LEVEL_NAME='ERROR' + ;; ( * ) _log 'error' "Call to '_log' with invalid log level argument '${1}'" @@ -95,7 +102,7 @@ function _log() { ;; esac - MESSAGE+="${RESET}] ${2}" + MESSAGE="$(date --rfc-3339='seconds') ${!LOG_COLOR}${LOG_LEVEL_NAME}${RESET} $(basename "${0}"): ${2}" if [[ ${1} =~ ^(warn|error)$ ]]; then echo -e "${MESSAGE}" >&2 @@ -104,11 +111,6 @@ function _log() { fi } -# Like `_log` but adds a timestamp in front of the message. -function _log_with_date() { - _log "${1}" "$(date '+%Y-%m-%d %H:%M:%S') ${2}" -} - # Get the value of the environment variable LOG_LEVEL if # it is set. Otherwise, try to query the common environment # variables file. If this does not yield a value either, diff --git a/target/scripts/update-check.sh b/target/scripts/update-check.sh index 257fc37d..f00bfcb3 100755 --- a/target/scripts/update-check.sh +++ b/target/scripts/update-check.sh @@ -10,8 +10,8 @@ CHANGELOG_URL='https://github.com/docker-mailserver/docker-mailserver/blob/maste # check for correct syntax # number + suffix. suffix must be 's' for seconds, 'm' for minutes, 'h' for hours or 'd' for days. if [[ ! ${UPDATE_CHECK_INTERVAL} =~ ^[0-9]+[smhd]{1}$ ]]; then - _log_with_date 'warn' "Invalid 'UPDATE_CHECK_INTERVAL' value '${UPDATE_CHECK_INTERVAL}'" - _log_with_date 'warn' 'Falling back to daily update checks' + _log 'warn' "Invalid 'UPDATE_CHECK_INTERVAL' value '${UPDATE_CHECK_INTERVAL}'" + _log 'warn' 'Falling back to daily update checks' UPDATE_CHECK_INTERVAL='1d' fi @@ -22,7 +22,7 @@ while true; do # did we get a valid response? if [[ ${LATEST} =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then - _log_with_date 'debug' 'Remote version information fetched' + _log 'debug' 'Remote version information fetched' # compare versions if dpkg --compare-versions "${VERSION}" lt "${LATEST}"; then @@ -38,15 +38,15 @@ Latest version: ${LATEST} Changelog: ${CHANGELOG_URL}#END EOF - _log_with_date 'info' "Update available [ ${VERSION} --> ${LATEST} ]" + _log 'info' "Update available [ ${VERSION} --> ${LATEST} ]" # only notify once echo "${MAIL}" | mail -s "Mailserver update available! [ ${VERSION} --> ${LATEST} ]" "${POSTMASTER_ADDRESS}" && exit 0 else - _log_with_date 'info' 'No update available' + _log 'info' 'No update available' fi else - _log_with_date 'warn' 'Update check failed' + _log 'warn' 'Update check failed' fi # check again in 'UPDATE_CHECK_INTERVAL' time diff --git a/test/tests/parallel/set1/spam_virus/rspamd_partly.bats b/test/tests/parallel/set1/spam_virus/rspamd_partly.bats index b647f562..a1afc9a3 100644 --- a/test/tests/parallel/set1/spam_virus/rspamd_partly.bats +++ b/test/tests/parallel/set1/spam_virus/rspamd_partly.bats @@ -44,7 +44,7 @@ function teardown_file() { _default_teardown ; } run docker logs "${CONTAINER_NAME}" assert_success for SERVICE in 'Amavis/SA' 'OpenDKIM' 'OpenDMARC' 'policyd-spf'; do - assert_output --regexp ".*WARNING.*Running ${SERVICE} & Rspamd at the same time is discouraged" + assert_output --regexp ".*WARN.*Running ${SERVICE} & Rspamd at the same time is discouraged" done } diff --git a/test/tests/parallel/set1/tls/dhparams.bats b/test/tests/parallel/set1/tls/dhparams.bats index 8b3047d1..86586f28 100644 --- a/test/tests/parallel/set1/tls/dhparams.bats +++ b/test/tests/parallel/set1/tls/dhparams.bats @@ -50,7 +50,7 @@ function teardown() { _default_teardown ; } # Should emit a warning: run docker logs "${CONTAINER_NAME}" assert_success - assert_output --partial '[ WARNING ] Using self-generated dhparams is considered insecure - unless you know what you are doing, please remove' + assert_output --partial 'Using self-generated dhparams is considered insecure - unless you know what you are doing, please remove' } # Ensures the docker image services (Postfix and Dovecot) have the expected DH files: From e78d5c61eefcef4bc47695e4879c6ba4f0787da8 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Mon, 3 Jun 2024 08:35:07 +0200 Subject: [PATCH 106/267] docs: updated `CONTRIBUTORS.md` (#4043) --- CONTRIBUTORS.md | 278 +++++++++++++++++++++++++----------------------- 1 file changed, 147 insertions(+), 131 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index c627f750..67bd247a 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -34,13 +34,6 @@ Thanks goes to these wonderful people ✨ NorseGaud - - - williamdes -
- williamdes - - wernerfred @@ -48,8 +41,6 @@ Thanks goes to these wonderful people ✨ wernerfred - - georglauterbach @@ -57,6 +48,8 @@ Thanks goes to these wonderful people ✨ georglauterbach + + tomav @@ -92,8 +85,6 @@ Thanks goes to these wonderful people ✨ Josef-Friedrich - - johansmitsnl @@ -101,6 +92,8 @@ Thanks goes to these wonderful people ✨ johansmitsnl + + youtous @@ -136,8 +129,6 @@ Thanks goes to these wonderful people ✨ ap-wtioit - - 00angus @@ -145,6 +136,8 @@ Thanks goes to these wonderful people ✨ 00angus + + alinmear @@ -180,14 +173,21 @@ Thanks goes to these wonderful people ✨ swiesend - - stonemaster
stonemaster + + + + + + williamdes +
+ williamdes + @@ -687,29 +687,6 @@ Thanks goes to these wonderful people ✨ frugan-dev - - - ubenmackin -
- ubenmackin - - - - - craue -
- craue - - - - - abh -
- abh - - - - andrewlow @@ -730,6 +707,29 @@ Thanks goes to these wonderful people ✨
elbracht + + + + + + abh +
+ abh + + + + + craue +
+ craue + + + + + ubenmackin +
+ ubenmackin + @@ -813,17 +813,17 @@ Thanks goes to these wonderful people ✨ - - fl42 + + millaguie
- fl42 + millaguie - - ipernet + + jamesfryer
- ipernet + jamesfryer @@ -834,26 +834,19 @@ Thanks goes to these wonderful people ✨ - - millaguie + + ipernet
- millaguie + ipernet - - jamesfryer + + fl42
- jamesfryer - - - - - radicand -
- radicand + fl42 @@ -884,8 +877,6 @@ Thanks goes to these wonderful people ✨ syl20bnr - - sylvaindumont @@ -893,6 +884,8 @@ Thanks goes to these wonderful people ✨ sylvaindumont + + TechnicLab @@ -921,6 +914,13 @@ Thanks goes to these wonderful people ✨ tobiabocchi + + + tknecht +
+ tknecht + + tweibert @@ -995,6 +995,20 @@ Thanks goes to these wonderful people ✨ ShiriNmi1520 + + + neuralp +
+ neuralp + + + + + radicand +
+ radicand + + nilshoell @@ -1002,6 +1016,8 @@ Thanks goes to these wonderful people ✨ nilshoell + + nknapp @@ -1016,8 +1032,6 @@ Thanks goes to these wonderful people ✨ pcqnt - - OrvilleQ @@ -1046,6 +1060,8 @@ Thanks goes to these wonderful people ✨ p3dda + + peter-hartmann @@ -1053,15 +1069,6 @@ Thanks goes to these wonderful people ✨ peter-hartmann - - - neuralp -
- neuralp - - - - piwai @@ -1097,6 +1104,8 @@ Thanks goes to these wonderful people ✨ norrs + + MightySCollins @@ -1104,8 +1113,6 @@ Thanks goes to these wonderful people ✨ MightySCollins - - 501st-alpha1 @@ -1141,6 +1148,8 @@ Thanks goes to these wonderful people ✨ shyim + + sjmudd @@ -1148,8 +1157,6 @@ Thanks goes to these wonderful people ✨ sjmudd - - simonsystem @@ -1185,6 +1192,8 @@ Thanks goes to these wonderful people ✨ mplx + + odinis @@ -1192,8 +1201,6 @@ Thanks goes to these wonderful people ✨ odinis - - okamidash @@ -1229,6 +1236,8 @@ Thanks goes to these wonderful people ✨ presocratics + + rhyst @@ -1236,8 +1245,6 @@ Thanks goes to these wonderful people ✨ rhyst - - rmlhuk @@ -1273,6 +1280,8 @@ Thanks goes to these wonderful people ✨ sportshead + + squash @@ -1280,8 +1289,6 @@ Thanks goes to these wonderful people ✨ squash - - strarsis @@ -1317,6 +1324,8 @@ Thanks goes to these wonderful people ✨ wolkenschieber + + worldworm @@ -1324,8 +1333,6 @@ Thanks goes to these wonderful people ✨ worldworm - - Zepmann @@ -1361,6 +1368,8 @@ Thanks goes to these wonderful people ✨ brainkiller + + cternes @@ -1368,8 +1377,6 @@ Thanks goes to these wonderful people ✨ cternes - - dborowy @@ -1405,6 +1412,8 @@ Thanks goes to these wonderful people ✨ ghnp5 + + helmutundarnold @@ -1412,8 +1421,6 @@ Thanks goes to these wonderful people ✨ helmutundarnold - - hnws @@ -1449,6 +1456,8 @@ Thanks goes to these wonderful people ✨ jjtt + + paralax @@ -1456,8 +1465,6 @@ Thanks goes to these wonderful people ✨ paralax - - jpduyx @@ -1492,13 +1499,6 @@ Thanks goes to these wonderful people ✨
matrixes - - - - 0xflotus -
- 0xflotus - @@ -1662,6 +1662,22 @@ Thanks goes to these wonderful people ✨ froks + + + JOduMonT +
+ JOduMonT + + + + + 0xflotus +
+ 0xflotus + + + + ifokeev @@ -1676,8 +1692,6 @@ Thanks goes to these wonderful people ✨ 20th - - 2b @@ -1706,6 +1720,8 @@ Thanks goes to these wonderful people ✨ acch + + vifino @@ -1720,8 +1736,6 @@ Thanks goes to these wonderful people ✨ kachkaev - - alexanderneu @@ -1750,6 +1764,8 @@ Thanks goes to these wonderful people ✨ groupmsl + + green-anger @@ -1764,8 +1780,6 @@ Thanks goes to these wonderful people ✨ iRhonin - - MrFreezeex @@ -1794,6 +1808,8 @@ Thanks goes to these wonderful people ✨ baxerus + + spock @@ -1808,8 +1824,6 @@ Thanks goes to these wonderful people ✨ erdos4d - - crash7 @@ -1824,13 +1838,6 @@ Thanks goes to these wonderful people ✨ auchri - - - fkefer -
- fkefer - - Kaan88 @@ -1845,6 +1852,8 @@ Thanks goes to these wonderful people ✨ akkumar + + thechubbypanda @@ -1852,8 +1861,6 @@ Thanks goes to these wonderful people ✨ thechubbypanda - - KCrawley @@ -1889,6 +1896,8 @@ Thanks goes to these wonderful people ✨ leowinterde + + linhandev @@ -1896,8 +1905,6 @@ Thanks goes to these wonderful people ✨ linhandev - - luke- @@ -1933,6 +1940,8 @@ Thanks goes to these wonderful people ✨ maxemann96 + + dragetd @@ -1940,8 +1949,6 @@ Thanks goes to these wonderful people ✨ dragetd - - michaeljensen @@ -1977,6 +1984,8 @@ Thanks goes to these wonderful people ✨ MohammedNoureldin + + mpldr @@ -1984,8 +1993,6 @@ Thanks goes to these wonderful people ✨ mpldr - - naveensrinivasan @@ -1993,6 +2000,13 @@ Thanks goes to these wonderful people ✨ naveensrinivasan + + + fkefer +
+ fkefer + + furstblumier @@ -2014,6 +2028,8 @@ Thanks goes to these wonderful people ✨ glandais + + GiovanH @@ -2021,6 +2037,13 @@ Thanks goes to these wonderful people ✨ GiovanH + + + Amphaal +
+ Amphaal + + harryyoud @@ -2028,8 +2051,6 @@ Thanks goes to these wonderful people ✨ harryyoud - - HeySora @@ -2051,6 +2072,8 @@ Thanks goes to these wonderful people ✨ Influencer + + in-seo @@ -2072,8 +2095,6 @@ Thanks goes to these wonderful people ✨ JacksonZ03 - - JamBalaya56562 @@ -2095,6 +2116,8 @@ Thanks goes to these wonderful people ✨ mivek + + init-js @@ -2116,8 +2139,6 @@ Thanks goes to these wonderful people ✨ jessp01 - - JiLleON @@ -2139,19 +2160,14 @@ Thanks goes to these wonderful people ✨ jmccl + + jurekbarth
jurekbarth - - - - JOduMonT -
- JOduMonT - From eb165ded65859794bbeca66e529a3466ceee298b Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Wed, 5 Jun 2024 17:14:35 +0200 Subject: [PATCH 107/267] prepare v14.0.0 release (#4013) Co-authored-by: Casper --- CHANGELOG.md | 4 +++- VERSION | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 30511983..779c92c1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,10 +2,12 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased](https://github.com/docker-mailserver/docker-mailserver/compare/v13.3.1...HEAD) +## [Unreleased](https://github.com/docker-mailserver/docker-mailserver/compare/v14.0.0...HEAD) > **Note**: Changes and additions listed here are contained in the `:edge` image tag. These changes may not be as stable as released changes. +## [v14.0.0](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v14.0.0) + The most noteworthy change of this release is the update of the container's base image from Debian 11 ("Bullseye") to Debian 12 ("Bookworm"). This update alone involves breaking changes and requires a careful update! ### Breaking diff --git a/VERSION b/VERSION index c3d10c59..4b964e96 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -13.3.1 +14.0.0 From f1df81a7e43cf2698bd3bd95e2e413bd5acc96e7 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Fri, 7 Jun 2024 18:42:05 +1200 Subject: [PATCH 108/267] docs: `mailserver.env` improve description for `SPAM_SUBJECT` (#4050) Co-authored-by: Casper --- mailserver.env | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mailserver.env b/mailserver.env index 0d00de0a..1ec88dd2 100644 --- a/mailserver.env +++ b/mailserver.env @@ -130,8 +130,9 @@ ENABLE_IMAP=1 # **0** => Disabled ENABLE_CLAMAV=0 -# Add the value as a prefix to the mail subject when spam is detected. -# NOTE: By default spam is delivered to a junk folder, reducing the value of a subject prefix for spam. +# Add the value of this ENV as a prefix to the mail subject when spam is detected. +# NOTE: This subject prefix may be redundant (by default spam is delivered to a junk folder). +# It provides value when your junk mail is stored alongside legitimate mail instead of a separate location (like with `SPAMASSASSIN_SPAM_TO_INBOX=1` or `MOVE_SPAM_TO_JUNK=0` or a POP3 only setup, without IMAP). # NOTE: When not using Docker Compose, other CRI may not support quote-wrapping the value here to preserve any trailing white-space. SPAM_SUBJECT= From 18d9d1adcc479b7a38a31780a363c8ed272da32b Mon Sep 17 00:00:00 2001 From: Casper Date: Sat, 8 Jun 2024 13:43:25 +0200 Subject: [PATCH 109/267] Fail2ban 1.1.0 (#4045) --- CHANGELOG.md | 5 ++++ Dockerfile | 2 -- target/scripts/build/packages.sh | 48 ++++++++++++++++++++++++++------ 3 files changed, 45 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 779c92c1..8fda6aab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,11 @@ All notable changes to this project will be documented in this file. The format > **Note**: Changes and additions listed here are contained in the `:edge` image tag. These changes may not be as stable as released changes. +### Updates + +- **Fail2ban**: + - Bump version to [1.1.0](https://github.com/fail2ban/fail2ban/releases/tag/1.1.0). For more information, check the [changelog](https://github.com/fail2ban/fail2ban/blob/1.1.0/ChangeLog). + ## [v14.0.0](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v14.0.0) The most noteworthy change of this release is the update of the container's base image from Debian 11 ("Bullseye") to Debian 12 ("Bookworm"). This update alone involves breaking changes and requires a careful update! diff --git a/Dockerfile b/Dockerfile index 854c2332..da9aa83d 100644 --- a/Dockerfile +++ b/Dockerfile @@ -185,8 +185,6 @@ COPY target/fail2ban/fail2ban.d/fixes.local /etc/fail2ban/fail2ban.d/fixes.local RUN <&1 + + curl -Lkso fail2ban.deb "${FAIL2BAN_DEB_URL}" + curl -Lkso fail2ban.deb.asc "${FAIL2BAN_DEB_ASC_URL}" + + FINGERPRINT=$(LANG=C gpg --verify fail2ban.deb.asc fail2ban.deb |& sed -n 's#Primary key fingerprint: \(.*\)#\1#p') + + if [[ -z ${FINGERPRINT} ]]; then + echo 'ERROR: Invalid GPG signature!' >&2 + exit 1 + fi + + if [[ ${FINGERPRINT} != "${FAIL2BAN_GPG_FINGERPRINT}" ]]; then + echo "ERROR: Wrong GPG fingerprint!" >&2 + exit 1 + fi + + dpkg -i fail2ban.deb 2>&1 + rm fail2ban.deb fail2ban.deb.asc + + _log 'debug' 'Patching Fail2ban to enable network bans' + # Enable network bans + # https://github.com/docker-mailserver/docker-mailserver/issues/2669 + # https://github.com/fail2ban/fail2ban/issues/3125 + sedfile -i -r 's/^_nft_add_set = .+/_nft_add_set = add set \\{ type \\; flags interval\\; \\}/' /etc/fail2ban/action.d/nftables.conf +} + function _post_installation_steps() { _log 'debug' 'Running post-installation steps (cleanup)' _log 'debug' 'Deleting sensitive files (secrets)' @@ -189,11 +225,6 @@ function _post_installation_steps() { _log 'trace' 'Removing leftovers from APT' apt-get "${QUIET}" clean rm -rf /var/lib/apt/lists/* - - _log 'debug' 'Patching Fail2ban to enable network bans' - # Enable network bans - # https://github.com/docker-mailserver/docker-mailserver/issues/2669 - sedfile -i -r 's/^_nft_add_set = .+/_nft_add_set = add set
\\{ type \\; flags interval\\; \\}/' /etc/fail2ban/action.d/nftables.conf } _pre_installation_steps @@ -202,4 +233,5 @@ _install_postfix _install_packages _install_dovecot _install_rspamd +_install_fail2ban _post_installation_steps From 5c798e68294f1e3c0fc133a1e4b6c745373836b8 Mon Sep 17 00:00:00 2001 From: mmehnert Date: Sun, 9 Jun 2024 15:12:49 +0200 Subject: [PATCH 110/267] Update logwatch ignore.conf to exclude Xapian messages about pending documents (#4060) Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- CHANGELOG.md | 4 ++++ target/logwatch/ignore.conf | 1 + 2 files changed, 5 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8fda6aab..1aa3391e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,10 @@ All notable changes to this project will be documented in this file. The format - **Fail2ban**: - Bump version to [1.1.0](https://github.com/fail2ban/fail2ban/releases/tag/1.1.0). For more information, check the [changelog](https://github.com/fail2ban/fail2ban/blob/1.1.0/ChangeLog). +#### Fixes +- **Dovecot:** + - `logwatch` Update logwatch `ignore.conf` to exclude Xapian messages about pending documents + ## [v14.0.0](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v14.0.0) The most noteworthy change of this release is the update of the container's base image from Debian 11 ("Bullseye") to Debian 12 ("Bookworm"). This update alone involves breaking changes and requires a careful update! diff --git a/target/logwatch/ignore.conf b/target/logwatch/ignore.conf index 5b7e9312..e3138434 100644 --- a/target/logwatch/ignore.conf +++ b/target/logwatch/ignore.conf @@ -1,2 +1,3 @@ # ignore output from dovecot-fts-xapian about successfully indexed messages dovecot: indexer-worker\([^\)]+\).*Indexed +dovecot: indexer-worker\([^\)]+\).*FTS Xapian: Waiting for all pending documents to be processed From e6713a0aecb0ee449862e2a9d9c76200872c461e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 11 Jun 2024 14:33:57 +1200 Subject: [PATCH 111/267] chore(deps): Bump docker/build-push-action from 5.3.0 to 5.4.0 (#4062) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 5.3.0 to 5.4.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/v5.3.0...v5.4.0) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index da5d51c1..7f1b7c98 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -83,7 +83,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v5.3.0 + uses: docker/build-push-action@v5.4.0 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index ead38a3f..bd54c428 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v5.3.0 + uses: docker/build-push-action@v5.4.0 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 4e648e32..a021e79f 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v5.3.0 + uses: docker/build-push-action@v5.4.0 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 2f1a3803..2b15929b 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v5.3.0 + uses: docker/build-push-action@v5.4.0 with: context: . tags: mailserver-testing:ci From 8e32635993d196c2268c7bd97f8f101a66e2e52d Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Sun, 16 Jun 2024 00:26:33 +1200 Subject: [PATCH 112/267] docs: Document fix for PROXY protocol with `postscreen` (#4066) --- docs/content/examples/tutorials/mailserver-behind-proxy.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/content/examples/tutorials/mailserver-behind-proxy.md b/docs/content/examples/tutorials/mailserver-behind-proxy.md index 60e4c539..0c39c125 100644 --- a/docs/content/examples/tutorials/mailserver-behind-proxy.md +++ b/docs/content/examples/tutorials/mailserver-behind-proxy.md @@ -259,6 +259,12 @@ The below guidance is focused on configuring [Traefik][traefik-web], but the adv postconf -P 12525/inet/postscreen_upstream_proxy_protocol=haproxy 12525/inet/syslog_name=smtp-proxyprotocol ``` + Supporting port 25 with an additional PROXY protocol port will also require a `postfix-main.cf` override line for `postscreen` to work correctly: + + ```cf title="docker-data/dms/config/postfix-main.cf" + postscreen_cache_map = proxy:btree:$data_directory/postscreen_cache + ``` + --- Dovecot is mostly the same as before: From e4aff5531e1eab2721e7161fb2b2497a8a95a714 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sun, 16 Jun 2024 18:36:10 +0200 Subject: [PATCH 113/267] docs: updated `CONTRIBUTORS.md` (#4069) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- CONTRIBUTORS.md | 268 ++++++++++++++++++++++++------------------------ 1 file changed, 134 insertions(+), 134 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 67bd247a..c4a42c2e 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -387,10 +387,10 @@ Thanks goes to these wonderful people ✨ - - - + + + - - - - + + - - - - + + - - + + + + + + + + + - - + + - - + + + - - + + - - + + - - + + - - + + - - + + - - + + - - - + + - - + + - - + + - - + + - - + + + - - + + - - + + - - + + - - - - - - - + + - - + + - - + + - - + + - - + + - - + + - - + + + - - - + + + + + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - -
- - voordev + + citec
- voordev + citec 		@@ -461,10 +461,10 @@ Thanks goes to these wonderful people ✨ - - Birkenstab + + Starbix
- Birkenstab + Starbix 		@@ -475,21 +475,12 @@ Thanks goes to these wonderful people ✨ - - Starbix + + Birkenstab
- Starbix + Birkenstab 		- - citec -
- citec - -
yajo @@ -497,25 +488,27 @@ Thanks goes to these wonderful people ✨ yajo
- - weo + + voordev
- weo + voordev 		- - analogue + + MakerMatrix
- analogue + MakerMatrix 		- - Rubytastic2 + + pbek
- Rubytastic2 + pbek 		@@ -526,40 +519,33 @@ Thanks goes to these wonderful people ✨ - - pbek + + Rubytastic2
- pbek + Rubytastic2 + + 		+ + analogue +
+ analogue
- - MakerMatrix + + weo
- MakerMatrix + weo 		- - mpanneck + + andrewlow
- mpanneck - - 		- - keslerm -
- keslerm - - 		- - willtho89 -
- willtho89 + andrewlow 		@@ -576,8 +562,6 @@ Thanks goes to these wonderful people ✨ p-fruck
rahilarious @@ -592,6 +576,8 @@ Thanks goes to these wonderful people ✨ Rillke
bobbravo2 @@ -606,28 +592,12 @@ Thanks goes to these wonderful people ✨ r-pufky - - yogo1212 -
- yogo1212 - - 		vincentDcmps
vincentDcmps -
- - tbutter -
- tbutter - 		@@ -650,6 +620,8 @@ Thanks goes to these wonderful people ✨ engelant
j-marz @@ -664,8 +636,13 @@ Thanks goes to these wonderful people ✨ lokipo
+ + mmehnert +
+ mmehnert + + 		msheakoski @@ -687,11 +664,34 @@ Thanks goes to these wonderful people ✨ frugan-dev
- - andrewlow + + tbutter
- andrewlow + tbutter + + 		+ + yogo1212 +
+ yogo1212 + + 		+ + willtho89 +
+ willtho89 + + 		+ + mpanneck +
+ mpanneck 		@@ -768,6 +768,13 @@ Thanks goes to these wonderful people ✨ emazzotta + + keslerm +
+ keslerm + + 		nueaf @@ -789,6 +796,8 @@ Thanks goes to these wonderful people ✨ artonge
spacecowboy @@ -796,8 +805,6 @@ Thanks goes to these wonderful people ✨ spacecowboy
jiriks74 @@ -833,6 +840,8 @@ Thanks goes to these wonderful people ✨ H4R0
ipernet @@ -840,8 +849,6 @@ Thanks goes to these wonderful people ✨ ipernet
fl42 @@ -849,6 +856,13 @@ Thanks goes to these wonderful people ✨ fl42 + + simonsystem +
+ simonsystem + + 		stephan-devop @@ -870,6 +884,8 @@ Thanks goes to these wonderful people ✨ 5ven
syl20bnr @@ -884,8 +900,6 @@ Thanks goes to these wonderful people ✨ sylvaindumont
TechnicLab @@ -914,6 +928,8 @@ Thanks goes to these wonderful people ✨ tobiabocchi
tknecht @@ -928,8 +944,6 @@ Thanks goes to these wonderful people ✨ tweibert
torus @@ -958,6 +972,8 @@ Thanks goes to these wonderful people ✨ k3it
Drakulix @@ -972,8 +988,6 @@ Thanks goes to these wonderful people ✨ vilisas
forzagreen @@ -989,10 +1003,10 @@ Thanks goes to these wonderful people ✨ - - ShiriNmi1520 + + arkanovicz
- ShiriNmi1520 + arkanovicz 		@@ -1002,6 +1016,8 @@ Thanks goes to these wonderful people ✨ neuralp
radicand @@ -1016,8 +1032,6 @@ Thanks goes to these wonderful people ✨ nilshoell
nknapp @@ -1046,6 +1060,8 @@ Thanks goes to these wonderful people ✨ ovidiucp
mrPjer @@ -1060,8 +1076,6 @@ Thanks goes to these wonderful people ✨ p3dda
peter-hartmann @@ -1090,6 +1104,8 @@ Thanks goes to these wonderful people ✨ robbertkl
romansey @@ -1104,8 +1120,6 @@ Thanks goes to these wonderful people ✨ norrs
MightySCollins @@ -1134,6 +1148,8 @@ Thanks goes to these wonderful people ✨ svdb0
3ap @@ -1148,8 +1164,6 @@ Thanks goes to these wonderful people ✨ shyim
sjmudd @@ -1158,10 +1172,10 @@ Thanks goes to these wonderful people ✨ - - simonsystem + + matrixes
- simonsystem + matrixes 		@@ -1178,13 +1192,8 @@ Thanks goes to these wonderful people ✨ millerjason - - mmehnert -
- mmehnert - -
mplx @@ -1192,8 +1201,6 @@ Thanks goes to these wonderful people ✨ mplx
odinis @@ -1229,6 +1236,8 @@ Thanks goes to these wonderful people ✨ pravynandas
presocratics @@ -1236,8 +1245,6 @@ Thanks goes to these wonderful people ✨ presocratics
rhyst @@ -1273,6 +1280,8 @@ Thanks goes to these wonderful people ✨ smargold476
sportshead @@ -1280,8 +1289,6 @@ Thanks goes to these wonderful people ✨ sportshead
squash @@ -1317,6 +1324,8 @@ Thanks goes to these wonderful people ✨ wligtenberg
wolkenschieber @@ -1324,8 +1333,6 @@ Thanks goes to these wonderful people ✨ wolkenschieber
worldworm @@ -1333,6 +1340,13 @@ Thanks goes to these wonderful people ✨ worldworm + + ShiriNmi1520 +
+ ShiriNmi1520 + + 		Zepmann @@ -1354,6 +1368,8 @@ Thanks goes to these wonderful people ✨ arcaine2
awb99 @@ -1368,8 +1384,6 @@ Thanks goes to these wonderful people ✨ brainkiller
cternes @@ -1398,6 +1412,8 @@ Thanks goes to these wonderful people ✨ eleith
fanqiaojun @@ -1412,8 +1428,6 @@ Thanks goes to these wonderful people ✨ ghnp5
helmutundarnold @@ -1442,6 +1456,8 @@ Thanks goes to these wonderful people ✨ idaadi
ixeft @@ -1456,8 +1472,6 @@ Thanks goes to these wonderful people ✨ jjtt
paralax @@ -1486,6 +1500,8 @@ Thanks goes to these wonderful people ✨ callmemagnus
marios88 @@ -1493,22 +1509,6 @@ Thanks goes to these wonderful people ✨ marios88 - - matrixes -
- matrixes - -
- - arkanovicz -
- arkanovicz - - 		CBeerta From 40aab6bd1808c521db1e193e970f267ba16ace2f Mon Sep 17 00:00:00 2001 From: beertje44 <101147154+beertje44@users.noreply.github.com> Date: Tue, 18 Jun 2024 08:46:34 +0200 Subject: [PATCH 114/267] docs: Add tutorial for configuring Dovecot FTS with Solr (#4070) Describe how to use Apache Solr as a Dovecot FTS backend. --------- Co-authored-by: Casper Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- .../examples/tutorials/dovecot-solr.md | 153 ++++++++++++++++++ docs/mkdocs.yml | 1 + 2 files changed, 154 insertions(+) create mode 100644 docs/content/examples/tutorials/dovecot-solr.md diff --git a/docs/content/examples/tutorials/dovecot-solr.md b/docs/content/examples/tutorials/dovecot-solr.md new file mode 100644 index 00000000..95287c5f --- /dev/null +++ b/docs/content/examples/tutorials/dovecot-solr.md @@ -0,0 +1,153 @@ +# Dovecot Full Text Search (FTS) using the Solr Backend + +Dovecot supports several FTS backends for providing fast and efficient full text searching of e-mails directly from the IMAP server. + +As the size of your mail storage grows, the benefits of FTS are especially notable: + +- Without FTS, Dovecot would perform a search query by checking each individual email stored for a match, and then repeat this process again from scratch for the exact same query in future. +- Some mail clients (_like Thunderbird_) may provide their own indexing and search features when all mail to search is stored locally, otherwise Dovecot needs to handle the search query (_for example webmail and mobile clients, like Gmail_). +- FTS indexes each mail into a database for querying instead, where it can skip the cost of inspecting irrelevant emails for a query. + +!!! warning "This is a community contributed guide" + + It extends [our official docs for Dovecot FTS][docs::dovecot::full-text-search] with a focus on Apache Solr. DMS does not officially support this integration. + +## Setup Solr for DMS + +An FTS backend supported by Dovecot is [Apache Solr][github-solr], a fast and efficient multi-purpose search indexer. + +### Add the required `dovecot-solr` package + +As the official DMS image does not provide `dovecot-solr`, you'll need to include the package in your own image (_extending a DMS release as a base image_), or via our [`user-patches.sh` feature][docs::user-patches]: + + +!!! quote "" + + === "`user-patches.sh`" + + If you'd prefer to avoid a custom image build. This approach is simpler but with the caveat that any time the container is restarted, you'll have a delay as the package is installed each time. + + ```bash + #!/bin/bash + + apt-get update && apt-get install dovecot-solr + ``` + + === "`compose.yaml`" + + A custom DMS image does not add much friction. You do not need a separate `Dockerfile` as Docker Compose supports building from an inline `Dockerfile` in your `compose.yaml`. + + The `image` key of the service is swapped for the `build` key instead, as shown below: + + ```yaml + services: + mailserver: + hostname: mail.example.com + # Do not use `image` anymore, unless referring to the tagged image build below + # Add this `build` section to your real `compose.yaml` for your DMS service: + build: + tags: + - local/dms:14.0 + dockerfile_inline: | + FROM docker.io/mailserver/docker-mailserver:14.0 + RUN apt-get update && apt-get install dovecot-solr + ``` + + - Just run `docker compose up` and it will pull DMS and build your custom image to run a container. + - Updating to a new DMS release is straight-forward, just adjust the version tag as you normally would. If you make future changes that don't apply, you may need to force a rebuild. + - This approach only needs to install the package once with the image build itself. This minimizes delay of container startup. + +!!! note "Why doesn't DMS include `dovecot-solr`?" + + This integration is not officially supported in DMS as no maintainer is able to provide troubleshooting support. + + Prior to v14, the package was included but the community contributed guide had been outdated for several years that it was non-functional. It was decided that it was better to drop support and docs, however some DMS users voiced active use of Solr and it's benefits over Xapian for FTS which led to these revised docs. + + **ARM64 builds do not have support for `dovecot-solr`**. Additionally the [user demand for including `dovecot-solr` is presently too low][gh-dms::feature-request::dovecot-solr-package] to justify vs the minimal effort to add additional packages as shown above. + +### `compose.yaml` config + +Firstly you need a working Solr container, for this the [official docker image][dockerhub-solr] will do: + +```yaml +services: + solr: + image: solr:latest + container_name: dms-solr + environment: + # As Solr can be quite resource hungry, raise the memory limit to 2GB. + # The default is 512MB, which may be exhausted quickly. + SOLR_JAVA_MEM: "-Xms2g -Xmx2g" + volumes: + - ./docker-data/solr:/var/solr + restart: always +``` + +DMS will connect internally to the `solr` service above. Either have both services in the same `compose.yaml` file, or ensure that the containers are connected to the same docker network. + +### Configure Solr for Dovecot + +1. Once the Solr container is started, you need to configure a "Solr core" for Dovecot: + + ```bash + docker exec -it dms-solr /bin/sh + solr create -c dovecot + cp -R /opt/solr/contrib/analysis-extras/lib /var/solr/data/dovecot + ``` + + Stop the `dms-solr` container and you should now have a `./data/dovecot` folder in the local bind mount volume. + +2. Solr needs a schema that is specifically tailored for Dovecot FTS. + + As of writing of this guide, Solr 9 is the current release. [Dovecot provides the required schema configs][github-dovecot::core-docs] for Solr, copy the following two v9 config files to `./data/dovecot` and rename them accordingly: + + - `solr-config-9.xml` (_rename to `solrconfig.xml`_) + - `solr-schema-9.xml` (_rename to `schema.xml`_) + + Additionally, remove the `managed-schema.xml` file from `./data/dovecot` and ensure the two files you copied have a [UID and GID of `8983`][dockerfile-solr-uidgid] assigned. + + Start the Solr container once again, you should now have a working Solr core specifically for Dovecot FTS. + +3. Configure Dovecot in DMS to connect to this Solr core: + + Create a `10-plugin.conf` file in your `./config/dovecot` folder with this contents: + + ```config + mail_plugins = $mail_plugins fts fts_solr + + plugin { + fts = solr + fts_autoindex = yes + fts_solr = url=http://dms-solr:8983/solr/dovecot/ + } + ``` + + Add a volume mount for that config to your DMS service in `compose.yaml`: + + ```yaml + services: + mailserver: + volumes: + - ./docker-data/config/dovecot/10-plugin.conf:/etc/dovecot/conf.d/10-plugin.conf:ro + ``` + +### Trigger Dovecot FTS indexing + +After following the previous steps, restart DMS and run this command to have Dovecot re-index all mail: + +```bash +docker compose exec mailserver doveadm fts rescan -A +``` + +!!! info "Indexing will take a while depending on how large your mail folders" + + Usually within 15 minutes or so, you should be able to search your mail using the Dovecot FTS feature! :tada: + +[docs::user-patches]: ../../config/advanced/override-defaults/user-patches.md +[docs::dovecot::full-text-search]: ../../config/advanced/full-text-search.md +[gh-dms::feature-request::dovecot-solr-package]: https://github.com/docker-mailserver/docker-mailserver/issues/4052 + +[dockerhub-solr]: https://hub.docker.com/_/solr +[dockerfile-solr-uidgid]: https://github.com/apache/solr-docker/blob/9cd850b72309de05169544395c83a85b329d6b86/9.6/Dockerfile#L89-L92 +[github-solr]: https://github.com/apache/solr +[github-dovecot::core-docs]: https://github.com/dovecot/core/tree/main/doc diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 59263443..b04631d2 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -174,6 +174,7 @@ nav: - 'Crowdsec': examples/tutorials/crowdsec.md - 'Building your own Docker image': examples/tutorials/docker-build.md - 'Blog Posts': examples/tutorials/blog-posts.md + - 'Dovecot FTS with Apache Solr': examples/tutorials/dovecot-solr.md - 'Use Cases': - 'Forward-Only Mail-Server with LDAP': examples/use-cases/forward-only-mailserver-with-ldap-authentication.md - 'Customize IMAP Folders': examples/use-cases/imap-folders.md From 98cbcfc17187805131f0efc66af50cd2a7a7f62b Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 18 Jun 2024 22:30:24 +0000 Subject: [PATCH 115/267] chore(deps): Bump docker/build-push-action from 5.4.0 to 6.0.0 (#4074) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Casper --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 7f1b7c98..0584238c 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -83,7 +83,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v5.4.0 + uses: docker/build-push-action@v6.0.0 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index bd54c428..db6fbd0c 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v5.4.0 + uses: docker/build-push-action@v6.0.0 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index a021e79f..46a294e6 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v5.4.0 + uses: docker/build-push-action@v6.0.0 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 2b15929b..aaeb5849 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v5.4.0 + uses: docker/build-push-action@v6.0.0 with: context: . tags: mailserver-testing:ci From d7dab2d20d25abecf42dc0e213b408465b38b3d8 Mon Sep 17 00:00:00 2001 From: Casper Date: Wed, 19 Jun 2024 08:10:00 +0200 Subject: [PATCH 116/267] feat: Add password confirmation (#4072) Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> --- CHANGELOG.md | 5 +++++ target/bin/adddovecotmasteruser | 3 +-- target/bin/addmailuser | 3 +-- target/bin/addsaslpassword | 3 +-- target/bin/updatedovecotmasteruser | 3 +-- target/bin/updatemailuser | 3 +-- target/scripts/helpers/database/manage/postfix-accounts.sh | 5 +++++ 7 files changed, 15 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1aa3391e..2205ed9b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,11 @@ All notable changes to this project will be documented in this file. The format > **Note**: Changes and additions listed here are contained in the `:edge` image tag. These changes may not be as stable as released changes. +### Added + +- **Internal:** + - Add password confirmation to several `setup.sh` commands ([#4072](https://github.com/docker-mailserver/docker-mailserver/pull/4072)) + ### Updates - **Fail2ban**: diff --git a/target/bin/adddovecotmasteruser b/target/bin/adddovecotmasteruser index 41041db2..d327478d 100755 --- a/target/bin/adddovecotmasteruser +++ b/target/bin/adddovecotmasteruser @@ -7,8 +7,7 @@ function _main() { _require_n_parameters_or_print_usage 1 "${@}" local MAIL_ACCOUNT="${1}" - shift - local PASSWD="${*}" + local PASSWD="${2}" _manage_accounts_dovecotmaster_create "${MAIL_ACCOUNT}" "${PASSWD}" } diff --git a/target/bin/addmailuser b/target/bin/addmailuser index 15691c89..0eee9d82 100755 --- a/target/bin/addmailuser +++ b/target/bin/addmailuser @@ -7,8 +7,7 @@ function _main() { _require_n_parameters_or_print_usage 1 "${@}" local MAIL_ACCOUNT="${1}" - shift - local PASSWD="${*}" + local PASSWD="${2}" _manage_accounts_create "${MAIL_ACCOUNT}" "${PASSWD}" diff --git a/target/bin/addsaslpassword b/target/bin/addsaslpassword index 9461c112..7cb82dc2 100755 --- a/target/bin/addsaslpassword +++ b/target/bin/addsaslpassword @@ -8,8 +8,7 @@ function _main() { local DOMAIN="${1}" local RELAY_ACCOUNT="${2}" - shift 2 - local PASSWD="${*}" + local PASSWD="${3}" _validate_parameters _add_relayhost_credentials diff --git a/target/bin/updatedovecotmasteruser b/target/bin/updatedovecotmasteruser index eeff21a5..cc1ae930 100755 --- a/target/bin/updatedovecotmasteruser +++ b/target/bin/updatedovecotmasteruser @@ -7,8 +7,7 @@ function _main() { _require_n_parameters_or_print_usage 1 "${@}" local MAIL_ACCOUNT="${1}" - shift - local PASSWD="${*}" + local PASSWD="${2}" _manage_accounts_dovecotmaster_update "${MAIL_ACCOUNT}" "${PASSWD}" } diff --git a/target/bin/updatemailuser b/target/bin/updatemailuser index ce45533e..be1b981c 100755 --- a/target/bin/updatemailuser +++ b/target/bin/updatemailuser @@ -7,8 +7,7 @@ function _main() { _require_n_parameters_or_print_usage 1 "${@}" local MAIL_ACCOUNT="${1}" - shift - local PASSWD="${*}" + local PASSWD="${2}" _manage_accounts_update "${MAIL_ACCOUNT}" "${PASSWD}" } diff --git a/target/scripts/helpers/database/manage/postfix-accounts.sh b/target/scripts/helpers/database/manage/postfix-accounts.sh index 987254fc..e92b2555 100644 --- a/target/scripts/helpers/database/manage/postfix-accounts.sh +++ b/target/scripts/helpers/database/manage/postfix-accounts.sh @@ -98,9 +98,14 @@ function __account_already_exists() { # Also used by addsaslpassword function _password_request_if_missing() { + local PASSWD_CONFIRM if [[ -z ${PASSWD} ]]; then read -r -s -p 'Enter Password: ' PASSWD echo [[ -z ${PASSWD} ]] && _exit_with_error 'Password must not be empty' + + read -r -s -p 'Confirm Password: ' PASSWD_CONFIRM + echo + [[ ${PASSWD} != "${PASSWD_CONFIRM}" ]] && _exit_with_error 'Passwords do not match!' fi } From 8a082be714d165e954f0ad05e65810d706c6b4ec Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Wed, 19 Jun 2024 20:39:06 +1200 Subject: [PATCH 117/267] docs: Add info regarding DKIM key rotation and non-expiry (#4076) Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Co-authored-by: Casper --- .../config/best-practices/dkim_dmarc_spf.md | 21 +++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/docs/content/config/best-practices/dkim_dmarc_spf.md b/docs/content/config/best-practices/dkim_dmarc_spf.md index d035556f..d6ba06b7 100644 --- a/docs/content/config/best-practices/dkim_dmarc_spf.md +++ b/docs/content/config/best-practices/dkim_dmarc_spf.md @@ -33,6 +33,21 @@ When DKIM is enabled: DKIM requires a public/private key pair to enable **signing (_via private key_)** your outgoing mail, while the receiving end must query DNS to **verify (_via public key_)** that the signature is trustworthy. +??? info "Verification expiry" + + Unlike your TLS certificate, your DKIM keypair does not have a fixed expiry associated to it. + + + Instead, an expiry may be included in your DKIM signature for each mail sent, where a receiver will [refuse to validate the signature for an email after that expiry date][dkim-verification-expiry-refusal]. This is an added precaution to mitigate malicious activity like "DKIM replay attacks", where an already delivered email from a third-party with a trustworthy DKIM signature is leveraged by a spammer when sending mail to an MTA which verifies the DKIM signature successfully, enabling the spammer to bypass spam protections. + + Unlike a TLS handshake where you are authenticating trust with future communications, with DKIM once the mail has been received and trust of the signature has been verified, the value of verifying the signature again at a later date is less meaningful since the signature was to ensure no tampering had occurred during delivery through the network. + +??? tip "DKIM key rotation" + + You can rotate your DKIM keypair by switching to a new DKIM selector (_and DNS updates_), while the previous key and selector remains valid for verification until the last mail signed with that key reaches it's expiry. + + DMS does not provide any automation or support for key rotation, [nor is it likely to provide a notable security benefit][gh-discussion::dkim-key-rotation-expiry] to the typical small scale DMS deployment. + ### Generating Keys You'll need to repeat this process if you add any new domains. @@ -72,7 +87,7 @@ You should have: According to [RFC 8301][rfc-8301], keys are preferably between 1024 and 2048 bits. Keys of size 4096-bit or larger may not be compatible to all systems your mail is intended for. - You [should not need a key length beyond 2048-bit][github-issue-dkimlength]. If 2048-bit does not meet your security needs, you may want to instead consider adopting key rotation or switching from RSA to ECC keys for DKIM. + You [should not need a key length beyond 2048-bit][gh-issue::dkim-length]. If 2048-bit does not meet your security needs, you may want to instead consider adopting key rotation or switching from RSA to ECC keys for DKIM. ??? note "You may need to specify mail domains explicitly" @@ -352,7 +367,8 @@ volumes: [docs-rspamd-config-dropin]: ../security/rspamd.md#manually [cloudflare-dkim-dmarc-spf]: https://www.cloudflare.com/learning/email-security/dmarc-dkim-spf/ [rfc-8301]: https://datatracker.ietf.org/doc/html/rfc8301#section-3.2 -[github-issue-dkimlength]: https://github.com/docker-mailserver/docker-mailserver/issues/1854#issuecomment-806280929 +[gh-discussion::dkim-key-rotation-expiry]: https://github.com/orgs/docker-mailserver/discussions/4068#discussioncomment-9784263 +[gh-issue::dkim-length]: https://github.com/docker-mailserver/docker-mailserver/issues/1854#issuecomment-806280929 [rspamd-docs-dkim-checks]: https://www.rspamd.com/doc/modules/dkim.html [rspamd-docs-dkim-signing]: https://www.rspamd.com/doc/modules/dkim_signing.html [dns::example-webui]: https://www.vultr.com/docs/introduction-to-vultr-dns/ @@ -360,6 +376,7 @@ volumes: [dns::wikipedia-zonefile]: https://en.wikipedia.org/wiki/Zone_file [dns::webui-dkim]: https://serverfault.com/questions/763815/route-53-doesnt-allow-adding-dkim-keys-because-length-is-too-long [dkim-ed25519-support]: https://serverfault.com/questions/1023674/is-ed25519-well-supported-for-the-dkim-validation/1074545#1074545 +[dkim-verification-expiry-refusal]: https://mxtoolbox.com/problem/dkim/dkim-signature-expiration [mxtoolbox-dkim-verifier]: https://mxtoolbox.com/dkim.aspx [dmarc-howto-configtags]: https://github.com/internetstandards/toolbox-wiki/blob/master/DMARC-how-to.md#overview-of-dmarc-configuration-tags [dmarc-tool-gca]: https://dmarcguide.globalcyberalliance.org From e370c0c96a28acfbd66b20ffac327528eda78104 Mon Sep 17 00:00:00 2001 From: Casper Date: Wed, 19 Jun 2024 18:34:18 +0200 Subject: [PATCH 118/267] fail2ban install: remove -k (--insecure) from curl options (#4080) --- CHANGELOG.md | 4 ++++ target/scripts/build/packages.sh | 4 ++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2205ed9b..832e25d2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ All notable changes to this project will be documented in this file. The format > **Note**: Changes and additions listed here are contained in the `:edge` image tag. These changes may not be as stable as released changes. +### Security + +- **Fail2ban**: + - Ensure a secure connection, when downloading the fail2ban package ([#4080](https://github.com/docker-mailserver/docker-mailserver/pull/4080)) ### Added - **Internal:** diff --git a/target/scripts/build/packages.sh b/target/scripts/build/packages.sh index 004c3b8c..4469f508 100644 --- a/target/scripts/build/packages.sh +++ b/target/scripts/build/packages.sh @@ -189,8 +189,8 @@ function _install_fail2ban() { gpg --keyserver "${FAIL2BAN_GPG_PUBLIC_KEY_SERVER}" --recv-keys "${FAIL2BAN_GPG_PUBLIC_KEY_ID}" 2>&1 - curl -Lkso fail2ban.deb "${FAIL2BAN_DEB_URL}" - curl -Lkso fail2ban.deb.asc "${FAIL2BAN_DEB_ASC_URL}" + curl -fsSLo fail2ban.deb "${FAIL2BAN_DEB_URL}" + curl -fsSLo fail2ban.deb.asc "${FAIL2BAN_DEB_ASC_URL}" FINGERPRINT=$(LANG=C gpg --verify fail2ban.deb.asc fail2ban.deb |& sed -n 's#Primary key fingerprint: \(.*\)#\1#p') From ccaa02b8b5f1841b127fe51ac6efe30620173ba6 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 25 Jun 2024 22:43:10 +0200 Subject: [PATCH 119/267] chore(deps): Bump docker/build-push-action from 6.0.0 to 6.1.0 (#4086) --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 0584238c..ac87d337 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -83,7 +83,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v6.0.0 + uses: docker/build-push-action@v6.1.0 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index db6fbd0c..b575249b 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v6.0.0 + uses: docker/build-push-action@v6.1.0 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 46a294e6..52d3d00d 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.0.0 + uses: docker/build-push-action@v6.1.0 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index aaeb5849..b2f74a44 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.0.0 + uses: docker/build-push-action@v6.1.0 with: context: . tags: mailserver-testing:ci From 5a4a136ec5631f9d3bffaf5c40eed571c30da0f1 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sat, 29 Jun 2024 12:44:00 +0200 Subject: [PATCH 120/267] docs: updated `CONTRIBUTORS.md` (#4084) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- CONTRIBUTORS.md | 133 +++++++++++++++++++++++++----------------------- 1 file changed, 70 insertions(+), 63 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index c4a42c2e..9a270918 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -856,13 +856,6 @@ Thanks goes to these wonderful people ✨ fl42 - - simonsystem -
- simonsystem - - 		stephan-devop @@ -884,8 +877,6 @@ Thanks goes to these wonderful people ✨ 5ven
syl20bnr @@ -893,6 +884,8 @@ Thanks goes to these wonderful people ✨ syl20bnr
sylvaindumont @@ -928,8 +921,6 @@ Thanks goes to these wonderful people ✨ tobiabocchi
tknecht @@ -937,6 +928,8 @@ Thanks goes to these wonderful people ✨ tknecht
tweibert @@ -972,8 +965,6 @@ Thanks goes to these wonderful people ✨ k3it
Drakulix @@ -981,6 +972,8 @@ Thanks goes to these wonderful people ✨ Drakulix
vilisas @@ -1003,10 +996,10 @@ Thanks goes to these wonderful people ✨ - - arkanovicz + + ShiriNmi1520
- arkanovicz + ShiriNmi1520 		@@ -1016,8 +1009,6 @@ Thanks goes to these wonderful people ✨ neuralp
radicand @@ -1025,6 +1016,8 @@ Thanks goes to these wonderful people ✨ radicand
nilshoell @@ -1060,8 +1053,6 @@ Thanks goes to these wonderful people ✨ ovidiucp
mrPjer @@ -1069,6 +1060,8 @@ Thanks goes to these wonderful people ✨ mrPjer
p3dda @@ -1104,8 +1097,6 @@ Thanks goes to these wonderful people ✨ robbertkl
romansey @@ -1113,6 +1104,8 @@ Thanks goes to these wonderful people ✨ romansey
norrs @@ -1148,8 +1141,6 @@ Thanks goes to these wonderful people ✨ svdb0
3ap @@ -1157,6 +1148,8 @@ Thanks goes to these wonderful people ✨ 3ap
shyim @@ -1171,6 +1164,13 @@ Thanks goes to these wonderful people ✨ sjmudd + + simonsystem +
+ simonsystem + + 		matrixes @@ -1340,13 +1340,6 @@ Thanks goes to these wonderful people ✨ worldworm - - ShiriNmi1520 -
- ShiriNmi1520 - - 		Zepmann @@ -1368,14 +1361,21 @@ Thanks goes to these wonderful people ✨ arcaine2
awb99
awb99 +
+ + beertje44 +
+ beertje44 + 		@@ -1509,6 +1509,13 @@ Thanks goes to these wonderful people ✨ marios88 + + arkanovicz +
+ arkanovicz + + 		CBeerta @@ -1537,6 +1544,8 @@ Thanks goes to these wonderful people ✨ dkarski
dbellavista @@ -1544,8 +1553,6 @@ Thanks goes to these wonderful people ✨ dbellavista
danielvandenberg95 @@ -1581,6 +1588,8 @@ Thanks goes to these wonderful people ✨ doominator42
aydodo @@ -1588,8 +1597,6 @@ Thanks goes to these wonderful people ✨ aydodo
vedtam @@ -1625,6 +1632,8 @@ Thanks goes to these wonderful people ✨ ErikEngerd
huncode @@ -1632,8 +1641,6 @@ Thanks goes to these wonderful people ✨ huncode
gitfeber @@ -1669,6 +1676,8 @@ Thanks goes to these wonderful people ✨ JOduMonT
0xflotus @@ -1676,8 +1685,6 @@ Thanks goes to these wonderful people ✨ 0xflotus
ifokeev @@ -1713,6 +1720,8 @@ Thanks goes to these wonderful people ✨ aspettl
acch @@ -1720,8 +1729,6 @@ Thanks goes to these wonderful people ✨ acch
vifino @@ -1757,6 +1764,8 @@ Thanks goes to these wonderful people ✨ eglia
groupmsl @@ -1764,8 +1773,6 @@ Thanks goes to these wonderful people ✨ groupmsl
green-anger @@ -1801,6 +1808,8 @@ Thanks goes to these wonderful people ✨ astrocket
baxerus @@ -1808,8 +1817,6 @@ Thanks goes to these wonderful people ✨ baxerus
spock @@ -1845,6 +1852,8 @@ Thanks goes to these wonderful people ✨ Kaan88
akkumar @@ -1852,8 +1861,6 @@ Thanks goes to these wonderful people ✨ akkumar
thechubbypanda @@ -1889,6 +1896,8 @@ Thanks goes to these wonderful people ✨ JustAnother1
leowinterde @@ -1896,8 +1905,6 @@ Thanks goes to these wonderful people ✨ leowinterde
linhandev @@ -1933,6 +1940,8 @@ Thanks goes to these wonderful people ✨ madmath03
maxemann96 @@ -1940,8 +1949,6 @@ Thanks goes to these wonderful people ✨ maxemann96
dragetd @@ -1977,6 +1984,8 @@ Thanks goes to these wonderful people ✨ mcchots
MohammedNoureldin @@ -1984,8 +1993,6 @@ Thanks goes to these wonderful people ✨ MohammedNoureldin
mpldr @@ -2021,6 +2028,8 @@ Thanks goes to these wonderful people ✨ Marsu31
glandais @@ -2028,8 +2037,6 @@ Thanks goes to these wonderful people ✨ glandais
GiovanH @@ -2065,6 +2072,8 @@ Thanks goes to these wonderful people ✨ sirgantrithon
Influencer @@ -2072,8 +2081,6 @@ Thanks goes to these wonderful people ✨ Influencer
in-seo @@ -2109,6 +2116,8 @@ Thanks goes to these wonderful people ✨ jcalfee
mivek @@ -2116,8 +2125,6 @@ Thanks goes to these wonderful people ✨ mivek
init-js @@ -2153,6 +2160,8 @@ Thanks goes to these wonderful people ✨ jirislav
jmccl @@ -2160,8 +2169,6 @@ Thanks goes to these wonderful people ✨ jmccl
jurekbarth From 22383c28e7cf6d38b674f8a0d28ff522bbad3929 Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Sun, 30 Jun 2024 11:34:38 +0200 Subject: [PATCH 121/267] CI: Remove reviewer assignment from Dependabot config (#4088) --- .github/dependabot.yml | 4 ---- 1 file changed, 4 deletions(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 99a0fb33..327014c5 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -4,8 +4,6 @@ updates: directory: "/" schedule: interval: "weekly" - reviewers: - - "docker-mailserver/maintainers" labels: - "area/ci" - "kind/update" @@ -15,8 +13,6 @@ updates: directory: / schedule: interval: "weekly" - reviewers: - - "docker-mailserver/maintainers" labels: - "area/ci" - "kind/update" From b3a5e9e4e8eb0de311940c86d233c66fa3d72354 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 3 Jul 2024 12:17:19 +1200 Subject: [PATCH 122/267] chore(deps): Bump docker/build-push-action from 6.1.0 to 6.2.0 (#4089) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6.1.0 to 6.2.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/v6.1.0...v6.2.0) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index ac87d337..51d300e4 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -83,7 +83,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v6.1.0 + uses: docker/build-push-action@v6.2.0 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index b575249b..7b7c4fe5 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v6.1.0 + uses: docker/build-push-action@v6.2.0 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 52d3d00d..03bebcc9 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.1.0 + uses: docker/build-push-action@v6.2.0 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index b2f74a44..5f7a1014 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.1.0 + uses: docker/build-push-action@v6.2.0 with: context: . tags: mailserver-testing:ci From 9175424d0f3c67622c030311cc720d7477f1c2b6 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Wed, 3 Jul 2024 22:16:02 +1200 Subject: [PATCH 123/267] fix: Update `dovecot-fts-xapian` to `1.7.13` (#4095) * fix: Update `dovecot-fts-xapian` to `1.7.13` Contains a fix to a regression introduced that broke indexing --------- Co-authored-by: casperklein --- CHANGELOG.md | 3 ++- Dockerfile | 4 ++-- target/scripts/build/compile.sh | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 832e25d2..2d9e6ceb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,7 +22,8 @@ All notable changes to this project will be documented in this file. The format #### Fixes - **Dovecot:** - - `logwatch` Update logwatch `ignore.conf` to exclude Xapian messages about pending documents + - `logwatch` Update logwatch `ignore.conf` to exclude Xapian messages about pending documents ([#4060](https://github.com/docker-mailserver/docker-mailserver/pull/4060)) + - `dovecot-fts-xapian` plugin was updated to `1.7.13`, fixing a regression with indexing ([#4095](https://github.com/docker-mailserver/docker-mailserver/pull/4095)) ## [v14.0.0](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v14.0.0) diff --git a/Dockerfile b/Dockerfile index da9aa83d..f6199fd4 100644 --- a/Dockerfile +++ b/Dockerfile @@ -82,8 +82,8 @@ EOF # install fts_xapian plugin -COPY --from=stage-compile dovecot-fts-xapian-1.7.12_1.7.12_*.deb / -RUN dpkg -i /dovecot-fts-xapian-1.7.12_1.7.12_*.deb && rm /dovecot-fts-xapian-1.7.12_1.7.12_*.deb +COPY --from=stage-compile dovecot-fts-xapian-*.deb / +RUN dpkg -i /dovecot-fts-xapian-*.deb && rm /dovecot-fts-xapian-*.deb COPY target/dovecot/*.inc target/dovecot/*.conf /etc/dovecot/conf.d/ COPY target/dovecot/dovecot-purge.cron /etc/cron.d/dovecot-purge.disabled diff --git a/target/scripts/build/compile.sh b/target/scripts/build/compile.sh index 4c6fba82..299ba7c8 100644 --- a/target/scripts/build/compile.sh +++ b/target/scripts/build/compile.sh @@ -16,7 +16,7 @@ function _compile_dovecot_fts_xapian() { apt-get "${QUIET}" --no-install-recommends install \ automake libtool pkg-config libicu-dev libsqlite3-dev libxapian-dev make build-essential dh-make devscripts dovecot-dev - local XAPIAN_VERSION='1.7.12' + local XAPIAN_VERSION='1.7.13' curl -sSfL -o dovecot-fts-xapian.tar.gz \ "https://github.com/grosjo/fts-xapian/releases/download/${XAPIAN_VERSION}/dovecot-fts-xapian-${XAPIAN_VERSION}.tar.gz" tar xf dovecot-fts-xapian.tar.gz From 2d12bbb7fd410393d0b5419f86e91287613d613c Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Fri, 5 Jul 2024 23:48:42 +1200 Subject: [PATCH 124/267] docs: Update `compose.yaml` for `dovecot-solr` guide (#4099) The `image` field is used for the default tag, if it's not specified Compose will infer one in addition to any extra `tags` provided. Better to use `image` for the tag assignment, and a clear `pull_policy` to prevent trying to pull a remote image of the same name. --- docs/content/examples/tutorials/dovecot-solr.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/content/examples/tutorials/dovecot-solr.md b/docs/content/examples/tutorials/dovecot-solr.md index 95287c5f..be4c91f6 100644 --- a/docs/content/examples/tutorials/dovecot-solr.md +++ b/docs/content/examples/tutorials/dovecot-solr.md @@ -43,11 +43,12 @@ As the official DMS image does not provide `dovecot-solr`, you'll need to includ services: mailserver: hostname: mail.example.com - # Do not use `image` anymore, unless referring to the tagged image build below + # The `image` setting now represents the tag for the local build configured below: + image: local/dms:14.0 + # Local build (no need to try pull `image` remotely): + pull_policy: build # Add this `build` section to your real `compose.yaml` for your DMS service: build: - tags: - - local/dms:14.0 dockerfile_inline: | FROM docker.io/mailserver/docker-mailserver:14.0 RUN apt-get update && apt-get install dovecot-solr From 19d52d9dcc5489f6dc8ac9aa51f8a224b48e411b Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 9 Jul 2024 11:23:02 +1200 Subject: [PATCH 125/267] chore(deps): Bump docker/build-push-action from 6.2.0 to 6.3.0 (#4103) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6.2.0 to 6.3.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/v6.2.0...v6.3.0) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 51d300e4..2938b522 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -83,7 +83,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v6.2.0 + uses: docker/build-push-action@v6.3.0 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 7b7c4fe5..11fd1f2e 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v6.2.0 + uses: docker/build-push-action@v6.3.0 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 03bebcc9..4c36b22b 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.2.0 + uses: docker/build-push-action@v6.3.0 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 5f7a1014..b07ae3c3 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.2.0 + uses: docker/build-push-action@v6.3.0 with: context: . tags: mailserver-testing:ci From 2cca907615697cbca2ecc20a91f7664e61abfd11 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 8 Jul 2024 23:24:48 +0000 Subject: [PATCH 126/267] chore(deps): Bump docker/setup-buildx-action from 3.3.0 to 3.4.0 (#4104) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3.3.0 to 3.4.0. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](https://github.com/docker/setup-buildx-action/compare/v3.3.0...v3.4.0) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 2938b522..29d78151 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -79,7 +79,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.3.0 + uses: docker/setup-buildx-action@v3.4.0 # NOTE: AMD64 can build within 2 minutes - name: 'Build images' diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 11fd1f2e..68c423c1 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -40,7 +40,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.3.0 + uses: docker/setup-buildx-action@v3.4.0 # Try get the cached build layers from a prior `generic_build.yml` job. # NOTE: Until adopting `type=gha` scoped cache exporter (in `docker/build-push-action`), diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 4c36b22b..9d7c6902 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -38,7 +38,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.3.0 + uses: docker/setup-buildx-action@v3.4.0 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index b07ae3c3..fe290de3 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -37,7 +37,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.3.0 + uses: docker/setup-buildx-action@v3.4.0 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. From 3a40c457fc7903cbf409609b446632e69bf346ab Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 8 Jul 2024 23:26:20 +0000 Subject: [PATCH 127/267] chore(deps): Bump docker/setup-qemu-action from 3.0.0 to 3.1.0 (#4105) Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 3.0.0 to 3.1.0. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](https://github.com/docker/setup-qemu-action/compare/v3.0.0...v3.1.0) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 29d78151..c54826ae 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -74,7 +74,7 @@ jobs: cache-buildx- - name: 'Set up QEMU' - uses: docker/setup-qemu-action@v3.0.0 + uses: docker/setup-qemu-action@v3.1.0 with: platforms: arm64 diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 68c423c1..bd6f5fa9 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -35,7 +35,7 @@ jobs: type=semver,pattern={{major}}.{{minor}}.{{patch}} - name: 'Set up QEMU' - uses: docker/setup-qemu-action@v3.0.0 + uses: docker/setup-qemu-action@v3.1.0 with: platforms: arm64 From 4778f15fdaf5e94aee364396166fdc2821066c46 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Wed, 10 Jul 2024 05:44:09 +1200 Subject: [PATCH 128/267] docs: TLS typo fix (#4106) - Caddy admonition - Missing `[` for annotating a link. - Traefik - `docker-compose` => "Docker Compose" --- docs/content/config/security/ssl.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/content/config/security/ssl.md b/docs/content/config/security/ssl.md index a5ba005a..eeb39a5f 100644 --- a/docs/content/config/security/ssl.md +++ b/docs/content/config/security/ssl.md @@ -567,7 +567,7 @@ DSM-generated letsencrypt certificates get auto-renewed every three months. !!! warning "Caddy certificate location varies" - The path contains the certificate provisioner used. This path may be different from the example above for you and may change over time when multiple provisioner services are used][dms-pr-feedback::caddy-provisioning-gotcha]. + The path contains the certificate provisioner used. This path may be different from the example above for you and may change over time when [multiple ACME provisioner services are used][dms-pr-feedback::caddy-provisioning-gotcha]. This can make the volume mounting for DMS to find the certificates non-deterministic, but you can [restrict provisioning to single service via the `acme_ca` setting][caddy::restrict-acme-provisioner]. @@ -583,7 +583,7 @@ This setup only comes with one caveat - The domain has to be configured on anoth ???+ example "Example Code" - Here is an example setup for [`docker-compose`](https://docs.docker.com/compose/): + Here is an example setup for [`Docker Compose`](https://docs.docker.com/compose/): ```yaml services: From 755540cacf50193780ee6697f6569efa9c8745d1 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Sun, 14 Jul 2024 21:24:33 +1200 Subject: [PATCH 129/267] docs: `docker-build.md` - Update `DOVECOT_COMMUNITY_REPO` default (#4111) --- docs/content/examples/tutorials/docker-build.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/examples/tutorials/docker-build.md b/docs/content/examples/tutorials/docker-build.md index 538da822..63241e67 100644 --- a/docs/content/examples/tutorials/docker-build.md +++ b/docs/content/examples/tutorials/docker-build.md @@ -32,7 +32,7 @@ We make use of build features that require a recent version of Docker. v23.0 or The `Dockerfile` includes several build [`ARG`][docker-docs::builder-arg] instructions that can be configured: -- `DOVECOT_COMMUNITY_REPO`: Install Dovecot from the community repo instead of from Debian (default = 1) +- `DOVECOT_COMMUNITY_REPO`: Install Dovecot from the community repo instead of from Debian (default = 0) - `DMS_RELEASE`: The image version (default = edge) - `VCS_REVISION`: The git commit hash used for the build (default = unknown) From bf4ebc2a410691af82315605d3b385ba3a28e1c6 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 16 Jul 2024 13:42:20 +1200 Subject: [PATCH 130/267] chore(deps): Bump anchore/scan-action from 3.6.4 to 4.0.0 (#4114) Bumps [anchore/scan-action](https://github.com/anchore/scan-action) from 3.6.4 to 4.0.0. - [Release notes](https://github.com/anchore/scan-action/releases) - [Changelog](https://github.com/anchore/scan-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/anchore/scan-action/compare/v3.6.4...v4.0.0) --- updated-dependencies: - dependency-name: anchore/scan-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_vulnerability-scan.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index fe290de3..1cb5f0ed 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -55,7 +55,7 @@ jobs: provenance: false - name: 'Run the Anchore Grype scan action' - uses: anchore/scan-action@v3.6.4 + uses: anchore/scan-action@v4.0.0 id: scan with: image: mailserver-testing:ci From 34423c2f665b8c7839c40fbc7d1fe8cda6342456 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 16 Jul 2024 01:44:17 +0000 Subject: [PATCH 131/267] chore(deps): Bump docker/build-push-action from 6.3.0 to 6.4.0 (#4113) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6.3.0 to 6.4.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/v6.3.0...v6.4.0) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index c54826ae..5b10ef87 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -83,7 +83,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v6.3.0 + uses: docker/build-push-action@v6.4.0 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index bd6f5fa9..42600990 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v6.3.0 + uses: docker/build-push-action@v6.4.0 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 9d7c6902..1cab38b8 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.3.0 + uses: docker/build-push-action@v6.4.0 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 1cb5f0ed..95b6c4f6 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.3.0 + uses: docker/build-push-action@v6.4.0 with: context: . tags: mailserver-testing:ci From c5f125c973d65d0f51000061952ec378b7449c6e Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Thu, 18 Jul 2024 10:31:50 +1200 Subject: [PATCH 132/267] tests: Update `curl` note for XOAUTH2 support (#4118) --- test/tests/serial/mail_with_oauth2.bats | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/test/tests/serial/mail_with_oauth2.bats b/test/tests/serial/mail_with_oauth2.bats index c9a02eea..968d63ab 100644 --- a/test/tests/serial/mail_with_oauth2.bats +++ b/test/tests/serial/mail_with_oauth2.bats @@ -59,9 +59,14 @@ function teardown_file() { } @test "should authenticate with XOAUTH2" { - # curl packaged in Debian 12 (and the latest release as of Jan 2024) broke XOAUTH2 support + # curl 7.80.0 (Nov 2021) broke XOAUTH2 support (DMS v14 release with Debian 12 packages curl 7.88.1) # https://github.com/docker-mailserver/docker-mailserver/pull/3403#issuecomment-1907100624 - skip 'unable to test XOAUTH mechanism due to bug since curl 7.80' + # + # Fixed in curl 8.6.0 (Jan 31 2024): + # - https://github.com/curl/curl/issues/10259 + # - https://github.com/curl/curl/commit/7b2d98dfadf209108aa7772ee21ae42e3dab219f (referenced in release changelog by commit title) + # - https://github.com/curl/curl/releases/tag/curl-8_6_0 + skip 'unable to test XOAUTH2 mechanism due to bug in curl versions 7.80.0 --> 8.5.0' __should_login_successfully_with 'XOAUTH2' } From 0698ad9370de780cccd46c0c6ca9113a45afc775 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Mon, 22 Jul 2024 10:00:53 +1200 Subject: [PATCH 133/267] docs: Refactor pages for Account Management (#4122) * docs: Relocate account / auth pages into a common section * docs: Update references to relocated pages * docs: Add account management overview page Updates remaining links to account sections on this page instead (_for `accounts`, `aliases`, `quotas`_). This page will cover the features and defer to separate pages for more specific content where relevant. * docs: Correct relocated pages titles and links * docs: Accounts (Dovecot Master) - Minor revisions * docs: Fix highlighting roundcube PHP snippet in OAuth2 page * docs: Accounts (File) - Refactor - Manual method not necessary to document. - Condense `setup` example guidance. - Quotas / Aliases content migrated to Overview when not specific about file provisioner. Some of the content is this commit is not a complete revision. * chore: Temporary commit * docs(refactor): Sub-addressing section Much better docs on the sub-addressing feature supported by Postfix and Dovecot, along with the guidance with usage in Sieve. * docs: Revise accounts section Add some context regarding DMS accounts and their distinction/overlap from the email address functionality, and it's relevant context for receiving/sending. File provisioner, minor revisions to referencing associated config files and account management. * docs: Minor adjustments * docs: Refactor the quota section Better documented with links and coverage over the workaround details we've implemented. * docs: Revise the quota section Minor revisions with phrasing, admonitions for structure and better explanation of the feature functionality/purpose. * docs: Alias section refactor Extensively covers known issues and technical details that have been discussed often enough. The improvements should benefit both users and maintainers. * docs: Refactor master accounts page This rewrite should more clearly document the feature, along with a better example and additional links for reference. * docs: OAuth2 revision Minor update to this page: - Links extracted to bottom of page as per convention. - ENV file example converted to preferred `compose.yaml` ENV settings. * docs: Sieve minor revisions - Correct link to subaddressing section - Make the config file example snippets intended filename less ambiguous. - Minor rephrasng. * docs: Revise accounts overview section Revised the account section and added additional clarity for common confusion with relation to sender address and multi-domain support. Top of the page now clarifies it's a technical reference and directs users to the related pages for configuration / caveats. Technical Overview links to Dovecot docs were missing. * docs: Another revision pass File based provisioner docs: - Sections indent with info admonitions. - Accounts section expanded with config format and example. - Quotas section expanded and shifted to bottom (alphabetical sort). - Split into `setup` CLI and config reference groups. Overview page: - Sections indent with info admonitions. - Revised content. * docs(chore): Shift sub-addressing section This is related to accounts and aliases, but not provisioners, thus extract out of the accounts parent section. * docs: Document `postfix-accounts.cf` third column This lacked documentation but was community contributed feature to allow further customization of a Dovecot Account. It has caveats as DMS does not take these into consideration anywhere in scripts. Documenting officially for better awareness. * docs: Revise and expand supplementary pages Better outline the OAuth2 login process, the two supported login mechanisms and their docs/rfcs, along with documenting caveat with mail client compatibility. Add a verification tip for the OAuth2 support, showing how `curl` can be used, along with caveat presently affecting the `curl` in DMS v14. Additionally note the feature still isn't documented fully, providing the user with additional references for more information. `ACCOUNT_PROVISIONER` ENV docs minimized. No `OIDC` provisioner plans, the OAuth2 docs page now mentions SCIM 2.0 API as the next step towards resolving that concern. The tip admonition was removed as it no longer provides value, instead we link to the Account Management overview page. Dovecot Master Accounts docs page now lightly document the `setup` CLI and config format for the feature. * docs: Fix broken anchor links Some anchor links to different parts of our docs have gone stale. This branch also broke a few itself that I missed. The build now only reports issues with anchor links to Content Tabs, which it must not be aware of during the build (_MKDocs Material specific feature?_) * docs(lint): Fix indentation level * chore: Add entry to `CHANGELOG.md` + corrections --- CHANGELOG.md | 41 +-- README.md | 4 +- .../config/account-management/overview.md | 252 ++++++++++++++++++ .../account-management/provisioner/file.md | 206 ++++++++++++++ .../provisioner/ldap.md} | 6 +- .../supplementary/master-accounts.md | 70 +++++ .../supplementary/oauth2.md | 145 ++++++++++ docs/content/config/advanced/auth-oauth2.md | 69 ----- .../advanced/dovecot-master-accounts.md | 21 -- docs/content/config/advanced/mail-sieve.md | 94 +++++-- .../config/advanced/optional-config.md | 19 +- docs/content/config/advanced/podman.md | 6 +- .../config/best-practices/dkim_dmarc_spf.md | 2 +- docs/content/config/debugging.md | 2 +- docs/content/config/environment.md | 21 +- docs/content/config/security/fail2ban.md | 2 +- docs/content/config/security/ssl.md | 2 +- .../security/understanding-the-ports.md | 2 +- docs/content/config/user-management.md | 86 ------ .../contributing/issues-and-pull-requests.md | 8 +- .../tutorials/mailserver-behind-proxy.md | 4 +- docs/content/examples/use-cases/auth-lua.md | 2 +- docs/content/usage.md | 2 +- docs/mkdocs.yml | 19 +- mailserver.env | 2 +- 25 files changed, 828 insertions(+), 259 deletions(-) create mode 100644 docs/content/config/account-management/overview.md create mode 100644 docs/content/config/account-management/provisioner/file.md rename docs/content/config/{advanced/auth-ldap.md => account-management/provisioner/ldap.md} (99%) create mode 100644 docs/content/config/account-management/supplementary/master-accounts.md create mode 100644 docs/content/config/account-management/supplementary/oauth2.md delete mode 100644 docs/content/config/advanced/auth-oauth2.md delete mode 100755 docs/content/config/advanced/dovecot-master-accounts.md delete mode 100644 docs/content/config/user-management.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 2d9e6ceb..b39f60d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,8 +8,9 @@ All notable changes to this project will be documented in this file. The format ### Security -- **Fail2ban**: +- **Fail2ban:** - Ensure a secure connection, when downloading the fail2ban package ([#4080](https://github.com/docker-mailserver/docker-mailserver/pull/4080)) + ### Added - **Internal:** @@ -17,10 +18,14 @@ All notable changes to this project will be documented in this file. The format ### Updates -- **Fail2ban**: +- **Fail2ban:** - Bump version to [1.1.0](https://github.com/fail2ban/fail2ban/releases/tag/1.1.0). For more information, check the [changelog](https://github.com/fail2ban/fail2ban/blob/1.1.0/ChangeLog). +- **Documentation:** + - Rewritten and organized the pages for Account Management and Authentication ([#4122](https://github.com/docker-mailserver/docker-mailserver/pull/4122)) + #### Fixes + - **Dovecot:** - `logwatch` Update logwatch `ignore.conf` to exclude Xapian messages about pending documents ([#4060](https://github.com/docker-mailserver/docker-mailserver/pull/4060)) - `dovecot-fts-xapian` plugin was updated to `1.7.13`, fixing a regression with indexing ([#4095](https://github.com/docker-mailserver/docker-mailserver/pull/4095)) @@ -64,7 +69,7 @@ The most noteworthy change of this release is the update of the container's base - **Removed support for Solr integration:** ([#4025](https://github.com/docker-mailserver/docker-mailserver/pull/4025)) - This was a community contributed feature for FTS (Full Text Search), the docs advise using an image that has not been maintained for over 2 years and lacks ARM64 support. Based on user engagement over the years this feature has very niche value to continue to support, thus is being removed. - If you use Solr, support can be restored if you're willing to contribute docs for the feature that resolves the concerns raised -- **Log**: +- **Log:** - The format of DMS specific logs (_from our scripts, not running services_) has been changed. The new format is ` : ` ([#4035](https://github.com/docker-mailserver/docker-mailserver/pull/4035)) - **rsyslog:** - Debian 12 adjusted the `rsyslog` configuration for the default file template from `RSYSLOG_TraditionalFileFormat` to `RSYSLOG_FileFormat` (_upstream default since 2012_). This change may affect you if you have any monitoring / analysis of log output (_eg: `mail.log` / `docker logs`_). @@ -81,7 +86,7 @@ The most noteworthy change of this release is the update of the container's base - `smtp_sasl_auth_enable = yes` (_SASL auth to outbound MTA connections is enabled_) - `smtp_sasl_security_options = noanonymous` (_credentials are mandatory for outbound mail delivery_) - `smtp_tls_security_level = encrypt` (_the outbound MTA connection must always be secure due to credentials sent_) -- **Environment Variables**: +- **Environment Variables:** - `SA_SPAM_SUBJECT` has been renamed into `SPAM_SUBJECT` to become anti-spam service agnostic. ([#3820](https://github.com/docker-mailserver/docker-mailserver/pull/3820)) - As this functionality is now handled in Dovecot via a Sieve script instead of the respective anti-spam service during Postfix processing, this feature will only apply to mail stored in Dovecot. If you have relied on this feature in a different context, it will no longer be available. - Rspamd previously handled this functionality via the `rewrite_subject` action which as now been disabled by default in favor of the new approach with `SPAM_SUBJECT`. @@ -89,16 +94,16 @@ The most noteworthy change of this release is the update of the container's base - The default has changed to not prepend any prefix to the subject unless configured to do so. If you relied on the implicit prefix, you will now need to provide one explicitly. - `undef` was previously supported as an opt-out with `SA_SPAM_SUBJECT`. This is no longer valid, the equivalent opt-out value is now an empty value (_or rather the omission of this ENV being configured_). - The feature to include [`_SCORE_` tag](https://spamassassin.apache.org/full/4.0.x/doc/Mail_SpamAssassin_Conf.html#rewrite_header-subject-from-to-STRING) in your value to be replaced by the associated spam score is no longer available. -- **Supervisord**: - - `supervisor-app.conf` renamed to `dms-services.conf` -- **Rspamd**: - - the Redis history key has been changed in order to not incorporate the hostname of the container (which is desirable in Kubernetes environments) ([#3927](https://github.com/docker-mailserver/docker-mailserver/pull/3927)) -- **Account Management** - - addresses (accounts) are now normalized to lowercase automatically and a warning is logged in case uppercase letters are supplied +- **Supervisord:** + - `supervisor-app.conf` renamed to `dms-services.conf` ([#3908](https://github.com/docker-mailserver/docker-mailserver/pull/3908)) +- **Rspamd:** + - The Redis history key has been changed in order to not incorporate the hostname of the container (which is desirable in Kubernetes environments) ([#3927](https://github.com/docker-mailserver/docker-mailserver/pull/3927)) +- **Account Management:** + - Addresses (accounts) are now normalized to lowercase automatically and a warning is logged in case uppercase letters are supplied ([#4033](https://github.com/docker-mailserver/docker-mailserver/pull/4033)) ### Added -- **Docs:** +- **Documentation:** - A guide for configuring a public server to relay inbound and outbound mail from DMS on a private server ([#3973](https://github.com/docker-mailserver/docker-mailserver/pull/3973)) - **Environment Variables:** - `LOGROTATE_COUNT` defines the number of files kept by logrotate ([#3907](https://github.com/docker-mailserver/docker-mailserver/pull/3907)) @@ -117,7 +122,7 @@ The most noteworthy change of this release is the update of the container's base - Enable spamassassin only, when amavis is enabled too. ([#3943](https://github.com/docker-mailserver/docker-mailserver/pull/3943)) - **Tests:** - Refactored helper methods for sending e-mails with specific `Message-ID` headers and the helpers for retrieving + filtering logs, which together help isolate logs relevant to specific mail when multiple mails have been processed within a single test. ([#3786](https://github.com/docker-mailserver/docker-mailserver/pull/3786)) -- **Rspamd**: +- **Rspamd:** - The `rewrite_subject` action, is now disabled by default. It has been replaced with the new `SPAM_SUBJECT` environment variable, which implements the functionality via a Sieve script instead which is anti-spam service agnostic ([#3820](https://github.com/docker-mailserver/docker-mailserver/pull/3820)) - `RSPAMD_NEURAL` was added and is disabled by default. If switched on it will enable the experimental Rspamd "Neural network" module to add a layer of analysis to spam detection ([#3833](https://github.com/docker-mailserver/docker-mailserver/pull/3833)) - The symbol weights of SPF, DKIM and DMARC have been adjusted again. Fixes a bug and includes more appropriate combinations of symbols ([#3913](https://github.com/docker-mailserver/docker-mailserver/pull/3913), [#3923](https://github.com/docker-mailserver/docker-mailserver/pull/3923)) @@ -167,12 +172,12 @@ The most noteworthy change of this release is the update of the container's base ### Added -- **Docs:** +- **Documentation:** - An example for how to bind outbound SMTP connections to a specific network interface ([#3465](https://github.com/docker-mailserver/docker-mailserver/pull/3465)) ### Updates -- **Tests**: +- **Tests:** - Revised OAuth2 test ([#3795](https://github.com/docker-mailserver/docker-mailserver/pull/3795)) - Replace `wc -l` with `grep -c` ([#3752](https://github.com/docker-mailserver/docker-mailserver/pull/3752)) - Revised testing of service process management (supervisord) to be more robust ([#3780](https://github.com/docker-mailserver/docker-mailserver/pull/3780)) @@ -184,9 +189,9 @@ The most noteworthy change of this release is the update of the container's base - `test/files/emails/existing/` files were removed similar to previous removal of SMTP auth files as they became redundant with `swaks`. - **Internal:** - Postfix is now configured with `smtputf8_enable = no` in our default `main.cf` config (_instead of during container startup_). ([#3750](https://github.com/docker-mailserver/docker-mailserver/pull/3750)) -- **Rspamd** ([#3726](https://github.com/docker-mailserver/docker-mailserver/pull/3726)): +- **Rspamd:** ([#3726](https://github.com/docker-mailserver/docker-mailserver/pull/3726)): - Symbol scores for SPF, DKIM & DMARC were updated to more closely align with [RFC7489](https://www.rfc-editor.org/rfc/rfc7489#page-24). Please note that complete alignment is undesirable as other symbols may be added as well, which changes the overall score calculation again, see [this issue](https://github.com/docker-mailserver/docker-mailserver/issues/3690#issuecomment-1866871996) -- **Docs:** +- **Documentation:** - Revised the SpamAssassin ENV docs to better communicate configuration and their relation to other ENV settings. ([#3756](https://github.com/docker-mailserver/docker-mailserver/pull/3756)) - Detailed how mail received is assigned a spam score by Rspamd and processed accordingly ([#3773](https://github.com/docker-mailserver/docker-mailserver/pull/3773)) @@ -235,7 +240,7 @@ DMS is now secured against the [recently published spoofing attack "SMTP Smuggli - ENV `ENABLE_IMAP` ([#3703](https://github.com/docker-mailserver/docker-mailserver/pull/3703)) - **Tests:** - You can now use `make run-local-instance` to run a DMS image that was built locally to test changes ([#3663](https://github.com/docker-mailserver/docker-mailserver/pull/3663)) -- **Internal**: +- **Internal:** - Log a warning when update-check is enabled, but no stable release image is used ([#3684](https://github.com/docker-mailserver/docker-mailserver/pull/3684)) ### Updates @@ -250,7 +255,7 @@ DMS is now secured against the [recently published spoofing attack "SMTP Smuggli ### Fixed -- **Internal**: +- **Internal:** - The container startup welcome log message now references `DMS_RELEASE` ([#3676](https://github.com/docker-mailserver/docker-mailserver/pull/3676)) - `VERSION` was incremented for prior releases to be notified of the v13.0.1 patch release ([#3676](https://github.com/docker-mailserver/docker-mailserver/pull/3676)) - `VERSION` is no longer included in the image ([#3711](https://github.com/docker-mailserver/docker-mailserver/pull/3711)) diff --git a/README.md b/README.md index e34aeb12..e4dbf44a 100644 --- a/README.md +++ b/README.md @@ -38,8 +38,8 @@ A production-ready fullstack but simple containerized mail server (SMTP, IMAP, L ## :package: Included Services -- [Postfix](http://www.postfix.org) with SMTP or LDAP authentication and support for [extension delimiters](https://docker-mailserver.github.io/docker-mailserver/v13.3/config/user-management/#address-tags-extension-delimiters-as-an-alternative-to-aliases) -- [Dovecot](https://www.dovecot.org) with SASL, IMAP, POP3, LDAP, [basic Sieve support](https://docker-mailserver.github.io/docker-mailserver/latest/config/advanced/mail-sieve) and [quotas](https://docker-mailserver.github.io/docker-mailserver/v13.3/config/user-management/#quotas) +- [Postfix](http://www.postfix.org) with SMTP or LDAP authentication and support for [extension delimiters](https://docker-mailserver.github.io/docker-mailserver/latest/config/account-management/overview/#aliases) +- [Dovecot](https://www.dovecot.org) with SASL, IMAP, POP3, LDAP, [basic Sieve support](https://docker-mailserver.github.io/docker-mailserver/latest/config/advanced/mail-sieve) and [quotas](https://docker-mailserver.github.io/docker-mailserver/latest/config/account-management/overview/#quotas) - [Rspamd](https://rspamd.com/) - [Amavis](https://www.amavis.org/) - [SpamAssassin](http://spamassassin.apache.org/) supporting custom rules diff --git a/docs/content/config/account-management/overview.md b/docs/content/config/account-management/overview.md new file mode 100644 index 00000000..d94104f2 --- /dev/null +++ b/docs/content/config/account-management/overview.md @@ -0,0 +1,252 @@ +# Account Management - Overview + +This page provides a technical reference for account management in DMS. + +!!! note "Account provisioners and alternative authentication support" + + Each [`ACCOUNT_PROVISIONER`][docs::env::account-provisioner] has a separate page for configuration guidance and caveats: + + - [`FILE` provisioner docs][docs::account-provisioner::file] + - [`LDAP` provisioner docs][docs::account-provisioner::ldap] + + Authentication from the provisioner can be supplemented with additional methods: + + - [OAuth2 / OIDC][docs::account-auth::oauth2] (_allow login from an external authentication service_) + - [Master Accounts][docs::account-auth::master-accounts] (_access the mailbox of any DMS account_) + + --- + + For custom authentication requirements, you could [implement this with Lua][docs::examples::auth-lua]. + +## Accounts + +!!! info + + To receive or send mail, you'll need to provision user accounts into DMS (_as each provisioner page documents_). + + --- + + A DMS account represents a user with their _login username_ + password, and optional config like aliases and quota. + + - Sending mail from different addresses **does not require** aliases or separate accounts. + - Each account is configured with a _primary email address_ that a mailbox is associated to. + +??? info "Primary email address" + + The email address associated to an account creates a mailbox. This address is relevant: + + - When DMS **receives mail** for that address as the recipient (_or an alias that resolves to it_), to identify which mailbox to deliver into. + - With **mail submission**: + - `SPOOF_PROTECTION=1` **restricts the sender address** to the DMS account email address (_unless additional sender addresses have been permitted via supported config_). + - `SPOOF_PROTECTION=0` allows DMS accounts to **use any sender address** (_only a single DMS account is necessary to send mail with different sender addresses_). + + --- + + For more details, see the [Technical Overview](#technical-overview) section. + +??? note "Support for multiple mail domains" + + No extra configuration in DMS is required after provisioning an account with an email address. + + - The DNS records for a domain should direct mail to DMS and allow DMS to send mail on behalf of that domain. + - DMS does not need TLS certificates for your mail domains, only for the DMS FQDN (_the `hostname` setting_). + +??? warning "Choosing a compatible email address" + + An email address should conform to the standard [permitted charset and format][email-syntax::valid-charset-format] (`local-part@domain-part`). + + --- + + DMS has features that need to reserve special characters to work correctly. Ensure those characters are not present in email addresses you configure for DMS, otherwise disable / opt-out of the feature. + + - [Sub-addressing](#sub-addressing) is enabled by default with `+` as the _tag delimiter_. The tag can be changed, feature opt-out when the tag is explicitly unset. + +### Aliases + +!!! info + + Aliases allow receiving mail: + + - As an alternative delivery address for a DMS account mailbox. + - To redirect / forward to an external address outside of DMS like `@gmail.com`. + +??? abstract "Technical Details (_Local vs Virtual aliases_)" + + Aliases are managed through Postfix which supports _local_ and _virtual_ aliases: + + - **Local aliases** are for mail routed to the [`local` delivery agent][postfix::delivery-agent::local] (see [associated alias config format][postfix::config-table::local-alias]) + - You rarely need to configure this. It is used internally for system unix accounts belonging to the services running in DMS (_including `root`_). + - `postmaster` may be a local alias to `root`, and `root` to a virtual alias or real email address. + - Any mail sent through the `local` delivery agent will not be delivered to an inbox managed by Dovecot (_unless you have configured a local alias to redirect mail to a valid address or alias_). + - The domain-part of an these aliases belongs to your DMS FQDN (_`hostname: mail.example.com`, thus `user@mail.example.com`_). Technically there is no domain-part at this point, that context is used when routing delivery, the local delivery agent only knows of the local-part (_an alias or unix account_). + - [**Virtual aliases**][postfix-docs::virtual-alias] are for mail routed to the [`virtual` delivery agent][postfix::delivery-agent::virtual] (see [associated alias config format][postfix::config-table::virtual-alias]) + - When alias support in DMS is discussed without the context of being a local or virtual alias, it's likely the virtual kind (_but could also be agnostic_). + - The domain-part of an these aliases belongs to a mail domain managed by DMS (_like `user@example.com`_). + + !!! tip "Verify alias resolves correctly" + + You can run `postmap -q ` in the container to verify an alias resolves to the expected target. If the target is also an alias, the command will not expand that alias to resolve the actual recipient(s). + + For the `FILE` provisioner, an example would be: `postmap -q alias1@example.com /etc/postfix/virtual`. For the `LDAP` provisioner you'd need to adjust the table path. + + !!! info "Side effect - Dovecot Quotas (`ENABLE_QUOTAS=1`)" + + As a side effect of the alias workaround for the `FILE` provisioner with this feature, aliases can be used for account login. This is not intentional. + +### Quotas + +!!! info + + Enables mail clients with the capability to query a mailbox for disk-space used and capacity limit. + + - This feature is enabled by default, opt-out via [`ENABLE_QUOTAS=0`][docs::env::enable-quotas] + - **Not implemented** for the LDAP provisioner (_PR welcome! View the [feature request for implementation advice][gh-issue::dms-feature-request::dovecot-quotas-ldap]_) + +??? tip "How are quotas useful?" + + Without quota limits for disk storage, a mailbox could fill up the available storage which would cause delivery failures to all mailboxes. + + Quotas help by preventing that abuse, so that only a mailbox exceeding the assigned quota experiences a delivery failure instead of negatively impacting others (_provided disk space is available_). + +??? abstract "Technical Details" + + The [Dovecot Quotas feature][gh-pr::dms-feature::dovecot-quotas] is configured by enabling the [Dovecot `imap-quota` plugin][dovecot-docs::plugin::imap-quota] and using the [`count` quota backend][dovecot-docs::config::quota-backend-count]. + + --- + + **Dovecot workaround for Postfix aliases** + + When mail is delivered to DMS, Postfix will query Dovecot with the recipient(s) to verify quota has not been exceeded. + + This allows early rejection of mail arriving to DMS, preventing a spammer from taking advantage of a [backscatter][wikipedia::backscatter] source if the mail was accepted by Postfix, only to later be rejected by Dovecot for storage when the quota limit was already reached. + + However, Postfix does not resolve aliases until after the incoming mail is accepted. + + 1. Postfix queries Dovecot (_a [`check_policy_service` restriction tied to the Dovecot `quota-status` service][dms::workaround::dovecot-quotas::notes-1]_) with the recipient (_the alias_). + 2. `dovecot: auth: passwd-file(alias@example.com): unknown user` is logged, Postfix is then informed that the recipient mailbox is not full even if it actually was (_since no such user exists in the Dovecot UserDB_). + 3. However, when the real mailbox address that the alias would later resolve into does have a quota that exceeded the configured limit, Dovecot will refuse the mail delivery from Postfix which introduces a backscatter source for spammers. + + As a [workaround to this problem with the `ENABLE_QUOTAS=1` feature][dms::workaround::dovecot-quotas::summary], DMS will add aliases as fake users into Dovecot UserDB (_that are configured with the same data as the real address the alias would resolve to, thus sharing the same mailbox location and quota limit_). This allows Postfix to properly be aware of an aliased mailbox having exceeded the allowed quota. + + **NOTE:** This workaround **only supports** aliases to a single target recipient of a real account address / mailbox. + + - Additionally, aliases that resolve to another alias or to an external address would both fail the UserDB lookup, unable to determine if enough storage is available. + - A proper fix would [implement a Postfix policy service][dms::workaround::dovecot-quotas::notes-2] that could correctly resolve aliases to valid entries in the Dovecot UserDB, querying the `quota-status` service and returning that response to Postfix. + +## Sub-addressing + +!!! info + + [Subaddressing][wikipedia::subaddressing] (_aka **Plus Addressing** or **Address Tags**_) is a feature that allows you to receive mail to an address which includes a tag appended to the `local-part` of a valid account address. + + - A subaddress has a tag delimiter (_default: `+`_), followed by the tag: `+@` + - The subaddress `user+github@example.com` would deliver mail to the same mailbox as `user@example.com`. + - Tags are dynamic. Anything between the `+` and `@` is understood as the tag, no additional configuration required. + - Only the first occurence of the tag delimiter is recognized. Any additional occurences become part of the tag value itself. + +??? tip "When is subaddressing useful?" + + A common use-case is to use a unique tag for each service you register your email address with. + + - Routing delivery to different folders in your mailbox based on the tag (_via a [Sieve filter][docs::sieve::subaddressing]_). + - Data leaks or bulk sales of email addresses. + - If spam / phishing mail you receive has not removed the tag, you will have better insight into where your address was compromised from. + - When the expected tag is missing, this additionally helps identify bad actors. Especially when mail delivery is routed to subfolders by tag. + - For more use-cases, view the end of [this article][web::subaddress-use-cases]. + +??? tip "Changing the tag delimiter" + + Add `recipient_delimiter = +` to these config override files (_replacing `+` with your preferred delimiter_): + + - Postfix: `docker-data/dms/config/postfix-main.cf` + - Dovecot: `docker-data/dms/config/dovecot.cf` + +??? tip "Opt-out of subaddressing" + + Follow the advice to change the tag delimiter, but instead set an empty value (`recipient_delimiter =`). + +??? warning "Only for receiving, not sending" + + Do not attempt to send mail from these tagged addresses, they are not equivalent to aliases. + + This feature is only intended to be used when a mail client sends to a DMS managed recipient address. While DMS does not restrict the sender address you choose to send mail from (_provided `SPOOF_PROTECTION` has not been enabled_), it is often [forbidden by mail services][ms-exchange-docs::limitations]. + +??? abstract "Technical Details" + + The configured tag delimiter (`+`) allows both Postfix and Dovecot to recognize subaddresses. Without this feature configured, the subaddresses would be considered as separate mail accounts rather than routed to a common account address. + + --- + + Internally DMS has the tag delimiter configured by: + + - Applying the Postfix `main.cf` setting: [`recipient_delimiter = +`][postfix-docs::recipient-delimiter] + - Dovecot has the equivalent setting set as `+` by default: [`recipient_delimiter = +`][dovecot-docs::config::recipient-delimiter] + +## Technical Overview + +!!! info + + This section provides insight for understanding how Postfix and Dovecot services are involved. It is intended as a reference for maintainers and contributors. + + - **Postfix** - Handles when mail is delivered (inbound) to DMS, or sent (outbound) from DMS. + - **Dovecot** - Manages access and storage for mail delivered to the DMS account mailboxes of your users. + +??? abstract "Technical Details - Postfix (Inbound vs Outbound)" + + Postfix needs to know how to handle inbound and outbound mail by asking these queries: + + === "Inbound" + + - What mail domains is DMS responsible for handling? (_for accepting mail delivered_) + - What are valid mail addresses for those mail domains? (_reject delivery for users that don't exist_) + - Are there any aliases to redirect mail to 1 or more users, or forward to externally? + + === "Outbound" + + - When `SPOOF_PROTECTION=1`, how should DMS restrict the sender address? (_eg: Users may only send mail from their associated mailbox address_) + +??? abstract "Technical Details - Dovecot (Authentication)" + + Dovecot additionally handles authenticating user accounts for sending and retrieving mail: + + - Over the ports for IMAP and POP3 connections (_110, 143, 993, 995_). + - As the default configured SASL provider, which Postfix delegates user authentication through (_for the submission(s) ports 465 & 587_). Saslauthd can be configured as an alternative SASL provider. + + Dovecot splits all authentication lookups into two categories: + + - A [PassDB][dovecot::docs::passdb] lookup most importantly authenticates the user. It may also provide any other necessary pre-login information. + - A [UserDB][dovecot::docs::userdb] lookup retrieves post-login information specific to a user. + +[docs::env::account-provisioner]: ../environment.md#account_provisioner +[docs::account-provisioner::file]: ./provisioner/file.md +[docs::account-provisioner::ldap]: ./provisioner/ldap.md +[docs::account-auth::oauth2]: ./supplementary/oauth2.md +[docs::account-auth::master-accounts]: ./supplementary/master-accounts.md +[docs::examples::auth-lua]: ../../examples/use-cases/auth-lua.md +[email-syntax::valid-charset-format]: https://stackoverflow.com/questions/2049502/what-characters-are-allowed-in-an-email-address/2049510#2049510 + +[postfix-docs::virtual-alias]: http://www.postfix.org/VIRTUAL_README.html#virtual_alias +[postfix-docs::recipient-delimiter]: http://www.postfix.org/postconf.5.html#recipient_delimiter +[dovecot-docs::config::recipient-delimiter]: https://doc.dovecot.org/settings/core/#core_setting-recipient_delimiter +[postfix::delivery-agent::local]: https://www.postfix.org/local.8.html +[postfix::delivery-agent::virtual]: https://www.postfix.org/virtual.8.html +[postfix::config-table::local-alias]: https://www.postfix.org/aliases.5.html +[postfix::config-table::virtual-alias]: https://www.postfix.org/virtual.5.html + +[docs::env::enable-quotas]: ../environment.md#enable_quotas +[gh-issue::dms-feature-request::dovecot-quotas-ldap]: https://github.com/docker-mailserver/docker-mailserver/issues/2957 +[dovecot-docs::config::quota-backend-count]: https://doc.dovecot.org/configuration_manual/quota/quota_count/#quota-backend-count +[dovecot-docs::plugin::imap-quota]: https://doc.dovecot.org/settings/plugin/imap-quota-plugin/ +[gh-pr::dms-feature::dovecot-quotas]: https://github.com/docker-mailserver/docker-mailserver/pull/1469 +[wikipedia::backscatter]: https://en.wikipedia.org/wiki/Backscatter_%28email%29 +[dms::workaround::dovecot-quotas::notes-1]: https://github.com/docker-mailserver/docker-mailserver/issues/2091#issuecomment-954298788 +[dms::workaround::dovecot-quotas::notes-2]: https://github.com/docker-mailserver/docker-mailserver/pull/2248#issuecomment-953754532 +[dms::workaround::dovecot-quotas::summary]: https://github.com/docker-mailserver/docker-mailserver/pull/2248#issuecomment-955088677 + +[docs::sieve::subaddressing]: ../advanced/mail-sieve.md#subaddress-mailbox-routing +[web::subaddress-use-cases]: https://www.codetwo.com/admins-blog/plus-addressing +[wikipedia::subaddressing]: https://en.wikipedia.org/wiki/Email_address#Sub-addressing +[ms-exchange-docs::limitations]: https://learn.microsoft.com/en-us/exchange/recipients-in-exchange-online/plus-addressing-in-exchange-online#using-plus-addresses + +[dovecot::docs::passdb]: https://doc.dovecot.org/configuration_manual/authentication/password_databases_passdb +[dovecot::docs::userdb]: https://doc.dovecot.org/configuration_manual/authentication/user_databases_userdb diff --git a/docs/content/config/account-management/provisioner/file.md b/docs/content/config/account-management/provisioner/file.md new file mode 100644 index 00000000..5b74ffcc --- /dev/null +++ b/docs/content/config/account-management/provisioner/file.md @@ -0,0 +1,206 @@ +--- +title: 'Account Management | Provisioner (File)' +--- + +# Provisioner - File + +## Management via the `setup` CLI + +The best way to manage DMS accounts and related config files is through our `setup` CLI provided within the container. + +!!! example "Using the `setup` CLI" + + Try the following within the DMS container (`docker exec -it bash`): + + - Add an account: `setup email add ` + - Add an alias: `setup alias add ` + - Learn more about the available subcommands via: `setup help` + + ```bash + # Starts a basic DMS instance and then shells into the container to use the `setup` CLI: + docker run --rm -itd --name dms --hostname mail.example.com mailserver/docker-mailserver + docker exec -it dms bash + + # Create an account: + setup email add hello@example.com your-password-here + + # Create an alias: + setup alias add your-alias-here@example.com hello@example.com + + # Limit the mailbox capacity to 10 MiB: + setup quota set hello@example.com 10M + ``` + + ??? tip "Secure password input" + + When you don't provide a password to the command, you will be prompted for one. This avoids the password being captured in your shell history. + + ```bash + # As you input your password it will not update. + # Press the ENTER key to apply the hidden password input. + $ setup email add hello@example.com + Enter Password: + Confirm Password: + ``` + +!!! note "Account removal via `setup email del`" + + When you remove a DMS account with this command, it will also remove any associated aliases and quota. + + The command will also prompt for deleting the account mailbox from disk, or can be forced with the `-y` flag. + +## Config Reference + +These config files belong to the [Config Volume][docs::volumes::config]. + +### Accounts + +!!! info + + **Config file:** `docker-data/dms/config/postfix-accounts.cf` + + --- + + The config format is line-based with two fields separated by the delimiter `|`: + + - **User:** The primary email address for the account mailbox to use. + - **Password:** A SHA512-CRYPT hash of the account password (_in this example it is `secret`_). + + ??? tip "Password hash without the `setup email add` command" + + A compatible password hash can be generated with: + + ```bash + doveadm pw -s SHA512-CRYPT -u hello@example.com -p secret + ``` + +!!! example "`postfix-accounts.cf` config file" + + In this example DMS manages mail for the domain `example.com`: + + ```cf title="postfix-accounts.cf" + hello@example.com|{SHA512-CRYPT}$6$W4rxRQwI6HNMt9n3$riCi5/OqUxnU8eZsOlZwoCnrNgu1gBGPkJc.ER.LhJCu7sOg9i1kBrRIistlBIp938GdBgMlYuoXYUU5A4Qiv0 + ``` + + --- + + **Dovecot "extra fields"** + + [Appending a third column will customize "extra fields"][gh-issue::provisioner-file::accounts-extra-fields] when converting account data into a Dovecot UserDB entry. + + DMS is not aware of these customizations beyond carrying them over, expect potential for bugs when this feature breaks any assumed conventions used in the scripts (_such as changing the mailbox path or type_). + +!!! note + + Account creation will normalize the provided email address to lowercase, as DMS does not support multiple case-sensitive address variants. + + The email address chosen will also represent the _login username_ credential for mail clients to authenticate with. + +### Aliases + +!!! info + + **Config file:** `docker-data/dms/config/postfix-virtual.cf` + + --- + + The config format is line-based with key value pairs (**alias** --> **target address**), with white-space as a delimiter. + +!!! example "`postfix-virtual.cf` config file" + + In this example DMS manages mail for the domain `example.com`: + + ```cf-extra title="postfix-virtual.cf" + # Alias delivers to an existing account: + alias1@example.com hello@example.com + + # Alias forwards to an external email address: + alias2@example.com external-account@gmail.com + ``` + +??? warning "Known Issues" + + **`setup` CLI prevents an alias and account sharing an address:** + + You cannot presently add a new account (`setup email add`) or alias (`setup alias add`) with an address which already exists as an alias or account in DMS. + + This [restriction was enforced][gh-issue::bugs::account-alias-overlap] due to [problems it could cause][gh-issue::bugs::account-alias-overlap-problem], although there are [use-cases where you may legitimately require this functionality][gh-issue::feature-request::allow-account-alias-overlap]. + + For now you must manually edit the `postfix-virtual.cf` file as a workaround. There are no run-time checks outside of the `setup` CLI related to this restriction. + + --- + + **Wildcard catch-all support (`@example.com`):** + + While this type of alias without a local-part is supported, you must keep in mind that aliases in Postfix have a higher precedence than a real address associated to a DMS account. + + As a result, the wildcard is matched first and will direct mail for that entire domain to the alias target address. To work around this, [you will need an alias for each non-alias address of that domain][gh-issue::bugs::wildcard-catchall]. + + Additionally, Postfix will read the alias config and choose the alias value that matches the recipient address first. Ensure your more specific aliases for the domain are declared above the wildcard alias in the config file. + + --- + + **Aliasing to another alias or multiple recipients:** + + [While aliasing to multiple recipients is possible][gh-discussions::no-support::alias-multiple-targets], DMS does not officially support that. + + - You may experience issues when our feature integrations don't expect more than one target per alias. + - These concerns also apply to the usage of nested aliases (_where the recipient target provided is to an alias instead of a real address_). An example is the [incompatibility with `setup alias add`][gh-issue::bugs::alias-nested]. + +#### Configuring RegEx aliases + +!!! info + + **Config file:** `docker-data/dms/config/postfix-regexp.cf` + + --- + + This config file is similar to the above `postfix-virtual.cf`, but the alias value is instead configured with a regex pattern. + + There is **no `setup` CLI support** for this feature, it is config only. + +!!! example "`postfix-regexp.cf` config file" + + Deliver all mail for `test` users to `qa@example.com` instead: + + ```cf-extra title="postfix-regexp.cf" + # Remember to escape regex tokens like `.` => `\.`, otherwise + # your alias pattern may be more permissive than you intended: + /^test[0-9][0-9]*@example\.com/ qa@example.com + ``` + +??? abstract "Technical Details" + + `postfix-virtual.cf` has precedence, `postfix-regexp.cf` will only be checked if no alias match was found in `postfix-virtual.cf`. + + These files are both copied internally to `/etc/postfix/` and configured in `main.cf` for the `virtual_alias_maps` setting. As `postfix-virtual.cf` is declared first for that setting, it will be processed before using `postfix-regexp.cf` as a fallback. + +### Quotas + +!!! info + + **Config file:** `docker-data/dms/config/dovecot-quotas.cf` + + ---- + + The config format is line-based with two fields separated by the delimiter `:`: + + - **Dovecot UserDB account:** The user DMS account. It should have a matching field in `postfix-accounts.cf`. + - **Quota limit:** Expressed in bytes (_binary unit suffix is supported: `M` => `MiB`, `G` => `GiB`_). + +!!! example "`dovecot-quotas.cf` config file" + + For the account with the mailbox address of `hello@example.com`, it may not exceed 5 GiB in storage: + + ```cf-extra title="dovecot-quotas.cf" + hello@example.com:5G + ``` + +[docs::volumes::config]: ../../advanced/optional-config.md#volumes-config +[gh-issue::provisioner-file::accounts-extra-fields]: https://github.com/docker-mailserver/docker-mailserver/issues/4117 +[gh-issue::feature-request::allow-account-alias-overlap]: https://github.com/docker-mailserver/docker-mailserver/issues/3528 +[gh-issue::bugs::account-alias-overlap-problem]: https://github.com/docker-mailserver/docker-mailserver/issues/3350#issuecomment-1550528898 +[gh-issue::bugs::account-alias-overlap]: https://github.com/docker-mailserver/docker-mailserver/issues/3022#issuecomment-1807816689 +[gh-issue::bugs::wildcard-catchall]: https://github.com/docker-mailserver/docker-mailserver/issues/3022#issuecomment-1610452561 +[gh-issue::bugs::alias-nested]: https://github.com/docker-mailserver/docker-mailserver/issues/3622#issuecomment-1794504849 +[gh-discussions::no-support::alias-multiple-targets]: https://github.com/orgs/docker-mailserver/discussions/3805#discussioncomment-8215417 diff --git a/docs/content/config/advanced/auth-ldap.md b/docs/content/config/account-management/provisioner/ldap.md similarity index 99% rename from docs/content/config/advanced/auth-ldap.md rename to docs/content/config/account-management/provisioner/ldap.md index 397e42eb..607ee3d5 100644 --- a/docs/content/config/advanced/auth-ldap.md +++ b/docs/content/config/account-management/provisioner/ldap.md @@ -1,5 +1,5 @@ --- -title: 'Advanced | LDAP Authentication' +title: 'Account Management | Provisioner (LDAP)' --- ## Introduction @@ -304,5 +304,5 @@ The changes on the configurations necessary to work with Active Directory (**onl - NET_ADMIN ``` -[docs-environment]: ../environment.md -[docs-userpatches]: ./override-defaults/user-patches.md +[docs-environment]: ../../environment.md +[docs-userpatches]: ../../advanced/override-defaults/user-patches.md diff --git a/docs/content/config/account-management/supplementary/master-accounts.md b/docs/content/config/account-management/supplementary/master-accounts.md new file mode 100644 index 00000000..a8665a83 --- /dev/null +++ b/docs/content/config/account-management/supplementary/master-accounts.md @@ -0,0 +1,70 @@ +--- +title: 'Account Management | Master Accounts (Dovecot)' +hide: + - toc # Hide Table of Contents for this page +--- + +This feature is useful for administrative tasks like hot backups. + +!!! note + + This feature is presently [not supported with `ACCOUNT_PROVISIONER=LDAP`][dms::feature::dovecot-master-accounts::caveat-ldap]. + +!!! info + + A _Master Account_: + + - Can login as any user (DMS account) and access their mailbox. + - Is not associated to a separate DMS account, nor is it a DMS account itself. + + --- + + **`setup` CLI support** + + Use the `setup dovecot-master ` commands. These are roughly equivalent to the `setup email` subcommands. + + --- + + **Config file:** `docker-data/dms/config/dovecot-masters.cf` + + The config format is the same as [`postfix-accounts.cf` for `ACCOUNT_PROVISIONER=FILE`][docs::account-management::file::accounts]. + + The only difference is the account field has no `@domain-part` suffix, it is only a username. + +??? abstract "Technical Details" + + [The _Master Accounts_ feature][dms::feature::dovecot-master-accounts] in DMS configures the [Dovecot Master Users][dovecot-docs::auth::master-users] feature with the Dovecot setting [`auth_master_user_separator`][dovecot-docs::config::auth-master-user-separator] (_where the default value is `*`_). + +## Login via Master Account + +!!! info + + To login as another DMS account (`user@example.com`) with POP3 or IMAP, use the following credentials format: + + - Username: `*` (`user@example.com*admin`) + - Password: `` + +!!! example "Verify login functionality" + + In the DMS container, you can verify with the `testsaslauthd` command: + + ```bash + # Prerequisites: + # A regular DMS account to test login through a Master Account: + setup email add user@example.com secret + # Add a new Master Account: + setup dovecot-master add admin top-secret + ``` + + ```bash + # Login with credentials format as described earlier: + testsaslauthd -u 'user@example.com*admin' -p 'top-secret' + ``` + + Alternatively, any mail client should be able to login the equivalent credentials. + +[dms::feature::dovecot-master-accounts]: https://github.com/docker-mailserver/docker-mailserver/pull/2535 +[dms::feature::dovecot-master-accounts::caveat-ldap]: https://github.com/docker-mailserver/docker-mailserver/pull/2535#issuecomment-1118056745 +[dovecot-docs::auth::master-users]: https://doc.dovecot.org/configuration_manual/authentication/master_users/ +[dovecot-docs::config::auth-master-user-separator]: https://doc.dovecot.org/settings/core/#core_setting-auth_master_user_separator +[docs::account-management::file::accounts]: ../provisioner/file.md#accounts diff --git a/docs/content/config/account-management/supplementary/oauth2.md b/docs/content/config/account-management/supplementary/oauth2.md new file mode 100644 index 00000000..fb74aeec --- /dev/null +++ b/docs/content/config/account-management/supplementary/oauth2.md @@ -0,0 +1,145 @@ +--- +title: 'Account Management | OAuth2 Support' +hide: + - toc # Hide Table of Contents for this page +--- + +# Authentication - OAuth2 / OIDC + +This feature enables support for delegating DMS account authentication through to an external _Identity Provider_ (IdP). + +!!! warning "Receiving mail requires a DMS account to exist" + + If you expect DMS to receive mail, you must provision an account into DMS in advance. Otherwise DMS has no awareness of your externally manmaged users and will reject delivery. + + There are [plans to implement support to provision users through a SCIM 2.0 API][dms-feature-request::scim-api]. An IdP that can operate as a SCIM Client (eg: Authentik) would then integrate with DMS for user provisioning. Until then you must keep your user accounts in sync manually via your configured [`ACCOUNT_PROVISIONER`][docs::env::account-provisioner]. + +??? info "How the feature works" + + 1. A **mail client must have support** to acquire an OAuth2 token from your IdP (_however many clients lack generic OAuth2 / OIDC provider support_). + 2. The mail client then provides that token as the user password via the login mechanism `XOAUTH2` or `OAUTHBEARER`. + 3. DMS (Dovecot) will then check the validity of that token against the Authentication Service it was configured with. + 4. If the response returned is valid for the user account, authentication is successful. + + [**XOAUTH2**][google::xoauth2-docs] (_Googles widely adopted implementation_) and **OAUTHBEARER** (_the newer variant standardized by [RFC 7628][rfc::7628] in 2015_) are supported as standards for verifying that a OAuth Bearer Token (_[RFC 6750][rfc::6750] from 2012_) is valid at the identity provider that created the token. The token itself in both cases is expected to be can an opaque _Access Token_, but it is possible to use a JWT _ID Token_ (_which encodes additional information into the token itself_). + + A mail client like Thunderbird has limited OAuth2 / OIDC support. The software maintains a hard-coded list of providers supported. Roundcube is a webmail client that does have support for generic providers, allowing you to integrate with a broader range of IdP services. + + --- + + **Documentation for this feature is WIP** + + See the [initial feature support][dms-feature::oauth2-pr] and [existing issues][dms-feature::oidc-issues] for guidance that has not yet been documented officially. + +??? tip "Verify authentication works" + + If you have a compatible mail client you can verify login through that. + + --- + + ??? example "CLI - Verify with `curl`" + + ```bash + # Shell into your DMS container: + docker exec -it dms bash + + # Adjust these variables for the methods below to use: + export AUTH_METHOD='OAUTHBEARER' USER_ACCOUNT='hello@example.com' ACCESS_TOKEN='DMS_YWNjZXNzX3Rva2Vu' + + # Authenticate via IMAP (Dovecot): + curl --silent --url 'imap://localhost:143' \ + --login-options "AUTH=${AUTH_METHOD}" --user "${USER_ACCOUNT}" --oauth2-bearer "${ACCESS_TOKEN}" \ + --request 'LOGOUT' \ + && grep "dovecot: imap-login: Login: user=<${USER_ACCOUNT}>, method=${AUTH_METHOD}" /var/log/mail/mail.log + + # Authenticate via SMTP (Postfix), sending a mail with the same sender(from) and recipient(to) address: + # NOTE: `curl` seems to require `--upload-file` with some mail content provided to test SMTP auth. + curl --silent --url 'smtp://localhost:587' \ + --login-options "AUTH=${AUTH_METHOD}" --user "${USER_ACCOUNT}" --oauth2-bearer "${ACCESS_TOKEN}" \ + --mail-from "${USER_ACCOUNT}" --mail-rcpt "${USER_ACCOUNT}" --upload-file - <<< 'RFC 5322 content - not important' \ + && grep "postfix/submission/smtpd.*, sasl_method=${AUTH_METHOD}, sasl_username=${USER_ACCOUNT}" /var/log/mail/mail.log + ``` + + --- + + **Troubleshooting:** + + - Add `--verbose` to the curl options. This will output the protocol exchange which includes if authentication was successful or failed. + - The above example chains the `curl` commands with `grep` on DMS logs (_for Dovecot and Postfix services_). When not running `curl` from the DMS container, ensure you check the logs correctly, or inspect the `--verbose` output instead. + + !!! warning "`curl` bug with `XOAUTH2`" + + [Older releases of `curl` have a bug with `XOAUTH2` support][gh-issue::curl::xoauth2-bug] since `7.80.0` (Nov 2021) but fixed from `8.6.0` (Jan 2024). It treats `XOAUTH2` as `OAUTHBEARER`. + + If you use `docker exec` to run `curl` from within DMS, the current DMS v14 release (_Debian 12 with curl `7.88.1`_) is affected by this bug. + +## Config Examples + +### Authentik with Roundcube + +This example assumes you have already set up: + +- A working DMS server +- An Authentik server ([documentation][authentik::docs::install]) +- A Roundcube server ([docker image][roundcube::dockerhub-image] or [bare metal install][roundcube::docs::install]) + +!!! example "Setup Instructions" + + === "1. Docker Mailserver" + + Update your Docker Compose ENV config to include: + + ```env title="compose.yaml" + services: + mailserver: + env: + # Enable the feature: + - ENABLE_OAUTH2=1 + # Specify the user info endpoint URL of the oauth2 server for token inspection: + - OAUTH2_INTROSPECTION_URL=https://authentik.example.com/application/o/userinfo/ + ``` + + === "2. Authentik" + + 1. Create a new OAuth2 provider. + 2. Note the client id and client secret. Roundcube will need this. + 3. Set the allowed redirect url to the equivalent of `https://roundcube.example.com/index.php/login/oauth` for your RoundCube instance. + + === "3. Roundcube" + + Add the following to `oauth2.inc.php` ([documentation][roundcube::docs::config]): + + ```php + $config['oauth_provider'] = 'generic'; + $config['oauth_provider_name'] = 'Authentik'; + $config['oauth_client_id'] = ''; + $config['oauth_client_secret'] = ''; + $config['oauth_auth_uri'] = 'https://authentik.example.com/application/o/authorize/'; + $config['oauth_token_uri'] = 'https://authentik.example.com/application/o/token/'; + $config['oauth_identity_uri'] = 'https://authentik.example.com/application/o/userinfo/'; + + // Optional: disable SSL certificate check on HTTP requests to OAuth server. For possible values, see: + // http://docs.guzzlephp.org/en/stable/request-options.html#verify + $config['oauth_verify_peer'] = false; + + $config['oauth_scope'] = 'email openid profile'; + $config['oauth_identity_fields'] = ['email']; + + // Boolean: automatically redirect to OAuth login when opening Roundcube without a valid session + $config['oauth_login_redirect'] = false; + ``` + +[dms-feature::oauth2-pr]: https://github.com/docker-mailserver/docker-mailserver/pull/3480 +[dms-feature::oidc-issues]: https://github.com/docker-mailserver/docker-mailserver/issues?q=label%3Afeature%2Fauth-oidc +[docs::env::account-provisioner]: ../../environment.md#account_provisioner +[dms-feature-request::scim-api]: https://github.com/docker-mailserver/docker-mailserver/issues/4090 + +[google::xoauth2-docs]: https://developers.google.com/gmail/imap/xoauth2-protocol#the_sasl_xoauth2_mechanism +[rfc::6750]: https://datatracker.ietf.org/doc/html/rfc6750 +[rfc::7628]: https://datatracker.ietf.org/doc/html/rfc7628 +[gh-issue::curl::xoauth2-bug]: https://github.com/curl/curl/issues/10259#issuecomment-1907192556 + +[authentik::docs::install]: https://goauthentik.io/docs/installation/ +[roundcube::dockerhub-image]: https://hub.docker.com/r/roundcube/roundcubemail +[roundcube::docs::install]: https://github.com/roundcube/roundcubemail/wiki/Installation +[roundcube::docs::config]: https://github.com/roundcube/roundcubemail/wiki/Configuration diff --git a/docs/content/config/advanced/auth-oauth2.md b/docs/content/config/advanced/auth-oauth2.md deleted file mode 100644 index 963a6c2c..00000000 --- a/docs/content/config/advanced/auth-oauth2.md +++ /dev/null @@ -1,69 +0,0 @@ ---- -title: 'Advanced | Basic OAuth2 Authentication' ---- - -## Introduction - -!!! warning "This is only a supplement to the existing account provisioners" - - Accounts must still be managed via the configured [`ACCOUNT_PROVISIONER`][env::account-provisioner] (FILE or LDAP). - - Reasoning for this can be found in [#3480][gh-pr::oauth2]. Future iterations on this feature may allow it to become a full account provisioner. - -[gh-pr::oauth2]: https://github.com/docker-mailserver/docker-mailserver/pull/3480 -[env::account-provisioner]: ../environment.md#account_provisioner - -The present OAuth2 support provides the capability for 3rd-party applications such as Roundcube to authenticate with DMS (dovecot) by using a token obtained from an OAuth2 provider, instead of passing passwords around. - -## Example (Authentik & Roundcube) - -This example assumes you have: - -- A working DMS server set up -- An Authentik server set up ([documentation](https://goauthentik.io/docs/installation/)) -- A Roundcube server set up (either [docker](https://hub.docker.com/r/roundcube/roundcubemail/) or [bare metal](https://github.com/roundcube/roundcubemail/wiki/Installation)) - -!!! example "Setup Instructions" - - === "1. Docker Mailserver" - Edit the following values in `mailserver.env`: - ```env - # ----------------------------------------------- - # --- OAUTH2 Section ---------------------------- - # ----------------------------------------------- - - # empty => OAUTH2 authentication is disabled - # 1 => OAUTH2 authentication is enabled - ENABLE_OAUTH2=1 - - # Specify the user info endpoint URL of the oauth2 provider - OAUTH2_INTROSPECTION_URL=https://authentik.example.com/application/o/userinfo/ - ``` - - === "2. Authentik" - 1. Create a new OAuth2 provider - 2. Note the client id and client secret - 3. Set the allowed redirect url to the equivalent of `https://roundcube.example.com/index.php/login/oauth` for your RoundCube instance. - - === "3. Roundcube" - Add the following to `oauth2.inc.php` ([documentation](https://github.com/roundcube/roundcubemail/wiki/Configuration)): - - ```php - $config['oauth_provider'] = 'generic'; - $config['oauth_provider_name'] = 'Authentik'; - $config['oauth_client_id'] = ''; - $config['oauth_client_secret'] = ''; - $config['oauth_auth_uri'] = 'https://authentik.example.com/application/o/authorize/'; - $config['oauth_token_uri'] = 'https://authentik.example.com/application/o/token/'; - $config['oauth_identity_uri'] = 'https://authentik.example.com/application/o/userinfo/'; - - // Optional: disable SSL certificate check on HTTP requests to OAuth server. For possible values, see: - // http://docs.guzzlephp.org/en/stable/request-options.html#verify - $config['oauth_verify_peer'] = false; - - $config['oauth_scope'] = 'email openid profile'; - $config['oauth_identity_fields'] = ['email']; - - // Boolean: automatically redirect to OAuth login when opening Roundcube without a valid session - $config['oauth_login_redirect'] = false; - ``` diff --git a/docs/content/config/advanced/dovecot-master-accounts.md b/docs/content/config/advanced/dovecot-master-accounts.md deleted file mode 100755 index 3b2afe2a..00000000 --- a/docs/content/config/advanced/dovecot-master-accounts.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: 'Advanced | Dovecot master accounts' ---- - -## Introduction - -A dovecot master account is able to login as any configured user. This is useful for administrative tasks like hot backups. - -## Configuration - -It is possible to create, update, delete and list dovecot master accounts using `setup.sh`. See `setup.sh help` for usage. - -This feature is presently [not supported with LDAP](https://github.com/docker-mailserver/docker-mailserver/pull/2535). - -## Logging in - -Once a master account is configured, it is possible to connect to any users mailbox using this account. Log in over POP3/IMAP using the following credential scheme: - -Username: `*` - -Password: `` diff --git a/docs/content/config/advanced/mail-sieve.md b/docs/content/config/advanced/mail-sieve.md index 759e4f78..93ac8601 100644 --- a/docs/content/config/advanced/mail-sieve.md +++ b/docs/content/config/advanced/mail-sieve.md @@ -81,22 +81,76 @@ For more examples or a detailed description of the Sieve language have a look at [sieve-info::examples]: http://sieve.info/examplescripts [third-party::sieve-examples]: https://support.tigertech.net/sieve#sieve-example-rules-jmp -## Automatic Sorting Based on Subaddresses +## Automatic Sorting Based on Sub-addresses { #subaddress-mailbox-routing } -It is possible to sort subaddresses such as `user+mailing-lists@example.com` into a corresponding folder (here: `INBOX/Mailing-lists`) automatically. +When mail is delivered to your account, it is possible to organize storing mail into folders by the [subaddress (tag)][docs::accounts-subaddressing] used. -```sieve -require ["envelope", "fileinto", "mailbox", "subaddress", "variables"]; +!!! example "Example: `user+@example.com` to `INBOX/`" -if envelope :detail :matches "to" "*" { - set :lower :upperfirst "tag" "${1}"; - if mailboxexists "INBOX.${1}" { - fileinto "INBOX.${1}"; - } else { - fileinto :create "INBOX.${tag}"; - } -} -``` + This example sorts mail into inbox folders by their tag: + + ```sieve title="docker-data/dms/config/user@example.com.dovecot.sieve" + require ["envelope", "fileinto", "mailbox", "subaddress", "variables"]; + + # Check if the mail recipient address has a tag (:detail) + if envelope :detail :matches "to" "*" { + # Create a variable `tag`, with the the captured `to` value normalized (SoCIAL => Social) + set :lower :upperfirst "tag" "${1}"; + + # Store the mail into a folder with the tag name, nested under your inbox folder: + if mailboxexists "INBOX.${tag}" { + fileinto "INBOX.${tag}"; + } else { + fileinto :create "INBOX.${tag}"; + } + } + ``` + + When receiving mail for `user+social@example.com` it would be delivered into the `INBOX/Social` folder. + +??? tip "Only redirect mail for specific tags" + + If you want to only handle specific tags, you could replace the envelope condition and tag assignment from the prior example with: + + ```sieve title="docker-data/dms/config/user@example.com.dovecot.sieve" + # Instead of `:matches`, use the default comparator `:is` (exact match) + if envelope :detail "to" "social" { + set "tag" "Social"; + ``` + + ```sieve title="docker-data/dms/config/user@example.com.dovecot.sieve" + # Alternatively you can also provide a list of values to match: + if envelope :detail "to" ["azure", "aws"] { + set "tag" "Cloud"; + ``` + + ```sieve title="docker-data/dms/config/user@example.com.dovecot.sieve" + # Similar to `:matches`, except `:regex` provides enhanced pattern matching. + # NOTE: This example needs you to `require` the "regex" extension + if envelope :detail :regex "to" "^cloud-(azure|aws)$" { + # Normalize the captured azure/aws tag as the resolved value is no longer fixed: + set :lower :upperfirst "vendor" "${1}"; + # If a `.` exists in the tag, it will create nested folders: + set "tag" "Cloud.${vendor}"; + ``` + + **NOTE:** There is no need to lowercase the tag in the conditional as the [`to` value is a case-insensitive check][sieve-docs::envelope]. + +??? abstract "Technical Details" + + - Dovecot supports this feature via the _Sieve subaddress extension_ ([RFC 5233][rfc::5233::sieve-subaddress]). + - Only a single tag per subaddress is supported. Any additional tag delimiters are part of the tag value itself. + - The Dovecot setting [`recipient_delimiter`][dovecot-docs::config::recipient_delimiter] (default: `+`) configures the tag delimiter. This is where the `local-part` of the recipient address will split at, providing the `:detail` (tag) value for Sieve. + + --- + + `INBOX` is the [default namespace configured by Dovecot][dovecot-docs::namespace]. + + - If you omit the `INBOX.` prefix from the sieve script above, the mailbox (folder) for that tag is created at the top-level alongside your Trash and Junk folders. + - The `.` between `INBOX` and `${tag}` is important as a [separator to distinguish mailbox names][dovecot-docs::mailbox-names]. This can vary by mailbox format or configuration. DMS uses [`Maildir`][dovecot-docs::mailbox-formats::maildir] by default, which uses `.` as the separator. + - [`lmtp_save_to_detail_mailbox = yes`][dovecot-docs::config::lmtp_save_to_detail_mailbox] can be set in `/etc/dovecot/conf.d/20-lmtp.conf`: + - This implements the feature globally, except for the tag normalization and `INBOX.` prefix parts of the example script. + - However, if the sieve script is also present, the script has precedence and will handle this task instead when the condition is successful, otherwise falling back to the global feature. ## Manage Sieve @@ -104,8 +158,7 @@ The [Manage Sieve](https://doc.dovecot.org/admin_manual/pigeonhole_managesieve_s !!! example - ```yaml - # compose.yaml + ```yaml title="compose.yaml" ports: - "4190:4190" environment: @@ -122,3 +175,14 @@ The extension is known to work with the following ManageSieve clients: - **[Sieve Editor](https://github.com/thsmi/sieve)** a portable standalone application based on the former Thunderbird plugin. - **[Kmail](https://kontact.kde.org/components/kmail/)** the mail client of [KDE](https://kde.org/)'s Kontact Suite. + +[docs::accounts-subaddressing]: ../account-management/overview.md#sub-addressing + +[dovecot-docs::namespace]: https://doc.dovecot.org/configuration_manual/namespace/ +[dovecot-docs::mailbox-names]: https://doc.dovecot.org/configuration_manual/sieve/usage/#mailbox-names +[dovecot-docs::mailbox-formats::maildir]: https://doc.dovecot.org/admin_manual/mailbox_formats/maildir/#maildir-mbox-format +[dovecot-docs::config::lmtp_save_to_detail_mailbox]: https://doc.dovecot.org/settings/core/#core_setting-lmtp_save_to_detail_mailbox +[dovecot-docs::config::recipient_delimiter]: https://doc.dovecot.org/settings/core/#core_setting-recipient_delimiter + +[rfc::5233::sieve-subaddress]: https://datatracker.ietf.org/doc/html/rfc5233 +[sieve-docs::envelope]: https://thsmi.github.io/sieve-reference/en/test/core/envelope.html diff --git a/docs/content/config/advanced/optional-config.md b/docs/content/config/advanced/optional-config.md index fb1ab29d..57fa68de 100644 --- a/docs/content/config/advanced/optional-config.md +++ b/docs/content/config/advanced/optional-config.md @@ -18,10 +18,7 @@ DMS has several locations in the container which may be worth persisting externa - [Config](#volumes-config): `docker-data/dms/config/` => `/tmp/docker-mailserver/` - [Mail Storage](#volumes-mail): `docker-data/dms/mail-data/` => `/var/mail/` - [State](#volumes-state): `docker-data/dms/mail-state/` => `/var/mail-state/` - - [Logs](#volumes-logs): `docker-data/dms/mail-logs/` => `/var/log/mail/` - -[docker-docs::volumes]: https://docs.docker.com/storage/volumes/ -[docker-docs::volumes::bind-mount]: https://docs.docker.com/storage/bind-mounts/ + - [Logs](#volumes-log): `docker-data/dms/mail-logs/` => `/var/log/mail/` ### Mail Storage Volume { #volumes-mail } @@ -80,8 +77,8 @@ This is a list of all configuration files and directories which are optional, au - **postfix-send-access.cf:** List of users denied sending. Modify via [`setup.sh email restrict`][docs-setupsh]. - **postfix-receive-access.cf:** List of users denied receiving. Modify via [`setup.sh email restrict`][docs-setupsh]. - **postfix-virtual.cf:** Alias configuration file. Modify via [`setup.sh alias`][docs-setupsh]. -- **postfix-sasl-password.cf:** listing of relayed domains with their respective `:`. Modify via `setup.sh relay add-auth []`. (Docs: [Relay-Hosts Auth][docs-relayhosts-senderauth]) -- **postfix-relaymap.cf:** domain-specific relays and exclusions. Modify via `setup.sh relay add-domain` and `setup.sh relay exclude-domain`. (Docs: [Relay-Hosts Senders][docs-relayhosts-senderhost]) +- **postfix-sasl-password.cf:** listing of relayed domains with their respective `:`. Modify via `setup.sh relay add-auth []`. (Docs: [Relay-Hosts Auth][docs::relay-hosts::advanced]) +- **postfix-relaymap.cf:** domain-specific relays and exclusions. Modify via `setup.sh relay add-domain` and `setup.sh relay exclude-domain`. (Docs: [Relay-Hosts Senders][docs::relay-hosts::advanced]) - **postfix-regexp.cf:** Regular expression alias file. (Docs: [Aliases][docs-aliases-regex]) - **ldap-users.cf:** Configuration for the virtual user mapping `virtual_mailbox_maps`. See the [`setup-stack.sh`][github-commit-setup-stack.sh-L411] script. - **ldap-groups.cf:** Configuration for the virtual alias mapping `virtual_alias_maps`. See the [`setup-stack.sh`][github-commit-setup-stack.sh-L411] script. @@ -97,16 +94,18 @@ This is a list of all configuration files and directories which are optional, au - **user-patches.sh:** this file will be run after all configuration files are set up, but before the postfix, amavis and other daemons are started. (Docs: [FAQ - How to adjust settings with the `user-patches.sh` script][docs-faq-userpatches]) - **rspamd/custom-commands.conf:** list of simple commands to adjust Rspamd modules in an easy way (Docs: [Rspamd][docs-rspamd-commands]) -[docs-accounts-quota]: ../../config/user-management.md#quotas -[docs-aliases-regex]: ../../config/user-management.md#configuring-regexp-aliases +[docker-docs::volumes]: https://docs.docker.com/storage/volumes/ +[docker-docs::volumes::bind-mount]: https://docs.docker.com/storage/bind-mounts/ + +[docs-accounts-quota]: ../../config/account-management/provisioner/file.md#quotas +[docs-aliases-regex]: ../../config/account-management/provisioner/file.md#configuring-regex-aliases [docs-dkim]: ../../config/best-practices/dkim_dmarc_spf.md#dkim [docs-fail2ban]: ../../config/security/fail2ban.md [docs-faq-spamrules]: ../../faq.md#how-can-i-manage-my-custom-spamassassin-rules [docs-faq-userpatches]: ../../faq.md#how-to-adjust-settings-with-the-user-patchessh-script [docs-override-postfix]: ./override-defaults/postfix.md [docs-override-dovecot]: ./override-defaults/dovecot.md -[docs-relayhosts-senderauth]: ./mail-forwarding/relay-hosts.md#sender-dependent-authentication -[docs-relayhosts-senderhost]: ./mail-forwarding/relay-hosts.md#sender-dependent-relay-host +[docs::relay-hosts::advanced]: ./mail-forwarding/relay-hosts.md#advanced-configuration [docs-sieve]: ./mail-sieve.md [docs-setupsh]: ../../config/setup.sh.md [docs-ssl]: ../../config/security/ssl.md diff --git a/docs/content/config/advanced/podman.md b/docs/content/config/advanced/podman.md index 4cb60e77..236eeb62 100644 --- a/docs/content/config/advanced/podman.md +++ b/docs/content/config/advanced/podman.md @@ -107,7 +107,7 @@ The `PERMIT_DOCKER` variable in the `mailserver.env` file allows to specify trus #### Use the slip4netns network driver The second workaround is slightly more complicated because the `compose.yaml` has to be modified. -As shown in the [fail2ban section](../security/fail2ban.md#podman-with-slirp4netns-port-driver) the `slirp4netns` network driver has to be enabled. +As shown in the [fail2ban section][docs::fail2ban::rootless] the `slirp4netns` network driver has to be enabled. This network driver enables podman to correctly resolve IP addresses but it is not compatible with user defined networks which might be a problem depending on your setup. @@ -150,7 +150,7 @@ Remember to run this command as root user. ### Port Forwarding -When it comes to forwarding ports using `firewalld`, see for more information. +When it comes to forwarding ports using `firewalld`, see [these port forwarding docs][firewalld-port-forwarding] for more information. ```bash firewall-cmd --permanent --add-forward-port=port=<25|143|465|587|993>:proto=:toport=<10025|10143|10465|10587|10993> @@ -171,5 +171,7 @@ firewall-cmd --reload Just map all the privilege port with non-privilege port you set in compose.yaml before as root user. +[docs::fail2ban::rootless]: ../security/fail2ban.md#rootless-container [rootless::podman]: https://github.com/containers/podman/blob/v3.4.1/docs/source/markdown/podman-run.1.md#--networkmode---net [rootless::podman::interface]: https://github.com/containers/podman/blob/v3.4.1/libpod/networking_slirp4netns.go#L264 +[firewalld-port-forwarding]: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/securing_networks/using-and-configuring-firewalld_securing-networks#port-forwarding_using-and-configuring-firewalld diff --git a/docs/content/config/best-practices/dkim_dmarc_spf.md b/docs/content/config/best-practices/dkim_dmarc_spf.md index d6ba06b7..2ef8e902 100644 --- a/docs/content/config/best-practices/dkim_dmarc_spf.md +++ b/docs/content/config/best-practices/dkim_dmarc_spf.md @@ -359,7 +359,7 @@ volumes: - ./docker-data/dms/config/postfix-policyd-spf.conf:/etc/postfix-policyd-spf-python/policyd-spf.conf ``` -[docs-accounts-add]: ../user-management.md#adding-a-new-account +[docs-accounts]: ../account-management/overview.md#accounts [docs-volumes-config]: ../advanced/optional-config.md#volumes-config [docs-env-opendkim]: ../environment.md#enable_opendkim [docs-env-rspamd]: ../environment.md#enable_rspamd diff --git a/docs/content/config/debugging.md b/docs/content/config/debugging.md index 2c89948d..30fee383 100644 --- a/docs/content/config/debugging.md +++ b/docs/content/config/debugging.md @@ -123,7 +123,7 @@ This could be from outdated software, or running a system that isn't able to pro [docs::faq-bare-domain]: ../faq.md#can-i-use-a-nakedbare-domain-ie-no-hostname [docs-ipv6]: ./advanced/ipv6.md [docs-introduction]: ../introduction.md -[docs-rootless-portdriver]: ./security/fail2ban.md#running-inside-a-rootless-container +[docs::fail2ban::rootless-portdriver]: ./security/fail2ban.md#rootless-container [docs-usage]: ../usage.md [gh-issues]: https://github.com/docker-mailserver/docker-mailserver/issues diff --git a/docs/content/config/environment.md b/docs/content/config/environment.md index 867c7459..2ebd092c 100644 --- a/docs/content/config/environment.md +++ b/docs/content/config/environment.md @@ -47,24 +47,12 @@ The Group ID assigned to the static vmail group for `/var/mail` (_Mail storage m ##### ACCOUNT_PROVISIONER -Configures the provisioning source of user accounts (including aliases) for user queries and authentication by services managed by DMS (_Postfix and Dovecot_). +Configures the [provisioning source of user accounts][docs::account-management::overview] (including aliases) for user queries and authentication by services managed by DMS (_Postfix and Dovecot_). -!!! tip "OAuth2 Support" - - Presently DMS supports OAuth2 only as an supplementary authentication method. - - - A third-party service must provide a valid token for the user which Dovecot validates with the authentication service provider. To enable this feature reference the [OAuth2 configuration example guide][docs::auth::oauth2-config-guide]. - - User accounts must be provisioned to receive mail via one of the supported `ACCOUNT_PROVISIONER` providers. - - User provisioning via OIDC is planned for the future, see [this tracking issue](https://github.com/docker-mailserver/docker-mailserver/issues/2713). - -[docs::auth::oauth2-config-guide]: ./advanced/auth-oauth2.md - -- **empty** => use FILE +- **FILE** => use local files - LDAP => use LDAP authentication -- OIDC => use OIDC authentication (**not yet implemented**) -- FILE => use local files (this is used as the default) -A second container for the ldap service is necessary (e.g. [`bitnami/openldap`](https://hub.docker.com/r/bitnami/openldap/)). +LDAP requires an external service (e.g. [`bitnami/openldap`](https://hub.docker.com/r/bitnami/openldap/)). ##### PERMIT_DOCKER @@ -1140,7 +1128,8 @@ Provide the credentials to use with `RELAY_HOST` or `DEFAULT_RELAY_HOST`. [docs-tls-letsencrypt]: ./security/ssl.md#lets-encrypt-recommended [docs-tls-manual]: ./security/ssl.md#bring-your-own-certificates [docs-tls-selfsigned]: ./security/ssl.md#self-signed-certificates -[docs-accounts-quota]: ./user-management.md#quotas +[docs-accounts-quota]: ./account-management/overview.md#quotas +[docs::account-management::overview]: ./account-management/overview.md [docs::relay-host]: ./advanced/mail-forwarding/relay-hosts.md [docs::dms-volumes-state]: ./advanced/optional-config.md#volumes-state [postfix-config::relayhost]: https://www.postfix.org/postconf.5.html#relayhost diff --git a/docs/content/config/security/fail2ban.md b/docs/content/config/security/fail2ban.md index 375f440c..08852274 100644 --- a/docs/content/config/security/fail2ban.md +++ b/docs/content/config/security/fail2ban.md @@ -78,7 +78,7 @@ docker exec setup fail2ban [ ] docker exec setup fail2ban log ``` -## Running Inside A Rootless Container +## Running Inside A Rootless Container { #rootless-container } [`RootlessKit`][rootless::rootless-kit] is the _fakeroot_ implementation for supporting _rootless mode_ in Docker and Podman. By default, RootlessKit uses the [`builtin` port forwarding driver][rootless::port-drivers], which does not propagate source IP addresses. diff --git a/docs/content/config/security/ssl.md b/docs/content/config/security/ssl.md index eeb39a5f..dd633c9d 100644 --- a/docs/content/config/security/ssl.md +++ b/docs/content/config/security/ssl.md @@ -6,7 +6,7 @@ There are multiple options to enable SSL (via [`SSL_TYPE`][docs-env::ssl-type]): - Using [letsencrypt](#lets-encrypt-recommended) (recommended) - Using [Caddy](#caddy) -- Using [Traefik](#traefik-v2) +- Using [Traefik](#traefik) - Using [self-signed certificates](#self-signed-certificates) - Using [your own certificates](#bring-your-own-certificates) diff --git a/docs/content/config/security/understanding-the-ports.md b/docs/content/config/security/understanding-the-ports.md index ede8ca3b..47e46e95 100644 --- a/docs/content/config/security/understanding-the-ports.md +++ b/docs/content/config/security/understanding-the-ports.md @@ -145,7 +145,7 @@ Unlike with HTTP where a web browser client communicates directly with the serve Other machines that facilitate a connection that generally aren't taken into account can exist between a client and server, such as those where your connection passes through your ISP provider are capable of compromising a `cleartext` connection through interception. -[docs-accounts]: ../user-management.md#accounts +[docs-accounts]: ../account-management/overview.md#accounts [docs-relays]: ../advanced/mail-forwarding/relay-hosts.md [iana-services-465]: https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml?search=465 [starttls-policy-list]: https://github.com/EFForg/starttls-everywhere#email-security-database-starttls-policy-list diff --git a/docs/content/config/user-management.md b/docs/content/config/user-management.md deleted file mode 100644 index a909157e..00000000 --- a/docs/content/config/user-management.md +++ /dev/null @@ -1,86 +0,0 @@ -# User Management - -## Accounts - -Users (email accounts) are managed in `/tmp/docker-mailserver/postfix-accounts.cf`. The best way to manage accounts is to use the reliable `setup` command inside the container. Just run `docker exec setup help` and have a look at the section about subcommands, specifically the `email` subcommand. - -### Adding a new Account - -#### Via `setup` inside the container - -You can add an account by running `docker exec -ti setup email add `. This method is strongly preferred. - -#### Manually - -!!! warning - - This method is discouraged! - -Alternatively, you may directly add the full email address and its encrypted password, separated by a pipe. To generate a new mail account data, directly from your host, you could for example run the following: - -```sh -docker run --rm -it \ - --env MAIL_USER=user1@example.com \ - --env MAIL_PASS=mypassword \ - ghcr.io/docker-mailserver/docker-mailserver:latest \ - /bin/bash -c \ - 'echo "${MAIL_USER}|$(doveadm pw -s SHA512-CRYPT -u ${MAIL_USER} -p ${MAIL_PASS})" >>docker-data/dms/config/postfix-accounts.cf' -``` - -You will then be asked for a password, and be given back the data for a new account entry, as text. To actually _add_ this new account, just copy all the output text in `docker-data/dms/config/postfix-accounts.cf` file of your running container. - -The result could look like this: - -```cf -user1@example.com|{SHA512-CRYPT}$6$2YpW1nYtPBs2yLYS$z.5PGH1OEzsHHNhl3gJrc3D.YMZkvKw/vp.r5WIiwya6z7P/CQ9GDEJDr2G2V0cAfjDFeAQPUoopsuWPXLk3u1 -``` - -### Quotas - -- `imap-quota` is enabled and allow clients to query their mailbox usage. -- When the mailbox is deleted, the quota directive is deleted as well. -- Dovecot quotas support LDAP, **but it's not implemented** (_PRs are welcome!_). - -## Aliases - -The best way to manage aliases is to use the reliable `setup` script inside the container. Just run `docker exec setup help` and have a look at the section about subcommands, specifically the `alias`-subcommand. - -### About - -You may read [Postfix's documentation on virtual aliases][postfix-docs-alias] first. Aliases are managed in `/tmp/docker-mailserver/postfix-virtual.cf`. An alias is a full email address that will either be: - -- delivered to an existing account registered in `/tmp/docker-mailserver/postfix-accounts.cf` -- redirected to one or more other email addresses - -Alias and target are space separated. An example on a server with `example.com` as its domain: - -```cf -# Alias delivered to an existing account -alias1@example.com user1@example.com - -# Alias forwarded to an external email address -alias2@example.com external-account@gmail.com -``` - -### Configuring RegExp Aliases - -Additional regexp aliases can be configured by placing them into `docker-data/dms/config/postfix-regexp.cf`. The regexp aliases get evaluated after the virtual aliases (container path: `/tmp/docker-mailserver/postfix-virtual.cf`). For example, the following `docker-data/dms/config/postfix-regexp.cf` causes all email sent to "test" users to be delivered to `qa@example.com` instead: - -```cf -/^test[0-9][0-9]*@example.com/ qa@example.com -``` - -### Address Tags (Extension Delimiters) as an alternative to Aliases - -Postfix supports so-called address tags, in the form of plus (+) tags - i.e. `address+tag@example.com` will end up at `address@example.com`. This is configured by default and the (configurable!) separator is set to `+`. For more info, see [Postfix's official documentation][postfix-docs-extension-delimiters]. - -!!! note - - If you do decide to change the configurable separator, you must add the same line to *both* `docker-data/dms/config/postfix-main.cf` and `docker-data/dms/config/dovecot.cf`, because Dovecot is acting as the delivery agent. For example, to switch to `-`, add: - - ```cf - recipient_delimiter = - - ``` - -[postfix-docs-alias]: http://www.postfix.org/VIRTUAL_README.html#virtual_alias -[postfix-docs-extension-delimiters]: http://www.postfix.org/postconf.5.html#recipient_delimiter diff --git a/docs/content/contributing/issues-and-pull-requests.md b/docs/content/contributing/issues-and-pull-requests.md index 0a32baab..c454e720 100644 --- a/docs/content/contributing/issues-and-pull-requests.md +++ b/docs/content/contributing/issues-and-pull-requests.md @@ -45,15 +45,15 @@ The development workflow is the following: 1. Fork the project and clone your fork with `git clone --recurse-submodules ...` or run `git submodule update --init --recursive` after you cloned your fork 2. Write the code that is needed :D 3. Add integration tests if necessary -4. [Prepare your environment and run linting and tests][docs-general-tests] -5. Document your improvements if necessary (e.g. if you introduced new environment variables, describe those in the [ENV documentation][docs-environment]) and add your changes the changelog under the "Unreleased" section +4. [Prepare your environment and run linting and tests][docs::contributing::tests] +5. Document your improvements if necessary (e.g. if you introduced new environment variables, describe those in the [ENV documentation][docs::env]) and add your changes the changelog under the "Unreleased" section 6. [Commit][commit] (and [sign your commit][gpg]), push and create a pull-request to merge into `master`. Please **use the pull-request template** to provide a minimum of contextual information and make sure to meet the requirements of the checklist. Pull requests are automatically tested against the CI and will be reviewed when tests pass. When your changes are validated, your branch is merged. CI builds the new `:edge` image immediately and your changes will be includes in the next version release. [docs-latest]: https://docker-mailserver.github.io/docker-mailserver/latest [github-file-readme]: https://github.com/docker-mailserver/docker-mailserver/blob/master/README.md -[docs-environment]: ../config/environment.md -[docs-general-tests]: ./general.md#tests +[docs::env]: ../config/environment.md +[docs::contributing::tests]: ./tests.md [commit]: https://help.github.com/articles/closing-issues-via-commit-messages/ [gpg]: https://docs.github.com/en/github/authenticating-to-github/generating-a-new-gpg-key diff --git a/docs/content/examples/tutorials/mailserver-behind-proxy.md b/docs/content/examples/tutorials/mailserver-behind-proxy.md index 0c39c125..11487747 100644 --- a/docs/content/examples/tutorials/mailserver-behind-proxy.md +++ b/docs/content/examples/tutorials/mailserver-behind-proxy.md @@ -136,6 +136,8 @@ The below guidance is focused on configuring [Traefik][traefik-web], but the adv Postfix and Dovecot are both compatible with PROXY protocol v1 and v2. +#### Ports + ??? abstract "Technical Details - Ports (Traefik config)" !!! info "Explicit TLS (STARTTLS)" @@ -386,7 +388,7 @@ While PROXY protocol works well with the reverse proxy, you may have some contai [docs::overrides::postfix]: ../../config/advanced/override-defaults/postfix.md [docs::overrides::user-patches]: ../../config/advanced/override-defaults/user-patches.md [docs::ipv6::security-risks]: ../../config/advanced/ipv6.md#what-can-go-wrong -[docs::tls::traefik]: ../../config/security/ssl.md#traefik-v2 +[docs::tls::traefik]: ../../config/security/ssl.md#traefik [docs::env::permit_docker]: ../../config/environment.md#permit_docker [gh-dms::dns-rewrite-example]: https://github.com/docker-mailserver/docker-mailserver/issues/3866#issuecomment-1928877236 diff --git a/docs/content/examples/use-cases/auth-lua.md b/docs/content/examples/use-cases/auth-lua.md index e34ed3b0..a7fe50c6 100644 --- a/docs/content/examples/use-cases/auth-lua.md +++ b/docs/content/examples/use-cases/auth-lua.md @@ -156,7 +156,7 @@ If working with HTTP in Lua, setting `debug = true;` when initiating `dovecot.ht Note that Lua runs compiled bytecode, and that scripts will be compiled when they are initially started. Once compiled, the bytecode is cached and changes in the Lua script will not be processed automatically. Dovecot will reload its configuration and clear its cached Lua bytecode when running `docker exec CONTAINER_NAME dovecot reload`. A (changed) Lua script will be compiled to bytecode the next time it is executed after running the Dovecot reload command. -[docs::auth-ldap]: ../../config/advanced/auth-ldap.md +[docs::auth-ldap]: ../../config/account-management/provisioner/ldap.md [docs::dovecot-override-configuration]: ../../config/advanced/override-defaults/dovecot.md#override-configuration [docs::dovecot-add-configuration]: ../../config/advanced/override-defaults/dovecot.md#add-configuration [docs::faq-alter-running-dms-instance-without-container-relaunch]: ../../faq.md#how-to-alter-a-running-dms-instance-without-relaunching-the-container diff --git a/docs/content/usage.md b/docs/content/usage.md index 087547ea..ca4e8dfe 100644 --- a/docs/content/usage.md +++ b/docs/content/usage.md @@ -164,7 +164,7 @@ You definitely want to setup TLS. Please refer to [our documentation about TLS][ You should add at least one [alias][docs-aliases], the [_postmaster alias_][docs-env-postmaster]. This is a common convention, but not strictly required. -[docs-aliases]: ./config/user-management.md#aliases +[docs-aliases]: ./config/account-management/overview.md#aliases [docs-env-postmaster]: ./config/environment.md#postmaster_address ```bash diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index b04631d2..d8ece917 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -104,6 +104,13 @@ markdown_extensions: lang: cfg - name: env lang: properties + # We only show PHP snippets, requires config change to work: + # https://squidfunk.github.io/mkdocs-material/setup/extensions/python-markdown-extensions/#highlight + # https://facelessuser.github.io/pymdown-extensions/extensions/highlight/#extended-pygments-lexer-options + - name: php + lang: php + options: + startinline: true # A variant that sometimes has nicer syntax highlighting: - name: cf-extra lang: linuxconfig @@ -131,7 +138,14 @@ nav: - 'Usage': usage.md - 'Configuration': - 'Environment Variables': config/environment.md - - 'User Management': config/user-management.md + - 'Account Management': + - 'Overview': config/account-management/overview.md + - 'Provisioner': + - 'File Based': config/account-management/provisioner/file.md + - 'LDAP Service': config/account-management/provisioner/ldap.md + - 'Supplementary': + - 'Master Accounts': config/account-management/supplementary/master-accounts.md + - 'OAuth2 Authentication': config/account-management/supplementary/oauth2.md - 'Best Practices': - 'Auto-discovery': config/best-practices/autodiscover.md - 'DKIM, DMARC & SPF': config/best-practices/dkim_dmarc_spf.md @@ -153,8 +167,6 @@ nav: - 'Dovecot': config/advanced/override-defaults/dovecot.md - 'Postfix': config/advanced/override-defaults/postfix.md - 'Modifications via Script': config/advanced/override-defaults/user-patches.md - - 'LDAP Authentication': config/advanced/auth-ldap.md - - 'OAuth2 Authentication': config/advanced/auth-oauth2.md - 'Email Filtering with Sieve': config/advanced/mail-sieve.md - 'Email Gathering with Fetchmail': config/advanced/mail-fetchmail.md - 'Email Gathering with Getmail': config/advanced/mail-getmail.md @@ -166,7 +178,6 @@ nav: - 'Kubernetes': config/advanced/kubernetes.md - 'IPv6': config/advanced/ipv6.md - 'Podman': config/advanced/podman.md - - 'Dovecot Master Accounts': config/advanced/dovecot-master-accounts.md - 'Examples': - 'Tutorials': - 'Basic Installation': examples/tutorials/basic-installation.md diff --git a/mailserver.env b/mailserver.env index 1ec88dd2..77b863ff 100644 --- a/mailserver.env +++ b/mailserver.env @@ -267,7 +267,7 @@ POSTFIX_DAGENT= # empty => 0 POSTFIX_MAILBOX_SIZE_LIMIT= -# See https://docker-mailserver.github.io/docker-mailserver/edge/config/user-management/accounts/#notes +# See https://docker-mailserver.github.io/docker-mailserver/latest/config/account-management/overview/#quotas # 0 => Dovecot quota is disabled # 1 => Dovecot quota is enabled ENABLE_QUOTAS=1 From 009237cc26f76a091d5745e1a26706dd1b937fc7 Mon Sep 17 00:00:00 2001 From: Moritz Poldrack <~git@mp.gy> Date: Mon, 22 Jul 2024 13:43:11 +0200 Subject: [PATCH 134/267] chore: Prevent Microsoft MUAs from sending reactions (#4120) --------- Signed-off-by: Moritz Poldrack <~git@mp.gy> --- CHANGELOG.md | 3 ++- target/postfix/sender_header_filter.pcre | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b39f60d8..490848c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,7 +22,8 @@ All notable changes to this project will be documented in this file. The format - Bump version to [1.1.0](https://github.com/fail2ban/fail2ban/releases/tag/1.1.0). For more information, check the [changelog](https://github.com/fail2ban/fail2ban/blob/1.1.0/ChangeLog). - **Documentation:** - Rewritten and organized the pages for Account Management and Authentication ([#4122](https://github.com/docker-mailserver/docker-mailserver/pull/4122)) - +- **Postfix** + - Disable Microsoft reactions to outgoing mail. ([#4120](https://github.com/docker-mailserver/docker-mailserver/pull/4120)) #### Fixes diff --git a/target/postfix/sender_header_filter.pcre b/target/postfix/sender_header_filter.pcre index cc0a79ab..fe4ff045 100644 --- a/target/postfix/sender_header_filter.pcre +++ b/target/postfix/sender_header_filter.pcre @@ -8,4 +8,4 @@ /^\s*X-Mailer/ IGNORE /^\s*X-Originating-IP/ IGNORE /^\s*Received: from.*127.0.0.1/ IGNORE - +/^Content-Type:/i PREPEND X-MS-Reactions: disallow From 2edd6936c4af9550a54c49b18ff091c93b0a6ffd Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 22 Jul 2024 16:48:59 +0200 Subject: [PATCH 135/267] chore(deps): Bump docker/setup-buildx-action from 3.4.0 to 3.5.0 (#4128) --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 5b10ef87..a38357fd 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -79,7 +79,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.4.0 + uses: docker/setup-buildx-action@v3.5.0 # NOTE: AMD64 can build within 2 minutes - name: 'Build images' diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 42600990..39e1ca23 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -40,7 +40,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.4.0 + uses: docker/setup-buildx-action@v3.5.0 # Try get the cached build layers from a prior `generic_build.yml` job. # NOTE: Until adopting `type=gha` scoped cache exporter (in `docker/build-push-action`), diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 1cab38b8..5bdc88aa 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -38,7 +38,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.4.0 + uses: docker/setup-buildx-action@v3.5.0 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 95b6c4f6..7e675744 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -37,7 +37,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.4.0 + uses: docker/setup-buildx-action@v3.5.0 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. From 720a4a25343fe09743c2ba46a1a09ffaf6e3049c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 22 Jul 2024 16:53:57 +0200 Subject: [PATCH 136/267] chore(deps): Bump docker/setup-qemu-action from 3.1.0 to 3.2.0 (#4127) --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index a38357fd..db46fcc6 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -74,7 +74,7 @@ jobs: cache-buildx- - name: 'Set up QEMU' - uses: docker/setup-qemu-action@v3.1.0 + uses: docker/setup-qemu-action@v3.2.0 with: platforms: arm64 diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 39e1ca23..c10e2ee0 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -35,7 +35,7 @@ jobs: type=semver,pattern={{major}}.{{minor}}.{{patch}} - name: 'Set up QEMU' - uses: docker/setup-qemu-action@v3.1.0 + uses: docker/setup-qemu-action@v3.2.0 with: platforms: arm64 From 093e43480cedec0261fddc513e959e4fa5d112c7 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 23 Jul 2024 08:46:13 +1200 Subject: [PATCH 137/267] chore(deps): Bump docker/build-push-action from 6.4.0 to 6.5.0 (#4126) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6.4.0 to 6.5.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/v6.4.0...v6.5.0) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index db46fcc6..dee6ce93 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -83,7 +83,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v6.4.0 + uses: docker/build-push-action@v6.5.0 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index c10e2ee0..02ce4ffe 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v6.4.0 + uses: docker/build-push-action@v6.5.0 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 5bdc88aa..0c160ad3 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.4.0 + uses: docker/build-push-action@v6.5.0 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 7e675744..7ccfd995 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.4.0 + uses: docker/build-push-action@v6.5.0 with: context: . tags: mailserver-testing:ci From 32c0a346fcec975ba133443d98e3a1e7c0d78714 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 23 Jul 2024 09:12:47 +1200 Subject: [PATCH 138/267] chore(deps): Bump anchore/scan-action from 4.0.0 to 4.1.0 (#4125) Bumps [anchore/scan-action](https://github.com/anchore/scan-action) from 4.0.0 to 4.1.0. - [Release notes](https://github.com/anchore/scan-action/releases) - [Changelog](https://github.com/anchore/scan-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/anchore/scan-action/compare/v4.0.0...v4.1.0) --- updated-dependencies: - dependency-name: anchore/scan-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_vulnerability-scan.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 7ccfd995..7c231b8e 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -55,7 +55,7 @@ jobs: provenance: false - name: 'Run the Anchore Grype scan action' - uses: anchore/scan-action@v4.0.0 + uses: anchore/scan-action@v4.1.0 id: scan with: image: mailserver-testing:ci From c544d770e7f27867e56567bda26a6aa125494638 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue, 30 Jul 2024 18:34:17 +0200 Subject: [PATCH 139/267] docs: updated `CONTRIBUTORS.md` (#4139) --- CONTRIBUTORS.md | 432 ++++++++++++++++++++++++------------------------ 1 file changed, 216 insertions(+), 216 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 9a270918..d4cc387b 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -328,50 +328,6 @@ Thanks goes to these wonderful people ✨ kiliant - - - - - - - - + + + + + + + + - - + + - - - + + - - + + - - + + + + + + + + + + - - - - - - + + - - - - - - + + + + + + + - - + + - - + + - - + + - - - - + + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - - - +
- - robertdolca -
- robertdolca - - 		- - okainov -
- okainov - - 		- - lukecyca -
- lukecyca - - 		- - jsonn -
- jsonn - -
- - jamebus -
- jamebus - - 		- - mathuin -
- mathuin - - 		dashohoxha @@ -387,49 +343,65 @@ Thanks goes to these wonderful people ✨ - - citec + + mathuin
- citec + mathuin 		- - Zehir + + jamebus
- Zehir + jamebus
- - guardiande + + robertdolca
- guardiande + robertdolca 		- - kamuri + + okainov
- kamuri + okainov 		- - davidszp + + jsonn
- davidszp + jsonn 		- - andreasgerstmayr + + lukecyca
- andreasgerstmayr + lukecyca 		+ + m-schmoock +
+ m-schmoock + + 		+ + mjung +
+ mjung + +
VanVan @@ -443,35 +415,49 @@ Thanks goes to these wonderful people ✨
eltociear + 		+ + andreasgerstmayr +
+ andreasgerstmayr + + 		+ + voordev +
+ voordev + + 		+ + davidszp +
+ davidszp + + 		+ + kamuri +
+ kamuri +
- - mjung + + guardiande
- mjung + guardiande 		- - m-schmoock + + Zehir
- m-schmoock - - 		- - Starbix -
- Starbix - - 		- - BrandonSchmitt -
- BrandonSchmitt + Zehir 		@@ -482,19 +468,33 @@ Thanks goes to these wonderful people ✨ - - yajo + + BrandonSchmitt
- yajo + BrandonSchmitt + + 		+ + Starbix +
+ Starbix + + 		+ + citec +
+ citec
- - voordev + + yajo
- voordev + yajo 		@@ -505,17 +505,17 @@ Thanks goes to these wonderful people ✨ - - pbek + + weo
- pbek + weo 		- - reneploetz + + analogue
- reneploetz + analogue 		@@ -526,26 +526,19 @@ Thanks goes to these wonderful people ✨ - - analogue + + reneploetz
- analogue + reneploetz
- - weo + + pbek
- weo - - 		- - andrewlow -
- andrewlow + pbek 		@@ -576,8 +569,6 @@ Thanks goes to these wonderful people ✨ Rillke
bobbravo2 @@ -585,6 +576,8 @@ Thanks goes to these wonderful people ✨ bobbravo2
r-pufky @@ -620,8 +613,6 @@ Thanks goes to these wonderful people ✨ engelant
j-marz @@ -629,6 +620,8 @@ Thanks goes to these wonderful people ✨ j-marz
lokipo @@ -664,8 +657,6 @@ Thanks goes to these wonderful people ✨ frugan-dev
tbutter @@ -673,6 +664,8 @@ Thanks goes to these wonderful people ✨ tbutter
yogo1212 @@ -687,6 +680,13 @@ Thanks goes to these wonderful people ✨ willtho89 + + mpldr +
+ mpldr + + 		mpanneck @@ -694,6 +694,43 @@ Thanks goes to these wonderful people ✨ mpanneck + + craue +
+ craue + + 		+ + danielpanteleit +
+ danielpanteleit + +
+ + ubenmackin +
+ ubenmackin + + 		+ + abh +
+ abh + + 		+ + andrewlow +
+ andrewlow + + 		aminvakil @@ -707,36 +744,6 @@ Thanks goes to these wonderful people ✨
elbracht -
- - abh -
- abh - - 		- - craue -
- craue - - 		- - ubenmackin -
- ubenmackin - - 		- - danielpanteleit -
- danielpanteleit - 		@@ -745,6 +752,8 @@ Thanks goes to these wonderful people ✨ dmcgrandle
theomega @@ -752,8 +761,6 @@ Thanks goes to these wonderful people ✨ theomega
DuncanvR @@ -768,20 +775,6 @@ Thanks goes to these wonderful people ✨ emazzotta - - keslerm -
- keslerm - - 		- - nueaf -
- nueaf - - 		martinwepner @@ -796,8 +789,6 @@ Thanks goes to these wonderful people ✨ artonge
spacecowboy @@ -805,6 +796,8 @@ Thanks goes to these wonderful people ✨ spacecowboy
jiriks74 @@ -812,6 +805,13 @@ Thanks goes to these wonderful people ✨ jiriks74 + + nueaf +
+ nueaf + + 		jedateach @@ -819,6 +819,13 @@ Thanks goes to these wonderful people ✨ jedateach + + keslerm +
+ keslerm + + 		millaguie @@ -826,6 +833,15 @@ Thanks goes to these wonderful people ✨ millaguie + + fl42 +
+ fl42 + +
jamesfryer @@ -840,8 +856,6 @@ Thanks goes to these wonderful people ✨ H4R0
ipernet @@ -850,10 +864,10 @@ Thanks goes to these wonderful people ✨ - - fl42 + + arkanovicz
- fl42 + arkanovicz 		@@ -870,6 +884,8 @@ Thanks goes to these wonderful people ✨ stigok
5ven @@ -884,8 +900,6 @@ Thanks goes to these wonderful people ✨ syl20bnr
sylvaindumont @@ -914,6 +928,8 @@ Thanks goes to these wonderful people ✨ Thiritin
tobiabocchi @@ -928,8 +944,6 @@ Thanks goes to these wonderful people ✨ tknecht
tweibert @@ -958,6 +972,8 @@ Thanks goes to these wonderful people ✨ Twist235
k3it @@ -972,8 +988,6 @@ Thanks goes to these wonderful people ✨ Drakulix
vilisas @@ -995,20 +1009,6 @@ Thanks goes to these wonderful people ✨ 42wim - - ShiriNmi1520 -
- ShiriNmi1520 - - 		- - neuralp -
- neuralp - - 		radicand @@ -1171,6 +1171,13 @@ Thanks goes to these wonderful people ✨ simonsystem + + ShiriNmi1520 +
+ ShiriNmi1520 + + 		matrixes @@ -1185,6 +1192,8 @@ Thanks goes to these wonderful people ✨ mchamplain
millerjason @@ -1192,8 +1201,6 @@ Thanks goes to these wonderful people ✨ millerjason
mplx @@ -1229,6 +1236,8 @@ Thanks goes to these wonderful people ✨ ontheair81
pravynandas @@ -1236,8 +1245,6 @@ Thanks goes to these wonderful people ✨ pravynandas
presocratics @@ -1273,6 +1280,8 @@ Thanks goes to these wonderful people ✨ schnippl0r
smargold476 @@ -1280,8 +1289,6 @@ Thanks goes to these wonderful people ✨ smargold476
sportshead @@ -1317,6 +1324,8 @@ Thanks goes to these wonderful people ✨ vivacarvajalito
wligtenberg @@ -1324,8 +1333,6 @@ Thanks goes to these wonderful people ✨ wligtenberg
wolkenschieber @@ -1361,6 +1368,8 @@ Thanks goes to these wonderful people ✨ arcaine2
awb99 @@ -1368,8 +1377,6 @@ Thanks goes to these wonderful people ✨ awb99
beertje44 @@ -1405,6 +1412,8 @@ Thanks goes to these wonderful people ✨ dimalo
eleith @@ -1412,8 +1421,6 @@ Thanks goes to these wonderful people ✨ eleith
fanqiaojun @@ -1449,6 +1456,8 @@ Thanks goes to these wonderful people ✨ i-C-o-d-e-r
idaadi @@ -1456,8 +1465,6 @@ Thanks goes to these wonderful people ✨ idaadi
ixeft @@ -1493,6 +1500,8 @@ Thanks goes to these wonderful people ✨ landergate
callmemagnus @@ -1500,8 +1509,6 @@ Thanks goes to these wonderful people ✨ callmemagnus
marios88 @@ -1509,13 +1516,6 @@ Thanks goes to these wonderful people ✨ marios88 - - arkanovicz -
- arkanovicz - - 		CBeerta @@ -1993,13 +1993,6 @@ Thanks goes to these wonderful people ✨ MohammedNoureldin - - mpldr -
- mpldr - - 		naveensrinivasan @@ -2007,6 +2000,13 @@ Thanks goes to these wonderful people ✨ naveensrinivasan + + neuralp +
+ neuralp + + 		fkefer From 01194b7552ec974da58b691c1ee2e238de87333b Mon Sep 17 00:00:00 2001 From: Casper Date: Tue, 30 Jul 2024 18:57:33 +0200 Subject: [PATCH 140/267] docs: Add minimal `compose.yaml` examples that demonstrate specific features (#4138) Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- demo-setups/fetchmail-compose.yaml | 60 ++++++++++++ demo-setups/relay-compose.yaml | 145 +++++++++++++++++++++++++++++ 2 files changed, 205 insertions(+) create mode 100644 demo-setups/fetchmail-compose.yaml create mode 100644 demo-setups/relay-compose.yaml diff --git a/demo-setups/fetchmail-compose.yaml b/demo-setups/fetchmail-compose.yaml new file mode 100644 index 00000000..d8632228 --- /dev/null +++ b/demo-setups/fetchmail-compose.yaml @@ -0,0 +1,60 @@ +# Docs: https://docker-mailserver.github.io/docker-mailserver/v14.0/config/advanced/mail-fetchmail +# Additional context, with CLI commands for verification: +# https://github.com/orgs/docker-mailserver/discussions/3994#discussioncomment-9290570 + +services: + dms-fetch: + image: ghcr.io/docker-mailserver/docker-mailserver:latest # :14.0 + hostname: mail.example.test + environment: + ENABLE_FETCHMAIL: 1 + # We change this setting to 10 for quicker testing: + FETCHMAIL_POLL: 10 + # Link the DNS lookup `remote.test` to resolve to the `dms-remote` container IP (for `@remote.test` address): + # This is only for this example, since no real DNS service is configured, this is a Docker internal DNS feature: + links: + - "dms-remote:remote.test" + # NOTE: Optional, You only need to publish ports if you want to verify via your own mail client. + #ports: + # - "465:465" # ESMTP (implicit TLS) + # - "993:993" # IMAP4 (implicit TLS) + # You'd normally use `volumes` here but for simplicity of the example, all config is contained within `compose.yaml`: + configs: + - source: dms-accounts-fetch + target: /tmp/docker-mailserver/postfix-accounts.cf + - source: fetchmail + target: /tmp/docker-mailserver/fetchmail.cf + + dms-remote: + image: ghcr.io/docker-mailserver/docker-mailserver:latest # :14.0 + hostname: mail.remote.test + environment: + # Allows for us send a test mail easily by trusting any mail client run within this container (`swaks`): + PERMIT_DOCKER: container + # Alternatively, trust and accept any mail received from clients in same subnet of dms-fetch: + #PERMIT_DOCKER: connected-networks + configs: + - source: dms-accounts-remote + target: /tmp/docker-mailserver/postfix-accounts.cf + +# Using the Docker Compose `configs.content` feature instead of volume mounting separate files. +# NOTE: This feature requires Docker Compose v2.23.1 (Nov 2023) or newer: +# https://github.com/compose-spec/compose-spec/pull/446 +configs: + fetchmail: + content: | + poll 'mail.remote.test' proto imap + user 'jane.doe@remote.test' + pass 'secret' + is 'john.doe@example.test' + no sslcertck + + # DMS requires an account to complete setup, configure one for each instance: + # NOTE: Both accounts are configured with the same password (SHA512-CRYPT hashed), `secret`. + dms-accounts-fetch: + content: | + john.doe@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8. + + dms-accounts-remote: + content: | + jane.doe@remote.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8. diff --git a/demo-setups/relay-compose.yaml b/demo-setups/relay-compose.yaml new file mode 100644 index 00000000..f4df2de0 --- /dev/null +++ b/demo-setups/relay-compose.yaml @@ -0,0 +1,145 @@ +# Docs: https://docker-mailserver.github.io/docker-mailserver/v14.0/config/advanced/mail-forwarding/relay-hosts/ +# Additional context, with CLI commands for verification: +# https://github.com/docker-mailserver/docker-mailserver/issues/4136#issuecomment-2253693490 + +services: + # This would represent your actual DMS container: + dms-sender: + image: mailserver/docker-mailserver:latest # :14.0 + hostname: mail.example.test + environment: + # All outbound mail will be relayed through this host + # (change the port to 587 if you do not want the postfix-main.cf override) + - DEFAULT_RELAY_HOST=[smtp.relay-service.test]:465 + # Your relay host credentials. + # (since the relay in the example is DMS, the relay account username is a full email address) + - RELAY_USER=relay-user@relay-service.test + - RELAY_PASSWORD=secret + # The mail client (swaks) needs to connect with TLS: + - SSL_TYPE=manual + - SSL_KEY_PATH=/tmp/tls/key.pem + - SSL_CERT_PATH=/tmp/tls/cert.pem + # You would usually have `volumes` instead of this `configs`: + configs: + - source: dms-main + target: /tmp/docker-mailserver/postfix-main.cf + - source: dms-accounts + target: /tmp/docker-mailserver/postfix-accounts.cf + # Authenticating on port 587 or 465 enforces TLS requirement: + - source: tls-cert + target: /tmp/tls/cert.pem + - source: tls-key + target: /tmp/tls/key.pem + # This is only needed if you want to verify the TLS cert chain with swaks + # (normally with public CA providers like LetsEncrypt this file is already available to a mail client) + - source: tls-ca-cert + target: /tmp/tls/ca-cert.pem + + # Pretend this is your third-party relay service: + dms-relay: + image: mailserver/docker-mailserver:latest # :14.0 + hostname: smtp.relay-service.test + environment: + # WORKAROUND: Bypass security checks from the mail-client (dms-sender container) + # (avoids needing valid DNS for this example) + - PERMIT_DOCKER=connected-networks + # TLS is required when relaying to dms-relay via ports 587 / 465 + # (dms-relay will then relay the mail to dms-destination over port 25) + - SSL_TYPE=manual + - SSL_KEY_PATH=/tmp/tls/key.pem + - SSL_CERT_PATH=/tmp/tls/cert.pem + # WORKAROUND: `links` is required due to lack of properly configured DNS. + # (resolves destination.test to the IP of the dms-destination container) + links: + - "dms-destination:destination.test" + configs: + - source: dms-accounts-relay + target: /tmp/docker-mailserver/postfix-accounts.cf + - source: tls-cert + target: /tmp/tls/cert.pem + - source: tls-key + target: /tmp/tls/key.pem + + # Pretend this is another mail server that your target recipient belongs to (like Gmail): + dms-destination: + image: mailserver/docker-mailserver:latest # :14.0 + hostname: mail.destination.test + # Same workaround for purposes of the example, with the target recipient provisioned to accept mail + environment: + - PERMIT_DOCKER=connected-networks + configs: + - source: dms-accounts-destination + target: /tmp/docker-mailserver/postfix-accounts.cf + +# Using the Docker Compose `configs.content` feature instead of volume mounting separate files. +# NOTE: This feature requires Docker Compose v2.23.1 (Nov 2023) or newer: +# https://github.com/compose-spec/compose-spec/pull/446 +configs: + # DMS expects an account to be configured to run, this example provides accounts already created. + # Login credentials: + # user: "john.doe@example.test" password: "secret" + # user: "relay-user@relay-service.test" password: "secret" + # user: "jane.doe@destination.test" password: "secret" + dms-accounts: + # NOTE: `$` needed to be repeated to escape it, + # which opts out of the `compose.yaml` variable interpolation feature. + content: | + john.doe@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8. + + dms-accounts-relay: + content: | + relay-user@relay-service.test|{SHA512-CRYPT}$$6$$o65y1ZXC4ooOPLwZ$$7TF1nYowEtNJpH6BwJBgdj2pPAxaCvhIKQA6ww5zdHm/AA7aemY9eoHC91DOgYNaKj1HLxSeWNDdvrp6mbtUY. + + dms-accounts-destination: + content: | + jane.doe@destination.test|{SHA512-CRYPT}$$6$$o65y1ZXC4ooOPLwZ$$7TF1nYowEtNJpH6BwJBgdj2pPAxaCvhIKQA6ww5zdHm/AA7aemY9eoHC91DOgYNaKj1HLxSeWNDdvrp6mbtUY. + + # This is `postfix-main.cf`, single line change to make all outbound SMTP connections over port 465 instead of 25 (default) + # If you selectively relay mail, you would need to adjust this on the relay service in `/etc/postfix/master.cf`, + # However DMS presently modifies this when using the DMS Relay Host feature support, which may override `postfix-master.cf` or `user-patches.sh` due to `check-for-changes.sh`. + dms-main: + content: | + smtp_tls_wrappermode=yes + + # TLS files: + # - Use an ECDSA cert that's been signed by a self-signed CA for TLS cert verification. + # - This cert is only valid for mail.example.test, mail.destination.test, smtp.relay-service.test + + # `swaks` run in the container will need to reference this CA cert file for successful verficiation (optional). + tls-ca-cert: + content: | + -----BEGIN CERTIFICATE----- + MIIBfTCCASKgAwIBAgIRAMAZttlRlkcuSun0yV0z4RwwCgYIKoZIzj0EAwIwHDEa + MBgGA1UEAxMRU21hbGxzdGVwIFJvb3QgQ0EwHhcNMjEwMTAxMDAwMDAwWhcNMzEw + MTAxMDAwMDAwWjAcMRowGAYDVQQDExFTbWFsbHN0ZXAgUm9vdCBDQTBZMBMGByqG + SM49AgEGCCqGSM49AwEHA0IABJX2hCtoK3+bM5I3rmyApXLJ1gOcVhtoSSwM8XXR + SEl25Kkc0n6mINuMK8UrBkiBUgexf6CYayx3xVr9TmMkg4KjRTBDMA4GA1UdDwEB + /wQEAwIBBjASBgNVHRMBAf8ECDAGAQH/AgEBMB0GA1UdDgQWBBQD8sBrApbyYyqU + y+/TlwGynx2V5jAKBggqhkjOPQQDAgNJADBGAiEAi8N2eOETI+6hY3+G+kzNMd3K + Sd3Ke8b++/nlwr5Fb/sCIQDYAjpKp/MpTDWICeHC2tcB5ptxoTdWkTBuG4rKcktA + 0w== + -----END CERTIFICATE----- + + tls-key: + content: | + -----BEGIN EC PRIVATE KEY----- + MHcCAQEEIOc6wqZmSDmT336K4O26dMk1RCVc0+cmnsO2eK4P5K5yoAoGCCqGSM49 + AwEHoUQDQgAEFOWNgekKKvUZE89vJ7henUYxODYIvCiHitRc2ylwttjqt1KUY1cp + q3jof2fhURHfBUH3dHPXLHig5V9Jw5gqeg== + -----END EC PRIVATE KEY----- + + tls-cert: + content: | + -----BEGIN CERTIFICATE----- + MIIB9DCCAZqgAwIBAgIQE53a/y2c//YXRsz2kLm6gDAKBggqhkjOPQQDAjAcMRow + GAYDVQQDExFTbWFsbHN0ZXAgUm9vdCBDQTAeFw0yMTAxMDEwMDAwMDBaFw0zMTAx + MDEwMDAwMDBaMBkxFzAVBgNVBAMTDlNtYWxsc3RlcCBMZWFmMFkwEwYHKoZIzj0C + AQYIKoZIzj0DAQcDQgAEFOWNgekKKvUZE89vJ7henUYxODYIvCiHitRc2ylwttjq + t1KUY1cpq3jof2fhURHfBUH3dHPXLHig5V9Jw5gqeqOBwDCBvTAOBgNVHQ8BAf8E + BAMCB4AwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMB0GA1UdDgQWBBSz + w74g+O6dcBbwienD70D8A9ESmDAfBgNVHSMEGDAWgBQD8sBrApbyYyqUy+/TlwGy + nx2V5jBMBgNVHREERTBDghFtYWlsLmV4YW1wbGUudGVzdIIVbWFpbC5kZXN0aW5h + dGlvbi50ZXN0ghdzbXRwLnJlbGF5LXNlcnZpY2UudGVzdDAKBggqhkjOPQQDAgNI + ADBFAiEAoety5oClZtuBMkvlUIWRmWlyg1VIOZ544LSEbplsIhcCIHb6awMwNdXP + m/xHjFkuwH1+UjDDRW53Ih7KZoLrQ6Cp + -----END CERTIFICATE----- From 37e5203a6973956d9afadbd9bc575bc5662e7f56 Mon Sep 17 00:00:00 2001 From: Casper Date: Wed, 31 Jul 2024 09:04:51 +0200 Subject: [PATCH 141/267] ci: Remove CONTRIBUTORS.md (#4141) --- .all-contributorsrc | 26 - .gitattributes | 1 - .github/workflows/contributors.yml | 33 - CHANGELOG.md | 6 +- CONTRIBUTORS.md | 2182 ---------------------------- 5 files changed, 5 insertions(+), 2243 deletions(-) delete mode 100644 .all-contributorsrc delete mode 100644 .github/workflows/contributors.yml delete mode 100644 CONTRIBUTORS.md diff --git a/.all-contributorsrc b/.all-contributorsrc deleted file mode 100644 index c3d39c65..00000000 --- a/.all-contributorsrc +++ /dev/null @@ -1,26 +0,0 @@ -{ - "files": [ - "CONTRIBUTORS.md" - ], - "imageSize": 100, - "commit": false, - "contributors": [ - { - "login": "matrixes", - "name": "matrixes", - "avatar_url": "https://avatars.githubusercontent.com/u/46491408?v=4", - "profile": "https://github.com/matrixes", - "contributions": [ - "blog" - ] - } - ], - "contributorsPerLine": 7, - "badgeTemplate": "", - "projectName": "docker-mailserver", - "projectOwner": "docker-mailserver", - "repoType": "github", - "repoHost": "https://github.com", - "skipCi": false, - "commitConvention": "none" -} diff --git a/.gitattributes b/.gitattributes index da5dadda..8933edd9 100644 --- a/.gitattributes +++ b/.gitattributes @@ -28,7 +28,6 @@ Makefile *.yaml text ## PROJECT -.all-contributorsrc text export-ignore .editorconfig text export-ignore .gitattributes text export-ignore .gitignore text export-ignore diff --git a/.github/workflows/contributors.yml b/.github/workflows/contributors.yml deleted file mode 100644 index 6ca6027b..00000000 --- a/.github/workflows/contributors.yml +++ /dev/null @@ -1,33 +0,0 @@ -name: 'Update Contributors List' - -on: - workflow_dispatch: - schedule: - - cron: 0 4 * * 0 - -permissions: - contents: write - pull-requests: write - statuses: write - -jobs: - add-contributors: - name: 'Add Contributors' - runs-on: ubuntu-22.04 - steps: - - name: 'Checkout' - uses: actions/checkout@v4 - - - name: 'Update CONTRIBUTORS.md' - uses: akhilmhdh/contributors-readme-action@v2.3.10 - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - with: - readme_path: CONTRIBUTORS.md - collaborators: all - use_username: true - commit_message: 'docs: updated `CONTRIBUTORS.md`' - committer_username: github-actions[bot] - committer_email: 41898282+github-actions[bot]@users.noreply.github.com - pr_title_on_protected: 'docs: update `CONTRIBUTORS.md`' - auto_detect_branch_protection: true diff --git a/CHANGELOG.md b/CHANGELOG.md index 490848c5..97c48e2a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,12 +25,16 @@ All notable changes to this project will be documented in this file. The format - **Postfix** - Disable Microsoft reactions to outgoing mail. ([#4120](https://github.com/docker-mailserver/docker-mailserver/pull/4120)) -#### Fixes +### Fixes - **Dovecot:** - `logwatch` Update logwatch `ignore.conf` to exclude Xapian messages about pending documents ([#4060](https://github.com/docker-mailserver/docker-mailserver/pull/4060)) - `dovecot-fts-xapian` plugin was updated to `1.7.13`, fixing a regression with indexing ([#4095](https://github.com/docker-mailserver/docker-mailserver/pull/4095)) +### CI + +- Workflow for `CONTRIBUTORS.md` updates removed. `CONTRIBUTORS.md` file and dependencies removed. ([#4141](https://github.com/docker-mailserver/docker-mailserver/pull/4141)) + ## [v14.0.0](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v14.0.0) The most noteworthy change of this release is the update of the container's base image from Debian 11 ("Bullseye") to Debian 12 ("Bookworm"). This update alone involves breaking changes and requires a careful update! diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md deleted file mode 100644 index d4cc387b..00000000 --- a/CONTRIBUTORS.md +++ /dev/null @@ -1,2182 +0,0 @@ -# Contributors - -Thanks goes to these wonderful people ✨ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - casperklein -
- casperklein - - 		- - fbartels -
- fbartels - - 		- - polarathene -
- polarathene - - 		- - NorseGaud -
- NorseGaud - - 		- - wernerfred -
- wernerfred - - 		- - georglauterbach -
- georglauterbach - -
- - tomav -
- tomav - - 		- - erik-wramner -
- erik-wramner - - 		- - chikamichi -
- chikamichi - - 		- - martin-schulze-vireso -
- martin-schulze-vireso - - 		- - Josef-Friedrich -
- Josef-Friedrich - - 		- - johansmitsnl -
- johansmitsnl - -
- - youtous -
- youtous - - 		- - 17Halbe -
- 17Halbe - - 		- - tve -
- tve - - 		- - gmasse -
- gmasse - - 		- - ap-wtioit -
- ap-wtioit - - 		- - 00angus -
- 00angus - -
- - alinmear -
- alinmear - - 		- - dominikwinter -
- dominikwinter - - 		- - crazystick -
- crazystick - - 		- - svenyonson -
- svenyonson - - 		- - swiesend -
- swiesend - - 		- - stonemaster -
- stonemaster - -
- - williamdes -
- williamdes - - 		- - omarc1492 -
- omarc1492 - - 		- - phish108 -
- phish108 - - 		- - mwlczk -
- mwlczk - - 		- - tyranron -
- tyranron - - 		- - KyleOndy -
- KyleOndy - -
- - mindrunner -
- mindrunner - - 		- - MichaelSp -
- MichaelSp - - 		- - m-a-v -
- m-a-v - - 		- - bilak -
- bilak - - 		- - vortex852456 -
- vortex852456 - - 		- - hanscees -
- hanscees - -
- - chris54721 -
- chris54721 - - 		- - jrpear -
- jrpear - - 		- - pyy -
- pyy - - 		- - moqmar -
- moqmar - - 		- - arneke -
- arneke - - 		- - akmet -
- akmet - -
- - diiigle -
- diiigle - - 		- - kiliant -
- kiliant - - 		- - dashohoxha -
- dashohoxha - - 		- - egavard -
- egavard - - 		- - mathuin -
- mathuin - - 		- - jamebus -
- jamebus - -
- - robertdolca -
- robertdolca - - 		- - okainov -
- okainov - - 		- - jsonn -
- jsonn - - 		- - lukecyca -
- lukecyca - - 		- - m-schmoock -
- m-schmoock - - 		- - mjung -
- mjung - -
- - VanVan -
- VanVan - - 		- - eltociear -
- eltociear - - 		- - andreasgerstmayr -
- andreasgerstmayr - - 		- - voordev -
- voordev - - 		- - davidszp -
- davidszp - - 		- - kamuri -
- kamuri - -
- - guardiande -
- guardiande - - 		- - Zehir -
- Zehir - - 		- - Birkenstab -
- Birkenstab - - 		- - BrandonSchmitt -
- BrandonSchmitt - - 		- - Starbix -
- Starbix - - 		- - citec -
- citec - -
- - yajo -
- yajo - - 		- - MakerMatrix -
- MakerMatrix - - 		- - weo -
- weo - - 		- - analogue -
- analogue - - 		- - Rubytastic2 -
- Rubytastic2 - - 		- - reneploetz -
- reneploetz - -
- - pbek -
- pbek - - 		- - castorinop -
- castorinop - - 		- - p-fruck -
- p-fruck - - 		- - rahilarious -
- rahilarious - - 		- - Rillke -
- Rillke - - 		- - bobbravo2 -
- bobbravo2 - -
- - r-pufky -
- r-pufky - - 		- - vincentDcmps -
- vincentDcmps - - 		- - andymel123 -
- andymel123 - - 		- - bigpigeon -
- bigpigeon - - 		- - engelant -
- engelant - - 		- - j-marz -
- j-marz - -
- - lokipo -
- lokipo - - 		- - mmehnert -
- mmehnert - - 		- - msheakoski -
- msheakoski - - 		- - GoliathLabs -
- GoliathLabs - - 		- - frugan-dev -
- frugan-dev - - 		- - tbutter -
- tbutter - -
- - yogo1212 -
- yogo1212 - - 		- - willtho89 -
- willtho89 - - 		- - mpldr -
- mpldr - - 		- - mpanneck -
- mpanneck - - 		- - craue -
- craue - - 		- - danielpanteleit -
- danielpanteleit - -
- - ubenmackin -
- ubenmackin - - 		- - abh -
- abh - - 		- - andrewlow -
- andrewlow - - 		- - aminvakil -
- aminvakil - - 		- - elbracht -
- elbracht - - 		- - dmcgrandle -
- dmcgrandle - -
- - theomega -
- theomega - - 		- - DuncanvR -
- DuncanvR - - 		- - emazzotta -
- emazzotta - - 		- - martinwepner -
- martinwepner - - 		- - artonge -
- artonge - - 		- - spacecowboy -
- spacecowboy - -
- - jiriks74 -
- jiriks74 - - 		- - nueaf -
- nueaf - - 		- - jedateach -
- jedateach - - 		- - keslerm -
- keslerm - - 		- - millaguie -
- millaguie - - 		- - fl42 -
- fl42 - -
- - jamesfryer -
- jamesfryer - - 		- - H4R0 -
- H4R0 - - 		- - ipernet -
- ipernet - - 		- - arkanovicz -
- arkanovicz - - 		- - stephan-devop -
- stephan-devop - - 		- - stigok -
- stigok - -
- - 5ven -
- 5ven - - 		- - syl20bnr -
- syl20bnr - - 		- - sylvaindumont -
- sylvaindumont - - 		- - TechnicLab -
- TechnicLab - - 		- - thomasschmit -
- thomasschmit - - 		- - Thiritin -
- Thiritin - -
- - tobiabocchi -
- tobiabocchi - - 		- - tknecht -
- tknecht - - 		- - tweibert -
- tweibert - - 		- - torus -
- torus - - 		- - VictorKoenders -
- VictorKoenders - - 		- - Twist235 -
- Twist235 - -
- - k3it -
- k3it - - 		- - Drakulix -
- Drakulix - - 		- - vilisas -
- vilisas - - 		- - forzagreen -
- forzagreen - - 		- - 42wim -
- 42wim - - 		- - radicand -
- radicand - -
- - nilshoell -
- nilshoell - - 		- - nknapp -
- nknapp - - 		- - pcqnt -
- pcqnt - - 		- - OrvilleQ -
- OrvilleQ - - 		- - ovidiucp -
- ovidiucp - - 		- - mrPjer -
- mrPjer - -
- - p3dda -
- p3dda - - 		- - peter-hartmann -
- peter-hartmann - - 		- - piwai -
- piwai - - 		- - remoe -
- remoe - - 		- - robbertkl -
- robbertkl - - 		- - romansey -
- romansey - -
- - norrs -
- norrs - - 		- - MightySCollins -
- MightySCollins - - 		- - 501st-alpha1 -
- 501st-alpha1 - - 		- - klamann -
- klamann - - 		- - svdb0 -
- svdb0 - - 		- - 3ap -
- 3ap - -
- - shyim -
- shyim - - 		- - sjmudd -
- sjmudd - - 		- - simonsystem -
- simonsystem - - 		- - ShiriNmi1520 -
- ShiriNmi1520 - - 		- - matrixes -
- matrixes - - 		- - mchamplain -
- mchamplain - -
- - millerjason -
- millerjason - - 		- - mplx -
- mplx - - 		- - odinis -
- odinis - - 		- - okamidash -
- okamidash - - 		- - olaf-mandel -
- olaf-mandel - - 		- - ontheair81 -
- ontheair81 - -
- - pravynandas -
- pravynandas - - 		- - presocratics -
- presocratics - - 		- - rhyst -
- rhyst - - 		- - rmlhuk -
- rmlhuk - - 		- - rriski -
- rriski - - 		- - schnippl0r -
- schnippl0r - -
- - smargold476 -
- smargold476 - - 		- - sportshead -
- sportshead - - 		- - squash -
- squash - - 		- - strarsis -
- strarsis - - 		- - tamueller -
- tamueller - - 		- - vivacarvajalito -
- vivacarvajalito - -
- - wligtenberg -
- wligtenberg - - 		- - wolkenschieber -
- wolkenschieber - - 		- - worldworm -
- worldworm - - 		- - Zepmann -
- Zepmann - - 		- - allddd -
- allddd - - 		- - arcaine2 -
- arcaine2 - -
- - awb99 -
- awb99 - - 		- - beertje44 -
- beertje44 - - 		- - brainkiller -
- brainkiller - - 		- - cternes -
- cternes - - 		- - dborowy -
- dborowy - - 		- - dimalo -
- dimalo - -
- - eleith -
- eleith - - 		- - fanqiaojun -
- fanqiaojun - - 		- - ghnp5 -
- ghnp5 - - 		- - helmutundarnold -
- helmutundarnold - - 		- - hnws -
- hnws - - 		- - i-C-o-d-e-r -
- i-C-o-d-e-r - -
- - idaadi -
- idaadi - - 		- - ixeft -
- ixeft - - 		- - jjtt -
- jjtt - - 		- - paralax -
- paralax - - 		- - jpduyx -
- jpduyx - - 		- - landergate -
- landergate - -
- - callmemagnus -
- callmemagnus - - 		- - marios88 -
- marios88 - - 		- - CBeerta -
- CBeerta - - 		- - damianmoore -
- damianmoore - - 		- - espitall -
- espitall - - 		- - dkarski -
- dkarski - -
- - dbellavista -
- dbellavista - - 		- - danielvandenberg95 -
- danielvandenberg95 - - 		- - denisix -
- denisix - - 		- - mlatorre31 -
- mlatorre31 - - 		- - mazzz1y -
- mazzz1y - - 		- - doominator42 -
- doominator42 - -
- - aydodo -
- aydodo - - 		- - vedtam -
- vedtam - - 		- - edvorg -
- edvorg - - 		- - eliroca -
- eliroca - - 		- - ekkis -
- ekkis - - 		- - ErikEngerd -
- ErikEngerd - -
- - huncode -
- huncode - - 		- - gitfeber -
- gitfeber - - 		- - felixn -
- felixn - - 		- - flole -
- flole - - 		- - froks -
- froks - - 		- - JOduMonT -
- JOduMonT - -
- - 0xflotus -
- 0xflotus - - 		- - ifokeev -
- ifokeev - - 		- - 20th -
- 20th - - 		- - 2b -
- 2b - - 		- - askz -
- askz - - 		- - aspettl -
- aspettl - -
- - acch -
- acch - - 		- - vifino -
- vifino - - 		- - kachkaev -
- kachkaev - - 		- - alexanderneu -
- alexanderneu - - 		- - ch3sh1r -
- ch3sh1r - - 		- - eglia -
- eglia - -
- - groupmsl -
- groupmsl - - 		- - green-anger -
- green-anger - - 		- - iRhonin -
- iRhonin - - 		- - MrFreezeex -
- MrFreezeex - - 		- - arunvc -
- arunvc - - 		- - astrocket -
- astrocket - -
- - baxerus -
- baxerus - - 		- - spock -
- spock - - 		- - erdos4d -
- erdos4d - - 		- - crash7 -
- crash7 - - 		- - auchri -
- auchri - - 		- - Kaan88 -
- Kaan88 - -
- - akkumar -
- akkumar - - 		- - thechubbypanda -
- thechubbypanda - - 		- - KCrawley -
- KCrawley - - 		- - khuedoan -
- khuedoan - - 		- - UltraCoderRU -
- UltraCoderRU - - 		- - JustAnother1 -
- JustAnother1 - -
- - leowinterde -
- leowinterde - - 		- - linhandev -
- linhandev - - 		- - luke- -
- luke- - - 		- - LucidityCrash -
- LucidityCrash - - 		- - MadsRC -
- MadsRC - - 		- - madmath03 -
- madmath03 - -
- - maxemann96 -
- maxemann96 - - 		- - dragetd -
- dragetd - - 		- - michaeljensen -
- michaeljensen - - 		- - exhuma -
- exhuma - - 		- - milas -
- milas - - 		- - mcchots -
- mcchots - -
- - MohammedNoureldin -
- MohammedNoureldin - - 		- - naveensrinivasan -
- naveensrinivasan - - 		- - neuralp -
- neuralp - - 		- - fkefer -
- fkefer - - 		- - furstblumier -
- furstblumier - - 		- - Marsu31 -
- Marsu31 - -
- - glandais -
- glandais - - 		- - GiovanH -
- GiovanH - - 		- - Amphaal -
- Amphaal - - 		- - harryyoud -
- harryyoud - - 		- - HeySora -
- HeySora - - 		- - sirgantrithon -
- sirgantrithon - -
- - Influencer -
- Influencer - - 		- - in-seo -
- in-seo - - 		- - firefly-cpp -
- firefly-cpp - - 		- - JacksonZ03 -
- JacksonZ03 - - 		- - JamBalaya56562 -
- JamBalaya56562 - - 		- - jcalfee -
- jcalfee - -
- - mivek -
- mivek - - 		- - init-js -
- init-js - - 		- - Jeidnx -
- Jeidnx - - 		- - jessp01 -
- jessp01 - - 		- - JiLleON -
- JiLleON - - 		- - jirislav -
- jirislav - -
- - jmccl -
- jmccl - - 		- - jurekbarth -
- jurekbarth - -
- From a338c0663931d621105b42350e44608ffc8abf13 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Thu, 1 Aug 2024 08:57:36 +1200 Subject: [PATCH 142/267] docs: Add caveat for ENV `DMS_VMAIL_UID` value compatibility (#4143) --- CHANGELOG.md | 9 +++++---- docs/content/config/environment.md | 8 ++++++++ 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 97c48e2a..e28aa1a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,7 +14,7 @@ All notable changes to this project will be documented in this file. The format ### Added - **Internal:** - - Add password confirmation to several `setup.sh` commands ([#4072](https://github.com/docker-mailserver/docker-mailserver/pull/4072)) + - Add password confirmation to several `setup` CLI subcommands ([#4072](https://github.com/docker-mailserver/docker-mailserver/pull/4072)) ### Updates @@ -22,13 +22,14 @@ All notable changes to this project will be documented in this file. The format - Bump version to [1.1.0](https://github.com/fail2ban/fail2ban/releases/tag/1.1.0). For more information, check the [changelog](https://github.com/fail2ban/fail2ban/blob/1.1.0/ChangeLog). - **Documentation:** - Rewritten and organized the pages for Account Management and Authentication ([#4122](https://github.com/docker-mailserver/docker-mailserver/pull/4122)) -- **Postfix** - - Disable Microsoft reactions to outgoing mail. ([#4120](https://github.com/docker-mailserver/docker-mailserver/pull/4120)) + - Add caveat for `DMS_VMAIL_UID` not being compatible with `0` / root ([#4143](https://github.com/docker-mailserver/docker-mailserver/pull/4143)) +- **Postfix:** + - Disable Microsoft reactions to outgoing mail ([#4120](https://github.com/docker-mailserver/docker-mailserver/pull/4120)) ### Fixes - **Dovecot:** - - `logwatch` Update logwatch `ignore.conf` to exclude Xapian messages about pending documents ([#4060](https://github.com/docker-mailserver/docker-mailserver/pull/4060)) + - Update logwatch `ignore.conf` to exclude Xapian messages about pending documents ([#4060](https://github.com/docker-mailserver/docker-mailserver/pull/4060)) - `dovecot-fts-xapian` plugin was updated to `1.7.13`, fixing a regression with indexing ([#4095](https://github.com/docker-mailserver/docker-mailserver/pull/4095)) ### CI diff --git a/docs/content/config/environment.md b/docs/content/config/environment.md index 2ebd092c..0a65b1e4 100644 --- a/docs/content/config/environment.md +++ b/docs/content/config/environment.md @@ -39,6 +39,12 @@ Default: 5000 The User ID assigned to the static vmail user for `/var/mail` (_Mail storage managed by Dovecot_). +!!! warning "Incompatible UID values" + + - A value of [`0` (root) is not compatible][gh-issue::vmail-uid-cannot-be-root]. + - This feature will attempt to adjust the `uid` for the `docker` user (`/etc/passwd`), hence the error emitted to logs if the UID is already assigned to another user. + - The feature appears to work with other UID values that are already assigned in `/etc/passwd`, even though Dovecot by default has a setting for the minimum UID as `500`. + ##### DMS_VMAIL_GID Default: 5000 @@ -1123,6 +1129,8 @@ Provide the credentials to use with `RELAY_HOST` or `DEFAULT_RELAY_HOST`. - Add the exact relayhost value (`host:port` / `[host]:port`) from the generated `/etc/postfix/relayhost_map`, or `main.cf:relayhost` (`DEFAULT_RELAY_HOST`). - `setup relay ...` is missing support, you must instead add these manually to `postfix-sasl-password.cf`. +[gh-issue::vmail-uid-cannot-be-root]: https://github.com/docker-mailserver/docker-mailserver/issues/4098#issuecomment-2257201025 + [docs-rspamd]: ./security/rspamd.md [docs-tls]: ./security/ssl.md [docs-tls-letsencrypt]: ./security/ssl.md#lets-encrypt-recommended From 2f8ad142eccfe3d6291373629c5b1dbf89b71f52 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Fri, 2 Aug 2024 09:42:39 +1200 Subject: [PATCH 143/267] fix: Prevent `stderr` being written to `/etc/postfix/main.cf` (#4147) `stderr` is filtered by `grep` to discard unwanted (expected) log noise when appending the override `postfix-main.cf` content (_updated settings did not replace earlier defined instances_). That `grep` filter introduced a regression into DMS v14 release, since any other `stderr` content not being excluded was now blended into `stdout` and redirected with the original `stdout` output for the `postconf -n` command. The fix is to ensure the `grep` output is redirect to `stderr` to avoid that mishap. --- CHANGELOG.md | 2 ++ target/scripts/startup/setup.d/postfix.sh | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e28aa1a5..900e867a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,6 +31,8 @@ All notable changes to this project will be documented in this file. The format - **Dovecot:** - Update logwatch `ignore.conf` to exclude Xapian messages about pending documents ([#4060](https://github.com/docker-mailserver/docker-mailserver/pull/4060)) - `dovecot-fts-xapian` plugin was updated to `1.7.13`, fixing a regression with indexing ([#4095](https://github.com/docker-mailserver/docker-mailserver/pull/4095)) +- **Internal:** + - Fixed a regression introduced in v14 where `postfix-main.cf` appended `stderr` output into `/etc/postfix/main.cf`, causing Postfix startup to fail ([#4147](https://github.com/docker-mailserver/docker-mailserver/pull/4147)) ### CI diff --git a/target/scripts/startup/setup.d/postfix.sh b/target/scripts/startup/setup.d/postfix.sh index 3ebdeb9d..e140fbaa 100644 --- a/target/scripts/startup/setup.d/postfix.sh +++ b/target/scripts/startup/setup.d/postfix.sh @@ -129,7 +129,7 @@ function __postfix__setup_override_configuration() { # Do not directly output to 'main.cf' as this causes a read-write-conflict. # `postconf` output is filtered to skip expected warnings regarding overrides: # https://github.com/docker-mailserver/docker-mailserver/pull/3880#discussion_r1510414576 - postconf -n >/tmp/postfix-main-new.cf 2> >(grep -v 'overriding earlier entry') + postconf -n >/tmp/postfix-main-new.cf 2> >(grep -v 'overriding earlier entry' >&2) mv /tmp/postfix-main-new.cf /etc/postfix/main.cf _adjust_mtime_for_postfix_maincf From 526fd64d110ae792748a2baa6a582e8bd3e7d150 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Sat, 3 Aug 2024 12:04:21 +1200 Subject: [PATCH 144/267] fix: Ensure main log file is tailed from the start (#4146) Co-authored-by: Casper --- CHANGELOG.md | 1 + target/scripts/start-mailserver.sh | 12 +++++++++--- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 900e867a..a6edc099 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,7 @@ All notable changes to this project will be documented in this file. The format - Update logwatch `ignore.conf` to exclude Xapian messages about pending documents ([#4060](https://github.com/docker-mailserver/docker-mailserver/pull/4060)) - `dovecot-fts-xapian` plugin was updated to `1.7.13`, fixing a regression with indexing ([#4095](https://github.com/docker-mailserver/docker-mailserver/pull/4095)) - **Internal:** + - The main `mail.log` which is piped to stdout via `tail` now correctly begins from the first log line of the active container run. Previously some daemon logs and potential warnings/errors were omitted. ([#4146](https://github.com/docker-mailserver/docker-mailserver/pull/4146)) - Fixed a regression introduced in v14 where `postfix-main.cf` appended `stderr` output into `/etc/postfix/main.cf`, causing Postfix startup to fail ([#4147](https://github.com/docker-mailserver/docker-mailserver/pull/4147)) ### CI diff --git a/target/scripts/start-mailserver.sh b/target/scripts/start-mailserver.sh index cbe38da9..1c43616d 100755 --- a/target/scripts/start-mailserver.sh +++ b/target/scripts/start-mailserver.sh @@ -194,10 +194,16 @@ fi # marker to check if container was restarted date >/CONTAINER_START +# Container logs will receive updates from this log file: +MAIN_LOGFILE=/var/log/mail/mail.log +# NOTE: rsyslogd would usually create this later during `_start_daemons`, however it would already exist if the container was restarted. +touch "${MAIN_LOGFILE}" +# Ensure `tail` follows the correct position of the log file for this container start (new logs begin once `_start_daemons` is called) +TAIL_START=$(( $(wc -l < "${MAIN_LOGFILE}") + 1 )) + [[ ${LOG_LEVEL} =~ (debug|trace) ]] && print-environment _start_daemons +# Container start-up scripts completed. `tail` will now pipe the log updates to stdout: _log 'info' "${HOSTNAME} is up and running" - -touch /var/log/mail/mail.log -exec tail -Fn 0 /var/log/mail/mail.log +exec tail -Fn "+${TAIL_START}" "${MAIN_LOGFILE}" From d61909bdea85604a99ce9e211af0c69dadb03073 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 6 Aug 2024 19:15:35 +1200 Subject: [PATCH 145/267] chore(deps): Bump docker/setup-buildx-action from 3.5.0 to 3.6.1 (#4152) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3.5.0 to 3.6.1. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](https://github.com/docker/setup-buildx-action/compare/v3.5.0...v3.6.1) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index dee6ce93..8442546c 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -79,7 +79,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.5.0 + uses: docker/setup-buildx-action@v3.6.1 # NOTE: AMD64 can build within 2 minutes - name: 'Build images' diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 02ce4ffe..52691317 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -40,7 +40,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.5.0 + uses: docker/setup-buildx-action@v3.6.1 # Try get the cached build layers from a prior `generic_build.yml` job. # NOTE: Until adopting `type=gha` scoped cache exporter (in `docker/build-push-action`), diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 0c160ad3..59e2998e 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -38,7 +38,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.5.0 + uses: docker/setup-buildx-action@v3.6.1 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 7c231b8e..a9b4144a 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -37,7 +37,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.5.0 + uses: docker/setup-buildx-action@v3.6.1 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. From fb57905aa3248ee215fc96aad16ec203a8014975 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 13 Aug 2024 13:26:05 +0200 Subject: [PATCH 146/267] chore(deps): Bump docker/build-push-action from 6.5.0 to 6.6.1 (#4158) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 8442546c..76f7014b 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -83,7 +83,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v6.5.0 + uses: docker/build-push-action@v6.6.1 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 52691317..ffb8a677 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v6.5.0 + uses: docker/build-push-action@v6.6.1 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 59e2998e..0f2f9031 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.5.0 + uses: docker/build-push-action@v6.6.1 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index a9b4144a..e33217a1 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.5.0 + uses: docker/build-push-action@v6.6.1 with: context: . tags: mailserver-testing:ci From b2978fd760f5ab4c33751edda8331e3d79aa6cf1 Mon Sep 17 00:00:00 2001 From: Casper Date: Sat, 17 Aug 2024 12:14:59 +0200 Subject: [PATCH 147/267] breaking: Refactor `getmail` support (#4156) Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> --- CHANGELOG.md | 13 ++++ Dockerfile | 3 +- .../getmail/getmailrc_general.cf | 4 ++ config-examples/getmail/imap-example.cf | 13 ++++ config-examples/getmail/pop3-example.cf | 13 ++++ docs/content/config/advanced/mail-getmail.md | 25 +++++--- docs/content/config/environment.md | 4 +- mailserver.env | 2 +- target/bin/debug-getmail | 19 ++++-- target/bin/getmail-cron | 8 --- target/getmail/getmail-service.sh | 47 ++++++++++++++ target/getmail/getmailrc_general | 11 ++++ target/scripts/start-mailserver.sh | 1 + target/scripts/startup/daemons-stack.sh | 1 + target/scripts/startup/setup.d/getmail.sh | 62 +++++++++++-------- target/scripts/startup/setup.d/mail_state.sh | 1 + target/supervisor/conf.d/dms-services.conf | 9 +++ .../getmail/{getmail-user3.cf => user3.cf} | 0 test/tests/parallel/set1/getmail.bats | 17 ++--- .../process_check_restart.bats | 4 ++ 20 files changed, 198 insertions(+), 59 deletions(-) rename target/getmail/getmailrc => config-examples/getmail/getmailrc_general.cf (59%) create mode 100644 config-examples/getmail/imap-example.cf create mode 100644 config-examples/getmail/pop3-example.cf delete mode 100644 target/bin/getmail-cron create mode 100644 target/getmail/getmail-service.sh create mode 100644 target/getmail/getmailrc_general rename test/config/getmail/{getmail-user3.cf => user3.cf} (100%) diff --git a/CHANGELOG.md b/CHANGELOG.md index a6edc099..1a253234 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,19 @@ All notable changes to this project will be documented in this file. The format > **Note**: Changes and additions listed here are contained in the `:edge` image tag. These changes may not be as stable as released changes. +### Breaking + +- **getmail6** has been refactored: ([#4156](https://github.com/docker-mailserver/docker-mailserver/pull/4156)) + - The [DMS config volume](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/advanced/optional-config/#volumes) now has support for `getmailrc_general.cf` for overriding [common default settings](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/advanced/mail-getmail/#common-options). If you previously mounted this config file directly to `/etc/getmailrc_general` you should switch to our config volume support. + - IMAP/POP3 example configs added to our [`config-examples`](https://github.com/docker-mailserver/docker-mailserver/tree/v15.0.0/config-examples/getmail). + - ENV [`GETMAIL_POLL`](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/environment/#getmail_poll) now supports values above 30 minutes. + - Added `getmail` as a new service for `supervisor` to manage, replacing cron for periodic polling. + - Generated getmail configuration files no longer set the `message_log` option. Instead of individual log files per config, the [default base settings DMS configures](https://github.com/docker-mailserver/docker-mailserver/tree/v15.0.0/target/getmail/getmailrc_general) now enables `message_log_syslog`. This aligns with how other services in DMS log to syslog where it is captured in `mail.log`. + - Getmail configurations have changed location from the base of the DMS Config Volume, to the `getmail/` subdirectory. Any existing configurations **must be migrated manually.** + - DMS v14 mistakenly relocated the getmail state directory to the DMS Config Volume as a `getmail/` subdirectory. + - This has been corrected to `/var/lib/getmail` (_if you have mounted a DMS State Volume to `/var/mail-state`, `/var/lib/getmail` will be symlinked to `/var/mail-state/lib-getmail`_). + - To preserve this state when upgrading to DMS v15, **you must manually migrate `getmail/` from the _DMS Config Volume_ to `lib-getmail/` in the _DMS State Volume_.** + ### Security - **Fail2ban:** diff --git a/Dockerfile b/Dockerfile index f6199fd4..02245ebc 100644 --- a/Dockerfile +++ b/Dockerfile @@ -212,7 +212,8 @@ EOF RUN echo 'Reason_Message = Message {rejectdefer} due to: {spf}.' >>/etc/postfix-policyd-spf-python/policyd-spf.conf COPY target/fetchmail/fetchmailrc /etc/fetchmailrc_general -COPY target/getmail/getmailrc /etc/getmailrc_general +COPY target/getmail/getmailrc_general /etc/getmailrc_general +COPY target/getmail/getmail-service.sh /usr/local/bin/ COPY target/postfix/main.cf target/postfix/master.cf /etc/postfix/ # DH parameters for DHE cipher suites, ffdhe4096 is the official standard 4096-bit DH params now part of TLS 1.3 diff --git a/target/getmail/getmailrc b/config-examples/getmail/getmailrc_general.cf similarity index 59% rename from target/getmail/getmailrc rename to config-examples/getmail/getmailrc_general.cf index 57c2b4a9..73fc443f 100644 --- a/target/getmail/getmailrc +++ b/config-examples/getmail/getmailrc_general.cf @@ -1,3 +1,5 @@ +# https://getmail6.org/configuration.html#conf-options + [options] verbose = 0 read_all = false @@ -5,3 +7,5 @@ delete = false max_messages_per_session = 500 received = false delivered_to = false +message_log_syslog = true + diff --git a/config-examples/getmail/imap-example.cf b/config-examples/getmail/imap-example.cf new file mode 100644 index 00000000..fa4fb7d6 --- /dev/null +++ b/config-examples/getmail/imap-example.cf @@ -0,0 +1,13 @@ +# https://getmail6.org/configuration.html + +[retriever] +type = SimpleIMAPSSLRetriever +server = imap.gmail.com +username = alice +password = notsecure + +[destination] +type = MDA_external +path = /usr/lib/dovecot/deliver +allow_root_commands = true +arguments =("-d","user1@example.com") diff --git a/config-examples/getmail/pop3-example.cf b/config-examples/getmail/pop3-example.cf new file mode 100644 index 00000000..dde60c87 --- /dev/null +++ b/config-examples/getmail/pop3-example.cf @@ -0,0 +1,13 @@ +# https://getmail6.org/configuration.html + +[retriever] +type = SimplePOP3SSLRetriever +server = pop3.gmail.com +username = alice +password = notsecure + +[destination] +type = MDA_external +path = /usr/lib/dovecot/deliver +allow_root_commands = true +arguments =("-d","user1@example.com") diff --git a/docs/content/config/advanced/mail-getmail.md b/docs/content/config/advanced/mail-getmail.md index f62388b0..d423b4f0 100644 --- a/docs/content/config/advanced/mail-getmail.md +++ b/docs/content/config/advanced/mail-getmail.md @@ -10,14 +10,19 @@ environment: - GETMAIL_POLL=5 ``` -In your DMS config volume (eg: `docker-data/dms/config/`), create a `getmail-.cf` file for each remote account that you want to retrieve mail and store into a local DMS account. `` should be replaced by you, and is just the rest of the filename (eg: `getmail-example.cf`). The contents of each file should be configuration like documented below. +In your DMS config volume (eg: `docker-data/dms/config/`), add a subdirectory `getmail/` for including your getmail config files (eg: `imap-example.cf`) for each remote account that you want to retrieve mail from and deliver to the mailbox of a DMS account. -The directory structure should similar to this: +The content of these config files is documented in the next section with an IMAP and POP3 example to reference. + +The directory structure should look similar to this: ```txt ├── docker-data/dms/config │   ├── dovecot.cf -│   ├── getmail-example.cf +│ ├── getmail +│   │ ├── getmailrc_general.cf +│   │ ├── remote-account1.cf +│   │ ├── remote-account2.cf │   ├── postfix-accounts.cf │   └── postfix-virtual.cf ├── docker-compose.yml @@ -42,9 +47,13 @@ received = false delivered_to = false ``` -If you want to use a different base config, mount a file to `/etc/getmailrc_general`. This file will replace the default "Common Options" base config above, that all `getmail-.cf` files will extend with their configs when used. +The DMS integration for Getmail generates a `getmailrc` config that prepends the common options of the base config to each remote account config file (`*.cf`) found in the DMS Config Volume `getmail/` directory. -??? example "IMAP Configuration" +!!! tip "Change the base options" + + Add your own base config as `getmail/getmailrc_general.cf` into the DMS Config Volume. It will replace the DMS defaults shown above. + +??? example "IMAP Configuration" This example will: @@ -54,7 +63,7 @@ If you want to use a different base config, mount a file to `/etc/getmailrc_gene ```getmailrc [retriever] - type = SimpleIMAPRetriever + type = SimpleIMAPSSLRetriever server = imap.gmail.com username = alice password = notsecure @@ -71,7 +80,7 @@ If you want to use a different base config, mount a file to `/etc/getmailrc_gene ```getmailrc [retriever] - type = SimplePOP3Retriever + type = SimplePOP3SSLRetriever server = pop3.gmail.com username = alice password = notsecure @@ -84,7 +93,7 @@ If you want to use a different base config, mount a file to `/etc/getmailrc_gene ### Polling Interval -By default the `getmail` service checks external mail accounts for new mail every 5 minutes. That polling interval is configurable via the `GETMAIL_POLL` ENV variable, with a value in minutes (_default: 5, min: 1, max: 30_): +By default the `getmail` service checks external mail accounts for new mail every 5 minutes. That polling interval is configurable via the `GETMAIL_POLL` ENV variable, with a value in minutes (_default: 5, min: 1_): ```yaml environment: diff --git a/docs/content/config/environment.md b/docs/content/config/environment.md index 0a65b1e4..5a766f53 100644 --- a/docs/content/config/environment.md +++ b/docs/content/config/environment.md @@ -450,7 +450,7 @@ Default: 6 (which corresponds to the `add_header` action) ##### RSPAMD_NEURAL -Can be used to enable or disable the [Neural network module][rspamd-docs-neural-network]. This is an experimental anti-spam weigh method using three neural networks in the configuration added here. As far as we can tell it trains itself by using other modules to find out what spam is. It will take a while (a week or more) to train its first neural network. The config trains new networks all the time and discards old networks. +Can be used to enable or disable the [Neural network module][rspamd-docs-neural-network]. This is an experimental anti-spam weigh method using three neural networks in the configuration added here. As far as we can tell it trains itself by using other modules to find out what spam is. It will take a while (a week or more) to train its first neural network. The config trains new networks all the time and discards old networks. Since it is experimental, it is switched off by default. - **0** => Disabled @@ -732,7 +732,7 @@ Enable or disable `getmail`. ##### GETMAIL_POLL -- **5** => `getmail` The number of minutes for the interval. Min: 1; Max: 30; Default: 5. +- **5** => `getmail` The number of minutes for the interval. Min: 1; Default: 5. #### OAUTH2 diff --git a/mailserver.env b/mailserver.env index 77b863ff..141b607a 100644 --- a/mailserver.env +++ b/mailserver.env @@ -430,7 +430,7 @@ FETCHMAIL_PARALLEL=0 # - 1 => Enabled ENABLE_GETMAIL=0 -# The number of minutes for the interval. Min: 1; Max: 30. +# The number of minutes for the interval. Min: 1; Default: 5. GETMAIL_POLL=5 # ----------------------------------------------- diff --git a/target/bin/debug-getmail b/target/bin/debug-getmail index 60270215..33d41893 100644 --- a/target/bin/debug-getmail +++ b/target/bin/debug-getmail @@ -5,9 +5,20 @@ source /usr/local/bin/helpers/log.sh # shellcheck source=../scripts/startup/setup-stack.sh source /usr/local/bin/setup.d/getmail.sh -_setup_getmail +# Setup getmail, even if not enabled. +ENABLE_GETMAIL=1 _setup_getmail -GETMAILDIR=/tmp/docker-mailserver/getmail -for FILE in /etc/getmailrc.d/getmailrc*; do - getmail --getmaildir "${GETMAILDIR}" --rcfile "${FILE}" --dump | tail -n +6 +# Directory, where "oldmail" files are stored. +# Getmail stores its state - its "memory" of what it has seen in your POP/IMAP account - in the oldmail files. +GETMAIL_DIR=/var/lib/getmail + +# If no matching filenames are found, and the shell option nullglob is disabled, the word is left unchanged. +# If the nullglob option is set, and no matches are found, the word is removed. +shopt -s nullglob + +# Dump configuration from each RC file. +for RC_FILE in /etc/getmailrc.d/*; do + echo "${RC_FILE##*/}:" + echo + getmail --getmaildir "${GETMAIL_DIR}" --rcfile "${RC_FILE}" --dump | tail -n +6 done diff --git a/target/bin/getmail-cron b/target/bin/getmail-cron deleted file mode 100644 index fec03906..00000000 --- a/target/bin/getmail-cron +++ /dev/null @@ -1,8 +0,0 @@ -#! /bin/bash - -GETMAILDIR=/tmp/docker-mailserver/getmail -for FILE in /etc/getmailrc.d/getmailrc*; do - if ! pgrep -f "${FILE}$" &>/dev/null; then - getmail --getmaildir "${GETMAILDIR}" --rcfile "${FILE}" - fi -done diff --git a/target/getmail/getmail-service.sh b/target/getmail/getmail-service.sh new file mode 100644 index 00000000..ff2cc571 --- /dev/null +++ b/target/getmail/getmail-service.sh @@ -0,0 +1,47 @@ +#!/bin/bash + +# shellcheck source=../scripts/helpers/log.sh +source /usr/local/bin/helpers/log.sh + +# Directory, where "oldmail" files are stored. +# getmail stores its state - its "memory" of what it has seen in your POP/IMAP account - in the oldmail files. +GETMAIL_DIR=/var/lib/getmail + +# Kill all child processes on EXIT. +# Otherwise 'supervisorctl restart getmail' leads to zombie 'sleep' processes. +trap 'pkill --parent ${$}' EXIT + +function _syslog_error() { + logger --priority mail.err --tag getmail "${1}" +} + +function _stop_service() { + _syslog_error "Stopping service" + exec supervisorctl stop getmail +} + +# Verify the correct value for GETMAIL_POLL. Valid are any numbers greater than 0. +if [[ ! ${GETMAIL_POLL} =~ ^[0-9]+$ ]] || [[ ${GETMAIL_POLL} -lt 1 ]]; then + _syslog_error "Invalid value for GETMAIL_POLL: ${GETMAIL_POLL}" + _stop_service +fi + +# If no matching filenames are found, and the shell option nullglob is disabled, the word is left unchanged. +# If the nullglob option is set, and no matches are found, the word is removed. +shopt -s nullglob + +# Run each getmailrc periodically. +while :; do + for RC_FILE in /etc/getmailrc.d/*; do + _log 'debug' "Processing ${RC_FILE}" + getmail --getmaildir "${GETMAIL_DIR}" --rcfile "${RC_FILE}" + done + + # Stop service if no configuration is found. + if [[ -z ${RC_FILE} ]]; then + _syslog_error 'No configuration found' + _stop_service + fi + + sleep "${GETMAIL_POLL}m" +done diff --git a/target/getmail/getmailrc_general b/target/getmail/getmailrc_general new file mode 100644 index 00000000..73fc443f --- /dev/null +++ b/target/getmail/getmailrc_general @@ -0,0 +1,11 @@ +# https://getmail6.org/configuration.html#conf-options + +[options] +verbose = 0 +read_all = false +delete = false +max_messages_per_session = 500 +received = false +delivered_to = false +message_log_syslog = true + diff --git a/target/scripts/start-mailserver.sh b/target/scripts/start-mailserver.sh index 1c43616d..71dc32f2 100755 --- a/target/scripts/start-mailserver.sh +++ b/target/scripts/start-mailserver.sh @@ -161,6 +161,7 @@ function _register_functions() { [[ ${ENABLE_CLAMAV} -eq 1 ]] && _register_start_daemon '_start_daemon_clamav' [[ ${ENABLE_AMAVIS} -eq 1 ]] && _register_start_daemon '_start_daemon_amavis' [[ ${ACCOUNT_PROVISIONER} == 'FILE' ]] && _register_start_daemon '_start_daemon_changedetector' + [[ ${ENABLE_GETMAIL} -eq 1 ]] && _register_start_daemon '_start_daemon_getmail' } # ------------------------------------------------------------ diff --git a/target/scripts/startup/daemons-stack.sh b/target/scripts/startup/daemons-stack.sh index a4cecf67..4cc9b2af 100644 --- a/target/scripts/startup/daemons-stack.sh +++ b/target/scripts/startup/daemons-stack.sh @@ -34,6 +34,7 @@ function _start_daemon_clamav { _default_start_daemon 'clamav' ; function _start_daemon_cron { _default_start_daemon 'cron' ; } function _start_daemon_dovecot { _default_start_daemon 'dovecot' ; } function _start_daemon_fail2ban { _default_start_daemon 'fail2ban' ; } +function _start_daemon_getmail { _default_start_daemon 'getmail' ; } function _start_daemon_opendkim { _default_start_daemon 'opendkim' ; } function _start_daemon_opendmarc { _default_start_daemon 'opendmarc' ; } function _start_daemon_postgrey { _default_start_daemon 'postgrey' ; } diff --git a/target/scripts/startup/setup.d/getmail.sh b/target/scripts/startup/setup.d/getmail.sh index c1345063..e63db138 100644 --- a/target/scripts/startup/setup.d/getmail.sh +++ b/target/scripts/startup/setup.d/getmail.sh @@ -4,38 +4,46 @@ function _setup_getmail() { if [[ ${ENABLE_GETMAIL} -eq 1 ]]; then _log 'trace' 'Preparing Getmail configuration' - local GETMAILRC ID CONFIGS + local GETMAIL_RC ID GETMAIL_DIR - GETMAILRC='/etc/getmailrc.d' - CONFIGS=0 + local GETMAIL_CONFIG_DIR='/tmp/docker-mailserver/getmail' + local GETMAIL_RC_DIR='/etc/getmailrc.d' + local GETMAIL_RC_GENERAL_CF="${GETMAIL_CONFIG_DIR}/getmailrc_general.cf" + local GETMAIL_RC_GENERAL='/etc/getmailrc_general' - mkdir -p "${GETMAILRC}" + # Create the directory /etc/getmailrc.d to place the user config in later. + mkdir -p "${GETMAIL_RC_DIR}" - # Generate getmailrc configs, starting with the `/etc/getmailrc_general` base config, - # Add a unique `message_log` config, then append users own config to the end. - for FILE in /tmp/docker-mailserver/getmail-*.cf; do - if [[ -f ${FILE} ]]; then - CONFIGS=1 - ID=$(cut -d '-' -f 3 <<< "${FILE}" | cut -d '.' -f 1) - local GETMAIL_CONFIG="${GETMAILRC}/getmailrc-${ID}" - - cat /etc/getmailrc_general >"${GETMAIL_CONFIG}" - echo -e "message_log = /var/log/mail/getmail-${ID}.log\n" >>"${GETMAIL_CONFIG}" - cat "${FILE}" >>"${GETMAIL_CONFIG}" - fi - done - - if [[ ${CONFIGS} -eq 1 ]]; then - cat >/etc/cron.d/getmail << EOF -*/${GETMAIL_POLL} * * * * root /usr/local/bin/getmail-cron -EOF - chmod -R 600 "${GETMAILRC}" + # Check if custom getmailrc_general.cf file is present. + if [[ -f "${GETMAIL_RC_GENERAL_CF}" ]]; then + _log 'debug' "Custom 'getmailrc_general.cf' found" + cp "${GETMAIL_RC_GENERAL_CF}" "${GETMAIL_RC_GENERAL}" fi - # Both the debug command and cron job (that runs getmail) for getmail - # expect this location to exist. - GETMAILDIR=/tmp/docker-mailserver/getmail - mkdir -p "${GETMAILDIR}" + # If no matching filenames are found, and the shell option nullglob is disabled, the word is left unchanged. + # If the nullglob option is set, and no matches are found, the word is removed. + shopt -s nullglob + + # Generate getmailrc configs, starting with the `/etc/getmailrc_general` base config, then appending users own config to the end. + for FILE in "${GETMAIL_CONFIG_DIR}"/*.cf; do + if [[ ${FILE} =~ /getmail/(.+)\.cf ]] && [[ ${FILE} != "${GETMAIL_RC_GENERAL_CF}" ]]; then + ID=${BASH_REMATCH[1]} + + _log 'debug' "Processing getmail config '${ID}'" + + GETMAIL_RC=${GETMAIL_RC_DIR}/${ID} + cat "${GETMAIL_RC_GENERAL}" "${FILE}" >"${GETMAIL_RC}" + fi + done + # Strip read access from non-root due to files containing secrets: + chmod -R 600 "${GETMAIL_RC_DIR}" + + # Directory, where "oldmail" files are stored. + # For more information see: https://getmail6.org/faq.html#faq-about-oldmail + # The debug command for getmail expects this location to exist. + GETMAIL_DIR=/var/lib/getmail + _log 'debug' "Creating getmail state-dir '${GETMAIL_DIR}'" + mkdir -p "${GETMAIL_DIR}" else _log 'debug' 'Getmail is disabled' fi diff --git a/target/scripts/startup/setup.d/mail_state.sh b/target/scripts/startup/setup.d/mail_state.sh index de15ee3b..e819c138 100644 --- a/target/scripts/startup/setup.d/mail_state.sh +++ b/target/scripts/startup/setup.d/mail_state.sh @@ -23,6 +23,7 @@ function _setup_save_states() { [[ ${ENABLE_CLAMAV} -eq 1 ]] && SERVICEDIRS+=('lib/clamav') [[ ${ENABLE_FAIL2BAN} -eq 1 ]] && SERVICEDIRS+=('lib/fail2ban') [[ ${ENABLE_FETCHMAIL} -eq 1 ]] && SERVICEDIRS+=('lib/fetchmail') + [[ ${ENABLE_GETMAIL} -eq 1 ]] && SERVICEDIRS+=('lib/getmail') [[ ${ENABLE_MTA_STS} -eq 1 ]] && SERVICEDIRS+=('lib/mta-sts') [[ ${ENABLE_POSTGREY} -eq 1 ]] && SERVICEDIRS+=('lib/postgrey') [[ ${ENABLE_RSPAMD} -eq 1 ]] && SERVICEDIRS+=('lib/rspamd') diff --git a/target/supervisor/conf.d/dms-services.conf b/target/supervisor/conf.d/dms-services.conf index 7f106456..f9db838c 100644 --- a/target/supervisor/conf.d/dms-services.conf +++ b/target/supervisor/conf.d/dms-services.conf @@ -170,3 +170,12 @@ stderr_logfile=/var/log/supervisor/%(program_name)s.log command=/usr/bin/mta-sts-daemon --config /etc/mta-sts-daemon.yml user=_mta-sts environment=HOME=/var/lib/mta-sts + +[program:getmail] +startsecs=0 +stopwaitsecs=55 +autostart=false +autorestart=true +stdout_logfile=/var/log/supervisor/%(program_name)s.log +stderr_logfile=/var/log/supervisor/%(program_name)s.log +command=/bin/bash -l -c /usr/local/bin/getmail-service.sh diff --git a/test/config/getmail/getmail-user3.cf b/test/config/getmail/user3.cf similarity index 100% rename from test/config/getmail/getmail-user3.cf rename to test/config/getmail/user3.cf diff --git a/test/tests/parallel/set1/getmail.bats b/test/tests/parallel/set1/getmail.bats index ca1fdf87..8d69ec3d 100644 --- a/test/tests/parallel/set1/getmail.bats +++ b/test/tests/parallel/set1/getmail.bats @@ -8,12 +8,13 @@ function setup_file() { local CUSTOM_SETUP_ARGUMENTS=(--env 'ENABLE_GETMAIL=1') _init_with_defaults - mv "${TEST_TMP_CONFIG}/getmail/getmail-user3.cf" "${TEST_TMP_CONFIG}/getmail-user3.cf" _common_container_setup 'CUSTOM_SETUP_ARGUMENTS' } function teardown_file() { _default_teardown ; } +#? The file used in the following tests is placed in test/config/getmail/user3.cf + @test 'default configuration exists and is correct' { _run_in_container cat /etc/getmailrc_general assert_success @@ -24,15 +25,16 @@ function teardown_file() { _default_teardown ; } assert_line 'max_messages_per_session = 500' assert_line 'received = false' assert_line 'delivered_to = false' + assert_line 'message_log_syslog = true' _run_in_container_bash '[[ -f /usr/local/bin/debug-getmail ]]' assert_success - _run_in_container_bash '[[ -f /usr/local/bin/getmail-cron ]]' + _run_in_container_bash '[[ -f /usr/local/bin/getmail-service.sh ]]' assert_success } @test 'debug-getmail works as expected' { - _run_in_container cat /etc/getmailrc.d/getmailrc-user3 + _run_in_container cat /etc/getmailrc.d/user3 assert_success assert_line '[options]' assert_line 'verbose = 0' @@ -41,7 +43,7 @@ function teardown_file() { _default_teardown ; } assert_line 'max_messages_per_session = 500' assert_line 'received = false' assert_line 'delivered_to = false' - assert_line 'message_log = /var/log/mail/getmail-user3.log' + assert_line 'message_log_syslog = true' assert_line '[retriever]' assert_line 'type = SimpleIMAPSSLRetriever' assert_line 'server = imap.remote-service.test' @@ -55,19 +57,18 @@ function teardown_file() { _default_teardown ; } _run_in_container /usr/local/bin/debug-getmail assert_success - assert_line --regexp 'retriever:.*SimpleIMAPSSLRetriever\(ca_certs="None", certfile="None", getmaildir="\/tmp\/docker-mailserver\/getmail", imap_on_delete="None", imap_search="None", keyfile="None", mailboxes="\(.*INBOX.*\)", move_on_delete="None", password="\*", password_command="\(\)", port="993", record_mailbox="True", server="imap.remote-service.test", ssl_cert_hostname="None", ssl_ciphers="None", ssl_fingerprints="\(\)", ssl_version="None", timeout="180", use_cram_md5="False", use_kerberos="False", use_peek="True", use_xoauth2="False", username="user3"\)' + assert_line --regexp 'retriever:.*SimpleIMAPSSLRetriever\(ca_certs="None", certfile="None", getmaildir="\/var\/lib\/getmail", imap_on_delete="None", imap_search="None", keyfile="None", mailboxes="\(.*INBOX.*\)", move_on_delete="None", password="\*", password_command="\(\)", port="993", record_mailbox="True", server="imap.remote-service.test", ssl_cert_hostname="None", ssl_ciphers="None", ssl_fingerprints="\(\)", ssl_version="None", timeout="180", use_cram_md5="False", use_kerberos="False", use_peek="True", use_xoauth2="False", username="user3"\)' assert_line --regexp 'destination:.*MDA_external\(allow_root_commands="True", arguments="\(.*-d.*user3@example.test.*\)", command="deliver", group="None", ignore_stderr="False", path="\/usr\/lib\/dovecot\/deliver", pipe_stdout="True", unixfrom="False", user="None"\)' assert_line ' delete : False' assert_line ' delete_after : 0' assert_line ' delete_bigger_than : 0' assert_line ' delivered_to : False' assert_line ' fingerprint : False' - assert_line ' logfile : logfile(filename="/var/log/mail/getmail-user3.log")' assert_line ' max_bytes_per_session : 0' assert_line ' max_message_size : 0' assert_line ' max_messages_per_session : 500' - assert_line ' message_log : /var/log/mail/getmail-user3.log' - assert_line ' message_log_syslog : False' + assert_line ' message_log : None' + assert_line ' message_log_syslog : True' assert_line ' message_log_verbose : False' assert_line ' netrc_file : None' assert_line ' read_all : False' diff --git a/test/tests/parallel/set3/container_configuration/process_check_restart.bats b/test/tests/parallel/set3/container_configuration/process_check_restart.bats index 78eabd7a..cc9cd5b3 100644 --- a/test/tests/parallel/set3/container_configuration/process_check_restart.bats +++ b/test/tests/parallel/set3/container_configuration/process_check_restart.bats @@ -18,6 +18,7 @@ function teardown() { _default_teardown ; } # dovecot (/usr/sbin/dovecot) # fetchmail (/usr/bin/fetchmail) # fail2ban-server (/usr/bin/python3 /usr/bin/fail2ban-server) - NOTE: python3 is due to the shebang +# getmail (/bin/bash /usr/local/bin/getmail-service.sh) # mta-sts-daemon (/usr/bin/bin/python3 /usr/bin/mta-sts-daemon) # postgrey (postgrey) - NOTE: This process command uses perl via shebang, but unlike python3 the context is missing # postsrsd (/usr/sbin/postsrsd) @@ -41,6 +42,7 @@ ENV_PROCESS_LIST=( dovecot fail2ban-server fetchmail + getmail mta-sts-daemon opendkim opendmarc @@ -56,6 +58,7 @@ ENV_PROCESS_LIST=( --env ENABLE_CLAMAV=0 --env ENABLE_FAIL2BAN=0 --env ENABLE_FETCHMAIL=0 + --env ENABLE_GETMAIL=0 --env ENABLE_MTA_STS=0 --env ENABLE_OPENDKIM=0 --env ENABLE_OPENDMARC=0 @@ -92,6 +95,7 @@ ENV_PROCESS_LIST=( --env ENABLE_AMAVIS=1 --env ENABLE_FAIL2BAN=1 --env ENABLE_FETCHMAIL=1 + --env ENABLE_GETMAIL=1 --env ENABLE_MTA_STS=1 --env ENABLE_OPENDKIM=1 --env ENABLE_OPENDMARC=1 From 310786453bb83b11ffab4f62f5be4a3274c58513 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Sat, 17 Aug 2024 22:55:31 +1200 Subject: [PATCH 148/267] chore(Dockerfile): COPY ClamAV database from debian images (#4160) Changes ClamAV image source from DockerHub clamav/clamav (Alpine) to clamav/clamav-debian. Only the Debian variant offers multi-platform images. This isn't too important since we are only interested in taking a copy of the database from the image. It should however resolve a CI warning. --- Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index 02245ebc..3edf6e95 100644 --- a/Dockerfile +++ b/Dockerfile @@ -62,7 +62,7 @@ SHELL ["/bin/bash", "-e", "-o", "pipefail", "-c"] # which would require an extra memory of 500MB+ during an image build. # When using `COPY --link`, the `--chown` option is only compatible with numeric ID values. # hadolint ignore=DL3021 -COPY --link --chown=200 --from=docker.io/clamav/clamav:latest /var/lib/clamav /var/lib/clamav +COPY --link --chown=200 --from=docker.io/clamav/clamav-debian:latest /var/lib/clamav /var/lib/clamav RUN < Date: Mon, 19 Aug 2024 00:51:44 +0200 Subject: [PATCH 149/267] chore: Add comments to `start-mailserver.sh` and stop using `inherit_errexit` (#4161) --- CHANGELOG.md | 1 + target/scripts/start-mailserver.sh | 6 +++++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1a253234..6621da18 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,7 @@ All notable changes to this project will be documented in this file. The format - **Internal:** - The main `mail.log` which is piped to stdout via `tail` now correctly begins from the first log line of the active container run. Previously some daemon logs and potential warnings/errors were omitted. ([#4146](https://github.com/docker-mailserver/docker-mailserver/pull/4146)) - Fixed a regression introduced in v14 where `postfix-main.cf` appended `stderr` output into `/etc/postfix/main.cf`, causing Postfix startup to fail ([#4147](https://github.com/docker-mailserver/docker-mailserver/pull/4147)) + - Unused `shopt -s inherit_errexit` removed from `start-mailserver.sh` ([#4161](https://github.com/docker-mailserver/docker-mailserver/pull/4161)) ### CI diff --git a/target/scripts/start-mailserver.sh b/target/scripts/start-mailserver.sh index 71dc32f2..f6ceadf5 100755 --- a/target/scripts/start-mailserver.sh +++ b/target/scripts/start-mailserver.sh @@ -1,7 +1,11 @@ #!/bin/bash +# When 'pipefail' is enabled, the exit status of the pipeline reflects the exit status of the last command that fails. +# Without 'pipefail', the exit status of a pipeline is determined by the exit status of the last command in the pipeline. set -o pipefail -shopt -s globstar inherit_errexit + +# Allows the usage of '**' in patterns, e.g. ls **/* +shopt -s globstar # ------------------------------------------------------------ # ? >> Sourcing helpers & stacks From 9589d2192bb41d0176c16b6868133c6edc00b019 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 19 Aug 2024 20:27:27 +0200 Subject: [PATCH 150/267] chore(deps): Bump anchore/scan-action from 4.1.0 to 4.1.1 (#4162) --- .github/workflows/generic_vulnerability-scan.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index e33217a1..02fbfab6 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -55,7 +55,7 @@ jobs: provenance: false - name: 'Run the Anchore Grype scan action' - uses: anchore/scan-action@v4.1.0 + uses: anchore/scan-action@v4.1.1 id: scan with: image: mailserver-testing:ci From cb963a9a8b68c4aa8191cf40c67162a5eb3043e0 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 19 Aug 2024 21:44:33 +0200 Subject: [PATCH 151/267] chore(deps): Bump docker/build-push-action from 6.6.1 to 6.7.0 (#4163) --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 76f7014b..203416fe 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -83,7 +83,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v6.6.1 + uses: docker/build-push-action@v6.7.0 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index ffb8a677..3a3d3749 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v6.6.1 + uses: docker/build-push-action@v6.7.0 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 0f2f9031..1658ddcc 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.6.1 + uses: docker/build-push-action@v6.7.0 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 02fbfab6..5ec5901b 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.6.1 + uses: docker/build-push-action@v6.7.0 with: context: . tags: mailserver-testing:ci From a87d49e8f856cebcb7799d22971a1fc133fa1d3b Mon Sep 17 00:00:00 2001 From: Vetu <59065425+Vetuska@users.noreply.github.com> Date: Sat, 24 Aug 2024 21:42:32 +0300 Subject: [PATCH 152/267] fix: typo in volume pathname (#4165) --- docs/content/config/security/ssl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/config/security/ssl.md b/docs/content/config/security/ssl.md index dd633c9d..96f4a717 100644 --- a/docs/content/config/security/ssl.md +++ b/docs/content/config/security/ssl.md @@ -240,7 +240,7 @@ After completing the steps above, your certificate should be ready to use. image: certbot/dns-cloudflare:latest command: renew --dns-cloudflare --dns-cloudflare-credentials /run/secrets/cloudflare-api-token volumes: - - ./docker-data/certbot/certs/:/etc/letsencrtypt/ + - ./docker-data/certbot/certs/:/etc/letsencrypt/ - ./docker-data/certbot/logs/:/var/log/letsencrypt/ secrets: - cloudflare-api-token From 3349bba1ffaf0a60d9305f073ec2b8d60246bc68 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 26 Aug 2024 18:30:22 +0200 Subject: [PATCH 153/267] chore(deps): Bump anchore/scan-action from 4.1.1 to 4.1.2 (#4166) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_vulnerability-scan.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 5ec5901b..0275fc98 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -55,7 +55,7 @@ jobs: provenance: false - name: 'Run the Anchore Grype scan action' - uses: anchore/scan-action@v4.1.1 + uses: anchore/scan-action@v4.1.2 id: scan with: image: mailserver-testing:ci From 4e85f799fc39365d87bbe7bc7b9151dae7f31a31 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Mon, 9 Sep 2024 09:58:12 +1200 Subject: [PATCH 154/267] fix: Dovecot LDAP config should exist --- Dockerfile | 3 ++- target/dovecot/auth-ldap.conf.ext | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Dockerfile b/Dockerfile index 3edf6e95..09295a22 100644 --- a/Dockerfile +++ b/Dockerfile @@ -109,14 +109,15 @@ COPY target/rspamd/local.d/ /etc/rspamd/local.d/ # --- OAUTH2 ------------------------------------ # ----------------------------------------------- -COPY target/dovecot/auth-oauth2.conf.ext /etc/dovecot/conf.d COPY target/dovecot/dovecot-oauth2.conf.ext /etc/dovecot +COPY target/dovecot/auth-oauth2.conf.ext /etc/dovecot/conf.d # ----------------------------------------------- # --- LDAP & SpamAssassin's Cron ---------------- # ----------------------------------------------- COPY target/dovecot/dovecot-ldap.conf.ext /etc/dovecot +COPY target/dovecot/auth-ldap.conf.ext /etc/dovecot/conf.d COPY \ target/postfix/ldap-users.cf \ target/postfix/ldap-groups.cf \ diff --git a/target/dovecot/auth-ldap.conf.ext b/target/dovecot/auth-ldap.conf.ext index 222769aa..a1cae923 100644 --- a/target/dovecot/auth-ldap.conf.ext +++ b/target/dovecot/auth-ldap.conf.ext @@ -6,7 +6,7 @@ passdb { driver = ldap - mechanism = plain login + mechanisms = plain login # Path for LDAP configuration file, see example-config/dovecot-ldap.conf.ext args = /etc/dovecot/dovecot-ldap.conf.ext From cace9c56d932bbd53a36f5bc36f21a676d92a708 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Mon, 9 Sep 2024 19:00:53 +1200 Subject: [PATCH 155/267] fix: Dovecot LDAP config should exist (#4175) The config was not copied over during image build, and the associated auth config had a typo for the `mechanisms` key. --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6621da18..7c9839db 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,6 +44,8 @@ All notable changes to this project will be documented in this file. The format - **Dovecot:** - Update logwatch `ignore.conf` to exclude Xapian messages about pending documents ([#4060](https://github.com/docker-mailserver/docker-mailserver/pull/4060)) - `dovecot-fts-xapian` plugin was updated to `1.7.13`, fixing a regression with indexing ([#4095](https://github.com/docker-mailserver/docker-mailserver/pull/4095)) +- **LDAP:** + - A previous compatibility fix for OAuth2 in v13.3.1 had not applied the actual LDAP config changes. This has been corrected ([#4175](https://github.com/docker-mailserver/docker-mailserver/pull/4175)) - **Internal:** - The main `mail.log` which is piped to stdout via `tail` now correctly begins from the first log line of the active container run. Previously some daemon logs and potential warnings/errors were omitted. ([#4146](https://github.com/docker-mailserver/docker-mailserver/pull/4146)) - Fixed a regression introduced in v14 where `postfix-main.cf` appended `stderr` output into `/etc/postfix/main.cf`, causing Postfix startup to fail ([#4147](https://github.com/docker-mailserver/docker-mailserver/pull/4147)) From 84180f879e5eaaa15930509d4154248c3248d765 Mon Sep 17 00:00:00 2001 From: GallowsDove <71831019+GallowsDove@users.noreply.github.com> Date: Wed, 18 Sep 2024 19:34:42 +0200 Subject: [PATCH 156/267] fix: fix incorrect link in README.md (#4184) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e4dbf44a..6e1d2735 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ A production-ready fullstack but simple containerized mail server (SMTP, IMAP, L ## :package: Included Services - [Postfix](http://www.postfix.org) with SMTP or LDAP authentication and support for [extension delimiters](https://docker-mailserver.github.io/docker-mailserver/latest/config/account-management/overview/#aliases) -- [Dovecot](https://www.dovecot.org) with SASL, IMAP, POP3, LDAP, [basic Sieve support](https://docker-mailserver.github.io/docker-mailserver/latest/config/advanced/mail-sieve) and [quotas](https://docker-mailserver.github.io/docker-mailserver/latest/config/account-management/overview/#quotas) +- [Dovecot](https://www.dovecot.org) with SASL, IMAP, POP3, LDAP, [basic Sieve support](https://docker-mailserver.github.io/docker-mailserver/latest/config/advanced/mail-sieve) and [quotas](https://docker-mailserver.github.io/docker-mailserver/latest/config/user-management/#quotas) - [Rspamd](https://rspamd.com/) - [Amavis](https://www.amavis.org/) - [SpamAssassin](http://spamassassin.apache.org/) supporting custom rules From 94751e00c9787c5223e39bd0cd5b139a0c1f6255 Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Sat, 21 Sep 2024 21:04:06 +0200 Subject: [PATCH 157/267] dependency: update `jaq` from `1.3.0` to `1.6.0` (#4190) --- CHANGELOG.md | 1 + target/scripts/build/packages.sh | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c9839db..21008a64 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,6 +38,7 @@ All notable changes to this project will be documented in this file. The format - Add caveat for `DMS_VMAIL_UID` not being compatible with `0` / root ([#4143](https://github.com/docker-mailserver/docker-mailserver/pull/4143)) - **Postfix:** - Disable Microsoft reactions to outgoing mail ([#4120](https://github.com/docker-mailserver/docker-mailserver/pull/4120)) +- bumped `jaq` version from 1.3.0 to 1.6.0 ([#4190](https://github.com/docker-mailserver/docker-mailserver/pull/4190)) ### Fixes diff --git a/target/scripts/build/packages.sh b/target/scripts/build/packages.sh index 4469f508..2ac40fba 100644 --- a/target/scripts/build/packages.sh +++ b/target/scripts/build/packages.sh @@ -38,7 +38,7 @@ function _pre_installation_steps() { function _install_utils() { _log 'debug' 'Installing utils sourced from Github' _log 'trace' 'Installing jaq' - local JAQ_TAG='v1.3.0' + local JAQ_TAG='v1.6.0' curl -sSfL "https://github.com/01mf02/jaq/releases/download/${JAQ_TAG}/jaq-${JAQ_TAG}-$(uname -m)-unknown-linux-gnu" -o /usr/bin/jaq chmod +x /usr/bin/jaq From 025a38d7366d7e08cf61b5106af762ec69e1505d Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Thu, 26 Sep 2024 20:01:35 +1200 Subject: [PATCH 158/267] chore: Add maintenance note for LMTP (#4199) --- target/scripts/startup/setup.d/postfix.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/target/scripts/startup/setup.d/postfix.sh b/target/scripts/startup/setup.d/postfix.sh index e140fbaa..e99e6607 100644 --- a/target/scripts/startup/setup.d/postfix.sh +++ b/target/scripts/startup/setup.d/postfix.sh @@ -79,6 +79,8 @@ EOF if [[ ${ACCOUNT_PROVISIONER} == 'FILE' ]]; then postconf 'virtual_mailbox_maps = texthash:/etc/postfix/vmailbox' fi + # Historical context regarding decision to use LMTP instead of LDA (do not change this): + # https://github.com/docker-mailserver/docker-mailserver/issues/4178#issuecomment-2375489302 postconf 'virtual_transport = lmtp:unix:/var/run/dovecot/lmtp' fi From 3bf32a6552a3e2ff2ec55bbdd5708e80413d9df1 Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Sat, 28 Sep 2024 12:52:49 +0200 Subject: [PATCH 159/267] add dedicated feature requests to new project automatically (#4198) --- .github/ISSUE_TEMPLATE/feature_request.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml index 0ee66a03..b2c0d4b8 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.yml +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -4,6 +4,8 @@ title: 'feature request: ' labels: - kind/new feature - meta/needs triage +projects: + - DMS Core Backlog body: - type: markdown From 1a938dfb15e94f43f1af5ddaf7b8276aba92a825 Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Sat, 28 Sep 2024 13:27:34 +0200 Subject: [PATCH 160/267] Rspamd: update GTube patters in tests (#4191) --- CHANGELOG.md | 1 + test/config/rspamd_full/user-patches.sh | 2 +- .../parallel/set1/spam_virus/rspamd_full.bats | 15 +++++++++------ 3 files changed, 11 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 21008a64..14b0b4c9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -39,6 +39,7 @@ All notable changes to this project will be documented in this file. The format - **Postfix:** - Disable Microsoft reactions to outgoing mail ([#4120](https://github.com/docker-mailserver/docker-mailserver/pull/4120)) - bumped `jaq` version from 1.3.0 to 1.6.0 ([#4190](https://github.com/docker-mailserver/docker-mailserver/pull/4190)) +- updated Rspamd GTube settings and tests ([#4191](https://github.com/docker-mailserver/docker-mailserver/pull/4191)) ### Fixes diff --git a/test/config/rspamd_full/user-patches.sh b/test/config/rspamd_full/user-patches.sh index f731ff36..56906064 100644 --- a/test/config/rspamd_full/user-patches.sh +++ b/test/config/rspamd_full/user-patches.sh @@ -5,7 +5,7 @@ # # We do not use `custom-commands.conf` because this a feature # we are testing too. -echo "enable_test_patterns = true;" >>/etc/rspamd/local.d/options.inc +echo 'gtube_patterns = "all"' >>/etc/rspamd/local.d/options.inc # We want Dovecot to be very detailed about what it is doing, # specifically for Sieve because we need to check whether the diff --git a/test/tests/parallel/set1/spam_virus/rspamd_full.bats b/test/tests/parallel/set1/spam_virus/rspamd_full.bats index 3bc9fb10..961731fa 100644 --- a/test/tests/parallel/set1/spam_virus/rspamd_full.bats +++ b/test/tests/parallel/set1/spam_virus/rspamd_full.bats @@ -45,8 +45,10 @@ function setup_file() { _wait_for_smtp_port_in_container # We will send 5 emails: - # 1. The first one should pass just fine + # 1. The first ones should pass just fine _send_email_with_msgid 'rspamd-test-email-pass' + _send_email_with_msgid 'rspamd-test-email-pass-gtube' \ + --body 'AJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X' # 2. The second one should be rejected (Rspamd-specific GTUBE pattern for rejection) _send_spam --expect-rejection # 3. The third one should be rejected due to a virus (ClamAV EICAR pattern) @@ -54,7 +56,7 @@ function setup_file() { _send_email_with_msgid 'rspamd-test-email-virus' --expect-rejection \ --body 'X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*' # 4. The fourth one will receive an added header (Rspamd-specific GTUBE pattern for adding a spam header) - # ref: https://rspamd.com/doc/gtube_patterns.html + # ref: https://rspamd.com/doc/other/gtube_patterns.html _send_email_with_msgid 'rspamd-test-email-header' \ --body "YJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X" # 5. The fifth one will have its subject rewritten, but now spam header is applied. @@ -134,11 +136,12 @@ function teardown_file() { _default_teardown ; } @test 'normal mail passes fine' { _service_log_should_contain_string 'rspamd' 'F (no action)' + _service_log_should_contain_string 'rspamd' 'S (no action)' _print_mail_log_for_msgid 'rspamd-test-email-pass' assert_output --partial "stored mail into mailbox 'INBOX'" - _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 2 + _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 3 } @test 'detects and rejects spam' { @@ -153,7 +156,7 @@ function teardown_file() { _default_teardown ; } refute_output --partial "stored mail into mailbox 'INBOX'" assert_failure - _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 2 + _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 3 } @test 'detects and rejects virus' { @@ -168,7 +171,7 @@ function teardown_file() { _default_teardown ; } refute_output --partial "stored mail into mailbox 'INBOX'" assert_failure - _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 2 + _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 3 } @test 'custom commands work correctly' { @@ -246,7 +249,7 @@ function teardown_file() { _default_teardown ; } _print_mail_log_for_msgid 'rspamd-test-email-header' assert_output --partial "fileinto action: stored mail into mailbox [SPECIAL-USE \\Junk]" - _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 2 + _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/new/ 3 _count_files_in_directory_in_container /var/mail/localhost.localdomain/user1/.Junk/new/ 1 } From 3937e1e719f4058e0a4a846bff39383071f8391d Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Sun, 29 Sep 2024 12:18:43 +0200 Subject: [PATCH 161/267] scripts: improve DKIM path scanning in Rspamd setup (#4201) --- CHANGELOG.md | 2 + .../startup/setup.d/security/rspamd.sh | 40 +++++++------------ 2 files changed, 17 insertions(+), 25 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 14b0b4c9..023758e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,6 +52,8 @@ All notable changes to this project will be documented in this file. The format - The main `mail.log` which is piped to stdout via `tail` now correctly begins from the first log line of the active container run. Previously some daemon logs and potential warnings/errors were omitted. ([#4146](https://github.com/docker-mailserver/docker-mailserver/pull/4146)) - Fixed a regression introduced in v14 where `postfix-main.cf` appended `stderr` output into `/etc/postfix/main.cf`, causing Postfix startup to fail ([#4147](https://github.com/docker-mailserver/docker-mailserver/pull/4147)) - Unused `shopt -s inherit_errexit` removed from `start-mailserver.sh` ([#4161](https://github.com/docker-mailserver/docker-mailserver/pull/4161)) +- **Rspamd:** + - DKIM private key path checking is now performed only on paths that do not contain "$" ([#4201](https://github.com/docker-mailserver/docker-mailserver/pull/4201)) ### CI diff --git a/target/scripts/startup/setup.d/security/rspamd.sh b/target/scripts/startup/setup.d/security/rspamd.sh index 37916282..18b9703b 100644 --- a/target/scripts/startup/setup.d/security/rspamd.sh +++ b/target/scripts/startup/setup.d/security/rspamd.sh @@ -76,8 +76,9 @@ function __rspamd__run_early_setup_and_checks() { mkdir -p /var/lib/rspamd/ : >/var/lib/rspamd/stats.ucl - if [[ -d ${RSPAMD_DMS_OVERRIDE_D} ]]; then - cp "${RSPAMD_DMS_OVERRIDE_D}"/* "${RSPAMD_OVERRIDE_D}" + # Copy if directory exists and is not empty + if [[ -d ${RSPAMD_DMS_OVERRIDE_D} ]] && [[ -z $(find "${RSPAMD_DMS_OVERRIDE_D}" -maxdepth 0 -empty) ]]; then + cp "${RSPAMD_DMS_OVERRIDE_D}/"* "${RSPAMD_OVERRIDE_D}" fi if [[ ${ENABLE_AMAVIS} -eq 1 ]] || [[ ${ENABLE_SPAMASSASSIN} -eq 1 ]]; then @@ -319,8 +320,7 @@ function __rspamd__setup_check_authenticated() { local MODULE_FILE="${RSPAMD_LOCAL_D}/settings.conf" readonly MODULE_FILE if _env_var_expect_zero_or_one 'RSPAMD_CHECK_AUTHENTICATED' \ - && [[ ${RSPAMD_CHECK_AUTHENTICATED} -eq 0 ]] - then + && [[ ${RSPAMD_CHECK_AUTHENTICATED} -eq 0 ]]; then __rspamd__log 'debug' 'Content checks for authenticated users are disabled' else __rspamd__log 'debug' 'Enabling content checks for authenticated users' @@ -332,32 +332,22 @@ function __rspamd__setup_check_authenticated() { # This function performs a simple check: go through DKIM configuration files, acquire # all private key file locations and check whether they exist and whether they can be -# accessed by Rspamd. +# accessed by Rspamd. We are not checking paths that conatain the '$' symbol. function __rspamd__check_dkim_permissions() { - local DKIM_CONF_FILES DKIM_KEY_FILES - [[ -f ${RSPAMD_LOCAL_D}/dkim_signing.conf ]] && DKIM_CONF_FILES+=("${RSPAMD_LOCAL_D}/dkim_signing.conf") - [[ -f ${RSPAMD_OVERRIDE_D}/dkim_signing.conf ]] && DKIM_CONF_FILES+=("${RSPAMD_OVERRIDE_D}/dkim_signing.conf") - - # Here, we populate DKIM_KEY_FILES which we later iterate over. DKIM_KEY_FILES - # contains all keys files configured by the user. - local FILE - for FILE in "${DKIM_CONF_FILES[@]}"; do - readarray -t DKIM_KEY_FILES_TMP < <(grep -o -E 'path = .*' "${FILE}" | cut -d '=' -f 2 | tr -d ' ";') - DKIM_KEY_FILES+=("${DKIM_KEY_FILES_TMP[@]}") - done - - for FILE in "${DKIM_KEY_FILES[@]}"; do - if [[ -f ${FILE} ]]; then - __rspamd__log 'trace' "Checking DKIM file '${FILE}'" + local KEY_FILE + while read -r KEY_FILE; do + if [[ -f ${KEY_FILE} ]]; then + __rspamd__log 'trace' "Checking DKIM file '${KEY_FILE}'" # See https://serverfault.com/a/829314 for an explanation on `-exec false {} +` # We additionally resolve symbolic links to check the permissions of the actual files - if find "$(realpath -eL "${FILE}")" \( -user _rspamd -or -group _rspamd -or -perm -o=r \) -exec false {} +; then - __rspamd__log 'warn' "Rspamd DKIM private key file '${FILE}' does not appear to have correct permissions/ownership for Rspamd to use it" + if find "$(realpath -L "${KEY_FILE}")" \( -user _rspamd -or -group _rspamd -or -perm -o=r \) \ + -exec false {} +; then + __rspamd__log 'warn' "Rspamd DKIM private key file '${KEY_FILE}' does not appear to have correct permissions/ownership for Rspamd to use it" else - __rspamd__log 'trace' "DKIM file '${FILE}' permissions and ownership appear correct" + __rspamd__log 'trace' "DKIM file '${KEY_FILE}' permissions and ownership appear correct" fi else - __rspamd__log 'warn' "Rspamd DKIM private key file '${FILE}' is configured for usage, but does not appear to exist" + __rspamd__log 'warn' "Rspamd DKIM private key file '${KEY_FILE}' is configured for usage, but does not appear to exist" fi - done + done < <(rspamadm configdump dkim_signing | grep 'path =' | grep -v -F '$' | awk '{print $3}' | tr -d ';"') } From 2bcc5cf9de61d8a4634d494ab31ad24b992dccfe Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Sun, 29 Sep 2024 12:53:10 +0200 Subject: [PATCH 162/267] Rspamd documentation: update Abusix signup link (#4204) --- docs/content/config/security/rspamd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/config/security/rspamd.md b/docs/content/config/security/rspamd.md index a98d3123..8705262c 100644 --- a/docs/content/config/security/rspamd.md +++ b/docs/content/config/security/rspamd.md @@ -251,7 +251,7 @@ There is a dedicated [section for setting up DKIM with Rspamd in our documentati This subsection provides information about the integration of [Abusix][abusix-web], "a set of blocklists that work as an additional email security layer for your existing mail environment". The setup is straight-forward and well documented: -1. [Create an account](https://app.abusix.com/signup) +1. [Create an account](https://app.abusix.com/) 2. Retrieve your API key 3. Navigate to the ["Getting Started" documentation for Rspamd][abusix-docs::rspamd-integration] and follow the steps described there 4. Make sure to change `` to your private API key From c29fe3ff0bc888016720536228878b3f19131056 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 1 Oct 2024 08:54:57 +1300 Subject: [PATCH 163/267] chore(deps): Bump docker/build-push-action from 6.7.0 to 6.9.0 (#4205) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6.7.0 to 6.9.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/v6.7.0...v6.9.0) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 203416fe..80e5b6d4 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -83,7 +83,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v6.7.0 + uses: docker/build-push-action@v6.9.0 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 3a3d3749..9c6828f0 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v6.7.0 + uses: docker/build-push-action@v6.9.0 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 1658ddcc..330b2ba5 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.7.0 + uses: docker/build-push-action@v6.9.0 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 0275fc98..b0e6b71a 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.7.0 + uses: docker/build-push-action@v6.9.0 with: context: . tags: mailserver-testing:ci From 26a44995a9bba825b1348dbf3551687c2100f7a3 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 8 Oct 2024 09:01:58 +1300 Subject: [PATCH 164/267] chore(deps): Bump docker/setup-buildx-action from 3.6.1 to 3.7.1 (#4216) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3.6.1 to 3.7.1. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](https://github.com/docker/setup-buildx-action/compare/v3.6.1...v3.7.1) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 80e5b6d4..43912357 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -79,7 +79,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.6.1 + uses: docker/setup-buildx-action@v3.7.1 # NOTE: AMD64 can build within 2 minutes - name: 'Build images' diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 9c6828f0..4f94cc97 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -40,7 +40,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.6.1 + uses: docker/setup-buildx-action@v3.7.1 # Try get the cached build layers from a prior `generic_build.yml` job. # NOTE: Until adopting `type=gha` scoped cache exporter (in `docker/build-push-action`), diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 330b2ba5..9ee070a1 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -38,7 +38,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.6.1 + uses: docker/setup-buildx-action@v3.7.1 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index b0e6b71a..d0088e1c 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -37,7 +37,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.6.1 + uses: docker/setup-buildx-action@v3.7.1 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. From 34eb54ac398c4c3846645cf79404d8528869891b Mon Sep 17 00:00:00 2001 From: pitilux <100941507+pitilux@users.noreply.github.com> Date: Sat, 12 Oct 2024 00:34:20 +0200 Subject: [PATCH 165/267] fix: Avoid alias being used as regex during dovecot dummy account userdb detection (#4222) Applies alternative approach previously suggested by @polarathene and adds test cases to prevent future regressions --- CHANGELOG.md | 1 + target/scripts/helpers/accounts.sh | 3 ++- test/config/postfix-virtual.cf | 10 ++++++++++ .../parallel/set3/mta/account_management.bats | 20 ++++++++++++++++++- 4 files changed, 32 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 023758e5..28a97c32 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -46,6 +46,7 @@ All notable changes to this project will be documented in this file. The format - **Dovecot:** - Update logwatch `ignore.conf` to exclude Xapian messages about pending documents ([#4060](https://github.com/docker-mailserver/docker-mailserver/pull/4060)) - `dovecot-fts-xapian` plugin was updated to `1.7.13`, fixing a regression with indexing ([#4095](https://github.com/docker-mailserver/docker-mailserver/pull/4095)) + - The Dovecot Quota support "dummy account" workaround no longer treats the alias as a regex when checking the Dovecot UserDB ([#4222](https://github.com/docker-mailserver/docker-mailserver/pull/4222)) - **LDAP:** - A previous compatibility fix for OAuth2 in v13.3.1 had not applied the actual LDAP config changes. This has been corrected ([#4175](https://github.com/docker-mailserver/docker-mailserver/pull/4175)) - **Internal:** diff --git a/target/scripts/helpers/accounts.sh b/target/scripts/helpers/accounts.sh index 78464b88..8510b6af 100644 --- a/target/scripts/helpers/accounts.sh +++ b/target/scripts/helpers/accounts.sh @@ -135,7 +135,8 @@ function _create_dovecot_alias_dummy_accounts() { fi DOVECOT_USERDB_LINE="${ALIAS}:${REAL_ACC[1]}:${DMS_VMAIL_UID}:${DMS_VMAIL_GID}::/var/mail/${REAL_DOMAINNAME}/${REAL_USERNAME}/home::${REAL_ACC[2]:-}" - if grep -qi "^${ALIAS}:" "${DOVECOT_USERDB_FILE}"; then + # Match a full line with `-xF` to avoid regex patterns introducing false positives matching `ALIAS`: + if grep -qixF "${DOVECOT_USERDB_LINE}" "${DOVECOT_USERDB_FILE}"; then _log 'warn' "Alias '${ALIAS}' will not be added to '${DOVECOT_USERDB_FILE}' twice" else echo "${DOVECOT_USERDB_LINE}" >>"${DOVECOT_USERDB_FILE}" diff --git a/test/config/postfix-virtual.cf b/test/config/postfix-virtual.cf index 4dec6bbb..88e29fb5 100644 --- a/test/config/postfix-virtual.cf +++ b/test/config/postfix-virtual.cf @@ -3,3 +3,13 @@ alias1@localhost.localdomain user1@localhost.localdomain # this is also a test comment, :O alias2@localhost.localdomain external1@otherdomain.tld @localdomain2.com user1@localhost.localdomain + +## Dovecot "dummy accounts" for quota support (handled in `helpers/accounts.sh`) +# Do not filter alias by substring condition (longer prefix must be before substring alias): +# https://github.com/docker-mailserver/docker-mailserver/issues/2639 +prefixtest@localhost.localdomain user2@otherdomain.tld +test@localhost.localdomain user2@otherdomain.tld +# Do not filter alias when input be treated as regex tokens (eg `.`): +# https://github.com/docker-mailserver/docker-mailserver/issues/4170 +first-name@localhost.localdomain user2@otherdomain.tld +first.name@localhost.localdomain user2@otherdomain.tld diff --git a/test/tests/parallel/set3/mta/account_management.bats b/test/tests/parallel/set3/mta/account_management.bats index f8d5f9de..f0d47968 100644 --- a/test/tests/parallel/set3/mta/account_management.bats +++ b/test/tests/parallel/set3/mta/account_management.bats @@ -29,7 +29,12 @@ function teardown_file() { _default_teardown ; } assert_line --index 5 'alias1@localhost.localdomain' # TODO: Probably not intentional?: assert_line --index 6 '@localdomain2.com' - _should_output_number_of_lines 7 + # Dovecot "dummy accounts" for quota support, see `test/config/postfix-virtual.cf` for more context + assert_line --index 7 'prefixtest@localhost.localdomain' + assert_line --index 8 'test@localhost.localdomain' + assert_line --index 9 'first-name@localhost.localdomain' + assert_line --index 10 'first.name@localhost.localdomain' + _should_output_number_of_lines 11 # Relevant log output from scripts/helpers/accounts.sh:_create_dovecot_alias_dummy_accounts(): # [ DEBUG ] Adding alias 'alias1@localhost.localdomain' for user 'user1@localhost.localdomain' to Dovecot's userdb @@ -37,6 +42,19 @@ function teardown_file() { _default_teardown ; } # [ DEBUG ] Adding alias '@localdomain2.com' for user 'user1@localhost.localdomain' to Dovecot's userdb } +# Dovecot "dummy accounts" for quota support, see `test/config/postfix-virtual.cf` for more context +@test "should create all dovecot dummy accounts" { + run docker logs "${CONTAINER_NAME}" + assert_success + assert_line --partial "Adding alias 'prefixtest@localhost.localdomain' for user 'user2@otherdomain.tld' to Dovecot's userdb" + assert_line --partial "Adding alias 'test@localhost.localdomain' for user 'user2@otherdomain.tld' to Dovecot's userdb" + refute_line --partial "Alias 'test@localhost.localdomain' will not be added to '/etc/dovecot/userdb' twice" + + assert_line --partial "Adding alias 'first-name@localhost.localdomain' for user 'user2@otherdomain.tld' to Dovecot's userdb" + assert_line --partial "Adding alias 'first.name@localhost.localdomain' for user 'user2@otherdomain.tld' to Dovecot's userdb" + refute_line --partial "Alias 'first.name@localhost.localdomain' will not be added to '/etc/dovecot/userdb' twice" +} + @test "should have created maildir for 'user1@localhost.localdomain'" { _run_in_container_bash '[[ -d /var/mail/localhost.localdomain/user1 ]]' assert_success From ff8fc8013b3d145f7bdb0210d168f973e4af9329 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 28 Oct 2024 19:23:45 +0100 Subject: [PATCH 166/267] chore(deps): Bump anchore/scan-action from 4.1.2 to 5.1.0 (#4239) --- .github/workflows/generic_vulnerability-scan.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index d0088e1c..42fa30f3 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -55,7 +55,7 @@ jobs: provenance: false - name: 'Run the Anchore Grype scan action' - uses: anchore/scan-action@v4.1.2 + uses: anchore/scan-action@v5.1.0 id: scan with: image: mailserver-testing:ci From 662afec1d115ad694acbaf869bdb6e24cc98e96c Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Wed, 30 Oct 2024 01:32:17 +0100 Subject: [PATCH 167/267] doc: add an example for using the keytype ed25519 when using DKIM & Rspamd (#4243) --- target/bin/rspamd-dkim | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/target/bin/rspamd-dkim b/target/bin/rspamd-dkim index 689aa7d5..b376271b 100755 --- a/target/bin/rspamd-dkim +++ b/target/bin/rspamd-dkim @@ -55,6 +55,10 @@ ${ORANGE}EXAMPLES${RESET} ${LWHITE}setup config dkim domain example.com${RESET} Generate the DKIM key for a different domain (example.com). + ${LWHITE}setup config dkim keytype ed25519 domain edward.com selector elliptic-test${RESET} + Generate the DKIM key using the ED25519 elliptic curve for the domain + edward.com and the selector elliptic-test. + ${ORANGE}EXIT STATUS${RESET} Exit status is 0 if command was successful. If wrong arguments are provided or arguments contain errors, the script will exit early with a non-zero exit status. From e6bd0b0a0932f11c60541322e769f3d10b711552 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Fri, 1 Nov 2024 14:11:33 +1300 Subject: [PATCH 168/267] ci(pr-docs): Acquire metadata from context (#4244) The metadata needed is available via context, prefer this approach instead. --- .github/workflows/docs-preview-deploy.yml | 29 ++++++++++++++++++---- .github/workflows/docs-preview-prepare.yml | 17 +------------ 2 files changed, 25 insertions(+), 21 deletions(-) diff --git a/.github/workflows/docs-preview-deploy.yml b/.github/workflows/docs-preview-deploy.yml index 7c924f86..37b5464e 100644 --- a/.github/workflows/docs-preview-deploy.yml +++ b/.github/workflows/docs-preview-deploy.yml @@ -16,14 +16,20 @@ jobs: preview: name: 'Deploy Preview' runs-on: ubuntu-22.04 - if: ${{ github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success' }} + # Requires a PR event triggered `docs-preview-prepare.yml` workflow run that was successful + ensure the head SHA belongs to an associated PR: + if: | + ${{ + github.event.workflow_run.conclusion == 'success' + && github.event.workflow_run.event == 'pull_request' + && contains(github.event.workflow_run.pull_requests.*.head.sha, github.event.workflow_run.head_sha) + }} steps: # ======================== # # Restore workflow context # # ======================== # - # Retrieve the artifact uploaded from `docs-preview-prepare.yml` workflow run that triggered this deployment + # Retrieve the build artifact uploaded from the `docs-preview-prepare.yml` workflow run (that triggered this deployment workflow): - name: 'Download build artifact' uses: actions/download-artifact@v4 with: @@ -34,8 +40,18 @@ jobs: - name: 'Extract build artifact' run: tar -xf artifact.tar.zst - - name: 'Restore preserved ENV' - run: cat pr.env >> "${GITHUB_ENV}" + # The `workflow_run` metadata contains an array of `pull_requests`, get the `workflow_run` equivalent of `github.event.pull_request.number`. + # There should only be one PR item in the array, verify that it shares the same `head_sha` (latest commit of PR). + - name: 'Get PR number' + env: + head_sha: ${{ github.event.workflow_run.head_sha }} + pull_requests: ${{ tojson(github.event.workflow_run.pull_requests) }} + run: | + PR_NUMBER=$(jq -r '[.[] | select(.head.sha == "${{ env.head_sha }}")][0].number' <<< '${{ env.pull_requests }}') + { + echo "PR_NUMBER=${PR_NUMBER}" + echo 'PR_HEADSHA=${{ env.head_sha }}' + } >> "${GITHUB_ENV}" # ==================== # # Deploy preview build # @@ -61,6 +77,9 @@ jobs: env: NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }} NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }} + # Keep these two ENV in sync with the `docs-preview-prepare.yml` workflow: + BUILD_DIR: docs/site + NETLIFY_SITE_PREFIX: pullrequest-${{ env.PR_NUMBER }} with: github-token: ${{ secrets.GITHUB_TOKEN }} # Fail the job early if credentials are missing / invalid: @@ -71,7 +90,7 @@ jobs: # Only publish the contents of the build output: publish-dir: ${{ env.BUILD_DIR }} # Custom message for the deploy log on Netlify: - deploy-message: '${{ env.PR_TITLE }} (PR #${{ env.PR_NUMBER }} @ commit: ${{ env.PR_HEADSHA }})' + deploy-message: 'Preview Build (PR #${{ env.PR_NUMBER }} @ commit: ${{ env.PR_HEADSHA }})' # Note: Split workflow incorrectly references latest primary branch commit for deployment. # Assign to non-default Deployment Environment for better management: diff --git a/.github/workflows/docs-preview-prepare.yml b/.github/workflows/docs-preview-prepare.yml index cee5562a..bf7ad135 100644 --- a/.github/workflows/docs-preview-prepare.yml +++ b/.github/workflows/docs-preview-prepare.yml @@ -54,23 +54,8 @@ jobs: # ============================== # # Minimize risk of upload failure by bundling files to a single compressed archive (tar + zstd). - # Bundles build dir and env file into a compressed archive, nested file paths will be preserved. - name: 'Prepare artifact for transfer' - env: - # As a precaution, reference this value by an interpolated ENV var; - # instead of interpolating user controllable input directly in the shell script.. - # https://github.com/docker-mailserver/docker-mailserver/issues/2332#issuecomment-998326798 - PR_TITLE: ${{ github.event.pull_request.title }} - run: | - # Save ENV for transfer - { - echo "PR_HEADSHA=${{ github.event.pull_request.head.sha }}" - echo "PR_NUMBER=${{ github.event.pull_request.number }}" - echo "PR_TITLE=${PR_TITLE}" - echo "NETLIFY_SITE_PREFIX=${{ env.NETLIFY_SITE_PREFIX }}" - echo "BUILD_DIR=${{ env.BUILD_DIR }}" - } >> pr.env - tar --zstd -cf artifact.tar.zst pr.env ${{ env.BUILD_DIR }} + run: tar --zstd -cf artifact.tar.zst ${{ env.BUILD_DIR }} - name: 'Upload artifact for workflow transfer' uses: actions/upload-artifact@v4 From dc0a6403b22536ce72841b48636abf7a254a4dbd Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 5 Nov 2024 09:21:53 +1300 Subject: [PATCH 169/267] chore(deps): Bump anchore/scan-action from 5.1.0 to 5.2.0 (#4249) Bumps [anchore/scan-action](https://github.com/anchore/scan-action) from 5.1.0 to 5.2.0. - [Release notes](https://github.com/anchore/scan-action/releases) - [Changelog](https://github.com/anchore/scan-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/anchore/scan-action/compare/v5.1.0...v5.2.0) --- updated-dependencies: - dependency-name: anchore/scan-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_vulnerability-scan.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 42fa30f3..b96d8ea5 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -55,7 +55,7 @@ jobs: provenance: false - name: 'Run the Anchore Grype scan action' - uses: anchore/scan-action@v5.1.0 + uses: anchore/scan-action@v5.2.0 id: scan with: image: mailserver-testing:ci From 0ff9c0132a8914d6756739a7a3b085e47870b93d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alvaro=20Mu=C3=B1oz?= Date: Tue, 5 Nov 2024 00:50:08 +0100 Subject: [PATCH 170/267] ci: Revise `docs-preview-deploy.yml` (#4247) - Fixes the `if` condition that was recently adjusted. - Better documents concerns for maintainers to be aware of. - Reference the `pull_requests` ENV at runtime instead of embedding content into the script via GHA context expression. This is a better practice which prevent exploits from untrusted inputs (_notably for context objects which might introduce new fields in future_). --------- Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- .github/workflows/docs-preview-deploy.yml | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/.github/workflows/docs-preview-deploy.yml b/.github/workflows/docs-preview-deploy.yml index 37b5464e..02dc4183 100644 --- a/.github/workflows/docs-preview-deploy.yml +++ b/.github/workflows/docs-preview-deploy.yml @@ -17,12 +17,12 @@ jobs: name: 'Deploy Preview' runs-on: ubuntu-22.04 # Requires a PR event triggered `docs-preview-prepare.yml` workflow run that was successful + ensure the head SHA belongs to an associated PR: + # NOTE: A multi-line `if` GHA expression must avoid wrapping with `${{ }}`, otherwise it is unintentionally parsed as a string: + # https://github.com/nikitastupin/pwnhub/blob/main/writings/if-condition.md if: | - ${{ - github.event.workflow_run.conclusion == 'success' - && github.event.workflow_run.event == 'pull_request' - && contains(github.event.workflow_run.pull_requests.*.head.sha, github.event.workflow_run.head_sha) - }} + github.event.workflow_run.conclusion == 'success' + && github.event.workflow_run.event == 'pull_request' + && contains(github.event.workflow_run.pull_requests.*.head.sha, github.event.workflow_run.head_sha) steps: # ======================== # @@ -42,12 +42,14 @@ jobs: # The `workflow_run` metadata contains an array of `pull_requests`, get the `workflow_run` equivalent of `github.event.pull_request.number`. # There should only be one PR item in the array, verify that it shares the same `head_sha` (latest commit of PR). + # NOTE: Careful when using GHA context expressions that may have untrusted input here. The expressions are evaluated before the script content itself is run: + # https://github.com/docker-mailserver/docker-mailserver/pull/4247#discussion_r1827067475 - name: 'Get PR number' env: head_sha: ${{ github.event.workflow_run.head_sha }} pull_requests: ${{ tojson(github.event.workflow_run.pull_requests) }} run: | - PR_NUMBER=$(jq -r '[.[] | select(.head.sha == "${{ env.head_sha }}")][0].number' <<< '${{ env.pull_requests }}') + PR_NUMBER=$(jq -r '[.[] | select(.head.sha == "${{ env.head_sha }}")][0].number' <<< "${pull_requests}") { echo "PR_NUMBER=${PR_NUMBER}" echo 'PR_HEADSHA=${{ env.head_sha }}' From a599936c4b5cfc81d00f7bc9d3c67ddff48f5f52 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Sun, 10 Nov 2024 18:57:31 +1300 Subject: [PATCH 171/267] ci: `docs-preview-deploy.yml` - Remove third job condition This condition was added as an additional guard but was preventing the workflow from running when PRs were from forked repos. --- .github/workflows/docs-preview-deploy.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/docs-preview-deploy.yml b/.github/workflows/docs-preview-deploy.yml index 02dc4183..e7ca9f12 100644 --- a/.github/workflows/docs-preview-deploy.yml +++ b/.github/workflows/docs-preview-deploy.yml @@ -22,7 +22,6 @@ jobs: if: | github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'pull_request' - && contains(github.event.workflow_run.pull_requests.*.head.sha, github.event.workflow_run.head_sha) steps: # ======================== # From b960efad748dbf6fdfe7be2e864eca431296cbf5 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 12 Nov 2024 08:49:29 +1300 Subject: [PATCH 172/267] chore(deps): Bump anchore/scan-action from 5.2.0 to 5.2.1 (#4260) Bumps [anchore/scan-action](https://github.com/anchore/scan-action) from 5.2.0 to 5.2.1. - [Release notes](https://github.com/anchore/scan-action/releases) - [Changelog](https://github.com/anchore/scan-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/anchore/scan-action/compare/v5.2.0...v5.2.1) --- updated-dependencies: - dependency-name: anchore/scan-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_vulnerability-scan.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index b96d8ea5..5b597d87 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -55,7 +55,7 @@ jobs: provenance: false - name: 'Run the Anchore Grype scan action' - uses: anchore/scan-action@v5.2.0 + uses: anchore/scan-action@v5.2.1 id: scan with: image: mailserver-testing:ci From 6b4627ceab44c65d04ae25926aab05956ca2bafb Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Fri, 15 Nov 2024 13:00:40 +1300 Subject: [PATCH 173/267] ci(docs-preview): Refactor workflows (#4262) **Overview of changes:** - Runner bumped from Ubuntu 22.04 => 24.04 - Revised inline documentation for maintainers. - The output of `build-docs.sh` is now grouped in the steps action log, and now hides the noise from pulling the image via `docker run`. - Removed the separate `tar` steps with ZSTD as there is only a directory to archive with recent changes to this workflow. The `upload` + `download` actions are sufficient. - The `workflow_run` job has had the PR context restore step extracted to a separate job to minimize noise. - `actions-netlify` is still effectively the same functionality. - `github-token` is no longer configured as it doesn't appear needed with the functions disabled. - Opt-out of the GH deployments feature which is not needed. --- .github/workflows/docs-preview-deploy.yml | 161 +++++++++++-------- .github/workflows/docs-preview-prepare.yml | 61 ++++--- .github/workflows/scripts/docs/build-docs.sh | 3 +- 3 files changed, 133 insertions(+), 92 deletions(-) diff --git a/.github/workflows/docs-preview-deploy.yml b/.github/workflows/docs-preview-deploy.yml index e7ca9f12..40b5cbd2 100644 --- a/.github/workflows/docs-preview-deploy.yml +++ b/.github/workflows/docs-preview-deploy.yml @@ -1,133 +1,164 @@ name: 'Documentation (run)' on: + # This workflow runs off the primary branch which provides access to the `secrets` context: workflow_run: workflows: ['Documentation (PR)'] types: - completed -# Note: If limiting concurrency is required for this workflow: -# 1. Add an additional job prior to `preview` to get the PR number make it an output. -# 2. Assign that new job as a `needs` dependency for the `preview` job. -# It is still required for `preview` job to download the artifact so that it can access the preview build files. +permissions: + # Required by `actions/download-artifact`: + actions: read + # Required by `marocchino/sticky-pull-request-comment`: + pull-requests: write + # Required by `myrotvorets/set-commit-status-action`: + statuses: write -# This workflow runs off the primary branch and has access to secrets as expected. jobs: - preview: - name: 'Deploy Preview' - runs-on: ubuntu-22.04 + # This could have been another step in the `deploy-preview` job and used `GITHUB_ENV` instead of `GITHUB_OUTPUT`. + # It was split out into a separate job for a cleaner overview of `deploy-preview` ENV inputs and to minimize noise + # from that job related to this workaround (_that is incompatible with PRs from forks_). + pr-context: + name: 'Restore PR Context' + runs-on: ubuntu-24.04 + outputs: + PR_HEADSHA: ${{ steps.set-pr-context.outputs.PR_HEADSHA }} + PR_NUMBER: ${{ steps.set-pr-context.outputs.PR_NUMBER }} # Requires a PR event triggered `docs-preview-prepare.yml` workflow run that was successful + ensure the head SHA belongs to an associated PR: - # NOTE: A multi-line `if` GHA expression must avoid wrapping with `${{ }}`, otherwise it is unintentionally parsed as a string: - # https://github.com/nikitastupin/pwnhub/blob/main/writings/if-condition.md + # NOTE: + # - The `contains` condition checks for event context that is not available when the PR is from a fork. An alternative method would be needed: + # https://stackoverflow.com/questions/59077079/how-to-get-pull-request-number-within-github-actions-workflow/79017997#79017997 + # - A multi-line `if` GHA expression must avoid wrapping with `${{ }}`, otherwise it is unintentionally parsed as a string: + # https://github.com/nikitastupin/pwnhub/blob/main/writings/if-condition.md if: | github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'pull_request' + && contains(github.event.workflow_run.pull_requests.*.head.sha, github.event.workflow_run.head_sha) steps: - - # ======================== # - # Restore workflow context # - # ======================== # - - # Retrieve the build artifact uploaded from the `docs-preview-prepare.yml` workflow run (that triggered this deployment workflow): - - name: 'Download build artifact' - uses: actions/download-artifact@v4 - with: - name: preview-build - github-token: ${{ secrets.GITHUB_TOKEN }} - run-id: ${{ github.event.workflow_run.id }} - - - name: 'Extract build artifact' - run: tar -xf artifact.tar.zst - - # The `workflow_run` metadata contains an array of `pull_requests`, get the `workflow_run` equivalent of `github.event.pull_request.number`. - # There should only be one PR item in the array, verify that it shares the same `head_sha` (latest commit of PR). - # NOTE: Careful when using GHA context expressions that may have untrusted input here. The expressions are evaluated before the script content itself is run: - # https://github.com/docker-mailserver/docker-mailserver/pull/4247#discussion_r1827067475 + # NOTE: + # - The `workflow_run` metadata contains an array of `pull_requests`: + # 1. Take the `workflow_run` equivalent of `github.event.pull_request.number`. + # 2. There should only be one PR item in the array, verify that it shares the same `head_sha` (latest commit of PR). + # - Careful when using GHA context expressions that may have untrusted input here. The expressions are evaluated before the script content itself is run: + # https://github.com/docker-mailserver/docker-mailserver/pull/4247#discussion_r1827067475 - name: 'Get PR number' + id: set-pr-context env: head_sha: ${{ github.event.workflow_run.head_sha }} pull_requests: ${{ tojson(github.event.workflow_run.pull_requests) }} run: | PR_NUMBER=$(jq -r '[.[] | select(.head.sha == "${{ env.head_sha }}")][0].number' <<< "${pull_requests}") { - echo "PR_NUMBER=${PR_NUMBER}" echo 'PR_HEADSHA=${{ env.head_sha }}' - } >> "${GITHUB_ENV}" + echo "PR_NUMBER=${PR_NUMBER}" + } >> "${GITHUB_OUTPUT}" + + deploy-preview: + name: 'Deploy Preview' + runs-on: ubuntu-24.04 + needs: [pr-context] + env: + # NOTE: Keep this in sync with the equivalent ENV in `docs-preview-prepare.yml`: + BUILD_DIR: docs/site/ + # PR head SHA (latest commit): + PR_HEADSHA: ${{ needs.pr-context.outputs.PR_HEADSHA }} + PR_NUMBER: ${{ needs.pr-context.outputs.PR_NUMBER }} + # Deploy URL preview prefix (the site name for this prefix is managed at Netlify): + PREVIEW_SITE_PREFIX: pullrequest-${{ needs.pr-context.outputs.PR_NUMBER }} + steps: + - name: 'Retrieve and extract the built docs preview' + uses: actions/download-artifact@v4 + with: + name: preview-build + path: ${{ env.BUILD_DIR }} + # These are needed due this approach relying on `workflow_run`, so that it can access the build artifact: + # (uploaded from the associated `docs-preview-prepare.yml` workflow run) + github-token: ${{ secrets.GITHUB_TOKEN }} + run-id: ${{ github.event.workflow_run.id }} # ==================== # # Deploy preview build # # ==================== # - # Manage workflow deployment status. `enable-commit-status` from `nwtgck/actions-netlify` would handle this, - # but presently does not work correctly via split workflow. It is useful in a split workflow as the 1st stage - # no longer indicates if the entire workflow/deployment was successful. - - name: 'Commit Status: Set Workflow Status as Pending' + # Manage workflow deployment status (Part 1/2): + # NOTE: + # - `workflow_run` trigger does not appear on the PR/commit checks status, only the initial prepare workflow triggered. + # This adds our own status check for this 2nd half of the workflow starting as `pending`, followed by `success` / `failure` at the end. + # - `enable-commit-status` from `nwtgck/actions-netlify` would have handled this, + # but the context `github.sha` that action tries to use references the primary branch commit that this workflow runs from, not the relevant PR commit. + - name: 'Commit Status (1/2) - Set Workflow Status as Pending' uses: myrotvorets/set-commit-status-action@v2.0.1 with: token: ${{ secrets.GITHUB_TOKEN }} status: pending - # Should match `env.PR_HEADSHA` when triggered by `pull_request` event workflow, - # Avoids failure of ENV being unavailable if job fails early: - sha: ${{ github.event.workflow_run.head_sha }} + sha: ${{ env.PR_HEADSHA }} context: 'Deploy Preview (pull_request => workflow_run)' - name: 'Send preview build to Netlify' uses: nwtgck/actions-netlify@v3.0 - id: preview + id: preview-netlify timeout-minutes: 1 env: NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }} - NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }} - # Keep these two ENV in sync with the `docs-preview-prepare.yml` workflow: - BUILD_DIR: docs/site - NETLIFY_SITE_PREFIX: pullrequest-${{ env.PR_NUMBER }} + NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }} with: - github-token: ${{ secrets.GITHUB_TOKEN }} - # Fail the job early if credentials are missing / invalid: + # Fail the job when the required Netlify credentials are missing from ENV: fails-without-credentials: true - # Sets/creates the Netlify deploy URL prefix. - # Uses the PR number for uniqueness: - alias: ${{ env.NETLIFY_SITE_PREFIX }} + # Set/create the Netlify deploy URL prefix: + alias: ${{ env.PREVIEW_SITE_PREFIX }} # Only publish the contents of the build output: publish-dir: ${{ env.BUILD_DIR }} # Custom message for the deploy log on Netlify: - deploy-message: 'Preview Build (PR #${{ env.PR_NUMBER }} @ commit: ${{ env.PR_HEADSHA }})' + deploy-message: 'Preview Build (PR #${{ env.PR_NUMBER }} @ commit: ${{ env.PR_HEADSHA }}' - # Note: Split workflow incorrectly references latest primary branch commit for deployment. - # Assign to non-default Deployment Environment for better management: - github-deployment-environment: documentation-previews - github-deployment-description: 'Preview deploy for documentation PRs' - - # Note - PR context used by this action is incorrect. These features are broken with split workflow: - # https://github.com/nwtgck/actions-netlify/issues/545 # Disable unwanted action defaults: - # Disable adding deploy comment on pre-merge commit (Github creates this for PR diff): + # This input does not fallback to the GITHUB_TOKEN taken from context, nor log that it will skip extra features of the action when this input is not set: + # https://github.com/nwtgck/actions-netlify/issues/1219 + # github-token: ${{ secrets.GITHUB_TOKEN }} + # NOTE: These features won't work correctly when the triggered workflow is not run from the PR branch due to assumed `pull_request` context: + # https://github.com/nwtgck/actions-netlify/issues/545 + # Disable adding a comment to the commit belonging to context `github.sha` about the successful deployment (redundant and often wrong commit): enable-commit-comment: false - # Disable adding a "Netlify - Netlify deployment" check status: + # Disable adding a "Netlify - Netlify deployment" PR check status (workflow job status is sufficient): enable-commit-status: false - # Disable. We provide a custom PR comment in the next action: + # Disable adding a comment about successful deployment status to the PR. + # Prefer `marocchino/sticky-pull-request-comment` instead (more flexible and allows custom message): enable-pull-request-comment: false + # Opt-out of deployment feature: + # NOTE: + # - When affected by `nwtgck/actions-netlify/issues/545`, the deployments published reference the wrong commit and thus information. + # - While the feature creates or assigns a deployment to associate the build with, it is unrelated to the related environments feature (secrets/vars): + # https://github.com/nwtgck/actions-netlify/issues/538#issuecomment-833983970 + # https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-deployments/viewing-deployment-history + # https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-deployments/managing-environments-for-deployment + enable-github-deployment: false + # Assign to non-default Deployment Environment for better management: + # github-deployment-environment: documentation-previews + # github-deployment-description: 'Preview deploy for documentation PRs' # If a `netlify.toml` config is ever needed, enable this: # netlify-config-path: ./docs/netlify.toml - # If ever switching from Github Pages, enable this conditionally (false by default): + # If ever switching from Github Pages, enable this only when not deploying a preview build (false by default): # production-deploy: false - - name: 'Comment on PR: Add/Update deployment status' + - name: 'Comment on PR with preview link' uses: marocchino/sticky-pull-request-comment@v2 with: number: ${{ env.PR_NUMBER }} header: preview-comment recreate: true message: | - [Documentation preview for this PR](${{ steps.preview.outputs.deploy-url }}) is ready! :tada: + [Documentation preview for this PR](${{ steps.preview-netlify.outputs.deploy-url }}) is ready! :tada: Built with commit: ${{ env.PR_HEADSHA }} - - name: 'Commit Status: Update deployment status' + # Manage workflow deployment status (Part 2/2): + - name: 'Commit Status (2/2) - Update deployment status' uses: myrotvorets/set-commit-status-action@v2.0.1 - # Always run this step regardless of job failing early: + # Always run this step regardless of the job failing early: if: ${{ always() }} + # Custom status descriptions: env: DEPLOY_SUCCESS: Successfully deployed preview. DEPLOY_FAILURE: Failed to deploy preview. diff --git a/.github/workflows/docs-preview-prepare.yml b/.github/workflows/docs-preview-prepare.yml index bf7ad135..40b586e0 100644 --- a/.github/workflows/docs-preview-prepare.yml +++ b/.github/workflows/docs-preview-prepare.yml @@ -7,59 +7,68 @@ on: - '.github/workflows/scripts/docs/build-docs.sh' - '.github/workflows/docs-preview-prepare.yml' -# If the workflow for a PR is triggered multiple times, previous existing runs will be canceled. -# eg: Applying multiple suggestions from a review directly via the Github UI. -# Instances of the 2nd phase of this workflow (via `workflow_run`) presently lack concurrency limits due to added complexity. +# If this workflow is triggered while already running for the PR, cancel any earlier running instances: +# Instances of the 2nd phase of this workflow (via `workflow_run`) lack any concurrency limits due to added complexity. concurrency: group: deploypreview-pullrequest-${{ github.event.pull_request.number }} cancel-in-progress: true +env: + # Build output directory (created by the mkdocs-material container, keep this in sync with `build-docs.sh`): + BUILD_DIR: docs/site/ + # These two are only needed to construct `PREVIEW_URL`: + PREVIEW_SITE_NAME: dms-doc-previews + PREVIEW_SITE_PREFIX: pullrequest-${{ github.event.pull_request.number }} + # `pull_request` workflow is unreliable alone: Non-collaborator contributions lack access to secrets for security reasons. # A separate workflow (docs-preview-deploy.yml) handles the deploy after the potentially untrusted code is first run in this workflow. # See: https://securitylab.github.com/research/github-actions-preventing-pwn-requests/ permissions: + # Required by `actions/checkout` for git checkout: contents: read jobs: prepare-preview: name: 'Build Preview' - runs-on: ubuntu-22.04 - env: - BUILD_DIR: docs/site - NETLIFY_SITE_PREFIX: pullrequest-${{ github.event.pull_request.number }} - NETLIFY_SITE_NAME: dms-doc-previews + runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v4 - - name: 'Build with mkdocs-material via Docker' - working-directory: docs - env: - PREVIEW_URL: 'https://${NETLIFY_SITE_PREFIX}--${NETLIFY_SITE_NAME}.netlify.app/' - NETLIFY_BRANDING: 'Deploys by Netlify' - run: | - # Adjust mkdocs.yml for preview build - sed -i "s|^site_url:.*|site_url: '${PREVIEW_URL}'|" mkdocs.yml + # ================== # + # Build docs preview # + # ================== # - # Insert sponsor branding into page content (Provider OSS plan requirement): - # Upstream does not provide a nicer maintainable way to do this.. - # Prepends HTML to copyright text and then aligns to the right side. + - name: 'Build with mkdocs-material via Docker' + working-directory: docs/ + env: + PREVIEW_URL: 'https://${{ env.PREVIEW_SITE_PREFIX }}--${{ env.PREVIEW_SITE_NAME }}.netlify.app/' + run: | + # Adjust `mkdocs.yml` for the preview build requirements: + # - Replace production `site_url` with the preview URL (only affects the canonical link: https://en.wikipedia.org/wiki/Canonical_link_element#HTML) + # - Prepend Netlify logo link to `copyright` content + sed -i "s|^site_url:.*|site_url: '${{ env.PREVIEW_URL }}'|" mkdocs.yml + + # Insert branding into page content (Netlify OSS plan requirement): + # - `mkdocs-material` does not provide a better way to do this. + # - Prepends HTML to the copyright text and then aligns the logo to the right-side of the page. + NETLIFY_BRANDING='Deploys by Netlify' sed -i "s|^copyright: '|copyright: '${NETLIFY_BRANDING}|" mkdocs.yml - # Need to override a CSS media query for parent element to always be full width: + # Override a CSS media query for the parent element to always be full width: echo '.md-footer-copyright { width: 100%; }' >> content/assets/css/customizations.css - ../.github/workflows/scripts/docs/build-docs.sh + # Build and prepare for upload: + echo "::group::Build (stdout)" + bash ../.github/workflows/scripts/docs/build-docs.sh + echo "::endgroup::" # ============================== # # Volley over to secure workflow # # ============================== # - # Minimize risk of upload failure by bundling files to a single compressed archive (tar + zstd). - - name: 'Prepare artifact for transfer' - run: tar --zstd -cf artifact.tar.zst ${{ env.BUILD_DIR }} - + # Archives directory `path` into a ZIP file: - name: 'Upload artifact for workflow transfer' uses: actions/upload-artifact@v4 with: name: preview-build - path: artifact.tar.zst + path: ${{ env.BUILD_DIR }} retention-days: 1 diff --git a/.github/workflows/scripts/docs/build-docs.sh b/.github/workflows/scripts/docs/build-docs.sh index 5d1cab52..d4384b5e 100755 --- a/.github/workflows/scripts/docs/build-docs.sh +++ b/.github/workflows/scripts/docs/build-docs.sh @@ -7,8 +7,9 @@ set -ex # `build --strict` ensures the build fails when any warnings are omitted. docker run \ --rm \ + --quiet \ --user "$(id -u):$(id -g)" \ - --volume "${PWD}:/docs" \ + --volume "./:/docs" \ --name "build-docs" \ squidfunk/mkdocs-material:9.5 build --strict From 02f1894f74b629d74dbcb8faac2908c2e95c5237 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Wed, 20 Nov 2024 16:37:34 +1300 Subject: [PATCH 174/267] ci(docs-preview): Acquire PR context via `gh` CLI (#4267) --- .github/workflows/docs-preview-deploy.yml | 62 +++++++++++------------ CHANGELOG.md | 1 + 2 files changed, 30 insertions(+), 33 deletions(-) diff --git a/.github/workflows/docs-preview-deploy.yml b/.github/workflows/docs-preview-deploy.yml index 40b5cbd2..ccb9c5f2 100644 --- a/.github/workflows/docs-preview-deploy.yml +++ b/.github/workflows/docs-preview-deploy.yml @@ -1,4 +1,4 @@ -name: 'Documentation (run)' +name: 'Documentation (Deploy)' on: # This workflow runs off the primary branch which provides access to the `secrets` context: @@ -10,49 +10,45 @@ on: permissions: # Required by `actions/download-artifact`: actions: read - # Required by `marocchino/sticky-pull-request-comment`: + # Required by `set-pr-context`: + contents: read + # Required by `marocchino/sticky-pull-request-comment` (write) + `set-pr-context` (read): pull-requests: write # Required by `myrotvorets/set-commit-status-action`: statuses: write jobs: - # This could have been another step in the `deploy-preview` job and used `GITHUB_ENV` instead of `GITHUB_OUTPUT`. - # It was split out into a separate job for a cleaner overview of `deploy-preview` ENV inputs and to minimize noise - # from that job related to this workaround (_that is incompatible with PRs from forks_). + # NOTE: This is handled as pre-requisite job to minimize the noise from acquiring these two outputs needed for `deploy-preview` ENV: pr-context: - name: 'Restore PR Context' + name: 'Acquire PR Context' runs-on: ubuntu-24.04 outputs: - PR_HEADSHA: ${{ steps.set-pr-context.outputs.PR_HEADSHA }} - PR_NUMBER: ${{ steps.set-pr-context.outputs.PR_NUMBER }} - # Requires a PR event triggered `docs-preview-prepare.yml` workflow run that was successful + ensure the head SHA belongs to an associated PR: - # NOTE: - # - The `contains` condition checks for event context that is not available when the PR is from a fork. An alternative method would be needed: - # https://stackoverflow.com/questions/59077079/how-to-get-pull-request-number-within-github-actions-workflow/79017997#79017997 - # - A multi-line `if` GHA expression must avoid wrapping with `${{ }}`, otherwise it is unintentionally parsed as a string: - # https://github.com/nikitastupin/pwnhub/blob/main/writings/if-condition.md - if: | - github.event.workflow_run.conclusion == 'success' - && github.event.workflow_run.event == 'pull_request' - && contains(github.event.workflow_run.pull_requests.*.head.sha, github.event.workflow_run.head_sha) + PR_HEADSHA: ${{ steps.set-pr-context.outputs.head-sha }} + PR_NUMBER: ${{ steps.set-pr-context.outputs.number }} + if: ${{ github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'pull_request' }} steps: - # NOTE: - # - The `workflow_run` metadata contains an array of `pull_requests`: - # 1. Take the `workflow_run` equivalent of `github.event.pull_request.number`. - # 2. There should only be one PR item in the array, verify that it shares the same `head_sha` (latest commit of PR). - # - Careful when using GHA context expressions that may have untrusted input here. The expressions are evaluated before the script content itself is run: - # https://github.com/docker-mailserver/docker-mailserver/pull/4247#discussion_r1827067475 - - name: 'Get PR number' + - name: 'Get PR context' id: set-pr-context env: - head_sha: ${{ github.event.workflow_run.head_sha }} - pull_requests: ${{ tojson(github.event.workflow_run.pull_requests) }} + # Token is required for the GH CLI: + GH_TOKEN: ${{ github.token }} + # Best practice for scripts is to reference via ENV at runtime. Avoid using GHA context expressions in the script content directly: + # https://github.com/docker-mailserver/docker-mailserver/pull/4247#discussion_r1827067475 + PR_TARGET_REPO: ${{ github.repository }} + # If the PR is from a fork, prefix it with `:`, otherwise only the PR branch name is relevant: + PR_BRANCH: |- + ${{ + (github.event.workflow_run.head_repository.owner.login != github.event.workflow_run.repository.owner.login) + && format('{0}:{1}', github.event.workflow_run.head_repository.owner.login, github.event.workflow_run.head_branch) + || github.event.workflow_run.head_branch + }} + # Use the GH CLI to query the PR branch, which provides the PR number and head SHA to assign as outputs: + # (`--jq` formats JSON to `key=value` pairs and renames `headRefOid` to `head-sha`) run: | - PR_NUMBER=$(jq -r '[.[] | select(.head.sha == "${{ env.head_sha }}")][0].number' <<< "${pull_requests}") - { - echo 'PR_HEADSHA=${{ env.head_sha }}' - echo "PR_NUMBER=${PR_NUMBER}" - } >> "${GITHUB_OUTPUT}" + gh pr view --repo "${PR_TARGET_REPO}" "${PR_BRANCH}" \ + --json 'number,headRefOid' \ + --jq '"number=\(.number)\nhead-sha=\(.headRefOid)"' \ + >> "${GITHUB_OUTPUT}" deploy-preview: name: 'Deploy Preview' @@ -165,6 +161,6 @@ jobs: with: token: ${{ secrets.GITHUB_TOKEN }} status: ${{ job.status == 'success' && 'success' || 'failure' }} - sha: ${{ github.event.workflow_run.head_sha }} + sha: ${{ env.PR_HEADSHA }} context: 'Deploy Preview (pull_request => workflow_run)' description: ${{ job.status == 'success' && env.DEPLOY_SUCCESS || env.DEPLOY_FAILURE }} diff --git a/CHANGELOG.md b/CHANGELOG.md index 28a97c32..0ba86566 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -59,6 +59,7 @@ All notable changes to this project will be documented in this file. The format ### CI - Workflow for `CONTRIBUTORS.md` updates removed. `CONTRIBUTORS.md` file and dependencies removed. ([#4141](https://github.com/docker-mailserver/docker-mailserver/pull/4141)) +- Refactored the workflows for generating documentation previews on PRs to be more secure ([#4267](https://github.com/docker-mailserver/docker-mailserver/pull/4267), [#4264](https://github.com/docker-mailserver/docker-mailserver/pull/4264), [#4262](https://github.com/docker-mailserver/docker-mailserver/pull/4262), [#4247](https://github.com/docker-mailserver/docker-mailserver/pull/4247), [#4244](https://github.com/docker-mailserver/docker-mailserver/pull/4244)) ## [v14.0.0](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v14.0.0) From c15354058fec085d331e6c01f0e3d1f498ea6f06 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Wed, 20 Nov 2024 17:19:58 +1300 Subject: [PATCH 175/267] fix: SASLAuth - Drop services for `mysql`, `shadow`, `pam` auth mechanisms (#4259) --- CHANGELOG.md | 1 + docs/content/config/environment.md | 22 ++++++++++-------- target/scripts/startup/variables-stack.sh | 5 ++-- target/supervisor/conf.d/saslauth.conf | 28 ----------------------- 4 files changed, 17 insertions(+), 39 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ba86566..44f8fd0d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ All notable changes to this project will be documented in this file. The format ### Breaking +- **saslauthd** mechanism support via ENV `SASLAUTHD_MECHANISMS` with `pam`, `shadow`, `mysql` values has been removed. Only `ldap` and `rimap` remain supported ([#4259](https://github.com/docker-mailserver/docker-mailserver/pull/4259)) - **getmail6** has been refactored: ([#4156](https://github.com/docker-mailserver/docker-mailserver/pull/4156)) - The [DMS config volume](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/advanced/optional-config/#volumes) now has support for `getmailrc_general.cf` for overriding [common default settings](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/advanced/mail-getmail/#common-options). If you previously mounted this config file directly to `/etc/getmailrc_general` you should switch to our config volume support. - IMAP/POP3 example configs added to our [`config-examples`](https://github.com/docker-mailserver/docker-mailserver/tree/v15.0.0/config-examples/getmail). diff --git a/docs/content/config/environment.md b/docs/content/config/environment.md index 5a766f53..b7231b10 100644 --- a/docs/content/config/environment.md +++ b/docs/content/config/environment.md @@ -910,22 +910,26 @@ Note: This postgrey setting needs `ENABLE_POSTGREY=1` ##### SASLAUTHD_MECHANISMS -- **empty** => pam -- `ldap` => authenticate against ldap server -- `shadow` => authenticate against local user db -- `mysql` => authenticate against mysql db -- `rimap` => authenticate against imap server -- NOTE: can be a list of mechanisms like pam ldap shadow +DMS only implements support for these mechanisms: + +- **`ldap`** => Authenticate against an LDAP server +- `rimap` => Authenticate against an IMAP server ##### SASLAUTHD_MECH_OPTIONS - **empty** => None -- e.g. with SASLAUTHD_MECHANISMS rimap you need to specify the ip-address/servername of the imap server ==> xxx.xxx.xxx.xxx + +!!! info + + With `SASLAUTHD_MECHANISMS=rimap` you need to specify the ip-address / servername of the IMAP server, such as `SASLAUTHD_MECH_OPTIONS=127.0.0.1`. ##### SASLAUTHD_LDAP_SERVER -- **empty** => same as `LDAP_SERVER_HOST` -- Note: You must include the desired URI scheme (`ldap://`, `ldaps://`, `ldapi://`). +- **empty** => Use the same value as `LDAP_SERVER_HOST` + +!!! note + + You must include the desired URI scheme (`ldap://`, `ldaps://`, `ldapi://`). ##### SASLAUTHD_LDAP_START_TLS diff --git a/target/scripts/startup/variables-stack.sh b/target/scripts/startup/variables-stack.sh index a3be72b8..3fa4d761 100644 --- a/target/scripts/startup/variables-stack.sh +++ b/target/scripts/startup/variables-stack.sh @@ -182,8 +182,9 @@ function _environment_variables_ldap() { function _environment_variables_saslauthd() { _log 'debug' 'Setting SASLAUTHD-related environment variables now' - # Only used by the supervisor service command (upstream default: `/etc/default/saslauthd`) - VARS[SASLAUTHD_MECHANISMS]="${SASLAUTHD_MECHANISMS:=pam}" + # This ENV is only used by the supervisor service config `saslauth.conf`: + # NOTE: `pam` is set as the upstream default in `/etc/default/saslauthd` + VARS[SASLAUTHD_MECHANISMS]="${SASLAUTHD_MECHANISMS:=ldap}" } # This function Writes the contents of the `VARS` map (associative array) diff --git a/target/supervisor/conf.d/saslauth.conf b/target/supervisor/conf.d/saslauth.conf index 508ff83c..e42aa198 100644 --- a/target/supervisor/conf.d/saslauth.conf +++ b/target/supervisor/conf.d/saslauth.conf @@ -7,24 +7,6 @@ stderr_logfile=/var/log/supervisor/%(program_name)s.log command=/usr/sbin/saslauthd -d -a ldap -O /etc/saslauthd.conf pidfile=/var/run/saslauthd/saslauthd.pid -[program:saslauthd_mysql] -startsecs=0 -autostart=false -autorestart=true -stdout_logfile=/var/log/supervisor/%(program_name)s.log -stderr_logfile=/var/log/supervisor/%(program_name)s.log -command=/usr/sbin/saslauthd -d -a mysql -O "%(ENV_SASLAUTHD_MECH_OPTIONS)s" -pidfile=/var/run/saslauthd/saslauthd.pid - -[program:saslauthd_pam] -startsecs=0 -autostart=false -autorestart=true -stdout_logfile=/var/log/supervisor/%(program_name)s.log -stderr_logfile=/var/log/supervisor/%(program_name)s.log -command=/usr/sbin/saslauthd -d -a pam -O "%(ENV_SASLAUTHD_MECH_OPTIONS)s" -pidfile=/var/run/saslauthd/saslauthd.pid - [program:saslauthd_rimap] startsecs=0 autostart=false @@ -33,13 +15,3 @@ stdout_logfile=/var/log/supervisor/%(program_name)s.log stderr_logfile=/var/log/supervisor/%(program_name)s.log command=/usr/sbin/saslauthd -d -a rimap -r -O "%(ENV_SASLAUTHD_MECH_OPTIONS)s" pidfile=/var/run/saslauthd/saslauthd.pid - -[program:saslauthd_shadow] -startsecs=0 -autostart=false -autorestart=true -stdout_logfile=/var/log/supervisor/%(program_name)s.log -stderr_logfile=/var/log/supervisor/%(program_name)s.log -command=/usr/sbin/saslauthd -d -a shadow -O "%(ENV_SASLAUTHD_MECH_OPTIONS)s" -pidfile=/var/run/saslauthd/saslauthd.pid - From ab087d28b398ee5d8f374b101a47e899dbf897ea Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 26 Nov 2024 09:08:10 +1300 Subject: [PATCH 176/267] chore(deps): Bump docker/metadata-action from 5.5.1 to 5.6.1 (#4273) Bumps [docker/metadata-action](https://github.com/docker/metadata-action) from 5.5.1 to 5.6.1. - [Release notes](https://github.com/docker/metadata-action/releases) - [Commits](https://github.com/docker/metadata-action/compare/v5.5.1...v5.6.1) --- updated-dependencies: - dependency-name: docker/metadata-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_publish.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 4f94cc97..596435b2 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -23,7 +23,7 @@ jobs: - name: 'Prepare tags' id: prep - uses: docker/metadata-action@v5.5.1 + uses: docker/metadata-action@v5.6.1 with: images: | ${{ secrets.DOCKER_REPOSITORY }} From edfecbceb163eda98124fdb10dd37bfe6fa0f46d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 26 Nov 2024 09:11:35 +1300 Subject: [PATCH 177/267] chore(deps): Bump anchore/scan-action from 5.2.1 to 5.3.0 (#4274) Bumps [anchore/scan-action](https://github.com/anchore/scan-action) from 5.2.1 to 5.3.0. - [Release notes](https://github.com/anchore/scan-action/releases) - [Changelog](https://github.com/anchore/scan-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/anchore/scan-action/compare/v5.2.1...v5.3.0) --- updated-dependencies: - dependency-name: anchore/scan-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- .github/workflows/generic_vulnerability-scan.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 5b597d87..3db8002c 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -55,7 +55,7 @@ jobs: provenance: false - name: 'Run the Anchore Grype scan action' - uses: anchore/scan-action@v5.2.1 + uses: anchore/scan-action@v5.3.0 id: scan with: image: mailserver-testing:ci From d07e6d67d6740ea9e4d43781d4df43a2aa4665af Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Fri, 29 Nov 2024 16:12:00 +1300 Subject: [PATCH 178/267] chore: Update `jaq` to `2.0.0` (#4277) - Bump to [`jaq` v2 release](https://github.com/01mf02/jaq/releases/tag/v2.0.0), artifact naming convention changed. - Tidied up the changelog a little bit unrelated to this `jaq` update. - Fixed a typo with an `rspamd.sh` comment + minor revision to the comment. --- CHANGELOG.md | 30 +++++++++---------- target/scripts/build/packages.sh | 4 +-- .../startup/setup.d/security/rspamd.sh | 6 ++-- 3 files changed, 20 insertions(+), 20 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 44f8fd0d..b8dc1830 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,7 +16,7 @@ All notable changes to this project will be documented in this file. The format - Added `getmail` as a new service for `supervisor` to manage, replacing cron for periodic polling. - Generated getmail configuration files no longer set the `message_log` option. Instead of individual log files per config, the [default base settings DMS configures](https://github.com/docker-mailserver/docker-mailserver/tree/v15.0.0/target/getmail/getmailrc_general) now enables `message_log_syslog`. This aligns with how other services in DMS log to syslog where it is captured in `mail.log`. - Getmail configurations have changed location from the base of the DMS Config Volume, to the `getmail/` subdirectory. Any existing configurations **must be migrated manually.** - - DMS v14 mistakenly relocated the getmail state directory to the DMS Config Volume as a `getmail/` subdirectory. + - DMS v14 mistakenly relocated the _getmail state directory_ to the _DMS Config Volume_ as a `getmail/` subdirectory. - This has been corrected to `/var/lib/getmail` (_if you have mounted a DMS State Volume to `/var/mail-state`, `/var/lib/getmail` will be symlinked to `/var/mail-state/lib-getmail`_). - To preserve this state when upgrading to DMS v15, **you must manually migrate `getmail/` from the _DMS Config Volume_ to `lib-getmail/` in the _DMS State Volume_.** @@ -33,34 +33,34 @@ All notable changes to this project will be documented in this file. The format ### Updates - **Fail2ban:** - - Bump version to [1.1.0](https://github.com/fail2ban/fail2ban/releases/tag/1.1.0). For more information, check the [changelog](https://github.com/fail2ban/fail2ban/blob/1.1.0/ChangeLog). + - Updated to version [`1.1.0`](https://github.com/fail2ban/fail2ban/releases/tag/1.1.0) ([#4045](https://github.com/docker-mailserver/docker-mailserver/pull/4045)) - **Documentation:** - - Rewritten and organized the pages for Account Management and Authentication ([#4122](https://github.com/docker-mailserver/docker-mailserver/pull/4122)) - - Add caveat for `DMS_VMAIL_UID` not being compatible with `0` / root ([#4143](https://github.com/docker-mailserver/docker-mailserver/pull/4143)) + - Account Management and Authentication pages have been rewritten and better organized ([#4122](https://github.com/docker-mailserver/docker-mailserver/pull/4122)) + - Add a caveat for `DMS_VMAIL_UID` not being compatible with `0` / root ([#4143](https://github.com/docker-mailserver/docker-mailserver/pull/4143)) - **Postfix:** - - Disable Microsoft reactions to outgoing mail ([#4120](https://github.com/docker-mailserver/docker-mailserver/pull/4120)) -- bumped `jaq` version from 1.3.0 to 1.6.0 ([#4190](https://github.com/docker-mailserver/docker-mailserver/pull/4190)) -- updated Rspamd GTube settings and tests ([#4191](https://github.com/docker-mailserver/docker-mailserver/pull/4191)) + - By default opt-out from _Microsoft reactions_ for outbound mail ([#4120](https://github.com/docker-mailserver/docker-mailserver/pull/4120)) +- Updated `jaq` version from `1.3.0` to `2.0.0` ([#4190](https://github.com/docker-mailserver/docker-mailserver/pull/4190)) +- Updated Rspamd GTube settings and tests ([#4191](https://github.com/docker-mailserver/docker-mailserver/pull/4191)) ### Fixes - **Dovecot:** - - Update logwatch `ignore.conf` to exclude Xapian messages about pending documents ([#4060](https://github.com/docker-mailserver/docker-mailserver/pull/4060)) + - The logwatch `ignore.conf` now also excludes Xapian messages about pending documents ([#4060](https://github.com/docker-mailserver/docker-mailserver/pull/4060)) - `dovecot-fts-xapian` plugin was updated to `1.7.13`, fixing a regression with indexing ([#4095](https://github.com/docker-mailserver/docker-mailserver/pull/4095)) - - The Dovecot Quota support "dummy account" workaround no longer treats the alias as a regex when checking the Dovecot UserDB ([#4222](https://github.com/docker-mailserver/docker-mailserver/pull/4222)) + - The "dummy account" workaround for _Dovecot Quota_ feature support no longer treats the alias as a regex when checking the Dovecot UserDB ([#4222](https://github.com/docker-mailserver/docker-mailserver/pull/4222)) - **LDAP:** - - A previous compatibility fix for OAuth2 in v13.3.1 had not applied the actual LDAP config changes. This has been corrected ([#4175](https://github.com/docker-mailserver/docker-mailserver/pull/4175)) + - Correctly apply a compatibility fix for OAuth2 introduced in DMS v13.3.1 which had not been applied to the actual LDAP config changes ([#4175](https://github.com/docker-mailserver/docker-mailserver/pull/4175)) - **Internal:** - - The main `mail.log` which is piped to stdout via `tail` now correctly begins from the first log line of the active container run. Previously some daemon logs and potential warnings/errors were omitted. ([#4146](https://github.com/docker-mailserver/docker-mailserver/pull/4146)) + - The main `mail.log` (_which is piped to stdout via `tail`_) now correctly begins from the first log line of the active container run. Previously some daemon logs and potential warnings/errors were omitted ([#4146](https://github.com/docker-mailserver/docker-mailserver/pull/4146)) - Fixed a regression introduced in v14 where `postfix-main.cf` appended `stderr` output into `/etc/postfix/main.cf`, causing Postfix startup to fail ([#4147](https://github.com/docker-mailserver/docker-mailserver/pull/4147)) - - Unused `shopt -s inherit_errexit` removed from `start-mailserver.sh` ([#4161](https://github.com/docker-mailserver/docker-mailserver/pull/4161)) + - `start-mailserver.sh` removed unused `shopt -s inherit_errexit` ([#4161](https://github.com/docker-mailserver/docker-mailserver/pull/4161)) - **Rspamd:** - - DKIM private key path checking is now performed only on paths that do not contain "$" ([#4201](https://github.com/docker-mailserver/docker-mailserver/pull/4201)) + - DKIM private key path checking is now performed only on paths that do not contain `$` ([#4201](https://github.com/docker-mailserver/docker-mailserver/pull/4201)) ### CI -- Workflow for `CONTRIBUTORS.md` updates removed. `CONTRIBUTORS.md` file and dependencies removed. ([#4141](https://github.com/docker-mailserver/docker-mailserver/pull/4141)) -- Refactored the workflows for generating documentation previews on PRs to be more secure ([#4267](https://github.com/docker-mailserver/docker-mailserver/pull/4267), [#4264](https://github.com/docker-mailserver/docker-mailserver/pull/4264), [#4262](https://github.com/docker-mailserver/docker-mailserver/pull/4262), [#4247](https://github.com/docker-mailserver/docker-mailserver/pull/4247), [#4244](https://github.com/docker-mailserver/docker-mailserver/pull/4244)) +- Removed `CONTRIBUTORS.md`, `.all-contributorsrc`, and workflow ([#4141](https://github.com/docker-mailserver/docker-mailserver/pull/4141)) +- Refactored the workflows to be more secure for generating documentation previews on PRs ([#4267](https://github.com/docker-mailserver/docker-mailserver/pull/4267), [#4264](https://github.com/docker-mailserver/docker-mailserver/pull/4264), [#4262](https://github.com/docker-mailserver/docker-mailserver/pull/4262), [#4247](https://github.com/docker-mailserver/docker-mailserver/pull/4247), [#4244](https://github.com/docker-mailserver/docker-mailserver/pull/4244)) ## [v14.0.0](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v14.0.0) diff --git a/target/scripts/build/packages.sh b/target/scripts/build/packages.sh index 2ac40fba..1e29efa4 100644 --- a/target/scripts/build/packages.sh +++ b/target/scripts/build/packages.sh @@ -38,8 +38,8 @@ function _pre_installation_steps() { function _install_utils() { _log 'debug' 'Installing utils sourced from Github' _log 'trace' 'Installing jaq' - local JAQ_TAG='v1.6.0' - curl -sSfL "https://github.com/01mf02/jaq/releases/download/${JAQ_TAG}/jaq-${JAQ_TAG}-$(uname -m)-unknown-linux-gnu" -o /usr/bin/jaq + local JAQ_TAG='v2.0.0' + curl -sSfL "https://github.com/01mf02/jaq/releases/download/${JAQ_TAG}/jaq-$(uname -m)-unknown-linux-gnu" -o /usr/bin/jaq chmod +x /usr/bin/jaq _log 'trace' 'Installing swaks' diff --git a/target/scripts/startup/setup.d/security/rspamd.sh b/target/scripts/startup/setup.d/security/rspamd.sh index 18b9703b..085197c2 100644 --- a/target/scripts/startup/setup.d/security/rspamd.sh +++ b/target/scripts/startup/setup.d/security/rspamd.sh @@ -330,9 +330,9 @@ function __rspamd__setup_check_authenticated() { fi } -# This function performs a simple check: go through DKIM configuration files, acquire -# all private key file locations and check whether they exist and whether they can be -# accessed by Rspamd. We are not checking paths that conatain the '$' symbol. +# This function performs a simple check on the queried rspamd DKIM configuration: +# - Acquire all private key file locations and check whether they exist and can be accessed by Rspamd. +# - We are not checking paths that contain the '$' symbol. function __rspamd__check_dkim_permissions() { local KEY_FILE while read -r KEY_FILE; do From 10882f97f23a1244e39771e9987d9443810ace69 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 2 Dec 2024 21:51:52 +0100 Subject: [PATCH 179/267] chore(deps): Bump docker/build-push-action from 6.9.0 to 6.10.0 (#4278) --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 43912357..70b5d256 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -83,7 +83,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v6.9.0 + uses: docker/build-push-action@v6.10.0 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 596435b2..a6d76a11 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v6.9.0 + uses: docker/build-push-action@v6.10.0 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 9ee070a1..3f1053da 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.9.0 + uses: docker/build-push-action@v6.10.0 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 3db8002c..9cad4293 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.9.0 + uses: docker/build-push-action@v6.10.0 with: context: . tags: mailserver-testing:ci From 9f0918c335f007e60cd9bc979a37c7ff8f61d049 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Sat, 7 Dec 2024 02:02:35 +1300 Subject: [PATCH 180/267] fix(`packages.sh`): `swaks --help` (#4282) This command requires the `perl-doc` package to work. --- CHANGELOG.md | 1 + target/scripts/build/packages.sh | 9 +++++---- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b8dc1830..b0409511 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -56,6 +56,7 @@ All notable changes to this project will be documented in this file. The format - `start-mailserver.sh` removed unused `shopt -s inherit_errexit` ([#4161](https://github.com/docker-mailserver/docker-mailserver/pull/4161)) - **Rspamd:** - DKIM private key path checking is now performed only on paths that do not contain `$` ([#4201](https://github.com/docker-mailserver/docker-mailserver/pull/4201)) +- The command `swaks --help` is now functional ([#4282](https://github.com/docker-mailserver/docker-mailserver/pull/4282)) ### CI diff --git a/target/scripts/build/packages.sh b/target/scripts/build/packages.sh index 1e29efa4..787c84ac 100644 --- a/target/scripts/build/packages.sh +++ b/target/scripts/build/packages.sh @@ -43,6 +43,7 @@ function _install_utils() { chmod +x /usr/bin/jaq _log 'trace' 'Installing swaks' + apt-get "${QUIET}" install --no-install-recommends perl-doc local SWAKS_VERSION='20240103.0' local SWAKS_RELEASE="swaks-${SWAKS_VERSION}" curl -sSfL "https://github.com/jetmore/swaks/releases/download/v${SWAKS_VERSION}/${SWAKS_RELEASE}.tar.gz" | tar -xz @@ -118,7 +119,7 @@ function _install_packages() { bind9-dnsutils iputils-ping less nano ) - apt-get "${QUIET}" --no-install-recommends install \ + apt-get "${QUIET}" install --no-install-recommends \ "${ANTI_VIRUS_SPAM_PACKAGES[@]}" \ "${CODECS_PACKAGES[@]}" \ "${MISCELLANEOUS_PACKAGES[@]}" \ @@ -154,10 +155,10 @@ function _install_dovecot() { fi _log 'debug' 'Installing Dovecot' - apt-get "${QUIET}" --no-install-recommends install "${DOVECOT_PACKAGES[@]}" + apt-get "${QUIET}" install --no-install-recommends "${DOVECOT_PACKAGES[@]}" # dependency for fts_xapian - apt-get "${QUIET}" --no-install-recommends install libxapian30 + apt-get "${QUIET}" install --no-install-recommends libxapian30 } function _install_rspamd() { @@ -185,7 +186,7 @@ function _install_fail2ban() { _log 'debug' 'Installing Fail2ban' # Dependencies (https://github.com/docker-mailserver/docker-mailserver/pull/3403#discussion_r1306581431) - apt-get "${QUIET}" --no-install-recommends install python3-pyinotify python3-dnspython python3-systemd + apt-get "${QUIET}" install --no-install-recommends python3-pyinotify python3-dnspython python3-systemd gpg --keyserver "${FAIL2BAN_GPG_PUBLIC_KEY_SERVER}" --recv-keys "${FAIL2BAN_GPG_PUBLIC_KEY_ID}" 2>&1 From cd225f1250267f868b0436791ad305d0a39616f5 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Sat, 7 Dec 2024 02:04:13 +1300 Subject: [PATCH 181/267] ci(`bug_report.yml`): Drop the feedback field (#4283) This input has not provided much value to us since it's introduction, removing as redundant. --- .github/ISSUE_TEMPLATE/bug_report.yml | 7 ------- 1 file changed, 7 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index c12439ce..83982b17 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -67,10 +67,3 @@ body: - This field expects only plain text (_rendered as a fenced code block_). - You can enable debug output by setting the environment variable `LOG_LEVEL` to `debug` or `trace`. render: Text - - type: input - id: form-improvements - attributes: - label: Improvements to this form? - description: If you have criticism or general feedback about this issue form, feel free to tell us so we can enhance the experience for everyone. - validations: - required: false From 96bffd79791e1cc85fe81a5e797384236fcca2e0 Mon Sep 17 00:00:00 2001 From: Casper Date: Fri, 6 Dec 2024 21:22:37 +0100 Subject: [PATCH 182/267] chore(compile.sh): Consistent apt-get install command --- target/scripts/build/compile.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/target/scripts/build/compile.sh b/target/scripts/build/compile.sh index 299ba7c8..c34a8833 100644 --- a/target/scripts/build/compile.sh +++ b/target/scripts/build/compile.sh @@ -13,7 +13,7 @@ _log_level_is 'trace' && QUIET='-y' || QUIET='-qq' function _compile_dovecot_fts_xapian() { apt-get "${QUIET}" update - apt-get "${QUIET}" --no-install-recommends install \ + apt-get "${QUIET}" install --no-install-recommends \ automake libtool pkg-config libicu-dev libsqlite3-dev libxapian-dev make build-essential dh-make devscripts dovecot-dev local XAPIAN_VERSION='1.7.13' From a03ff8ff7c4ecef502e56aa4d79b67c32a999772 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 16 Dec 2024 14:18:51 +0100 Subject: [PATCH 183/267] chore(deps): Bump docker/setup-buildx-action from 3.7.1 to 3.8.0 (#4293) --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 70b5d256..0f1b308e 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -79,7 +79,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.7.1 + uses: docker/setup-buildx-action@v3.8.0 # NOTE: AMD64 can build within 2 minutes - name: 'Build images' diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index a6d76a11..119809d0 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -40,7 +40,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.7.1 + uses: docker/setup-buildx-action@v3.8.0 # Try get the cached build layers from a prior `generic_build.yml` job. # NOTE: Until adopting `type=gha` scoped cache exporter (in `docker/build-push-action`), diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 3f1053da..31ec6165 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -38,7 +38,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.7.1 + uses: docker/setup-buildx-action@v3.8.0 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 9cad4293..1405df73 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -37,7 +37,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.7.1 + uses: docker/setup-buildx-action@v3.8.0 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. From 02415b03dce763a717747bb737e4698ddcd7aefd Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 17 Dec 2024 08:26:14 +0100 Subject: [PATCH 184/267] chore(deps): Bump anchore/scan-action from 5.3.0 to 6.0.0 (#4292) --- .github/workflows/generic_vulnerability-scan.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 1405df73..86246f2c 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -55,7 +55,7 @@ jobs: provenance: false - name: 'Run the Anchore Grype scan action' - uses: anchore/scan-action@v5.3.0 + uses: anchore/scan-action@v6.0.0 id: scan with: image: mailserver-testing:ci From 259f2031fca6ee8b2f55386cec8c55d30176cfd2 Mon Sep 17 00:00:00 2001 From: Pooyan Khanjankhani Date: Mon, 23 Dec 2024 05:08:46 +0330 Subject: [PATCH 185/267] docs: Fix typo on usage page (#4294) Co-authored-by: Casper --- docs/content/usage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/usage.md b/docs/content/usage.md index ca4e8dfe..1c31aa17 100644 --- a/docs/content/usage.md +++ b/docs/content/usage.md @@ -48,7 +48,7 @@ We will later dig into DKIM, DMARC & SPF, but for now, these are the records tha - The **MX record** tells everyone which (DNS) name is responsible for e-mails on your domain. Because you want to keep the option of running another service on the domain name itself, you run your mail server on `mail.example.com`. This does not imply your e-mails will look like `test@mail.example.com`, the DNS name of your mail server is decoupled of the domain it serves e-mails for. - In theory, you mail server could even serve e-mails for `test@some-other-domain.com`, if the MX record for `some-other-domain.com` points to `mail.example.com`. + Your mail server could also handle emails for `test@some-other-domain.com`, if the MX record for `some-other-domain.com` points to `mail.example.com`. - The **A record** tells everyone which IP address the DNS name `mail.example.com` resolves to. - The **PTR record** is the counterpart of the A record, telling everyone what name the IP address `11.22.33.44` resolves to. From 24fb65ce7b638b342c1c9f427912756a0e53ed3e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Wojciech=20Wo=C5=BAniak?= <92504295+zespere@users.noreply.github.com> Date: Sat, 28 Dec 2024 06:00:27 +0100 Subject: [PATCH 186/267] docs: Environment - Update Dovecot docs URL (#4296) Co-authored-by: Casper --- docs/content/config/environment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/config/environment.md b/docs/content/config/environment.md index b7231b10..a16f6bcb 100644 --- a/docs/content/config/environment.md +++ b/docs/content/config/environment.md @@ -295,7 +295,7 @@ Customize the update check interval. Number + Suffix. Suffix must be 's' for sec - sdbox => (experimental) uses Dovecot high-performance mailbox format, one file contains one message - mdbox ==> (experimental) uses Dovecot high-performance mailbox format, multiple messages per file and multiple files per box -This option has been added in November 2019. Using other format than Maildir is considered as experimental in docker-mailserver and should only be used for testing purpose. For more details, please refer to [Dovecot Documentation](https://wiki2.dovecot.org/MailboxFormat). +This option has been added in November 2019. Using other format than Maildir is considered as experimental in docker-mailserver and should only be used for testing purpose. For more details, please refer to [Dovecot Documentation](https://doc.dovecot.org/admin_manual/mailbox_formats/#mailbox-formats). ##### POSTFIX_REJECT_UNKNOWN_CLIENT_HOSTNAME From e6d519b6f82f1d3a1fada4b24edaed6f64f5d3f0 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Mon, 6 Jan 2025 09:02:02 +1300 Subject: [PATCH 187/267] docs: TLS (Caddy) - Revise advice on `tls internal` (#4305) --- docs/content/config/security/ssl.md | 36 ++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 6 deletions(-) diff --git a/docs/content/config/security/ssl.md b/docs/content/config/security/ssl.md index 96f4a717..9b1fd8f7 100644 --- a/docs/content/config/security/ssl.md +++ b/docs/content/config/security/ssl.md @@ -485,6 +485,8 @@ DSM-generated letsencrypt certificates get auto-renewed every three months. !!! example + While DMS does not need a webserver to work, this workaround will provision a TLS certificate for DMS to use by adding a dummy site block to trigger cert provisioning. + ```yaml title="compose.yaml" services: # Basic Caddy service to provision certs: @@ -510,9 +512,12 @@ DSM-generated letsencrypt certificates get auto-renewed every three months. - ${CADDY_DATA_DIR}/certificates/acme-v02.api.letsencrypt.org-directory/mail.example.com/mail.example.com.key:/etc/letsencrypt/live/mail.example.com/privkey.pem ``` + An explicit entry in your `Caddyfile` config will have Caddy provision and renew a certificate for your DMS FQDN: + ```caddyfile title="Caddyfile" mail.example.com { - tls internal { + # Optionally provision RSA 2048-bit certificate instead of ECDSA P-256: + tls { key_type rsa2048 } @@ -522,10 +527,12 @@ DSM-generated letsencrypt certificates get auto-renewed every three months. } ``` - While DMS does not need a webserver to work, this workaround will provision a TLS certificate for DMS to use. + !!! info - - [`tls internal`][caddy-docs::tls-internal] will create a local self-signed cert for testing. This targets only the site-address, unlike the global `local_certs` option. - - [`key_type`][caddy-docs::key-type] can be used in the `tls` block if you need to enforce RSA as the key type for certificates provisioned. The default is currently ECDSA (P-256). + An explicit `tls` directive affects only the site-address block it's used in: + + - Use [`tls internal { ... }`][caddy-docs::tls-internal] if wanting to create a local self-signed cert, which may be useful for testing. This allows opt-in to use self-signed certs unlike the global `local_certs` option. + - [`key_type`][caddy-docs::key-type] can be used in the `tls` block if you need to enforce RSA as the key type for certificates provisioned. The default is currently ECDSA (P-256). This may improve compatibility with legacy clients. ??? example "With `caddy-docker-proxy`" @@ -558,9 +565,9 @@ DSM-generated letsencrypt certificates get auto-renewed every three months. labels: # Set your DMS FQDN here to add the site-address into the generated Caddyfile: caddy_0: mail.example.com - # Add a dummy directive is required: + # Adding a dummy directive is required: caddy_0.respond: "Hello DMS" - # Uncomment to make a proxy for Rspamd + # Uncomment to make a proxy for Rspamd: # caddy_1: rspamd.example.com # caddy_1.reverse_proxy: "{{upstreams 11334}}" ``` @@ -571,6 +578,23 @@ DSM-generated letsencrypt certificates get auto-renewed every three months. This can make the volume mounting for DMS to find the certificates non-deterministic, but you can [restrict provisioning to single service via the `acme_ca` setting][caddy::restrict-acme-provisioner]. + --- + + **NOTE:** Bind mounting a file directly instead of a directory will mount by inode. If the file is updated at renewal and this modifies the inode on the host system, then the container will still point to the old certificate. + + If this happens, consider using our manual TLS type instead: + + ```yaml title="compose.yaml" + services: + mailserver: + environment: + SSL_TYPE: manual + SSL_CERT_PATH: /srv/tls/mail.example.com/mail.example.com.crt + SSL_KEY_PATH: /srv/tls/mail.example.com/mail.example.com.key + volumes: + - ${CADDY_DATA_DIR}/certificates/acme-v02.api.letsencrypt.org-directory/mail.example.com/:/srv/tls/mail.example.com/:ro + ``` + ### Traefik [Traefik][traefik::github] is an open-source application proxy using the [ACME protocol][ietf::rfc::acme]. Traefik can request certificates for domains and subdomains, and it will take care of renewals, challenge negotiations, etc. From a302bd79e3c76c2b501628cfbcee77fee03b9511 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 15 Jan 2025 12:36:44 +1300 Subject: [PATCH 188/267] chore(deps): Bump docker/setup-qemu-action from 3.2.0 to 3.3.0 (#4309) Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 3.2.0 to 3.3.0. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](https://github.com/docker/setup-qemu-action/compare/v3.2.0...v3.3.0) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 0f1b308e..c3dce52c 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -74,7 +74,7 @@ jobs: cache-buildx- - name: 'Set up QEMU' - uses: docker/setup-qemu-action@v3.2.0 + uses: docker/setup-qemu-action@v3.3.0 with: platforms: arm64 diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 119809d0..2520aade 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -35,7 +35,7 @@ jobs: type=semver,pattern={{major}}.{{minor}}.{{patch}} - name: 'Set up QEMU' - uses: docker/setup-qemu-action@v3.2.0 + uses: docker/setup-qemu-action@v3.3.0 with: platforms: arm64 From 8df1fba96e1efb1c28317f6e162323bf781918be Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 15 Jan 2025 12:38:15 +1300 Subject: [PATCH 189/267] chore(deps): Bump docker/build-push-action from 6.10.0 to 6.11.0 (#4310) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6.10.0 to 6.11.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/v6.10.0...v6.11.0) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index c3dce52c..0561587f 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -83,7 +83,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v6.10.0 + uses: docker/build-push-action@v6.11.0 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 2520aade..835e2a81 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v6.10.0 + uses: docker/build-push-action@v6.11.0 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 31ec6165..19324f0a 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.10.0 + uses: docker/build-push-action@v6.11.0 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 86246f2c..bf700139 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.10.0 + uses: docker/build-push-action@v6.11.0 with: context: . tags: mailserver-testing:ci From 3ebca5dabaea1469f482d22912e08ec33c16cfed Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Sat, 18 Jan 2025 21:26:07 +0100 Subject: [PATCH 190/267] Revert "fix: fix incorrect link in README.md (#4184)" (#4322) This reverts commit 84180f879e5eaaa15930509d4154248c3248d765. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6e1d2735..e4dbf44a 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ A production-ready fullstack but simple containerized mail server (SMTP, IMAP, L ## :package: Included Services - [Postfix](http://www.postfix.org) with SMTP or LDAP authentication and support for [extension delimiters](https://docker-mailserver.github.io/docker-mailserver/latest/config/account-management/overview/#aliases) -- [Dovecot](https://www.dovecot.org) with SASL, IMAP, POP3, LDAP, [basic Sieve support](https://docker-mailserver.github.io/docker-mailserver/latest/config/advanced/mail-sieve) and [quotas](https://docker-mailserver.github.io/docker-mailserver/latest/config/user-management/#quotas) +- [Dovecot](https://www.dovecot.org) with SASL, IMAP, POP3, LDAP, [basic Sieve support](https://docker-mailserver.github.io/docker-mailserver/latest/config/advanced/mail-sieve) and [quotas](https://docker-mailserver.github.io/docker-mailserver/latest/config/account-management/overview/#quotas) - [Rspamd](https://rspamd.com/) - [Amavis](https://www.amavis.org/) - [SpamAssassin](http://spamassassin.apache.org/) supporting custom rules From f0daa1c8abff606bd3fa6f5212b7e77c1e7b38a2 Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Sun, 19 Jan 2025 01:27:07 +0100 Subject: [PATCH 191/267] chore: remove `VERSION` file (#4321) --- .dockerignore | 1 - .github/workflows/generic_build.yml | 7 ++----- CHANGELOG.md | 1 + VERSION | 1 - 4 files changed, 3 insertions(+), 7 deletions(-) delete mode 100644 VERSION diff --git a/.dockerignore b/.dockerignore index b6eef669..50365994 100644 --- a/.dockerignore +++ b/.dockerignore @@ -1,3 +1,2 @@ * !target -!VERSION diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 0561587f..b81c94de 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -32,7 +32,7 @@ jobs: with: submodules: recursive - # Can potentially be replaced by: `${{ hashFiles('target/**', 'Dockerfile', 'VERSION') }}` + # Can potentially be replaced by: `${{ hashFiles('target/**', 'Dockerfile') }}` # Must not be affected by file metadata changes and have a consistent sort order: # https://docs.github.com/en/actions/learn-github-actions/expressions#hashfiles # Keying by the relevant build context is more re-usable than a commit SHA. @@ -40,10 +40,7 @@ jobs: id: derive-image-cache-key shell: bash run: | - ADDITIONAL_FILES=( - 'Dockerfile' - 'VERSION' - ) + ADDITIONAL_FILES=( 'Dockerfile' ) # Recursively collect file paths from `target/` and pipe a list of # checksums to be sorted (by hash value) and finally generate a checksum diff --git a/CHANGELOG.md b/CHANGELOG.md index b0409511..8ca4bdcb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ All notable changes to this project will be documented in this file. The format - DMS v14 mistakenly relocated the _getmail state directory_ to the _DMS Config Volume_ as a `getmail/` subdirectory. - This has been corrected to `/var/lib/getmail` (_if you have mounted a DMS State Volume to `/var/mail-state`, `/var/lib/getmail` will be symlinked to `/var/mail-state/lib-getmail`_). - To preserve this state when upgrading to DMS v15, **you must manually migrate `getmail/` from the _DMS Config Volume_ to `lib-getmail/` in the _DMS State Volume_.** +- **removed `VERSION`** file that was used for checking version updates ([#3677](https://github.com/docker-mailserver/docker-mailserver/issues/3677),[#4321](https://github.com/docker-mailserver/docker-mailserver/pull/4321)) ### Security diff --git a/VERSION b/VERSION deleted file mode 100644 index 4b964e96..00000000 --- a/VERSION +++ /dev/null @@ -1 +0,0 @@ -14.0.0 From 2d56210c522410a29d15d0c890667fad2fee1c75 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 22 Jan 2025 21:42:25 +0100 Subject: [PATCH 192/267] chore(deps): Bump docker/build-push-action from 6.11.0 to 6.12.0 (#4324) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6.11.0 to 6.12.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/v6.11.0...v6.12.0) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index b81c94de..5c819637 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -80,7 +80,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v6.11.0 + uses: docker/build-push-action@v6.12.0 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 835e2a81..edd642fb 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v6.11.0 + uses: docker/build-push-action@v6.12.0 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 19324f0a..190f1f89 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.11.0 + uses: docker/build-push-action@v6.12.0 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index bf700139..a74c9827 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.11.0 + uses: docker/build-push-action@v6.12.0 with: context: . tags: mailserver-testing:ci From 7cb2fc788f67a90319e2d7ecdd5422bc9cd33afc Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 29 Jan 2025 10:56:37 +0100 Subject: [PATCH 193/267] chore(deps): Bump anchore/scan-action from 6.0.0 to 6.1.0 (#4332) Bumps [anchore/scan-action](https://github.com/anchore/scan-action) from 6.0.0 to 6.1.0. - [Release notes](https://github.com/anchore/scan-action/releases) - [Changelog](https://github.com/anchore/scan-action/blob/main/RELEASE.md) - [Commits](https://github.com/anchore/scan-action/compare/v6.0.0...v6.1.0) --- updated-dependencies: - dependency-name: anchore/scan-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/generic_vulnerability-scan.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index a74c9827..2e73e0f2 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -55,7 +55,7 @@ jobs: provenance: false - name: 'Run the Anchore Grype scan action' - uses: anchore/scan-action@v6.0.0 + uses: anchore/scan-action@v6.1.0 id: scan with: image: mailserver-testing:ci From 4d8a56072a88cbacea373be88aa1d805e1eacc59 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 29 Jan 2025 11:02:25 +0100 Subject: [PATCH 194/267] chore(deps): Bump docker/build-push-action from 6.12.0 to 6.13.0 (#4331) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6.12.0 to 6.13.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/v6.12.0...v6.13.0) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 5c819637..6000f5ff 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -80,7 +80,7 @@ jobs: # NOTE: AMD64 can build within 2 minutes - name: 'Build images' - uses: docker/build-push-action@v6.12.0 + uses: docker/build-push-action@v6.13.0 with: context: . # Build at least the AMD64 image (which runs against the test suite). diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index edd642fb..42cde7ea 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -67,7 +67,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} - name: 'Build and publish images' - uses: docker/build-push-action@v6.12.0 + uses: docker/build-push-action@v6.13.0 with: context: . build-args: | diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index 190f1f89..f28dbc9b 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -43,7 +43,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.12.0 + uses: docker/build-push-action@v6.13.0 with: context: . tags: mailserver-testing:ci diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 2e73e0f2..3b20b2ed 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -42,7 +42,7 @@ jobs: # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. - name: 'Build AMD64 image from cache' - uses: docker/build-push-action@v6.12.0 + uses: docker/build-push-action@v6.13.0 with: context: . tags: mailserver-testing:ci From 3faa40bfb5442e4fd6638aa5878fe807feb7e94f Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Sun, 2 Feb 2025 22:35:50 +0100 Subject: [PATCH 195/267] docs: add ARC example to Rspamd documentation (#4328) Signed-off-by: georglauterbach <44545919+georglauterbach@users.noreply.github.com> --- docs/content/config/security/rspamd.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/content/config/security/rspamd.md b/docs/content/config/security/rspamd.md index 8705262c..c49baea6 100644 --- a/docs/content/config/security/rspamd.md +++ b/docs/content/config/security/rspamd.md @@ -247,6 +247,24 @@ Rspamd is running, but you want or need to adjust it? First, create a file named There is a dedicated [section for setting up DKIM with Rspamd in our documentation][docs::dkim-with-rspamd]. +### ARC (Authenticated Received Chain) + +ARC is not set up by default, but you can easily enable it by adding a file called `arc.conf` to `docker-data/dms/config/rspamd/override.d/`. ARC can use DKIM keys that you should have already created. The configuration file could then contain the following: + +```conf +sign_local = true; +sign_authenticated = true; + +domain { + { + # Change the path here to your actual private key + path = "/tmp/docker-mailserver/rspamd/dkim/rsa-2048-mail-.private.txt"; + # Changhe the selected if you chose a non-default one + selector = "mail"; + } +} +``` + ### _Abusix_ Integration This subsection provides information about the integration of [Abusix][abusix-web], "a set of blocklists that work as an additional email security layer for your existing mail environment". The setup is straight-forward and well documented: From 0e61f170fd552077e797483310a7d61500bbe020 Mon Sep 17 00:00:00 2001 From: Zlatibor Veljkovic Date: Tue, 4 Feb 2025 00:45:43 +0100 Subject: [PATCH 196/267] docs: `bind-smtp-network-interface.md` - Add bridge network config advice (#4330) Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> --- .../use-cases/bind-smtp-network-interface.md | 50 ++++++++++++++++--- 1 file changed, 44 insertions(+), 6 deletions(-) diff --git a/docs/content/examples/use-cases/bind-smtp-network-interface.md b/docs/content/examples/use-cases/bind-smtp-network-interface.md index b12e21de..0da33b19 100644 --- a/docs/content/examples/use-cases/bind-smtp-network-interface.md +++ b/docs/content/examples/use-cases/bind-smtp-network-interface.md @@ -21,10 +21,6 @@ This can be configured by [overriding the default Postfix configurations][docs:: In `postfix-main.cf` you'll have to set the [`smtp_bind_address`][postfix-docs::smtp-bind-address-ipv4] and [`smtp_bind_address6`][postfix-docs::smtp-bind-address-ipv6] to the respective IP-address on the server you want to use. -[docs::overrides-postfix]: ../../config/advanced/override-defaults/postfix.md -[postfix-docs::smtp-bind-address-ipv4]: https://www.postfix.org/postconf.5.html#smtp_bind_address -[postfix-docs::smtp-bind-address-ipv6]: https://www.postfix.org/postconf.5.html#smtp_bind_address6 - !!! example === "Contributed solution" @@ -55,14 +51,56 @@ to the respective IP-address on the server you want to use. ``` If that avoids the concern with `smtp-amavis`, you may still need to additionally override for the [`relay` transport][gh-src::postfix-master-cf::relay-transport] as well if you have configured DMS to relay mail. + + === "Bridged Networks" + + When your DMS container is using a bridge network, you'll instead need to restrict which IP address inbound and outbound traffic is routed through via the bridged interface. + + For **inbound** traffic, you may configure this at whatever scope is most appropriate for you: + + - **Daemon:** Change the [default bind address][inbound-ip::docker-docs::daemon] configured in `/etc/docker/daemon.json` (default `0.0.0.0`) + - **Network:** Assign the [`host_binding_ipv4` bridge driver option][inbound-ip::docker-docs::network] as shown in the below `compose.yaml` snippet. + - **Container:** Provide an explicit host IP address when [publishing a port][inbound-ip::docker-docs::container]. + + For **outbound** traffic, the bridge network will use the default route. You can change this by either: + + - [Manually routing networks][outbound-ip::route-manually] on the host. + - Use the [`host_ipv4` driver option][outbind-ip::host-ipv4] for Docker networks to force the SNAT (source IP) that the bridged network will route outbound traffic through. + - This IP address must belong to a network interface to be routed through it. + - IPv6 support via `host_ipv6` [requires at least Docker v25][outbind-ip::host-ipv6]. + + --- + + Here is a `compose.yaml` snippet that applies the inbound + outbound settings to the default bridge network Docker Compose creates (_if it already exists, you will need to ensure it's re-created to apply the updated settings_): + + ```yaml title="compose.yaml" + networks: + default: + driver_opts: + # Inbound IP (sets the host IP that published ports receive traffic from): + com.docker.network.bridge.host_binding_ipv4: 198.51.100.42 + # Outbound IP (sets the host IP that external hosts will receive connections from): + com.docker.network.host_ipv4: 198.51.100.42 + ``` !!! note "IP addresses for documentation" - IP addresses shown in above examples are placeholders, they are IP addresses reserved for documentation by IANA (_[RFC-5737 (IPv4)][rfc-5737] and [RFC-3849 (IPv6)][rfc-3849]_). Replace them with the IP addresses you want DMS to send mail through. - + IP addresses shown in above examples (`198.51.100.42` + `2001:DB8::42`) are placeholders, they are IP addresses reserved for documentation by IANA (_[RFC-5737 (IPv4)][rfc-5737] and [RFC-3849 (IPv6)][rfc-3849]_). Replace them with the IP addresses you want DMS to send mail through. + +[docs::overrides-postfix]: ../../config/advanced/override-defaults/postfix.md +[postfix-docs::smtp-bind-address-ipv4]: https://www.postfix.org/postconf.5.html#smtp_bind_address +[postfix-docs::smtp-bind-address-ipv6]: https://www.postfix.org/postconf.5.html#smtp_bind_address6 + [rfc-5737]: https://datatracker.ietf.org/doc/html/rfc5737 [rfc-3849]: https://datatracker.ietf.org/doc/html/rfc3849 [gh-pr::3465::comment-restrictions]: https://github.com/docker-mailserver/docker-mailserver/pull/3465#discussion_r1458114528 [gh-pr::3465::alternative-solution]: https://github.com/docker-mailserver/docker-mailserver/pull/3465#issuecomment-1678107233 [gh-src::postfix-master-cf::relay-transport]: https://github.com/docker-mailserver/docker-mailserver/blob/9cdbef2b369fb4fb0f1b4e534da8703daf92abc9/target/postfix/master.cf#L65 + +[inbound-ip::docker-docs::daemon]: https://docs.docker.com/reference/cli/dockerd/#default-network-options +[inbound-ip::docker-docs::network]: https://docs.docker.com/engine/network/drivers/bridge/#default-host-binding-address +[inbound-ip::docker-docs::container]: https://docs.docker.com/reference/compose-file/services/#ports +[outbound-ip::route-manually]: https://github.com/moby/moby/issues/30053#issuecomment-1077041045 +[outbind-ip::host-ipv4]: https://github.com/moby/libnetwork/pull/2454 +[outbind-ip::host-ipv6]: https://github.com/moby/moby/issues/46469 From e116920f4df64cd55c9cd91e8783180b3fa2aae0 Mon Sep 17 00:00:00 2001 From: Casper Date: Wed, 5 Feb 2025 23:10:57 +0100 Subject: [PATCH 197/267] Add missing "setup debug getmail" command and documentation (#4346) --- CHANGELOG.md | 1 + docs/content/config/advanced/mail-getmail.md | 8 ++++++++ target/bin/setup | 2 ++ 3 files changed, 11 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8ca4bdcb..781ef3e4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ All notable changes to this project will be documented in this file. The format - DMS v14 mistakenly relocated the _getmail state directory_ to the _DMS Config Volume_ as a `getmail/` subdirectory. - This has been corrected to `/var/lib/getmail` (_if you have mounted a DMS State Volume to `/var/mail-state`, `/var/lib/getmail` will be symlinked to `/var/mail-state/lib-getmail`_). - To preserve this state when upgrading to DMS v15, **you must manually migrate `getmail/` from the _DMS Config Volume_ to `lib-getmail/` in the _DMS State Volume_.** + - Added missing `debug getmail` subcommand to `setup` ([#4346](https://github.com/docker-mailserver/docker-mailserver/pull/4346)) - **removed `VERSION`** file that was used for checking version updates ([#3677](https://github.com/docker-mailserver/docker-mailserver/issues/3677),[#4321](https://github.com/docker-mailserver/docker-mailserver/pull/4321)) ### Security diff --git a/docs/content/config/advanced/mail-getmail.md b/docs/content/config/advanced/mail-getmail.md index d423b4f0..f4ce7e63 100644 --- a/docs/content/config/advanced/mail-getmail.md +++ b/docs/content/config/advanced/mail-getmail.md @@ -108,3 +108,11 @@ It is possible to utilize the `getmail-gmail-xoauth-tokens` helper to provide au [getmail-docs]: https://getmail6.org/configuration.html [getmail-docs-xoauth-12]: https://github.com/getmail6/getmail6/blob/1f95606156231f1e074ba62a9baa64f892a92ef8/docs/getmailrc-examples#L286 [getmail-docs-xoauth-13]: https://github.com/getmail6/getmail6/blob/1f95606156231f1e074ba62a9baa64f892a92ef8/docs/getmailrc-examples#L351 + +## Debugging + +To debug your `getmail` configurations, run this `setup debug` command: + +```sh +docker exec -it dms-container-name setup debug getmail +``` diff --git a/target/bin/setup b/target/bin/setup index ac0d6328..c05f59a7 100755 --- a/target/bin/setup +++ b/target/bin/setup @@ -64,6 +64,7 @@ ${RED}[${ORANGE}SUB${RED}]${ORANGE}COMMANDS${RESET} ${LBLUE}COMMAND${RESET} debug ${RED}:=${RESET} setup debug ${CYAN}fetchmail${RESET} + setup debug ${CYAN}getmail${RESET} setup debug ${CYAN}login${RESET} setup debug ${CYAN}show-mail-logs${RESET} @@ -150,6 +151,7 @@ function _main() { ( debug ) case ${2:-} in ( fetchmail ) debug-fetchmail ;; + ( getmail ) debug-getmail ;; ( show-mail-logs ) cat /var/log/mail/mail.log ;; ( login ) shift 2 From 85793988d6d81ec21e19ccab96070fcd1f3d6208 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Fri, 7 Feb 2025 10:10:37 +1300 Subject: [PATCH 198/267] chore: `demo-setups/relay-compose.yaml` should use network alias (#4347) --- demo-setups/relay-compose.yaml | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/demo-setups/relay-compose.yaml b/demo-setups/relay-compose.yaml index f4df2de0..cb4f0846 100644 --- a/demo-setups/relay-compose.yaml +++ b/demo-setups/relay-compose.yaml @@ -41,17 +41,13 @@ services: hostname: smtp.relay-service.test environment: # WORKAROUND: Bypass security checks from the mail-client (dms-sender container) - # (avoids needing valid DNS for this example) + # (avoids needing expected DNS records to run this example) - PERMIT_DOCKER=connected-networks # TLS is required when relaying to dms-relay via ports 587 / 465 # (dms-relay will then relay the mail to dms-destination over port 25) - SSL_TYPE=manual - SSL_KEY_PATH=/tmp/tls/key.pem - SSL_CERT_PATH=/tmp/tls/cert.pem - # WORKAROUND: `links` is required due to lack of properly configured DNS. - # (resolves destination.test to the IP of the dms-destination container) - links: - - "dms-destination:destination.test" configs: - source: dms-accounts-relay target: /tmp/docker-mailserver/postfix-accounts.cf @@ -64,8 +60,14 @@ services: dms-destination: image: mailserver/docker-mailserver:latest # :14.0 hostname: mail.destination.test - # Same workaround for purposes of the example, with the target recipient provisioned to accept mail + # WORKAROUND: dms-relay must be able to resolve DNS for `@destination.test` to the IP of this container: + # Normally a MX record would direct mail to the MTA (eg: `mail.destination.test`) + networks: + default: + aliases: + - destination.test environment: + # WORKAROUND: Same workaround as needed for dms-relay - PERMIT_DOCKER=connected-networks configs: - source: dms-accounts-destination @@ -75,6 +77,13 @@ services: # NOTE: This feature requires Docker Compose v2.23.1 (Nov 2023) or newer: # https://github.com/compose-spec/compose-spec/pull/446 configs: + # `postfix-main.cf`, a single line change to make all outbound SMTP connections over implicit TLS instead of the default explicit TLS (StartTLS). + # NOTE: If you need to only selectively relay mail, you would need to instead adjust this on the relay service in `/etc/postfix/master.cf`, + # However DMS presently modifies this when using the DMS Relay Host feature support, which may override `postfix-master.cf` or `user-patches.sh` due to `check-for-changes.sh`. + dms-main: + content: | + smtp_tls_wrappermode=yes + # DMS expects an account to be configured to run, this example provides accounts already created. # Login credentials: # user: "john.doe@example.test" password: "secret" @@ -94,13 +103,6 @@ configs: content: | jane.doe@destination.test|{SHA512-CRYPT}$$6$$o65y1ZXC4ooOPLwZ$$7TF1nYowEtNJpH6BwJBgdj2pPAxaCvhIKQA6ww5zdHm/AA7aemY9eoHC91DOgYNaKj1HLxSeWNDdvrp6mbtUY. - # This is `postfix-main.cf`, single line change to make all outbound SMTP connections over port 465 instead of 25 (default) - # If you selectively relay mail, you would need to adjust this on the relay service in `/etc/postfix/master.cf`, - # However DMS presently modifies this when using the DMS Relay Host feature support, which may override `postfix-master.cf` or `user-patches.sh` due to `check-for-changes.sh`. - dms-main: - content: | - smtp_tls_wrappermode=yes - # TLS files: # - Use an ECDSA cert that's been signed by a self-signed CA for TLS cert verification. # - This cert is only valid for mail.example.test, mail.destination.test, smtp.relay-service.test From 59a379aed781d8e66b9388491aed245a9f5d9af1 Mon Sep 17 00:00:00 2001 From: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> Date: Sat, 8 Feb 2025 22:23:06 +0100 Subject: [PATCH 199/267] scripts: restructure container restart behavior (#4323) Signed-off-by: georglauterbach <44545919+georglauterbach@users.noreply.github.com> Signed-off-by: dependabot[bot] Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- CHANGELOG.md | 9 +- target/scripts/start-mailserver.sh | 54 ++--- target/scripts/startup/check-stack.sh | 18 -- target/scripts/startup/setup-stack.sh | 4 +- target/scripts/startup/setup.d/mail_state.sh | 242 ++++++++++--------- target/scripts/startup/variables-stack.sh | 54 ++++- 6 files changed, 202 insertions(+), 179 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 781ef3e4..379992c3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,8 +19,6 @@ All notable changes to this project will be documented in this file. The format - DMS v14 mistakenly relocated the _getmail state directory_ to the _DMS Config Volume_ as a `getmail/` subdirectory. - This has been corrected to `/var/lib/getmail` (_if you have mounted a DMS State Volume to `/var/mail-state`, `/var/lib/getmail` will be symlinked to `/var/mail-state/lib-getmail`_). - To preserve this state when upgrading to DMS v15, **you must manually migrate `getmail/` from the _DMS Config Volume_ to `lib-getmail/` in the _DMS State Volume_.** - - Added missing `debug getmail` subcommand to `setup` ([#4346](https://github.com/docker-mailserver/docker-mailserver/pull/4346)) -- **removed `VERSION`** file that was used for checking version updates ([#3677](https://github.com/docker-mailserver/docker-mailserver/issues/3677),[#4321](https://github.com/docker-mailserver/docker-mailserver/pull/4321)) ### Security @@ -31,9 +29,11 @@ All notable changes to this project will be documented in this file. The format - **Internal:** - Add password confirmation to several `setup` CLI subcommands ([#4072](https://github.com/docker-mailserver/docker-mailserver/pull/4072)) + - Added a `debug getmail` subcommand to `setup` ([#4346](https://github.com/docker-mailserver/docker-mailserver/pull/4346)) ### Updates +- **Removed `VERSION` file** from the repo that releases of DMS prior to v13 (Nov 2023) would check to detect new releases ([#3677](https://github.com/docker-mailserver/docker-mailserver/issues/3677), [#4321](https://github.com/docker-mailserver/docker-mailserver/pull/4321)) - **Fail2ban:** - Updated to version [`1.1.0`](https://github.com/fail2ban/fail2ban/releases/tag/1.1.0) ([#4045](https://github.com/docker-mailserver/docker-mailserver/pull/4045)) - **Documentation:** @@ -54,11 +54,12 @@ All notable changes to this project will be documented in this file. The format - Correctly apply a compatibility fix for OAuth2 introduced in DMS v13.3.1 which had not been applied to the actual LDAP config changes ([#4175](https://github.com/docker-mailserver/docker-mailserver/pull/4175)) - **Internal:** - The main `mail.log` (_which is piped to stdout via `tail`_) now correctly begins from the first log line of the active container run. Previously some daemon logs and potential warnings/errors were omitted ([#4146](https://github.com/docker-mailserver/docker-mailserver/pull/4146)) - - Fixed a regression introduced in v14 where `postfix-main.cf` appended `stderr` output into `/etc/postfix/main.cf`, causing Postfix startup to fail ([#4147](https://github.com/docker-mailserver/docker-mailserver/pull/4147)) - `start-mailserver.sh` removed unused `shopt -s inherit_errexit` ([#4161](https://github.com/docker-mailserver/docker-mailserver/pull/4161)) + - Fixed a regression introduced in v14 where `postfix-main.cf` appended `stderr` output into `/etc/postfix/main.cf`, causing Postfix startup to fail ([#4147](https://github.com/docker-mailserver/docker-mailserver/pull/4147)) + - Fixed a regression introduced in v14 to better support running `start-mailserver.sh` with container restarts, which now only skip calling `_setup()` ([#4323](https://github.com/docker-mailserver/docker-mailserver/pull/4323#issuecomment-2629559254)) + - The command `swaks --help` is now functional ([#4282](https://github.com/docker-mailserver/docker-mailserver/pull/4282)) - **Rspamd:** - DKIM private key path checking is now performed only on paths that do not contain `$` ([#4201](https://github.com/docker-mailserver/docker-mailserver/pull/4201)) -- The command `swaks --help` is now functional ([#4282](https://github.com/docker-mailserver/docker-mailserver/pull/4282)) ### CI diff --git a/target/scripts/start-mailserver.sh b/target/scripts/start-mailserver.sh index f6ceadf5..e820f746 100755 --- a/target/scripts/start-mailserver.sh +++ b/target/scripts/start-mailserver.sh @@ -38,7 +38,6 @@ function _register_functions() { # ? >> Checks _register_check_function '_check_hostname' - _register_check_function '_check_log_level' _register_check_function '_check_spam_prefix' # ? >> Setup @@ -63,7 +62,6 @@ function _register_functions() { ;; ( 'LDAP' ) - _environment_variables_ldap _register_setup_function '_setup_ldap' ;; @@ -76,15 +74,8 @@ function _register_functions() { ;; esac - if [[ ${ENABLE_OAUTH2} -eq 1 ]]; then - _environment_variables_oauth2 - _register_setup_function '_setup_oauth2' - fi - - if [[ ${ENABLE_SASLAUTHD} -eq 1 ]]; then - _environment_variables_saslauthd - _register_setup_function '_setup_saslauthd' - fi + [[ ${ENABLE_OAUTH2} -eq 1 ]] && _register_setup_function '_setup_oauth2' + [[ ${ENABLE_SASLAUTHD} -eq 1 ]] && _register_setup_function '_setup_saslauthd' _register_setup_function '_setup_dovecot_inet_protocols' @@ -122,20 +113,23 @@ function _register_functions() { _register_setup_function '_setup_logwatch' _register_setup_function '_setup_save_states' - _register_setup_function '_setup_apply_fixes_after_configuration' - _register_setup_function '_environment_variables_export' + _register_setup_function '_setup_adjust_state_permissions' if [[ ${ENABLE_MTA_STS} -eq 1 ]]; then _register_setup_function '_setup_mta_sts' _register_start_daemon '_start_daemon_mta_sts_daemon' fi + # ! The following functions must be executed after all other setup functions + _register_setup_function '_setup_directory_and_file_permissions' + _register_setup_function '_setup_run_user_patches' + # ? >> Daemons _register_start_daemon '_start_daemon_cron' _register_start_daemon '_start_daemon_rsyslog' - [[ ${SMTP_ONLY} -ne 1 ]] && _register_start_daemon '_start_daemon_dovecot' + [[ ${SMTP_ONLY} -ne 1 ]] && _register_start_daemon '_start_daemon_dovecot' if [[ ${ENABLE_UPDATE_CHECK} -eq 1 ]]; then if [[ ${DMS_RELEASE} != 'edge' ]]; then @@ -174,26 +168,24 @@ function _register_functions() { # ? >> Executing all stacks / actual start of DMS # ------------------------------------------------------------ +_early_supervisor_setup +_early_variables_setup + +_log 'info' "Welcome to docker-mailserver ${DMS_RELEASE}" + +_register_functions +_check + # Ensure DMS only adjusts config files for a new container. # Container restarts should skip as they retain the modified config. -if [[ ! -f /CONTAINER_START ]]; then - _early_supervisor_setup - _early_variables_setup - - _log 'info' "Welcome to docker-mailserver ${DMS_RELEASE}" - - _register_functions - _check - _setup - _run_user_patches +if [[ -f /CONTAINER_START ]]; then + _log 'info' 'Container was restarted. Skipping most setup routines.' + # We cannot skip all setup routines because some need to run _after_ + # the initial setup (and hence, they cannot be moved to the check stack). + _setup_directory_and_file_permissions + _setup_adjust_state_permissions else - # container was restarted - _early_variables_setup - - _log 'info' 'Container was restarted. Skipping setup routines.' - _log 'info' "Welcome to docker-mailserver ${DMS_RELEASE}" - - _register_functions + _setup fi # marker to check if container was restarted diff --git a/target/scripts/startup/check-stack.sh b/target/scripts/startup/check-stack.sh index 766fcccf..52d83ad4 100644 --- a/target/scripts/startup/check-stack.sh +++ b/target/scripts/startup/check-stack.sh @@ -26,24 +26,6 @@ function _check_hostname() { fi } -function _check_log_level() { - if [[ ${LOG_LEVEL} == 'trace' ]] \ - || [[ ${LOG_LEVEL} == 'debug' ]] \ - || [[ ${LOG_LEVEL} == 'info' ]] \ - || [[ ${LOG_LEVEL} == 'warn' ]] \ - || [[ ${LOG_LEVEL} == 'error' ]] - then - return 0 - else - local DEFAULT_LOG_LEVEL='info' - _log 'warn' "Log level '${LOG_LEVEL}' is invalid (falling back to default '${DEFAULT_LOG_LEVEL}')" - - # shellcheck disable=SC2034 - VARS[LOG_LEVEL]="${DEFAULT_LOG_LEVEL}" - LOG_LEVEL="${DEFAULT_LOG_LEVEL}" - fi -} - function _check_spam_prefix() { # This check should be independent of ENABLE_POP3 and ENABLE_IMAP if [[ ${MOVE_SPAM_TO_JUNK} -eq 0 ]] \ diff --git a/target/scripts/startup/setup-stack.sh b/target/scripts/startup/setup-stack.sh index 8c8e6461..789dc0c7 100644 --- a/target/scripts/startup/setup-stack.sh +++ b/target/scripts/startup/setup-stack.sh @@ -82,7 +82,7 @@ function _setup_timezone() { fi } -function _setup_apply_fixes_after_configuration() { +function _setup_directory_and_file_permissions() { _log 'trace' 'Removing leftover PID files from a stop/start' find /var/run/ -not -name 'supervisord.pid' -name '*.pid' -delete touch /dev/shm/supervisor.sock @@ -103,7 +103,7 @@ function _setup_apply_fixes_after_configuration() { fi } -function _run_user_patches() { +function _setup_run_user_patches() { local USER_PATCHES='/tmp/docker-mailserver/user-patches.sh' if [[ -f ${USER_PATCHES} ]]; then diff --git a/target/scripts/startup/setup.d/mail_state.sh b/target/scripts/startup/setup.d/mail_state.sh index e819c138..931e3c05 100644 --- a/target/scripts/startup/setup.d/mail_state.sh +++ b/target/scripts/startup/setup.d/mail_state.sh @@ -3,123 +3,129 @@ # Consolidate all states into a single directory # (/var/mail-state) to allow persistence using docker volumes function _setup_save_states() { - local DEST DESTDIR STATEDIR SERVICEDIR SERVICEDIRS SERVICEFILE SERVICEFILES - - STATEDIR='/var/mail-state' - - if [[ -d ${STATEDIR} ]]; then - _log 'debug' "Consolidating all state onto ${STATEDIR}" - - # Always enabled features: - SERVICEDIRS=( - lib/logrotate - lib/postfix - spool/postfix - ) - - # Only consolidate state for services that are enabled - # Notably avoids copying over 200MB for the ClamAV database - [[ ${ENABLE_AMAVIS} -eq 1 ]] && SERVICEDIRS+=('lib/amavis') - [[ ${ENABLE_CLAMAV} -eq 1 ]] && SERVICEDIRS+=('lib/clamav') - [[ ${ENABLE_FAIL2BAN} -eq 1 ]] && SERVICEDIRS+=('lib/fail2ban') - [[ ${ENABLE_FETCHMAIL} -eq 1 ]] && SERVICEDIRS+=('lib/fetchmail') - [[ ${ENABLE_GETMAIL} -eq 1 ]] && SERVICEDIRS+=('lib/getmail') - [[ ${ENABLE_MTA_STS} -eq 1 ]] && SERVICEDIRS+=('lib/mta-sts') - [[ ${ENABLE_POSTGREY} -eq 1 ]] && SERVICEDIRS+=('lib/postgrey') - [[ ${ENABLE_RSPAMD} -eq 1 ]] && SERVICEDIRS+=('lib/rspamd') - [[ ${ENABLE_RSPAMD_REDIS} -eq 1 ]] && SERVICEDIRS+=('lib/redis') - [[ ${ENABLE_SPAMASSASSIN} -eq 1 ]] && SERVICEDIRS+=('lib/spamassassin') - [[ ${ENABLE_SRS} -eq 1 ]] && SERVICEDIRS+=('lib/postsrsd') - [[ ${SMTP_ONLY} -ne 1 ]] && SERVICEDIRS+=('lib/dovecot') - - # Single service files - [[ ${ENABLE_SRS} -eq 1 ]] && SERVICEFILES+=('/etc/postsrsd.secret') - - for SERVICEFILE in "${SERVICEFILES[@]}"; do - DEST="${STATEDIR}/${SERVICEFILE}" - DESTDIR="${DEST%/*}" - - mkdir -p "${DESTDIR}" - if [[ -f ${DEST} ]]; then - _log 'trace' "Destination ${DEST} exists, linking ${SERVICEFILE} to it" - # Original content from image no longer relevant, remove it: - rm -f "${SERVICEFILE}" - elif [[ -f "${SERVICEFILE}" ]]; then - _log 'trace' "Moving ${SERVICEFILE} to ${DEST}" - # Empty volume was mounted, or new content from enabling a feature ENV: - mv "${SERVICEFILE}" "${DEST}" - # Apply SELinux security context to match the state directory, so access - # is not restricted to the current running container: - chcon -R --reference="${STATEDIR}" "${DEST}" 2>/dev/null || true - fi - - # Symlink the original file in the container ($SERVICEFILE) to be - # sourced from assocaiated path in /var/mail-state/ ($DEST): - ln -s "${DEST}" "${SERVICEFILE}" - done - - for SERVICEDIR in "${SERVICEDIRS[@]}"; do - DEST="${STATEDIR}/${SERVICEDIR//\//-}" - SERVICEDIR="/var/${SERVICEDIR}" - - # If relevant content is found in /var/mail-state (presumably a volume mount), - # use it instead. Otherwise copy over any missing directories checked. - if [[ -d ${DEST} ]]; then - _log 'trace' "Destination ${DEST} exists, linking ${SERVICEDIR} to it" - # Original content from image no longer relevant, remove it: - rm -rf "${SERVICEDIR}" - elif [[ -d ${SERVICEDIR} ]]; then - _log 'trace' "Moving contents of ${SERVICEDIR} to ${DEST}" - # An empty volume was mounted, or new content dir now exists from enabling a feature ENV: - mv "${SERVICEDIR}" "${DEST}" - # Apply SELinux security context to match the state directory, so access - # is not restricted to the current running container: - chcon -R --reference="${STATEDIR}" "${DEST}" 2>/dev/null || true - else - _log 'error' "${SERVICEDIR} should exist but is missing" - fi - - # Symlink the original path in the container ($SERVICEDIR) to be - # sourced from assocaiated path in /var/mail-state/ ($DEST): - ln -s "${DEST}" "${SERVICEDIR}" - done - - # This ensures the user and group of the files from the external mount have their - # numeric ID values in sync. New releases where the installed packages order changes - # can change the values in the Docker image, causing an ownership mismatch. - # NOTE: More details about users and groups added during image builds are documented here: - # https://github.com/docker-mailserver/docker-mailserver/pull/3011#issuecomment-1399120252 - _log 'trace' "Fixing ${STATEDIR}/* permissions" - [[ ${ENABLE_AMAVIS} -eq 1 ]] && chown -R amavis:amavis "${STATEDIR}/lib-amavis" - [[ ${ENABLE_CLAMAV} -eq 1 ]] && chown -R clamav:clamav "${STATEDIR}/lib-clamav" - [[ ${ENABLE_FETCHMAIL} -eq 1 ]] && chown -R fetchmail:nogroup "${STATEDIR}/lib-fetchmail" - [[ ${ENABLE_MTA_STS} -eq 1 ]] && chown -R _mta-sts:_mta-sts "${STATEDIR}/lib-mta-sts" - [[ ${ENABLE_POSTGREY} -eq 1 ]] && chown -R postgrey:postgrey "${STATEDIR}/lib-postgrey" - [[ ${ENABLE_RSPAMD} -eq 1 ]] && chown -R _rspamd:_rspamd "${STATEDIR}/lib-rspamd" - [[ ${ENABLE_RSPAMD_REDIS} -eq 1 ]] && chown -R redis:redis "${STATEDIR}/lib-redis" - [[ ${ENABLE_SPAMASSASSIN} -eq 1 ]] && chown -R debian-spamd:debian-spamd "${STATEDIR}/lib-spamassassin" - - chown -R root:root "${STATEDIR}/lib-logrotate" - chown -R postfix:postfix "${STATEDIR}/lib-postfix" - - # NOTE: The Postfix spool location has mixed owner/groups to take into account: - # UID = postfix(101): active, bounce, corrupt, defer, deferred, flush, hold, incoming, maildrop, private, public, saved, trace - # UID = root(0): dev, etc, lib, pid, usr - # GID = postdrop(103): maildrop, public - # GID for all other directories is root(0) - # NOTE: `spool-postfix/private/` will be set to `postfix:postfix` when Postfix starts / restarts - # Set most common ownership: - chown -R postfix:root "${STATEDIR}/spool-postfix" - chown root:root "${STATEDIR}/spool-postfix" - - # These two require the postdrop(103) group: - chgrp -R postdrop "${STATEDIR}"/spool-postfix/{maildrop,public} - - # These permissions rely on the `postdrop` binary having the SGID bit set. - # Ref: https://github.com/docker-mailserver/docker-mailserver/pull/3625 - chmod 730 "${STATEDIR}/spool-postfix/maildrop" - chmod 710 "${STATEDIR}/spool-postfix/public" - else - _log 'debug' "'${STATEDIR}' is not present; Not consolidating state" + if [[ ! -d ${DMS_STATE_DIR:?DMS_STATE_DIR is not set} ]]; then + _log 'debug' "'${DMS_STATE_DIR}' is not present - not consolidating state" + return 0 fi + + _log 'debug' "Consolidating all state onto ${DMS_STATE_DIR}" + + local DEST SERVICEDIR SERVICEDIRS SERVICEFILE SERVICEFILES + + # Always enabled features: + SERVICEDIRS=( + 'lib/logrotate' + 'lib/postfix' + 'spool/postfix' + ) + + # Only consolidate state for services that are enabled + # Notably avoids copying over 200MB for the ClamAV database + [[ ${ENABLE_AMAVIS} -eq 1 ]] && SERVICEDIRS+=('lib/amavis') + [[ ${ENABLE_CLAMAV} -eq 1 ]] && SERVICEDIRS+=('lib/clamav') + [[ ${ENABLE_FAIL2BAN} -eq 1 ]] && SERVICEDIRS+=('lib/fail2ban') + [[ ${ENABLE_FETCHMAIL} -eq 1 ]] && SERVICEDIRS+=('lib/fetchmail') + [[ ${ENABLE_GETMAIL} -eq 1 ]] && SERVICEDIRS+=('lib/getmail') + [[ ${ENABLE_MTA_STS} -eq 1 ]] && SERVICEDIRS+=('lib/mta-sts') + [[ ${ENABLE_POSTGREY} -eq 1 ]] && SERVICEDIRS+=('lib/postgrey') + [[ ${ENABLE_RSPAMD} -eq 1 ]] && SERVICEDIRS+=('lib/rspamd') + [[ ${ENABLE_RSPAMD_REDIS} -eq 1 ]] && SERVICEDIRS+=('lib/redis') + [[ ${ENABLE_SPAMASSASSIN} -eq 1 ]] && SERVICEDIRS+=('lib/spamassassin') + [[ ${ENABLE_SRS} -eq 1 ]] && SERVICEDIRS+=('lib/postsrsd') + [[ ${SMTP_ONLY} -ne 1 ]] && SERVICEDIRS+=('lib/dovecot') + + # Single service files + [[ ${ENABLE_SRS} -eq 1 ]] && SERVICEFILES+=('/etc/postsrsd.secret') + + for SERVICEFILE in "${SERVICEFILES[@]}"; do + DEST="${DMS_STATE_DIR}/${SERVICEFILE}" + + # Append service parent dir(s) path to the state dir and ensure it exists: + mkdir -p "${DEST%/*}" + if [[ -f ${DEST} ]]; then + _log 'trace' "Destination ${DEST} exists, linking ${SERVICEFILE} to it" + # Original content from image no longer relevant, remove it: + rm -f "${SERVICEFILE}" + elif [[ -f "${SERVICEFILE}" ]]; then + _log 'trace' "Moving ${SERVICEFILE} to ${DEST}" + # Empty volume was mounted, or new content from enabling a feature ENV: + mv "${SERVICEFILE}" "${DEST}" + # Apply SELinux security context to match the state directory, so access + # is not restricted to the current running container: + chcon -R --reference="${DMS_STATE_DIR}" "${DEST}" 2>/dev/null || true + fi + + # Symlink the original file in the container ($SERVICEFILE) to be + # sourced from assocaiated path in /var/mail-state/ ($DEST): + ln -s "${DEST}" "${SERVICEFILE}" + done + + for SERVICEDIR in "${SERVICEDIRS[@]}"; do + DEST="${DMS_STATE_DIR}/${SERVICEDIR//\//-}" + SERVICEDIR="/var/${SERVICEDIR}" + + # If relevant content is found in /var/mail-state (presumably a volume mount), + # use it instead. Otherwise copy over any missing directories checked. + if [[ -d ${DEST} ]]; then + _log 'trace' "Destination ${DEST} exists, linking ${SERVICEDIR} to it" + # Original content from image no longer relevant, remove it: + rm -rf "${SERVICEDIR}" + elif [[ -d ${SERVICEDIR} ]]; then + _log 'trace' "Moving contents of ${SERVICEDIR} to ${DEST}" + # An empty volume was mounted, or new content dir now exists from enabling a feature ENV: + mv "${SERVICEDIR}" "${DEST}" + # Apply SELinux security context to match the state directory, so access + # is not restricted to the current running container: + # https://github.com/docker-mailserver/docker-mailserver/pull/3890 + chcon -R --reference="${DMS_STATE_DIR}" "${DEST}" 2>/dev/null || true + else + _log 'error' "${SERVICEDIR} should exist but is missing" + fi + + # Symlink the original path in the container ($SERVICEDIR) to be + # sourced from associated path in /var/mail-state/ ($DEST): + ln -s "${DEST}" "${SERVICEDIR}" + done +} + +# These corrections are to fix changes to UID/GID values between upgrades, +# or when ownership/permissions were altered externally on the host (eg: migration or system scripts) +function _setup_adjust_state_permissions() { + [[ ! -d ${DMS_STATE_DIR:?DMS_STATE_DIR is not set} ]] && return 0 + + # This ensures the user and group of the files from the external mount have their + # numeric ID values in sync. New releases where the installed packages order changes + # can change the values in the Docker image, causing an ownership mismatch. + # NOTE: More details about users and groups added during image builds are documented here: + # https://github.com/docker-mailserver/docker-mailserver/pull/3011#issuecomment-1399120252 + _log 'trace' "Ensuring correct ownership + permissions for DMS state dir: '${DMS_STATE_DIR}'" + [[ ${ENABLE_AMAVIS} -eq 1 ]] && chown -R amavis:amavis "${DMS_STATE_DIR}/lib-amavis" + [[ ${ENABLE_CLAMAV} -eq 1 ]] && chown -R clamav:clamav "${DMS_STATE_DIR}/lib-clamav" + [[ ${ENABLE_FETCHMAIL} -eq 1 ]] && chown -R fetchmail:nogroup "${DMS_STATE_DIR}/lib-fetchmail" + [[ ${ENABLE_MTA_STS} -eq 1 ]] && chown -R _mta-sts:_mta-sts "${DMS_STATE_DIR}/lib-mta-sts" + [[ ${ENABLE_POSTGREY} -eq 1 ]] && chown -R postgrey:postgrey "${DMS_STATE_DIR}/lib-postgrey" + [[ ${ENABLE_RSPAMD} -eq 1 ]] && chown -R _rspamd:_rspamd "${DMS_STATE_DIR}/lib-rspamd" + [[ ${ENABLE_RSPAMD_REDIS} -eq 1 ]] && chown -R redis:redis "${DMS_STATE_DIR}/lib-redis" + [[ ${ENABLE_SPAMASSASSIN} -eq 1 ]] && chown -R debian-spamd:debian-spamd "${DMS_STATE_DIR}/lib-spamassassin" + + chown -R root:root "${DMS_STATE_DIR}/lib-logrotate" + chown -R postfix:postfix "${DMS_STATE_DIR}/lib-postfix" + + # NOTE: The Postfix spool location has mixed owner/groups to take into account: + # UID = postfix(101): active, bounce, corrupt, defer, deferred, flush, hold, incoming, maildrop, private, public, saved, trace + # UID = root(0): dev, etc, lib, pid, usr + # GID = postdrop(103): maildrop, public + # GID for all other directories is root(0) + # NOTE: `spool-postfix/private/` will be set to `postfix:postfix` when Postfix starts / restarts + # Set most common ownership: + chown -R postfix:root "${DMS_STATE_DIR}/spool-postfix" + chown root:root "${DMS_STATE_DIR}/spool-postfix" + + # These two require the postdrop(103) group: + chgrp -R postdrop "${DMS_STATE_DIR}"/spool-postfix/{maildrop,public} + + # These permissions rely on the `postdrop` binary having the SGID bit set. + # Ref: https://github.com/docker-mailserver/docker-mailserver/pull/3625 + chmod 730 "${DMS_STATE_DIR}/spool-postfix/maildrop" + chmod 710 "${DMS_STATE_DIR}/spool-postfix/public" } diff --git a/target/scripts/startup/variables-stack.sh b/target/scripts/startup/variables-stack.sh index 3fa4d761..f8f5af5e 100644 --- a/target/scripts/startup/variables-stack.sh +++ b/target/scripts/startup/variables-stack.sh @@ -4,9 +4,27 @@ declare -A VARS function _early_variables_setup() { + __environment_variables_log_level _obtain_hostname_and_domainname __environment_variables_backwards_compatibility __environment_variables_general_setup + + [[ ${ACCOUNT_PROVISIONER} == 'LDAP' ]] && __environment_variables_ldap + [[ ${ENABLE_OAUTH2} -eq 1 ]] && __environment_variables_oauth2 + [[ ${ENABLE_SASLAUTHD} -eq 1 ]] && __environment_variables_saslauthd + + __environment_variables_export +} + +# Declare a variable as readonly if it is not already set. +function __declare_readonly() { + local VARIABLE_NAME=${1:?Variable name required when declaring a variable as readonly} + local VARIABLE_VALUE=${2:?Variable value required when declaring a variable as readonly} + + if [[ ! -v ${VARIABLE_NAME} ]]; then + readonly "${VARIABLE_NAME}=${VARIABLE_VALUE}" + VARS[${VARIABLE_NAME}]="${VARIABLE_VALUE}" + fi } # This function handles variables that are deprecated. This allows a @@ -55,6 +73,12 @@ function __environment_variables_general_setup() { VARS[DMS_VMAIL_UID]="${DMS_VMAIL_UID:=5000}" VARS[DMS_VMAIL_GID]="${DMS_VMAIL_GID:=5000}" + # internal variables are next + + __declare_readonly 'DMS_STATE_DIR' '/var/mail-state' + + # user-customizable are last + _log 'trace' 'Setting anti-spam & anti-virus environment variables' VARS[AMAVIS_LOGLEVEL]="${AMAVIS_LOGLEVEL:=0}" @@ -159,15 +183,27 @@ function __environment_variables_general_setup() { VARS[UPDATE_CHECK_INTERVAL]="${UPDATE_CHECK_INTERVAL:=1d}" } -function _environment_variables_oauth2() { - _log 'debug' 'Setting OAUTH2-related environment variables now' +function __environment_variables_log_level() { + if [[ ${LOG_LEVEL} == 'trace' ]] \ + || [[ ${LOG_LEVEL} == 'debug' ]] \ + || [[ ${LOG_LEVEL} == 'info' ]] \ + || [[ ${LOG_LEVEL} == 'warn' ]] \ + || [[ ${LOG_LEVEL} == 'error' ]] + then + return 0 + else + local DEFAULT_LOG_LEVEL='info' + _log 'warn' "Log level '${LOG_LEVEL}' is invalid (falling back to default '${DEFAULT_LOG_LEVEL}')" - VARS[OAUTH2_INTROSPECTION_URL]="${OAUTH2_INTROSPECTION_URL:=}" + # shellcheck disable=SC2034 + VARS[LOG_LEVEL]="${DEFAULT_LOG_LEVEL}" + LOG_LEVEL="${DEFAULT_LOG_LEVEL}" + fi } # This function handles environment variables related to LDAP. # NOTE: SASLAuthd and Dovecot LDAP support inherit these common ENV. -function _environment_variables_ldap() { +function __environment_variables_ldap() { _log 'debug' 'Setting LDAP-related environment variables now' VARS[LDAP_BIND_DN]="${LDAP_BIND_DN:=}" @@ -177,9 +213,15 @@ function _environment_variables_ldap() { VARS[LDAP_START_TLS]="${LDAP_START_TLS:=no}" } +function __environment_variables_oauth2() { + _log 'debug' 'Setting OAUTH2-related environment variables now' + + VARS[OAUTH2_INTROSPECTION_URL]="${OAUTH2_INTROSPECTION_URL:=}" +} + # This function handles environment variables related to SASLAUTHD # LDAP specific ENV handled in: `startup/setup.d/saslauthd.sh:_setup_saslauthd()` -function _environment_variables_saslauthd() { +function __environment_variables_saslauthd() { _log 'debug' 'Setting SASLAUTHD-related environment variables now' # This ENV is only used by the supervisor service config `saslauth.conf`: @@ -190,7 +232,7 @@ function _environment_variables_saslauthd() { # This function Writes the contents of the `VARS` map (associative array) # to locations where they can be sourced from (e.g. `/etc/dms-settings`) # or where they can be used by Bash directly (e.g. `/root/.bashrc`). -function _environment_variables_export() { +function __environment_variables_export() { _log 'debug' "Exporting environment variables now (creating '/etc/dms-settings')" : >/root/.bashrc # make DMS variables available in login shells and their subprocesses From c66d8ce40b705f7ce9d67e12e48a26ca1a67be9d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 11 Feb 2025 10:36:20 +0100 Subject: [PATCH 200/267] chore(deps): Bump docker/setup-qemu-action from 3.3.0 to 3.4.0 (#4353) --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 6000f5ff..53cb6893 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -71,7 +71,7 @@ jobs: cache-buildx- - name: 'Set up QEMU' - uses: docker/setup-qemu-action@v3.3.0 + uses: docker/setup-qemu-action@v3.4.0 with: platforms: arm64 diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index 42cde7ea..d5966a2b 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -35,7 +35,7 @@ jobs: type=semver,pattern={{major}}.{{minor}}.{{patch}} - name: 'Set up QEMU' - uses: docker/setup-qemu-action@v3.3.0 + uses: docker/setup-qemu-action@v3.4.0 with: platforms: arm64 From 83bfe72d489beda875717130eba9b348218f3bda Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Wed, 12 Feb 2025 11:56:51 +1300 Subject: [PATCH 201/267] chore: Migrate dovecot config from Dockerfile (#4350) --- Dockerfile | 10 ---- target/scripts/start-mailserver.sh | 1 - target/scripts/startup/setup.d/dovecot.sh | 59 ++++++++++++++++++++--- 3 files changed, 51 insertions(+), 19 deletions(-) diff --git a/Dockerfile b/Dockerfile index 09295a22..32ac9ae8 100644 --- a/Dockerfile +++ b/Dockerfile @@ -88,16 +88,6 @@ RUN dpkg -i /dovecot-fts-xapian-*.deb && rm /dovecot-fts-xapian-*.deb COPY target/dovecot/*.inc target/dovecot/*.conf /etc/dovecot/conf.d/ COPY target/dovecot/dovecot-purge.cron /etc/cron.d/dovecot-purge.disabled RUN chmod 0 /etc/cron.d/dovecot-purge.disabled -WORKDIR /usr/share/dovecot - -# hadolint ignore=SC2016,SC2086,SC2069 -RUN < Date: Wed, 12 Feb 2025 20:34:22 +0100 Subject: [PATCH 202/267] docs: Fix typo in DKIM and utils.sh (#4358) --- docs/content/config/best-practices/dkim_dmarc_spf.md | 4 ++-- target/scripts/helpers/utils.sh | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/content/config/best-practices/dkim_dmarc_spf.md b/docs/content/config/best-practices/dkim_dmarc_spf.md index 2ef8e902..c04d4b7e 100644 --- a/docs/content/config/best-practices/dkim_dmarc_spf.md +++ b/docs/content/config/best-practices/dkim_dmarc_spf.md @@ -29,7 +29,7 @@ Cloudflare has written an [article about DKIM, DMARC and SPF][cloudflare-dkim-dm When DKIM is enabled: 1. Inbound mail will verify any included DKIM signatures -2. Outbound mail is signed (_when you're sending domain has a configured DKIM key_) +2. Outbound mail is signed (_when your sending domain has a configured DKIM key_) DKIM requires a public/private key pair to enable **signing (_via private key_)** your outgoing mail, while the receiving end must query DNS to **verify (_via public key_)** that the signature is trustworthy. @@ -76,7 +76,7 @@ You should have: ??? info "Changing the key size" The keypair generated for using with DKIM presently defaults to RSA-2048. This is a good size but you can lower the security to `1024-bit`, or increase it to `4096-bit` (_discouraged as that is excessive_). - + To generate a key with different size (_for RSA 1024-bit_) run: ```sh diff --git a/target/scripts/helpers/utils.sh b/target/scripts/helpers/utils.sh index c848e13e..90fd8f9b 100644 --- a/target/scripts/helpers/utils.sh +++ b/target/scripts/helpers/utils.sh @@ -122,7 +122,7 @@ function _reload_postfix() { # you can set the environment variable `POSTFIX_README_DIRECTORY='/new/dir/'` # (`POSTFIX_` is an arbitrary prefix, you can choose the one you like), # and then call this function: -# `_replace_by_env_in_file 'POSTFIX_' 'PATH TO POSTFIX's main.cf>` +# `_replace_by_env_in_file 'POSTFIX_' '` # # ## Panics # From 425d1162aeeeb532e12eeccd9548d31213c9f07f Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Fri, 14 Feb 2025 01:16:31 +1300 Subject: [PATCH 203/267] chore: `packages.sh` - Bump versions + housekeeping (#4357) --- CHANGELOG.md | 38 ++++++++++--------- target/scripts/build/compile.sh | 51 +++++++++++++++---------- target/scripts/build/packages.sh | 64 +++++++++++++++++++------------- 3 files changed, 90 insertions(+), 63 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 379992c3..794c2e64 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,20 +11,12 @@ All notable changes to this project will be documented in this file. The format - **saslauthd** mechanism support via ENV `SASLAUTHD_MECHANISMS` with `pam`, `shadow`, `mysql` values has been removed. Only `ldap` and `rimap` remain supported ([#4259](https://github.com/docker-mailserver/docker-mailserver/pull/4259)) - **getmail6** has been refactored: ([#4156](https://github.com/docker-mailserver/docker-mailserver/pull/4156)) - The [DMS config volume](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/advanced/optional-config/#volumes) now has support for `getmailrc_general.cf` for overriding [common default settings](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/advanced/mail-getmail/#common-options). If you previously mounted this config file directly to `/etc/getmailrc_general` you should switch to our config volume support. - - IMAP/POP3 example configs added to our [`config-examples`](https://github.com/docker-mailserver/docker-mailserver/tree/v15.0.0/config-examples/getmail). - - ENV [`GETMAIL_POLL`](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/environment/#getmail_poll) now supports values above 30 minutes. - - Added `getmail` as a new service for `supervisor` to manage, replacing cron for periodic polling. - Generated getmail configuration files no longer set the `message_log` option. Instead of individual log files per config, the [default base settings DMS configures](https://github.com/docker-mailserver/docker-mailserver/tree/v15.0.0/target/getmail/getmailrc_general) now enables `message_log_syslog`. This aligns with how other services in DMS log to syslog where it is captured in `mail.log`. - Getmail configurations have changed location from the base of the DMS Config Volume, to the `getmail/` subdirectory. Any existing configurations **must be migrated manually.** - - DMS v14 mistakenly relocated the _getmail state directory_ to the _DMS Config Volume_ as a `getmail/` subdirectory. + - **DMS v14 mistakenly** relocated the _getmail state directory_ to the _DMS Config Volume_ as a `getmail/` subdirectory. - This has been corrected to `/var/lib/getmail` (_if you have mounted a DMS State Volume to `/var/mail-state`, `/var/lib/getmail` will be symlinked to `/var/mail-state/lib-getmail`_). - To preserve this state when upgrading to DMS v15, **you must manually migrate `getmail/` from the _DMS Config Volume_ to `lib-getmail/` in the _DMS State Volume_.** -### Security - -- **Fail2ban:** - - Ensure a secure connection, when downloading the fail2ban package ([#4080](https://github.com/docker-mailserver/docker-mailserver/pull/4080)) - ### Added - **Internal:** @@ -33,30 +25,40 @@ All notable changes to this project will be documented in this file. The format ### Updates -- **Removed `VERSION` file** from the repo that releases of DMS prior to v13 (Nov 2023) would check to detect new releases ([#3677](https://github.com/docker-mailserver/docker-mailserver/issues/3677), [#4321](https://github.com/docker-mailserver/docker-mailserver/pull/4321)) -- **Fail2ban:** - - Updated to version [`1.1.0`](https://github.com/fail2ban/fail2ban/releases/tag/1.1.0) ([#4045](https://github.com/docker-mailserver/docker-mailserver/pull/4045)) +**Internal:** + - **Removed `VERSION` file** from the repo. Releases of DMS prior to v13 (Nov 2023) would check this to detect new releases ([#3677](https://github.com/docker-mailserver/docker-mailserver/issues/3677), [#4321](https://github.com/docker-mailserver/docker-mailserver/pull/4321)) + - During image build, ensure a secure connection when downloading the `fail2ban` package ([#4080](https://github.com/docker-mailserver/docker-mailserver/pull/4080)) - **Documentation:** - Account Management and Authentication pages have been rewritten and better organized ([#4122](https://github.com/docker-mailserver/docker-mailserver/pull/4122)) - Add a caveat for `DMS_VMAIL_UID` not being compatible with `0` / root ([#4143](https://github.com/docker-mailserver/docker-mailserver/pull/4143)) +- **Getmail:** ([#4156](https://github.com/docker-mailserver/docker-mailserver/pull/4156)) + - Added `getmail` as a new service for `supervisor` to manage, replacing cron for periodic polling. + - IMAP/POP3 example configs added to our [`config-examples`](https://github.com/docker-mailserver/docker-mailserver/tree/v15.0.0/config-examples/getmail). + - ENV [`GETMAIL_POLL`](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/environment/#getmail_poll) now supports values above 30 minutes. - **Postfix:** - By default opt-out from _Microsoft reactions_ for outbound mail ([#4120](https://github.com/docker-mailserver/docker-mailserver/pull/4120)) -- Updated `jaq` version from `1.3.0` to `2.0.0` ([#4190](https://github.com/docker-mailserver/docker-mailserver/pull/4190)) -- Updated Rspamd GTube settings and tests ([#4191](https://github.com/docker-mailserver/docker-mailserver/pull/4191)) +- **Rspamd:** + - Updated GTube settings and tests ([#4191](https://github.com/docker-mailserver/docker-mailserver/pull/4191)) +- Updated externally installed software ([#4357](https://github.com/docker-mailserver/docker-mailserver/pull/4357)): + - `DOVECOT_COMMUNITY_REPO=1` custom image build ARG now supports the latest Dovecot [`2.4.x`](https://github.com/dovecot/core/releases/tag/2.4.0) (_DMS provides Dovecot `2.3.19` by default_) + - Dovecot FTS Xapian module (`1.7.12` => [`1.9.0`](https://github.com/grosjo/fts-xapian/releases/tag/1.9)) + - `jaq` (`1.3.0` => [`2.1.0`](https://github.com/01mf02/jaq/releases/tag/v2.1.0)) + - Fail2Ban (`1.0.2-2` => [`1.1.0`](https://github.com/fail2ban/fail2ban/releases/tag/1.1.0)) ([#4045](https://github.com/docker-mailserver/docker-mailserver/pull/4045)) + - Rspamd (`3.8.4` => [`3.11.0`](https://github.com/rspamd/rspamd/releases/tag/3.11.0)) - Implicitly upgraded during image build, as the third-party repo lacks version pinning support. ### Fixes - **Dovecot:** - The logwatch `ignore.conf` now also excludes Xapian messages about pending documents ([#4060](https://github.com/docker-mailserver/docker-mailserver/pull/4060)) - - `dovecot-fts-xapian` plugin was updated to `1.7.13`, fixing a regression with indexing ([#4095](https://github.com/docker-mailserver/docker-mailserver/pull/4095)) + - `dovecot-fts-xapian` plugin was updated, fixing a regression with indexing ([#4095](https://github.com/docker-mailserver/docker-mailserver/pull/4095)) - The "dummy account" workaround for _Dovecot Quota_ feature support no longer treats the alias as a regex when checking the Dovecot UserDB ([#4222](https://github.com/docker-mailserver/docker-mailserver/pull/4222)) - **LDAP:** - - Correctly apply a compatibility fix for OAuth2 introduced in DMS v13.3.1 which had not been applied to the actual LDAP config changes ([#4175](https://github.com/docker-mailserver/docker-mailserver/pull/4175)) + - Correctly apply a compatibility fix for OAuth2 introduced in DMS `v13.3.1` which had not been applied to the actual LDAP config changes ([#4175](https://github.com/docker-mailserver/docker-mailserver/pull/4175)) - **Internal:** - The main `mail.log` (_which is piped to stdout via `tail`_) now correctly begins from the first log line of the active container run. Previously some daemon logs and potential warnings/errors were omitted ([#4146](https://github.com/docker-mailserver/docker-mailserver/pull/4146)) - `start-mailserver.sh` removed unused `shopt -s inherit_errexit` ([#4161](https://github.com/docker-mailserver/docker-mailserver/pull/4161)) - - Fixed a regression introduced in v14 where `postfix-main.cf` appended `stderr` output into `/etc/postfix/main.cf`, causing Postfix startup to fail ([#4147](https://github.com/docker-mailserver/docker-mailserver/pull/4147)) - - Fixed a regression introduced in v14 to better support running `start-mailserver.sh` with container restarts, which now only skip calling `_setup()` ([#4323](https://github.com/docker-mailserver/docker-mailserver/pull/4323#issuecomment-2629559254)) + - Fixed a regression introduced in DMS v14 where `postfix-main.cf` appended `stderr` output into `/etc/postfix/main.cf`, causing Postfix startup to fail ([#4147](https://github.com/docker-mailserver/docker-mailserver/pull/4147)) + - Fixed a regression introduced in DMS v14 to better support running `start-mailserver.sh` with container restarts, which now only skip calling `_setup()` ([#4323](https://github.com/docker-mailserver/docker-mailserver/pull/4323#issuecomment-2629559254)) - The command `swaks --help` is now functional ([#4282](https://github.com/docker-mailserver/docker-mailserver/pull/4282)) - **Rspamd:** - DKIM private key path checking is now performed only on paths that do not contain `$` ([#4201](https://github.com/docker-mailserver/docker-mailserver/pull/4201)) diff --git a/target/scripts/build/compile.sh b/target/scripts/build/compile.sh index c34a8833..bad84f23 100644 --- a/target/scripts/build/compile.sh +++ b/target/scripts/build/compile.sh @@ -11,28 +11,39 @@ source /usr/local/bin/helpers/log.sh # shellcheck disable=SC2310 _log_level_is 'trace' && QUIET='-y' || QUIET='-qq' -function _compile_dovecot_fts_xapian() { +function _install_build_deps() { apt-get "${QUIET}" update apt-get "${QUIET}" install --no-install-recommends \ automake libtool pkg-config libicu-dev libsqlite3-dev libxapian-dev make build-essential dh-make devscripts dovecot-dev - - local XAPIAN_VERSION='1.7.13' - curl -sSfL -o dovecot-fts-xapian.tar.gz \ - "https://github.com/grosjo/fts-xapian/releases/download/${XAPIAN_VERSION}/dovecot-fts-xapian-${XAPIAN_VERSION}.tar.gz" - tar xf dovecot-fts-xapian.tar.gz - - cd "fts-xapian-${XAPIAN_VERSION}" - USER=root dh_make -p "dovecot-fts-xapian-${XAPIAN_VERSION}" --single --native --copyright gpl2 -y - - rm debian/*.ex - cp PACKAGES/DEB/control debian/ - cp PACKAGES/DEB/changelog debian/ - cp PACKAGES/DEB/compat debian/ - - sed -i -E "s|(dovecot-fts-xapian)-[1-9\.-]+|\1-${XAPIAN_VERSION}|g" debian/control - sed -i -E "s|(dovecot-fts-xapian)-[1-9\.-]+ \(.*\)(.*)|\1-${XAPIAN_VERSION} (${XAPIAN_VERSION})\2|g" debian/changelog - - debuild -us -uc -B | tee /tmp/debuild.log 2>&1 } -_compile_dovecot_fts_xapian +function _build_package() { + local XAPIAN_VERSION='1.9' + curl -fsSL "https://github.com/grosjo/fts-xapian/releases/download/${XAPIAN_VERSION}/dovecot-fts-xapian-${XAPIAN_VERSION}.tar.gz" \ + | tar -xz + cd "fts-xapian-${XAPIAN_VERSION}" + + # Prepare for building DEB source package: + # https://manpages.debian.org/bookworm/dh-make/dh_make.1.en.html + # License LGPL 2.1: https://github.com/grosjo/fts-xapian/issues/174#issuecomment-2422404568 + USER=root dh_make --packagename "dovecot-fts-xapian-${XAPIAN_VERSION}" --single --native --copyright lgpl2 -y + # Remove generated example files: + rm debian/*.ex + # Add required package metadata: + # https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#control + curl -fsSL https://raw.githubusercontent.com/grosjo/fts-xapian/refs/tags/1.7.16/PACKAGES/DEB/control > debian/control + # Replace version number: + sed -i -E "s|(dovecot-fts-xapian)-[1-9\.-]+|\1-${XAPIAN_VERSION}|g" debian/control + # Required to proceed with debuild: + # https://www.debian.org/doc/manuals/maint-guide/dother.en.html#compat + # (13 is the default debhelper version from the original `dh_make` generated `debian/control`): + echo '13' > debian/compat + + # Build arch specific binary package via debuild: + # https://manpages.debian.org/bookworm/devscripts/debuild.1.en.html + # https://manpages.debian.org/bookworm/dpkg-dev/dpkg-buildpackage.1.en.html + debuild --no-sign --build=any | tee /tmp/debuild.log 2>&1 +} + +_install_build_deps +_build_package diff --git a/target/scripts/build/packages.sh b/target/scripts/build/packages.sh index 787c84ac..bcfdcbcb 100644 --- a/target/scripts/build/packages.sh +++ b/target/scripts/build/packages.sh @@ -24,13 +24,14 @@ function _pre_installation_steps() { apt-get "${QUIET}" upgrade _log 'trace' 'Installing packages that are needed early' - # add packages usually required by apt to - # - not log unnecessary warnings - # - be able to add PPAs early (e.g., Rspamd) + # Add packages usually required by apt to: local EARLY_PACKAGES=( - apt-utils # avoid useless warnings - apt-transport-https ca-certificates curl gnupg # required for adding PPAs - systemd-standalone-sysusers # avoid problems with SA / Amavis (https://github.com/docker-mailserver/docker-mailserver/pull/3403#pullrequestreview-1596689953) + # Avoid logging unnecessary warnings: + apt-utils + # Required for adding third-party repos (/etc/apt/sources.list.d) as alternative package sources (eg: Dovecot CE and Rspamd): + apt-transport-https ca-certificates curl gnupg + # Avoid problems with SA / Amavis (https://github.com/docker-mailserver/docker-mailserver/pull/3403#pullrequestreview-1596689953): + systemd-standalone-sysusers ) apt-get "${QUIET}" install --no-install-recommends "${EARLY_PACKAGES[@]}" 2>/dev/null } @@ -38,7 +39,7 @@ function _pre_installation_steps() { function _install_utils() { _log 'debug' 'Installing utils sourced from Github' _log 'trace' 'Installing jaq' - local JAQ_TAG='v2.0.0' + local JAQ_TAG='v2.1.0' curl -sSfL "https://github.com/01mf02/jaq/releases/download/${JAQ_TAG}/jaq-$(uname -m)-unknown-linux-gnu" -o /usr/bin/jaq chmod +x /usr/bin/jaq @@ -136,43 +137,56 @@ function _install_dovecot() { dovecot-pop3d dovecot-sieve ) - # Dovecot packages for community supported features. + # Additional Dovecot packages for supporting the DMS community (docs-only guide contributions). DOVECOT_PACKAGES+=(dovecot-auth-lua) - # Dovecot's deb community repository only provides x86_64 packages, so do not include it - # when building for another architecture. + # (Opt-in via ENV) Change repo source for dovecot packages to a third-party repo maintained by Dovecot. + # NOTE: AMD64 / x86_64 is the only supported arch from the Dovecot CE repo (thus noDMS built for ARM64 / aarch64) + # Repo: https://repo.dovecot.org/ce-2.4-latest/debian/bookworm/dists/bookworm/main/ + # Docs: https://repo.dovecot.org/#debian if [[ ${DOVECOT_COMMUNITY_REPO} -eq 1 ]] && [[ "$(uname --machine)" == "x86_64" ]]; then - _log 'trace' 'Using Dovecot community repository' - curl -sSfL https://repo.dovecot.org/DOVECOT-REPO-GPG | gpg --import - gpg --export ED409DA1 > /etc/apt/trusted.gpg.d/dovecot.gpg - echo "deb https://repo.dovecot.org/ce-2.3-latest/debian/${VERSION_CODENAME} ${VERSION_CODENAME} main" > /etc/apt/sources.list.d/dovecot.list + # WARNING: Repo only provides Debian Bookworm package support for Dovecot CE 2.4+. + # As Debian Bookworm only packages Dovecot 2.3.x, building DMS with this alternative package repo may not yet be compatible with DMS: + # - 2.3.19: https://salsa.debian.org/debian/dovecot/-/tree/stable/bookworm + # - 2.3.21: https://salsa.debian.org/debian/dovecot/-/tree/stable/bookworm-backports - _log 'trace' 'Updating Dovecot package signatures' + _log 'trace' 'Adding third-party package repository (Dovecot)' + curl -fsSL https://repo.dovecot.org/DOVECOT-REPO-GPG-2.4 | gpg --dearmor > /usr/share/keyrings/upstream-dovecot.gpg + echo \ + "deb [signed-by=/usr/share/keyrings/upstream-dovecot.gpg] https://repo.dovecot.org/ce-2.4-latest/debian/${VERSION_CODENAME} ${VERSION_CODENAME} main" \ + > /etc/apt/sources.list.d/upstream-dovecot.list + + # Refresh package index: apt-get "${QUIET}" update - # Additional community package needed for Lua support if the Dovecot community repository is used. + # This repo instead provides `dovecot-auth-lua` as a transitional package to `dovecot-lua`, + # thus this extra package is required to retain lua support: DOVECOT_PACKAGES+=(dovecot-lua) fi _log 'debug' 'Installing Dovecot' apt-get "${QUIET}" install --no-install-recommends "${DOVECOT_PACKAGES[@]}" - # dependency for fts_xapian + # Runtime dependency for fts_xapian (built via `compile.sh`): apt-get "${QUIET}" install --no-install-recommends libxapian30 } function _install_rspamd() { - _log 'debug' 'Installing Rspamd' - _log 'trace' 'Adding Rspamd PPA' - curl -sSfL https://rspamd.com/apt-stable/gpg.key | gpg --dearmor >/etc/apt/trusted.gpg.d/rspamd.gpg - echo \ - "deb [signed-by=/etc/apt/trusted.gpg.d/rspamd.gpg] http://rspamd.com/apt-stable/ ${VERSION_CODENAME} main" \ - >/etc/apt/sources.list.d/rspamd.list + # NOTE: DMS only supports the rspamd package via using the third-party repo maintained by Rspamd (AMD64 + ARM64): + # Repo: https://rspamd.com/apt-stable/dists/bookworm/main/ + # Docs: https://rspamd.com/downloads.html#debian-and-ubuntu-linux + # NOTE: Debian 12 provides Rspamd 3.4 (too old) and Rspamd discourages it's use - _log 'trace' 'Updating package index after adding PPAs' + _log 'trace' 'Adding third-party package repository (Rspamd)' + curl -fsSL https://rspamd.com/apt-stable/gpg.key | gpg --dearmor > /usr/share/keyrings/upstream-rspamd.gpg + echo \ + "deb [signed-by=/usr/share/keyrings/upstream-rspamd.gpg] https://rspamd.com/apt-stable/ ${VERSION_CODENAME} main" \ + > /etc/apt/sources.list.d/upstream-rspamd.list + + # Refresh package index: apt-get "${QUIET}" update - _log 'trace' 'Installing actual package' + _log 'debug' 'Installing Rspamd' apt-get "${QUIET}" install rspamd redis-server } From f2fedff251a91da3bfc1f780a17b30e1346a53a3 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 13 Feb 2025 21:46:09 +0100 Subject: [PATCH 204/267] chore(deps): Bump docker/setup-buildx-action from 3.8.0 to 3.9.0 (#4352) --- .github/workflows/generic_build.yml | 2 +- .github/workflows/generic_publish.yml | 2 +- .github/workflows/generic_test.yml | 2 +- .github/workflows/generic_vulnerability-scan.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/generic_build.yml b/.github/workflows/generic_build.yml index 53cb6893..30d8df46 100644 --- a/.github/workflows/generic_build.yml +++ b/.github/workflows/generic_build.yml @@ -76,7 +76,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.8.0 + uses: docker/setup-buildx-action@v3.9.0 # NOTE: AMD64 can build within 2 minutes - name: 'Build images' diff --git a/.github/workflows/generic_publish.yml b/.github/workflows/generic_publish.yml index d5966a2b..72228d51 100644 --- a/.github/workflows/generic_publish.yml +++ b/.github/workflows/generic_publish.yml @@ -40,7 +40,7 @@ jobs: platforms: arm64 - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.8.0 + uses: docker/setup-buildx-action@v3.9.0 # Try get the cached build layers from a prior `generic_build.yml` job. # NOTE: Until adopting `type=gha` scoped cache exporter (in `docker/build-push-action`), diff --git a/.github/workflows/generic_test.yml b/.github/workflows/generic_test.yml index f28dbc9b..1303c8ad 100644 --- a/.github/workflows/generic_test.yml +++ b/.github/workflows/generic_test.yml @@ -38,7 +38,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.8.0 + uses: docker/setup-buildx-action@v3.9.0 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. diff --git a/.github/workflows/generic_vulnerability-scan.yml b/.github/workflows/generic_vulnerability-scan.yml index 3b20b2ed..511ade7a 100644 --- a/.github/workflows/generic_vulnerability-scan.yml +++ b/.github/workflows/generic_vulnerability-scan.yml @@ -37,7 +37,7 @@ jobs: # Ensures consistent BuildKit version (not coupled to Docker Engine), # and increased compatibility of the build cache vs mixing buildx drivers. - name: 'Set up Docker Buildx' - uses: docker/setup-buildx-action@v3.8.0 + uses: docker/setup-buildx-action@v3.9.0 # Importing from the cache should create the image within approx 30 seconds: # NOTE: `qemu` step is not needed as we only test for AMD64. From aba92b7bb83929fab296a4c5340133d3f5dadaeb Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Sun, 16 Feb 2025 21:53:01 +1300 Subject: [PATCH 205/267] ci: Upgrade `mkdocs-material` to 9.6 (#4368) --- .github/workflows/scripts/docs/build-docs.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/scripts/docs/build-docs.sh b/.github/workflows/scripts/docs/build-docs.sh index d4384b5e..4eac8c4d 100755 --- a/.github/workflows/scripts/docs/build-docs.sh +++ b/.github/workflows/scripts/docs/build-docs.sh @@ -11,7 +11,7 @@ docker run \ --user "$(id -u):$(id -g)" \ --volume "./:/docs" \ --name "build-docs" \ - squidfunk/mkdocs-material:9.5 build --strict + squidfunk/mkdocs-material:9.6 build --strict # Remove unnecessary build artifacts: https://github.com/squidfunk/mkdocs-material/issues/2519 # site/ is the build output folder. From 07e558e4bee524262c8ab18ad04e685b941e782d Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Sun, 16 Feb 2025 22:22:40 +1300 Subject: [PATCH 206/267] docs: Fix broken ref links (#4366) --- docs/content/config/best-practices/dkim_dmarc_spf.md | 2 +- docs/content/config/debugging.md | 2 +- docs/content/contributing/general.md | 11 +++++++++-- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/docs/content/config/best-practices/dkim_dmarc_spf.md b/docs/content/config/best-practices/dkim_dmarc_spf.md index c04d4b7e..daea09d3 100644 --- a/docs/content/config/best-practices/dkim_dmarc_spf.md +++ b/docs/content/config/best-practices/dkim_dmarc_spf.md @@ -54,7 +54,7 @@ You'll need to repeat this process if you add any new domains. You should have: -- At least one [email account setup][docs-accounts-add] +- At least one [email account setup][docs-accounts] - Attached a [volume for config][docs-volumes-config] to persist the generated files to local storage !!! example "Creating DKIM Keys" diff --git a/docs/content/config/debugging.md b/docs/content/config/debugging.md index 30fee383..fca962f1 100644 --- a/docs/content/config/debugging.md +++ b/docs/content/config/debugging.md @@ -111,7 +111,7 @@ This could be from outdated software, or running a system that isn't able to pro - **Container runtime:** Docker and Podman for example have subtle differences. DMS docs are primarily focused on Docker, but we try to document known issues where relevant. - **Rootless containers:** Introduces additional differences in behavior or requirements: - cgroup v2 is required for supporting rootless containers. - - Differences such as for container networking which may further affect support for IPv6 and preserving the client IP (Remote address). Example with Docker rootless are [binding a port to a specific interface][docker-rootless-interface] and the choice of [port forwarding driver][docs-rootless-portdriver]. + - Differences such as for container networking which may further affect support for IPv6 and preserving the client IP (Remote address). Example with Docker rootless are [binding a port to a specific interface][docker-rootless-interface] and the choice of [port forwarding driver][docs::fail2ban::rootless-portdriver]. [network::docker-userlandproxy]: https://github.com/moby/moby/issues/44721 [network::docker-nftables]: https://github.com/moby/moby/issues/26824 diff --git a/docs/content/contributing/general.md b/docs/content/contributing/general.md index ea88dc85..a540f257 100644 --- a/docs/content/contributing/general.md +++ b/docs/content/contributing/general.md @@ -15,13 +15,20 @@ When refactoring, writing or altering scripts or other files, adhere to these ru Make sure to select `edge` in the dropdown menu at the top. Navigate to the page you would like to edit and click the edit button in the top right. This allows you to make changes and create a pull-request. -Alternatively you can make the changes locally. For that you'll need to have Docker installed. Navigate into the `docs/` directory. Then run: +Alternatively you can make the changes locally. For that you'll need to have Docker installed and run: ```sh -docker run --rm -it -p 8000:8000 -v "${PWD}:/docs" squidfunk/mkdocs-material +# From the root directory of the git clone: +docker run --rm -it -p 8000:8000 -v "./docs:/docs" squidfunk/mkdocs-material ``` This serves the documentation on your local machine on port `8000`. Each change will be hot-reloaded onto the page you view, just edit, save and look at the result. +!!! note + + The container logs will inform you of invalid links detected, but a [few are false-positives][gh-dms::mkdocs-link-error-false-positives] due to our usage of linking to specific [content tabs][mkdocs::content-tabs]. + [get-docker]: https://docs.docker.com/get-docker/ [docs-bats-parallel]: https://bats-core.readthedocs.io/en/v1.8.2/usage.html#parallel-execution +[gh-dms::mkdocs-link-error-false-positives]: https://github.com/docker-mailserver/docker-mailserver/pull/4366 +[mkdocs::content-tabs]: https://squidfunk.github.io/mkdocs-material/reference/content-tabs/#anchor-links From 0ebf820b00fcc7c634229c5df50d5c36cb2a9c8f Mon Sep 17 00:00:00 2001 From: Casper Date: Sun, 16 Feb 2025 10:46:49 +0100 Subject: [PATCH 207/267] Make deletion of mailbox data opt-in (#4365) Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com> --- CHANGELOG.md | 1 + target/bin/delmailuser | 26 ++++++++++++++------------ 2 files changed, 15 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 794c2e64..36e51e41 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ All notable changes to this project will be documented in this file. The format - **DMS v14 mistakenly** relocated the _getmail state directory_ to the _DMS Config Volume_ as a `getmail/` subdirectory. - This has been corrected to `/var/lib/getmail` (_if you have mounted a DMS State Volume to `/var/mail-state`, `/var/lib/getmail` will be symlinked to `/var/mail-state/lib-getmail`_). - To preserve this state when upgrading to DMS v15, **you must manually migrate `getmail/` from the _DMS Config Volume_ to `lib-getmail/` in the _DMS State Volume_.** + - `setup email delete ` now requires explicit confirmation if the mailbox data should be deleted ([#4365](https://github.com/docker-mailserver/docker-mailserver/pull/4365)). ### Added diff --git a/target/bin/delmailuser b/target/bin/delmailuser index c4b68ff3..9488953e 100755 --- a/target/bin/delmailuser +++ b/target/bin/delmailuser @@ -19,7 +19,11 @@ function _main() { for MAIL_ACCOUNT in "${@}"; do _account_should_already_exist - [[ ${MAILDEL} -eq 1 ]] && _remove_maildir "${MAIL_ACCOUNT}" + if [[ ${MAILDEL} -eq 1 ]]; then + _remove_maildir "${MAIL_ACCOUNT}" + else + _log 'info' "The mailbox data will not be deleted." + fi _manage_virtual_aliases_delete '_' "${MAIL_ACCOUNT}" \ || _exit_with_error "Aliases for '${MAIL_ACCOUNT}' could not be deleted" @@ -31,7 +35,7 @@ function _main() { _manage_accounts_delete "${MAIL_ACCOUNT}" \ || _exit_with_error "'${MAIL_ACCOUNT}' could not be deleted" - _log 'info' "'${MAIL_ACCOUNT}' and associated data deleted" + _log 'info' "'${MAIL_ACCOUNT}' and associated data (aliases, quotas) deleted" done } @@ -43,14 +47,14 @@ ${ORANGE}USAGE${RESET} ${ORANGE}OPTIONS${RESET} -y - Skip prompt by approving to ${LWHITE}delete all mail storage${RESET} for the account(s). + Skip prompt by approving to ${LWHITE}delete all mail data${RESET} for the account(s). ${BLUE}Generic Program Information${RESET} help Print the usage information. ${ORANGE}DESCRIPTION${RESET} Delete a mail account, including associated data (aliases, quotas) and - optionally the mailbox storage for that account. + optionally the mailbox data for that account. ${ORANGE}EXAMPLES${RESET} ${LWHITE}./setup.sh email del user@example.com${RESET} @@ -87,12 +91,10 @@ function _parse_options() { function _maildel_request_if_missing() { if [[ ${MAILDEL} -eq 0 ]]; then local MAILDEL_CHOSEN - read -r -p "Do you want to delete the mailbox as well (removing all mails)? [Y/n] " MAILDEL_CHOSEN + read -r -p "Do you want to delete the mailbox data as well (removing all mails)? [y/N] " MAILDEL_CHOSEN - # TODO: Why would MAILDEL be set to true if MAILDEL_CHOSEN is empty? - if [[ ${MAILDEL_CHOSEN} =~ (y|Y|yes|Yes) ]] || [[ -z ${MAILDEL_CHOSEN} ]]; then - MAILDEL=1 - fi + # Delete mailbox data only if the user provides explicit confirmation. + [[ ${MAILDEL_CHOSEN,,} == "y" ]] && MAILDEL=1 fi } @@ -103,10 +105,10 @@ function _remove_maildir() { local DOMAIN_PART="${MAIL_ACCOUNT#*@}" local MAIL_ACCOUNT_STORAGE_DIR="/var/mail/${DOMAIN_PART}/${LOCAL_PART}" - [[ ! -d ${MAIL_ACCOUNT_STORAGE_DIR} ]] && _exit_with_error "Mailbox directory '${MAIL_ACCOUNT_STORAGE_DIR}' does not exist" + [[ ! -d ${MAIL_ACCOUNT_STORAGE_DIR} ]] && _exit_with_error "Mailbox data directory '${MAIL_ACCOUNT_STORAGE_DIR}' does not exist" - _log 'info' "Deleting Mailbox: '${MAIL_ACCOUNT_STORAGE_DIR}'" - rm -R "${MAIL_ACCOUNT_STORAGE_DIR}" || _exit_with_error 'Mailbox could not be deleted' + _log 'info' "Deleting mailbox data: '${MAIL_ACCOUNT_STORAGE_DIR}'" + rm -R "${MAIL_ACCOUNT_STORAGE_DIR}" || _exit_with_error 'Mailbox data could not be deleted' # Remove parent directory too if it's empty: rmdir "/var/mail/${DOMAIN_PART}" &>/dev/null } From bcee78e2c1c310c4815b3e03c5e9278ad8ef1343 Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Mon, 17 Feb 2025 11:17:29 +1300 Subject: [PATCH 208/267] docs: Revise Rspamd page (#4360) --- docs/content/config/security/rspamd.md | 257 +++++++++++++++---------- 1 file changed, 159 insertions(+), 98 deletions(-) diff --git a/docs/content/config/security/rspamd.md b/docs/content/config/security/rspamd.md index c49baea6..45f94c65 100644 --- a/docs/content/config/security/rspamd.md +++ b/docs/content/config/security/rspamd.md @@ -6,34 +6,60 @@ title: 'Security | Rspamd' Rspamd is a ["fast, free and open-source spam filtering system"][rspamd-web]. DMS integrates Rspamd like any other service. We provide a basic but easy to maintain setup of Rspamd. -If you want to take a look at the default configuration files for Rspamd that DMS packs, navigate to [`target/rspamd/` inside the repository][dms-repo::default-rspamd-configuration]. Please consult the [section "The Default Configuration"](#the-default-configuration) section down below for a written overview. +If you want to take a look at the default configuration files for Rspamd that DMS adds, navigate to [`target/rspamd/` inside the repository][dms-repo::default-rspamd-configuration]. Please consult the [section "The Default Configuration"](#the-default-configuration) section down below for a written overview. -## Related Environment Variables +### Enable Rspamd -The following environment variables are related to Rspamd: +Rspamd is presently opt-in for DMS, but intended to become the default anti-spam service in a future release. -1. [`ENABLE_RSPAMD`](../environment.md#enable_rspamd) -2. [`ENABLE_RSPAMD_REDIS`](../environment.md#enable_rspamd_redis) -3. [`RSPAMD_CHECK_AUTHENTICATED`](../environment.md#rspamd_check_authenticated) -4. [`RSPAMD_GREYLISTING`](../environment.md#rspamd_greylisting) -5. [`RSPAMD_HFILTER`](../environment.md#rspamd_hfilter) -6. [`RSPAMD_HFILTER_HOSTNAME_UNKNOWN_SCORE`](../environment.md#rspamd_hfilter_hostname_unknown_score) -7. [`RSPAMD_LEARN`](../environment.md#rspamd_learn) -8. [`SPAM_SUBJECT`](../environment.md#spam_subject) -9. [`MOVE_SPAM_TO_JUNK`][docs::spam-to-junk] -10. [`MARK_SPAM_AS_READ`](../environment.md#mark_spam_as_read) +DMS offers two anti-spam solutions: -With these variables, you can enable Rspamd itself, and you can enable / disable certain features related to Rspamd. +- Legacy (_Amavis, SpamAssassin, OpenDKIM, OpenDMARC_) +- Rspamd (_Provides equivalent features of software from the legacy solution_) -## The Default Configuration +While you could configure Rspamd to only replace some of the legacy services, it is advised to only use Rspamd with the legacy services disabled. -### Other Anti-Spam-Services +!!! example "Switch to Rspamd" -DMS packs other anti-spam services, like SpamAssassin or Amavis, next to Rspamd. There exist services, like ClamAV (`ENABLE_CLAMAV`), that Rspamd can utilize to improve the scanning. Except for ClamAV, we recommend disabling **all other** anti-spam services when using Rspamd. The [basic configuration shown below](#a-very-basic-configuration) provides a good starting point. + To use Rspamd add the following ENV config changes: + + ```env + ENABLE_RSPAMD=1 + + # Rspamd replaces the functionality of all these anti-spam services, disable them: + ENABLE_OPENDKIM=0 + ENABLE_OPENDMARC=0 + ENABLE_POLICYD_SPF=0 + ENABLE_AMAVIS=0 + ENABLE_SPAMASSASSIN=0 + # Greylisting is opt-in, if you had enabled Postgrey switch to the Rspamd equivalent: + ENABLE_POSTGREY=0 + RSPAMD_GREYLISTING=1 + + # Optional: Add anti-virus support with ClamAV (compatible with Rspamd): + ENABLE_CLAMAV=1 + ``` + +!!! info "Relevant Environment Variables" + + The following environment variables are related to Rspamd: + + 1. [`ENABLE_RSPAMD`](../environment.md#enable_rspamd) + 2. [`ENABLE_RSPAMD_REDIS`](../environment.md#enable_rspamd_redis) + 3. [`RSPAMD_CHECK_AUTHENTICATED`](../environment.md#rspamd_check_authenticated) + 4. [`RSPAMD_GREYLISTING`](../environment.md#rspamd_greylisting) + 5. [`RSPAMD_HFILTER`](../environment.md#rspamd_hfilter) + 6. [`RSPAMD_HFILTER_HOSTNAME_UNKNOWN_SCORE`](../environment.md#rspamd_hfilter_hostname_unknown_score) + 7. [`RSPAMD_LEARN`](../environment.md#rspamd_learn) + 8. [`SPAM_SUBJECT`](../environment.md#spam_subject) + 9. [`MOVE_SPAM_TO_JUNK`][docs::spam-to-junk] + 10. [`MARK_SPAM_AS_READ`](../environment.md#mark_spam_as_read) + +## Overview of Rspamd support ### Mode of Operation -!!! tip "Attention" +!!! note "Attention" Read this section carefully if you want to understand how Rspamd is integrated into DMS and how it works (on a surface level). @@ -79,13 +105,16 @@ DMS does not set a default password for the controller worker. You may want to d When Rspamd is enabled, we implicitly also start an instance of Redis in the container: - Redis is configured to persist its data via RDB snapshots to disk in the directory `/var/lib/redis` (_or the [`/var/mail-state/`][docs::dms-volumes-state] volume when present_). -- With the volume mount, the snapshot will restore the Redis data across container restarts, and provide a way to keep backup. +- With the volume mount, the snapshot will restore the Redis data across container updates, and provide a way to keep a backup. +- Without a volume mount a containers internal state will persist across restarts until the container is recreated due to changes like ENV or upgrading the image for the container. Redis uses `/etc/redis/redis.conf` for configuration: - We adjust this file when enabling the internal Redis service. - If you have an external instance of Redis to use, the internal Redis service can be opt-out via setting the ENV [`ENABLE_RSPAMD_REDIS=0`][docs::env::enable-redis] (_link also details required changes to the DMS Rspamd config_). +If you are interested in using Valkey instead of Redis, please refer to [this guidance][gh-dms::guide::valkey]. + ### Web Interface Rspamd provides a [web interface][rspamd-docs::web-ui], which contains statistics and data Rspamd collects. The interface is enabled by default and reachable on port 11334. @@ -96,7 +125,7 @@ To use the web interface you will need to configure a password, [otherwise you w ??? example "Set a custom password" - Add this line to [your rspamd `custom-commands.conf` config](#with-the-help-of-a-custom-file) which sets the `password` option of the _controller worker_: + Add this line to [your Rspamd `custom-commands.conf` config](#with-the-help-of-a-custom-file) which sets the `password` option of the _controller worker_: ``` set-option-for-controller password "your hashed password here" @@ -108,9 +137,13 @@ To use the web interface you will need to configure a password, [otherwise you w docker exec -it rspamadm pw ``` + --- + + **Related:** A minimal Rspamd `compose.yaml` [example with a reverse-proxy for web access][gh-dms:guide::rspamd-web]. + ### DNS -DMS does not supply custom values for DNS servers (to Rspamd). If you need to use custom DNS servers, which could be required when using [DNS-based deny/allowlists](#rbls-real-time-blacklists-dnsbls-dns-based-blacklists), you need to adjust [`options.inc`][rspamd-docs::basic-options] yourself. Make sure to also read our [FAQ page on DNS servers][docs::faq::dns-servers]. +DMS does not supply custom values for DNS servers (to Rspamd). If you need to use custom DNS servers, which could be required when using [DNS-based deny/allowlists](#rbls-real-time-blacklists-dnsbls-dns-based-blacklists), you need to adjust [`options.inc`][rspamd-docs::config::global] yourself. Make sure to also read our [FAQ page on DNS servers][docs::faq::dns-servers]. !!! warning @@ -142,7 +175,7 @@ You can choose to enable ClamAV, and Rspamd will then use it to check for viruse #### RBLs (Real-time Blacklists) / DNSBLs (DNS-based Blacklists) -The [RBL module][rspamd-docs::modules::rbl] is enabled by default. As a consequence, Rspamd will perform DNS lookups to various blacklists. Whether an RBL or a DNSBL is queried depends on where the domain name was obtained: RBL servers are queried with IP addresses extracted from message headers, DNSBL server are queried with domains and IP addresses extracted from the message body \[[source][www::rbl-vs-dnsbl]\]. +The [RBL module][rspamd-docs::modules::rbl] is enabled by default. As a consequence, Rspamd will perform DNS lookups to various blacklists. Whether an RBL or a DNSBL is queried depends on where the domain name was obtained: RBL servers are queried with IP addresses extracted from message headers, DNSBL server are queried with domains and IP addresses extracted from the message body ([source][www::rbl-vs-dnsbl]). !!! danger "Rspamd and DNS Block Lists" @@ -152,124 +185,141 @@ The [RBL module][rspamd-docs::modules::rbl] is enabled by default. As a conseque ## Providing Custom Settings & Overriding Settings -DMS brings sane default settings for Rspamd. They are located at `/etc/rspamd/local.d/` inside the container (or `target/rspamd/local.d/` in the repository). +!!! info "Rspamd config overriding precedence" -### Manually + Rspamd has a layered approach for configuration with [`local.d` and `override.d` config directories][rspamd-docs::config-directories]. -!!! question "What is [`docker-data/dms/config/`][docs::dms-volumes-config]?" + - DMS [extends the Rspamd default configs via `/etc/rspamd/local.d/`][dms-repo::default-rspamd-configuration]. + - User config changes should be handled separately as overrides via the [DMS Config Volume][docs::dms-volumes-config] (`docker-data/dms/config/`) with either: + - `./rspamd/override.d/` - Config files placed here are copied to `/etc/rspamd/override.d/` during container startup. + - [`./rspamd/custom-commands.conf`](#with-the-help-of-a-custom-file) - Applied after copying any provided configs from `rspamd/override.d/` (DMS Config volume) to `/etc/rspamd/override.d/`. -If you want to overwrite the default settings or provide your settings, you can place files at `docker-data/dms/config/rspamd/override.d/`. Files from this directory are copied to `/etc/rspamd/override.d/` during startup. These files [forcibly override][rspamd-docs::override-dir] Rspamd and DMS default settings. +!!! abstract "Reference docs for Rspamd config" -!!! question "What is the [`local.d` directory and how does it compare to `override.d`][rspamd-docs::config-directories]?" + - [Config Overview][rspamd-docs::config::overview], [Quickstart guide][rspamd-docs::config::quickstart], and [Config Syntax (UCL)][rspamd-docs::config::ucl-syntax] + - Global Options ([`options.inc`][rspamd-docs::config::global]) + - [Workers][rspamd-docs::config::workers] ([`worker-controller.inc`][rspamd-docs::config::worker-controller], [`worker-proxy.inc`][rspamd-docs::config::worker-proxy]) + - [Modules][rspamd-docs::modules] (_view each module page for their specific config options_) -!!! warning "Clashing Overrides" +!!! tip "View rendered config" - Note that when also [using the `custom-commands.conf` file](#with-the-help-of-a-custom-file), files in `override.d` may be overwritten in case you adjust them manually and with the help of the file. + `rspamadm configdump` will output the full rspamd configuration that is used should you need it for troubleshooting / inspection. -### With the Help of a Custom File + - You can also see which modules are enabled / disabled via `rspamadm configdump --modules-state` + - Specific config sections like `dkim` or `worker` can also be used to filter the output to just those sections: `rspamadm configdump dkim worker` + - Use `--show-help` to include inline documentation for many settings. -DMS provides the ability to do simple adjustments to Rspamd modules with the help of a single file. Just place a file called `custom-commands.conf` into `docker-data/dms/config/rspamd/`. If this file is present, DMS will evaluate it. The structure is simple, as each line in the file looks like this: +### Using `custom-commands.conf` { #with-the-help-of-a-custom-file } -```txt -COMMAND ARGUMENT1 ARGUMENT2 ARGUMENT3 -``` +For convenience DMS provides a single config file that will directly create or modify multiple configs at `/etc/rspamd/override.d/`. This is handled as the final rspamd configuration step during container startup. -where `COMMAND` can be: +DMS will apply this config when you provide `rspamd/custom-commands.conf` in your DMS Config volume. Configure it with directive lines as documented below. -1. `disable-module`: disables the module with name `ARGUMENT1` -2. `enable-module`: explicitly enables the module with name `ARGUMENT1` -3. `set-option-for-module`: sets the value for option `ARGUMENT2` to `ARGUMENT3` inside module `ARGUMENT1` -4. `set-option-for-controller`: set the value of option `ARGUMENT1` to `ARGUMENT2` for the controller worker -5. `set-option-for-proxy`: set the value of option `ARGUMENT1` to `ARGUMENT2` for the proxy worker -6. `set-common-option`: set the option `ARGUMENT1` that [defines basic Rspamd behavior][rspamd-docs::basic-options] to value `ARGUMENT2` -7. `add-line`: this will add the complete line after `ARGUMENT1` (with all characters) to the file `/etc/rspamd/override.d/` +!!! note "Only use this feature for `option = value` changes" -!!! example "An Example Is [Shown Down Below](#adjusting-and-extending-the-very-basic-configuration)" + `custom-commands.conf` is only suitable for adding or replacing simple `option = value` settings for configs at `/etc/rspamd/override.d/`. + + - New settings are appended to the associated config file. + - When replacing an existing setting in an override config, that setting may be any matching line (_allowing for nested scopes, instead of only top-level keys_). + + Any changes involving more advanced [UCL config syntax][rspamd-docs::config::ucl-syntax] should instead add UCL config files directly to `rspamd/override.d/` (_in the DMS Config volume_). -!!! note "File Names & Extensions" +!!! info "`custom-commands.conf` syntax" - For command 1 - 3, we append the `.conf` suffix to the module name to get the correct file name automatically. For commands 4 - 6, the file name is fixed (you don't even need to provide it). For command 7, you will need to provide the whole file name (including the suffix) yourself! + There are 7 directives available to manage custom Rspamd configurations. Add these directive lines into `custom-commands.conf`, they will be processed sequentially. -You can also have comments (the line starts with `#`) and blank lines in `custom-commands.conf` - they are properly handled and not evaluated. + **Directives:** -!!! tip "Adjusting Modules This Way" + ```txt + # For /etc/rspamd/override.d/{options.inc,worker-controller.inc,worker-proxy}.inc + set-common-option