chore: addmailuser - Remove delaying completion until /var/mail is ready (#2729)

## Quick Summary

Resolves a `TODO` task with `addmailuser`.

## Overview

The main change is adding three new methods in `common.bash`, which replace the completion delay in `addmailuser` / `setup email add` command.

Other than that:

- I swapped `sh -c 'addmailuser ...'` to `setup email add ...`.
- Improved three tests in `setup-cli.bats` for `setup email add|update|del` (_logic remains effectively the same still_).
- Rewrote the `TODO` comment for `setup-cli.bats` test on `setup email del` to better clarify the concern, but the test itself was no longer affected due to changes prior to this PR, so I enabled the commented out assertion.
- Removed unnecessary waits. The two `skip` tests in `test/tests.bats` could be enabled again after this PR.
- Additional fixes to tests were made during the PR (see discussion comments for details), resolving race conditions.

Individual commit messages of the PR provide additional details if helpful.

---

## Relevant commit messages

* chore: Remove creation delay in `addmailuser`

This was apparently only for supporting tests that need to wait on account creation being ready to test against.

As per the removed inline docs, it should be fine to remove once tests are updated to work correctly without it.

* tests(feat): Add two new common helper methods

`wait_until_account_maildir_exists()` provides the same logic `addmailuser` command was carrying, to wait upon the account dir creation in `/var/mail`.

As this was specifically to support tests, it makes more sense as a test method.

`add_mail_account_then_wait_until_ready()` was added to handle the common pattern of creating account and waiting on it. An internal assert will ensure the account was successfully created first during the test before attempting to wait.

* tests(feat): Add common helper for waiting on change event to be processed

The current helper is more complicated for no real benefit, it only detects when a change is made that would trigger a change event in the `changedetector` service. Our usage of this in tests however is only interested in waiting out the completion of the change event.

Remove unnecessary change event waits. These waits should not be necessary if handled correctly.

* tests: `addmailuser` to `add_mail_account_then_wait_until_ready mail()`

This helper method is used where appropriate.
- A password is not relevant (optional).
- We need to wait on the creation on the account (Dovecot and `/var/mail` directory).

* tests: `setup-cli` revise `add`, `update`, `del` tests

The delete test was failing as the `/var/mail` directory did not yet exist.

There is now a proper delay imposed in the `add` test now shares the same account for both `update` and `del` tests resolving that failure.

Additionally tests use better asserts where appropriate and the wait + sleep logic in `add` has been improved (now takes 10 seconds to complete, approx half the time than before).

The `del` test TODO while not technically addressed is no longer relevant due to the tests being switched to `-c` option (there is a separate `no container` test file, but it doesn't provide a `del` test).

* tests(fix): Ensure Postfix is reachable after waiting on ClamAV

There is not much reason to check before waiting on ClamAV.

It is more helpful to debug failures from `nc` mail send commands if we know that nothing went wrong inbetween the ClamAV wait time.

Additionally added an assertion which should provide more information if this part of the test setup fails again.

* tests(fix): Move health check to the top

This test is a bit fragile. It relies on defaults for the healthcheck with intervals of 30 seconds.

If the check occurs while Postfix is down due a change event from earlier tests and the healthcheck kicks in at that point, then if there is not enough time to refresh the health status from `unhealthy`, the test will fail with a false-positive as Postfix is actually working and up again..

* tests(fix): Wait on directory to be removed

Workaround that tries not to introduce heavier delays by waiting on a full change event to complete in the previous `email update` if possible.

There is a chance that the account has the folder deleted, but restored from an active change event (for password update, then the account delete).

test/test_helper/common.bash

@@ -173,6 +173,73 @@ function wait_for_changes_to_be_detected_in_container() {
   repeat_in_container_until_success_or_timeout "${TIMEOUT}" "${CONTAINER_NAME}" bash -c 'source /usr/local/bin/helpers/index.sh; _obtain_hostname_and_domainname; cmp --silent -- <(_monitored_files_checksums) "${CHKSUM_FILE}" >/dev/null'
 }
 
+# Relies on ENV `LOG_LEVEL=debug` or higher
+function wait_until_change_detection_event_completes() {
+  local CONTAINER_NAME="${1}"
+  # Ensure early failure if arg is missing:
+  assert_not_equal "${CONTAINER_NAME}" ""
+
+  # Ensure the container is configured with the required `LOG_LEVEL` ENV:
+  assert_regex \
+    $(docker exec "${CONTAINER_NAME}" env | grep '^LOG_LEVEL=') \
+    '=(debug|trace)$'
+
+  local CHANGE_EVENT_START='Change detected'
+  local CHANGE_EVENT_END='Completed handling of detected change' # debug log
+
+  function __change_event_status() {
+    docker exec "${CONTAINER_NAME}" \
+      grep -oE "${CHANGE_EVENT_START}|${CHANGE_EVENT_END}" /var/log/supervisor/changedetector.log \
+      | tail -1
+  }
+
+  function __is_changedetector_processing() {
+    [[ $(__change_event_status) == "${CHANGE_EVENT_START}" ]]
+  }
+
+  function __is_changedetector_finished() {
+    [[ $(__change_event_status) == "${CHANGE_EVENT_END}" ]]
+  }
+
+  if [[ ! $(__is_changedetector_processing) ]]
+  then
+    # A new change event is expected, wait for it:
+    repeat_until_success_or_timeout 60 __is_changedetector_processing
+  fi
+
+  # Change event is in progress, wait until it finishes:
+  repeat_until_success_or_timeout 60 __is_changedetector_finished
+
+  # NOTE: Although the change event has completed, services like Postfix and Dovecot
+  # may still be in the process of restarting.
+  # You may still want to wait longer if depending on those to be ready.
+}
+
+# An account added to `postfix-accounts.cf` must wait for the `changedetector` service
+# to process the update before Dovecot creates the mail account and associated storage dir:
+function wait_until_account_maildir_exists() {
+  local CONTAINER_NAME=$1
+  local MAIL_ACCOUNT=$2
+
+  local LOCAL_PART="${MAIL_ACCOUNT%@*}"
+  local DOMAIN_PART="${MAIL_ACCOUNT#*@}"
+  local MAIL_ACCOUNT_STORAGE_DIR="/var/mail/${DOMAIN_PART}/${LOCAL_PART}"
+
+  repeat_in_container_until_success_or_timeout 60 "${CONTAINER_NAME}" bash -c "[[ -d ${MAIL_ACCOUNT_STORAGE_DIR} ]]"
+}
+
+function add_mail_account_then_wait_until_ready() {
+  local CONTAINER_NAME=$1
+  local MAIL_ACCOUNT=$2
+  # Password is optional (omit when the password is not needed during the test)
+  local MAIL_PASS="${3:-password_not_relevant_to_test}"
+
+  run docker exec "${CONTAINER_NAME}" setup email add "${MAIL_ACCOUNT}" "${MAIL_PASS}"
+  assert_success
+
+  wait_until_account_maildir_exists "${CONTAINER_NAME}" "${MAIL_ACCOUNT}"
+}
+
 function wait_for_empty_mail_queue_in_container() {
   local CONTAINER_NAME="${1}"
   local TIMEOUT=${TEST_TIMEOUT_IN_SECONDS}