docker-mailserver.docker-ma.../test/tests/serial/mail_changedetector.bats
Brennan Kinney 8d80c6317f
tests(refactor): Adjust mail_changedetector + change detection helpers (#2997)
* tests(refactor): `mail_changedetector.bats` - Leverage DRY methods

`supervisorctl tail` is not the most reliably way to get logs for the latest change detection and has been known to be fragile in the past.

We can instead read the full log for the service directly with `tac` and `sed` to extract all log content since the last change detection.

Common asserts have also been extracted out into separate methods.

* tests(chore): Remove sleep and redundant change event

Container 1 is still blocked at this point from an existing lock and change event.

Make the lock stale immediately and no extra sleep is required when paired with the helper method to wait until the event is processed (which should remove the stale lock).

* tests(refactor): Add more DRY methods

- Simplify the test case so it's easier to grok.
- 2nd test case (blocking) extracts out initial setup into a separate method and merges the later service restart logic which is redundant.
- Additional comments for improved context of what is going on / expected.

* tests(chore): Revise the change detection helper method

- Add explicit counting arg to change detection support.
- Extract revised logic into it's own generic helper method.
- Utilize this for a separate method that monitors for a change event having started, but not waiting for completion.

This allows dropping the 40 sec of remaining `sleep` in `mail_changedetector` test. It was also required due to potentially missing the timing of a change event completing concurrently in a 2nd container that needed to be waited on and then checked.

* tests(chore): Migrate to current test conventions

- Switch to common container setup helpers
- Update container name and change usage to variables instead.
- Adopt the new convention of prefix variable for test cases (revised test case descriptions).

* tests(chore): Remove legacy change detection

This has since been replaced with the new helper watches the `changedetector` service logs directly instead of only detecting a change has occurred via checksum comparison.

No tests use this method anymore, it was originally for `tests.bats`. Thus the tests in `test_helper.bats` are being dropped too. The new helper has test coverage in `changedetector` tests.

* chore: Lock removal should not incur `sleep 5` afterwards

- A new lock should be created by this script after removal. The sleep doesn't help avoid a race condition with lock file creation after removal.
- Reduces test time as a bonus.
- Added some additional comments to test.

* tests(chore): `tls_letsencrypt.bats` leverage improved change detection

- No need to wait on the change detection service anymore during container startup.
- No need to count change events processed either as waiting a fixed duration is no longer relied on.
- This makes the reload count method redundant, dropped.

* tests(chore): Convert `setup-cli.bats` to new test conventions

This test file was already adapted to the original common setup helpers.

- `TEST_NAME` replaced with `CONTAINER_NAME`.
- Prefix var added, test case descriptions drop explicit prefix.
- No other changes.

* tests(chore): Extract out helpers related to change-detection

- New helper file for sharing these helpers to tests.
- Includes the helpful log method from changedetector tests.
- No longer need to maintain duplicate copies of these methods during the test migration. All tests that use them are now importing the separate helper file.
- `tls_letsencrypt.bats` has switched to using the log helper.
- Generic log count helper is removed from `test_helper/common.bash` as any test that needs it in future can adapt to `helper/common.bash`.

* tests(refactor): `tls_letsencrypt.bats` remove `_get_service_logs()`

This helper does not seem useful as moving away from `supervisorctl tail` and no other tests had a need for it.

* tests(chore): Remove common setup methods from `test_helper/common.bash`

No other tests depend on this. Future tests will adopt the revised versions from `helper/setup.bash`.

Additionally updates `helper/setup.bash` comments that are no longer applicable to `TEST_TMP_CONFIG` and `CONTAINER_NAME`.

* chore: Use `|| true` to simplify setting `EXPECTED_COUNT` correctly
2023-01-16 20:39:46 +13:00

139 lines
5.7 KiB
Bash

load "${REPOSITORY_ROOT}/test/helper/common"
load "${REPOSITORY_ROOT}/test/helper/change-detection"
load "${REPOSITORY_ROOT}/test/helper/setup"
BATS_TEST_NAME_PREFIX='[Change Detection] '
CONTAINER1_NAME='dms-test_changedetector_one'
CONTAINER2_NAME='dms-test_changedetector_two'
function setup_file() {
export CONTAINER_NAME
local CUSTOM_SETUP_ARGUMENTS=(
--env LOG_LEVEL=trace
)
CONTAINER_NAME=${CONTAINER1_NAME}
init_with_defaults
common_container_setup 'CUSTOM_SETUP_ARGUMENTS'
CONTAINER_NAME=${CONTAINER2_NAME}
# NOTE: No `init_with_defaults` used here,
# Intentionally sharing previous containers config instead.
common_container_setup 'CUSTOM_SETUP_ARGUMENTS'
# Set default implicit container fallback for helpers:
CONTAINER_NAME=${CONTAINER1_NAME}
}
function teardown_file() {
docker rm -f "${CONTAINER1_NAME}" "${CONTAINER2_NAME}"
}
@test "changedetector service is ready" {
wait_for_service "${CONTAINER1_NAME}" changedetector
wait_for_service "${CONTAINER2_NAME}" changedetector
}
# NOTE: Non-deterministic behaviour - One container will perform change detection before the other.
# Depending on the timing of the other container checking for a lock file, the lock may no longer be
# present, avoiding the 5 second delay. The first container to create a lock is not deterministic either.
# NOTE: Change detection at this point typically occurs at 2 or 4 seconds since the service was up,
# thus expect 2-8 seconds to complete.
@test "should detect and process changes (in both containers with shared config)" {
_create_change_event
_should_perform_standard_change_event "${CONTAINER1_NAME}"
_should_perform_standard_change_event "${CONTAINER2_NAME}"
}
# Both containers should acknowledge the foreign lock file added,
# blocking an attempt to process the pending change event detected.
@test "should find existing lock file and block processing changes (without removing lock)" {
_prepare_blocking_lock_test
# Wait until the 2nd change event attempts to process:
_should_block_change_event_from_processing "${CONTAINER1_NAME}" 2
# NOTE: Although the service is restarted, a change detection should still occur (previous checksum still exists):
_should_block_change_event_from_processing "${CONTAINER2_NAME}" 1
}
@test "should remove lock file when stale" {
# Avoid a race condition (to remove the lock file) by removing the 2nd container:
docker rm -f "${CONTAINER2_NAME}"
# Make the previously created lock file become stale:
docker exec "${CONTAINER1_NAME}" touch -d '60 seconds ago' /tmp/docker-mailserver/check-for-changes.sh.lock
# A 2nd change event should complete (or may already have if quick enough?):
wait_until_change_detection_event_completes "${CONTAINER1_NAME}" 2
# Should have removed the stale lock file, then handle the change event:
run _get_logs_since_last_change_detection "${CONTAINER1_NAME}"
assert_output --partial 'Lock file older than 1 minute - removing stale lock file'
_assert_has_standard_change_event_logs
}
function _should_perform_standard_change_event() {
local CONTAINER_NAME=${1}
# Wait for change detection event to start and complete processing:
# NOTE: An explicit count is provided as the 2nd container may have already completed processing.
wait_until_change_detection_event_completes "${CONTAINER_NAME}" 1
# Container should have created it's own lock file,
# and later removed it when finished processing:
run _get_logs_since_last_change_detection "${CONTAINER_NAME}"
_assert_has_standard_change_event_logs
}
function _should_block_change_event_from_processing() {
local CONTAINER_NAME=${1}
local EXPECTED_COUNT=${2}
# Once the next change event has started, the processing blocked log ('another execution') should be present:
wait_until_change_detection_event_begins "${CONTAINER_NAME}" "${EXPECTED_COUNT}"
run _get_logs_since_last_change_detection "${CONTAINER_NAME}"
_assert_foreign_lock_exists
# This additionally verifies that the change event processing is incomplete (blocked):
_assert_no_lock_actions_performed
}
function _assert_has_standard_change_event_logs() {
assert_output --partial "Creating lock '/tmp/docker-mailserver/check-for-changes.sh.lock'"
assert_output --partial 'Reloading services due to detected changes'
assert_output --partial 'Removed lock'
assert_output --partial 'Completed handling of detected change'
}
function _assert_foreign_lock_exists() {
assert_output --partial "Lock file '/tmp/docker-mailserver/check-for-changes.sh.lock' exists"
assert_output --partial "- another execution of 'check-for-changes.sh' is happening - trying again shortly"
}
function _assert_no_lock_actions_performed() {
refute_output --partial 'Lock file older than 1 minute - removing stale lock file'
refute_output --partial "Creating lock '/tmp/docker-mailserver/check-for-changes.sh.lock'"
refute_output --partial 'Removed lock'
}
function _prepare_blocking_lock_test {
# Temporarily disable the Container2 changedetector service:
docker exec "${CONTAINER2_NAME}" bash -c 'supervisorctl stop changedetector'
docker exec "${CONTAINER2_NAME}" bash -c 'rm -f /var/log/supervisor/changedetector.log'
# Create a foreign lock file to prevent change processing (in both containers):
docker exec "${CONTAINER1_NAME}" bash -c 'touch /tmp/docker-mailserver/check-for-changes.sh.lock'
# Create a new change to detect (that the foreign lock should prevent from processing):
_create_change_event
# Restore Container2 changedetector service:
# NOTE: The last known checksum is retained in Container2,
# It will be compared to and start a change event.
docker exec "${CONTAINER2_NAME}" bash -c 'supervisorctl start changedetector'
}
function _create_change_event() {
echo '' >>"${TEST_TMP_CONFIG}/postfix-accounts.cf"
}