Merge branch 'netwatch' into next

This commit is contained in:
Christian Hesse 2022-07-06 11:42:47 +02:00
commit 99feceda38
9 changed files with 74 additions and 53 deletions

View file

@ -57,7 +57,7 @@ Tips & Tricks
Netwatch entries can be created to work with both - this script and Netwatch entries can be created to work with both - this script and
[netwatch-notify](netwatch-notify.md). Just give options for both: [netwatch-notify](netwatch-notify.md). Just give options for both:
/tool/netwatch/add comment="doh, notify, hostname=cloudflare-dns" host=1.1.1.1; /tool/netwatch/add comment="doh, notify, name=cloudflare-dns" host=1.1.1.1;
Also this allows to update host address, see option `resolve`. Also this allows to update host address, see option `resolve`.

View file

@ -159,10 +159,10 @@
</tspan><tspan </tspan><tspan
x="180" x="180"
y="40.85" y="40.85"
id="tspan2281">Host example.com (93.184.216.34) is down since id="tspan2281">The host 'example.com' (93.184.216.34) is down
</tspan><tspan </tspan><tspan
x="180" x="180"
y="55.85" y="55.85"
id="tspan2283">jun/08/2021 06:55:03.</tspan></text> id="tspan2283">since jun/08/2021 06:55:03.</tspan></text>
</g> </g>
</svg> </svg>

Before

Width:  |  Height:  |  Size: 8.1 KiB

After

Width:  |  Height:  |  Size: 8.1 KiB

Before After
Before After

View file

@ -159,11 +159,11 @@
</tspan><tspan </tspan><tspan
x="180" x="180"
y="40.85" y="40.85"
id="tspan2246">Host example.com (93.184.216.34) is up since id="tspan2246">The host 'example.com' (93.184.216.34) is up
</tspan><tspan </tspan><tspan
x="180" x="180"
y="55.85" y="55.85"
id="tspan2248">jun/08/2021 07:01:00. id="tspan2248">since jun/08/2021 07:01:00.
</tspan><tspan </tspan><tspan
x="180" x="180"
y="70.85" y="70.85"

Before

Width:  |  Height:  |  Size: 8.2 KiB

After

Width:  |  Height:  |  Size: 8.2 KiB

Before After
Before After

View file

@ -36,7 +36,7 @@ Configuration
The hosts to be checked have to be added to netwatch with specific comment: The hosts to be checked have to be added to netwatch with specific comment:
/tool/netwatch/add comment="notify, hostname=example.com" host=[ :resolve "example.com" ]; /tool/netwatch/add comment="notify, name=example.com" host=[ :resolve "example.com" ];
### Hooks ### Hooks
@ -44,7 +44,7 @@ It is possible to run an up hook command (`up-hook`) or down hook command
(`down-hook`) when a notification is triggered. This has to be added in (`down-hook`) when a notification is triggered. This has to be added in
comment, note that some characters need extra escaping: comment, note that some characters need extra escaping:
/tool/netwatch/add comment=("notify, hostname=device, down-hook=/interface/ethernet \\{ disable \\\"en2\\\"; enable \\\"en2\\\"; \\}") host=10.0.0.20; /tool/netwatch/add comment=("notify, name=device, down-hook=/interface/ethernet \\{ disable \\\"en2\\\"; enable \\\"en2\\\"; \\}") host=10.0.0.20;
Also there is a `pre-down-hook` that fires at two thirds of failed checks Also there is a `pre-down-hook` that fires at two thirds of failed checks
required for the notification. The idea is to fix the issue before a required for the notification. The idea is to fix the issue before a
@ -54,15 +54,15 @@ notification is sent.
The count threshould (default is 5 checks) is configurable as well: The count threshould (default is 5 checks) is configurable as well:
/tool/netwatch/add comment="notify, hostname=example.com, count=10" host=104.18.144.11; /tool/netwatch/add comment="notify, name=example.com, count=10" host=104.18.144.11;
### Parents & dependencies ### Parents & dependencies
If the host is behind another checked host add a dependency, this will If the host is behind another checked host add a dependency, this will
suppress notification if the parent host is down: suppress notification if the parent host is down:
/tool/netwatch/add comment="notify, hostname=gateway" host=93.184.216.1; /tool/netwatch/add comment="notify, name=gateway" host=93.184.216.1;
/tool/netwatch/add comment="notify, hostname=example.com, parent=gateway" host=93.184.216.34; /tool/netwatch/add comment="notify, name=example.com, parent=gateway" host=93.184.216.34;
Note that every configured parent in a chain increases the check count Note that every configured parent in a chain increases the check count
threshould by one. threshould by one.
@ -72,7 +72,7 @@ threshould by one.
The host address can be updated dynamically. Give extra parameter `resolve` The host address can be updated dynamically. Give extra parameter `resolve`
with a resolvable name: with a resolvable name:
/tool/netwatch/add comment="notify, hostname=example.com, resolve=example.com"; /tool/netwatch/add comment="notify, name=example.com, resolve=example.com";
But be warned: Dynamic updates will probably cause issues if the name has But be warned: Dynamic updates will probably cause issues if the name has
more than one record in dns - a high rate of configuration changes (and flash more than one record in dns - a high rate of configuration changes (and flash
@ -84,7 +84,7 @@ Also suppressing the notification on host down is possible with parameter
`no-down-notification`. This may be desired for devices that are usually `no-down-notification`. This may be desired for devices that are usually
powered off, but accessibility is of interest. powered off, but accessibility is of interest.
/tool/netwatch/add comment="notify, hostname=printer, no-down-notification" host=10.0.0.30; /tool/netwatch/add comment="notify, name=printer, no-down-notification" host=10.0.0.30;
Go and get your coffee ☕️ before sending the print job. Go and get your coffee ☕️ before sending the print job.
@ -99,10 +99,10 @@ Tips & Tricks
Sometimes it is sufficient if one of a number of hosts is available. You can Sometimes it is sufficient if one of a number of hosts is available. You can
make `netwatch-notify` check for that by adding several items with same make `netwatch-notify` check for that by adding several items with same
`hostname`. Note that `count` has to be multiplied to keep the actual time. `name`. Note that `count` has to be multiplied to keep the actual time.
/tool/netwatch/add comment="notify, hostname=service, count=10" host=10.0.0.10; /tool/netwatch/add comment="notify, name=service, count=10" host=10.0.0.10;
/tool/netwatch/add comment="notify, hostname=service, count=10" host=10.0.0.20; /tool/netwatch/add comment="notify, name=service, count=10" host=10.0.0.20;
### Checking internet connectivity ### Checking internet connectivity
@ -112,11 +112,11 @@ check `1.1.1.1` (Cloudflare DNS), `9.9.9.9` (Quad-nine DNS), `8.8.8.8`
(Google DNS) or any other reliable address that indicates internet (Google DNS) or any other reliable address that indicates internet
connectivity. connectivity.
/tool/netwatch/add comment="notify, hostname=internet" host=1.1.1.1; /tool/netwatch/add comment="notify, name=internet" host=1.1.1.1;
A target like this suits well to be parent for other checks. A target like this suits well to be parent for other checks.
/tool/netwatch/add comment="notify, hostname=example.com, parent=internet" host=93.184.216.34; /tool/netwatch/add comment="notify, name=example.com, parent=internet" host=93.184.216.34;
### Checking specific ISP ### Checking specific ISP
@ -130,7 +130,7 @@ Create a route and firewall mangle rule.
Finally monitor the address with `netwatch-notify`. Finally monitor the address with `netwatch-notify`.
/tool/netwatch/add comment="notify, hostname=quad-one via isp1" host=1.0.0.1; /tool/netwatch/add comment="notify, name=quad-one via isp1" host=1.0.0.1;
Note that *all* traffic to the given address is routed that way. In case of Note that *all* traffic to the given address is routed that way. In case of
link failure this address is not available, so use something reliable but link failure this address is not available, so use something reliable but
@ -142,7 +142,7 @@ non-essential. In this example the address `1.0.0.1` is used, the same service
Netwatch entries can be created to work with both - this script and Netwatch entries can be created to work with both - this script and
[netwatch-dns](netwatch-dns.md). Just give options for both: [netwatch-dns](netwatch-dns.md). Just give options for both:
/tool/netwatch/add comment="doh, notify, hostname=cloudflare-dns" host=1.1.1.1; /tool/netwatch/add comment="doh, notify, name=cloudflare-dns" host=1.1.1.1;
See also See also
-------- --------

View file

@ -8,7 +8,7 @@
# Make sure all configuration properties are up to date and this # Make sure all configuration properties are up to date and this
# value is in sync with value in script 'global-functions'! # value is in sync with value in script 'global-functions'!
:global GlobalConfigVersion 81; :global GlobalConfigVersion 82;
# This is used for DNS and backup file. # This is used for DNS and backup file.
:global Domain "example.com"; :global Domain "example.com";

View file

@ -8,7 +8,7 @@
# Make sure all configuration properties are up to date and this # Make sure all configuration properties are up to date and this
# value is in sync with value in script 'global-functions'! # value is in sync with value in script 'global-functions'!
# Comment or remove to disable news and change notifications. # Comment or remove to disable news and change notifications.
:global GlobalConfigVersion 81; :global GlobalConfigVersion 82;
# Copy configuration from global-config here and modify it. # Copy configuration from global-config here and modify it.

View file

@ -90,6 +90,7 @@
79="Introduced new script 'backup-partition' to save configuration to fallback partition."; 79="Introduced new script 'backup-partition' to save configuration to fallback partition.";
80="The 'routeros-v7' branch will now freeze, and vanish any time in future. You already switched to 'main' branch, well done!"; 80="The 'routeros-v7' branch will now freeze, and vanish any time in future. You already switched to 'main' branch, well done!";
81="Dropped script 'rotate-ntp', as the limitation does no longer exist."; 81="Dropped script 'rotate-ntp', as the limitation does no longer exist.";
82="Renamed the comment parameter 'hostname' to just 'name' for 'netwatch-notify'.";
}; };
# Migration steps to be applied on script updates # Migration steps to be applied on script updates
@ -103,4 +104,5 @@
67=":global ScriptInstallUpdate; :global CharacterReplace; :foreach Script in=[ /system/script/find where name~\"^global-functions.d/\" ] do={ /system/script/set name=[ \$CharacterReplace [ /system/script/get \$Script name ] \"global-functions.d/\" \"mod/\" ] \$Script; }; \$ScriptInstallUpdate;"; 67=":global ScriptInstallUpdate; :global CharacterReplace; :foreach Script in=[ /system/script/find where name~\"^global-functions.d/\" ] do={ /system/script/set name=[ \$CharacterReplace [ /system/script/get \$Script name ] \"global-functions.d/\" \"mod/\" ] \$Script; }; \$ScriptInstallUpdate;";
73=":global ScriptInstallUpdate; :global CharacterReplace; :foreach Old,New in={ \"cloud-backup\"=\"backup-cloud\"; \"email-backup\"=\"backup-email\"; \"upload-backup\"=\"backup-upload\" } do={ /system/script/set name=\$New [ find where name=\$Old ]; :foreach Scheduler in=[ /system/scheduler/find where on-event~\$Old ] do={ /system/scheduler/set \$Scheduler name=[ \$CharacterReplace [ get \$Scheduler name ] \$Old \$New ] on-event=[ \$CharacterReplace [ get \$Scheduler on-event ] \$Old \$New ]; }; }; \$ScriptInstallUpdate;"; 73=":global ScriptInstallUpdate; :global CharacterReplace; :foreach Old,New in={ \"cloud-backup\"=\"backup-cloud\"; \"email-backup\"=\"backup-email\"; \"upload-backup\"=\"backup-upload\" } do={ /system/script/set name=\$New [ find where name=\$Old ]; :foreach Scheduler in=[ /system/scheduler/find where on-event~\$Old ] do={ /system/scheduler/set \$Scheduler name=[ \$CharacterReplace [ get \$Scheduler name ] \$Old \$New ] on-event=[ \$CharacterReplace [ get \$Scheduler on-event ] \$Old \$New ]; }; }; \$ScriptInstallUpdate;";
81=":global NtpPool; :if ([ :len [ /system/script/find where name=\"rotate-ntp\" ] ] > 0) do={ /system/script/remove [ find where name=\"rotate-ntp\" ]; /system/scheduler/remove [ find where name=\"rotate-ntp\" ]; /system/ntp/client/set servers=\$NtpPool; };"; 81=":global NtpPool; :if ([ :len [ /system/script/find where name=\"rotate-ntp\" ] ] > 0) do={ /system/script/remove [ find where name=\"rotate-ntp\" ]; /system/scheduler/remove [ find where name=\"rotate-ntp\" ]; /system/ntp/client/set servers=\$NtpPool; };";
82=":global CharacterReplace; :foreach Netwatch in=[ /tool/netwatch/find where comment~\"notify\" !disabled ] do={ /tool/netwatch/set \$Netwatch comment=[ \$CharacterReplace [ get \$Netwatch comment ] \"hostname=\" \"name=\" ]; };";
}; };

View file

@ -10,7 +10,7 @@
:local 0 "global-functions"; :local 0 "global-functions";
# expected configuration version # expected configuration version
:global ExpectedConfigVersion 81; :global ExpectedConfigVersion 82;
# global variables not to be changed by user # global variables not to be changed by user
:global GlobalFunctionsReady false; :global GlobalFunctionsReady false;

View file

@ -12,6 +12,7 @@
:global NetwatchNotify; :global NetwatchNotify;
:global EitherOr;
:global IfThenElse; :global IfThenElse;
:global IsDNSResolving; :global IsDNSResolving;
:global LogPrintExit2; :global LogPrintExit2;
@ -21,9 +22,10 @@
:global SymbolForNotification; :global SymbolForNotification;
:local NetwatchNotifyHook do={ :local NetwatchNotifyHook do={
:local Name [ :tostr $1 ]; :local Name [ :tostr $1 ];
:local Type [ :tostr $2 ]; :local Type [ :tostr $2 ];
:local Hook [ :tostr $3 ]; :local State [ :tostr $3 ];
:local Hook [ :tostr $4 ];
:global LogPrintExit2; :global LogPrintExit2;
:global ValidateSyntax; :global ValidateSyntax;
@ -32,15 +34,18 @@
:do { :do {
[ :parse $Hook ]; [ :parse $Hook ];
} on-error={ } on-error={
$LogPrintExit2 warning $0 ("The " . $Type . "-hook for host " . $Name . " failed to run.") false; $LogPrintExit2 warning $0 ("The " . $State . "-hook for " . $Type . " '" . $Name . \
"' failed to run.") false;
:return ("The hook failed to run."); :return ("The hook failed to run.");
} }
} else={ } else={
$LogPrintExit2 warning $0 ("The " . $Type . "-hook for host " . $Name . " failed syntax validation.") false; $LogPrintExit2 warning $0 ("The " . $State . "-hook for " . $Type . " '" . $Name . \
"' failed syntax validation.") false;
:return ("The hook failed syntax validation."); :return ("The hook failed syntax validation.");
} }
$LogPrintExit2 info $0 ("Ran hook on host " . $Name . " " . $Type . ": " . $Hook) false; $LogPrintExit2 info $0 ("Ran hook on " . $Type . " '" . $Name . "' " . $State . ": " . \
$Hook) false;
:return ("Ran hook:\n" . $Hook); :return ("Ran hook:\n" . $Hook);
} }
@ -54,16 +59,17 @@ $ScriptLock $0;
:set NetwatchNotify [ :toarray "" ]; :set NetwatchNotify [ :toarray "" ];
} }
:foreach Host in=[ /tool/netwatch/find where comment~"notify" disabled=no ] do={ :foreach Host in=[ /tool/netwatch/find where comment~"notify" !disabled ] do={
:local HostVal [ /tool/netwatch/get $Host ]; :local HostVal [ /tool/netwatch/get $Host ];
:local Type [ $IfThenElse ($HostVal->"type" ~ "^(http-get|tcp-conn)\$") "service" "host" ];
:local HostInfo [ $ParseKeyValueStore ($HostVal->"comment") ]; :local HostInfo [ $ParseKeyValueStore ($HostVal->"comment") ];
:if ($HostInfo->"notify" = true && $HostInfo->"disabled" != true) do={ :if ($HostInfo->"notify" = true && $HostInfo->"disabled" != true) do={
:local HostName ($HostInfo->"hostname"); :local Name [ $EitherOr ($HostInfo->"name") ($HostVal->"name") ];
:local Metric { "count"=0; "notified"=false }; :local Metric { "count"=0; "notified"=false };
:if ([ :typeof ($NetwatchNotify->$HostName) ] = "array") do={ :if ([ :typeof ($NetwatchNotify->$Name) ] = "array") do={
:set $Metric ($NetwatchNotify->$HostName); :set $Metric ($NetwatchNotify->$Name);
} }
:if ([ :typeof ($HostInfo->"resolve") ] = "str") do={ :if ([ :typeof ($HostInfo->"resolve") ] = "str") do={
@ -71,16 +77,18 @@ $ScriptLock $0;
:do { :do {
:local Resolve [ :resolve ($HostInfo->"resolve") ]; :local Resolve [ :resolve ($HostInfo->"resolve") ];
:if ($Resolve != $HostVal->"host") do={ :if ($Resolve != $HostVal->"host") do={
$LogPrintExit2 info $0 ("Name '" . $HostInfo->"resolve" . [ $IfThenElse ($HostInfo->"resolve" != \ $LogPrintExit2 info $0 ("Name '" . $HostInfo->"resolve" . [ $IfThenElse \
$HostInfo->"hostname") ("' for host '" . $HostInfo->"hostname") "" ] . \ ($HostInfo->"resolve" != $HostInfo->"name") ("' for " . $Type . " '" . \
"' resolves to different address " . $Resolve . ", updating.") false; $HostInfo->"name") "" ] . "' resolves to different address " . $Resolve . \
", updating.") false;
/tool/netwatch/set host=$Resolve $Host; /tool/netwatch/set host=$Resolve $Host;
:set ($Metric->"resolve-failed") false; :set ($Metric->"resolve-failed") false;
} }
} on-error={ } on-error={
:if ($Metric->"resolve-failed" != true) do={ :if ($Metric->"resolve-failed" != true) do={
$LogPrintExit2 warning $0 ("Resolving name '" . $HostInfo->"resolve" . [ $IfThenElse ($HostInfo->"resolve" != \ $LogPrintExit2 warning $0 ("Resolving name '" . $HostInfo->"resolve" . [ $IfThenElse \
$HostInfo->"hostname") ("' for host '" . $HostInfo->"hostname") "" ] . "' failed.") false; ($HostInfo->"resolve" != $HostInfo->"name") ("' for " . $Type . " '" . \
$HostInfo->"name") "" ] . "' failed.") false;
:set ($Metric->"resolve-failed") true; :set ($Metric->"resolve-failed") true;
} }
} }
@ -90,17 +98,21 @@ $ScriptLock $0;
:if ($HostVal->"status" = "up") do={ :if ($HostVal->"status" = "up") do={
:local Count ($Metric->"count"); :local Count ($Metric->"count");
:if ($Count > 0) do={ :if ($Count > 0) do={
$LogPrintExit2 info $0 ("Host " . $HostName . " (" . $HostVal->"host" . ") is up.") false; $LogPrintExit2 info $0 ("The " . $Type . " '" . $Name . "' (" . $HostVal->"host" . \
") is up.") false;
:set ($Metric->"count") 0; :set ($Metric->"count") 0;
} }
:if ($Metric->"notified" = true) do={ :if ($Metric->"notified" = true) do={
:local Message ("Host " . $HostName . " (" . $HostVal->"host" . ") is up since " . $HostVal->"since" . ".\n" . \ :local Message ("The " . $Type . " '" . $Name . "' (" . $HostVal->"host" . \
"It was down for " . $Count . " checks since " . ($Metric->"since") . "."); ") is up since " . $HostVal->"since" . ".\n" . \
"It was down for " . $Count . " checks since " . ($Metric->"since") . ".");
:if ([ :typeof ($HostInfo->"up-hook") ] = "str") do={ :if ([ :typeof ($HostInfo->"up-hook") ] = "str") do={
:set Message ($Message . "\n\n" . [ $NetwatchNotifyHook $HostName "up" ($HostInfo->"up-hook") ]); :set Message ($Message . "\n\n" . [ $NetwatchNotifyHook $Name $Type "up" \
($HostInfo->"up-hook") ]);
} }
$SendNotification2 ({ origin=$0; \ $SendNotification2 ({ origin=$0; \
subject=([ $SymbolForNotification "white-heavy-check-mark" ] . "Netwatch Notify: " . $HostName . " up"); \ subject=([ $SymbolForNotification "white-heavy-check-mark" ] . "Netwatch Notify: " . \
$Name . " up"); \
message=$Message }); message=$Message });
} }
:set ($Metric->"notified") false; :set ($Metric->"notified") false;
@ -119,32 +131,39 @@ $ScriptLock $0;
:set Parent ($HostInfo->"parent"); :set Parent ($HostInfo->"parent");
:local ParentNotified false; :local ParentNotified false;
:while ($ParentNotified = false && [ :len $Parent ] > 0) do={ :while ($ParentNotified = false && [ :len $Parent ] > 0) do={
:set ParentNotified [ $IfThenElse (($NetwatchNotify->$Parent->"notified") = true) true false ]; :set ParentNotified [ $IfThenElse (($NetwatchNotify->$Parent->"notified") = true) \
true false ];
:if ($ParentNotified = false) do={ :if ($ParentNotified = false) do={
:set Parent ($NetwatchNotify->$Parent->"parent"); :set Parent ($NetwatchNotify->$Parent->"parent");
} }
} }
$LogPrintExit2 [ $IfThenElse ($HostInfo->"no-down-notification" != true) info debug ] $0 \ $LogPrintExit2 [ $IfThenElse ($HostInfo->"no-down-notification" != true) info debug ] $0 \
("Host " . $HostName . " (" . $HostVal->"host" . ") is down for " . $Metric->"count" . " checks, " . \ ("The " . $Type . " '" . $Name . "' (" . $HostVal->"host" . ") is down for " . \
[ $IfThenElse ($ParentNotified = false) [ $IfThenElse ($Metric->"notified" = true) ("already notified.") \ $Metric->"count" . " checks, " . [ $IfThenElse ($ParentNotified = false) [ $IfThenElse \
($Count - $Metric->"count" . " to go.") ] ("parent host " . $Parent . " is down.") ]) false; ($Metric->"notified" = true) ("already notified.") ($Count - $Metric->"count" . \
:if ((($Count * 2) - ($Metric->"count" * 3)) / 2 = 0 && [ :typeof ($HostInfo->"pre-down-hook") ] = "str") do={ " to go.") ] ("parent " . $Type . " " . $Parent . " is down.") ]) false;
$NetwatchNotifyHook $HostName "pre-down" ($HostInfo->"pre-down-hook"); :if ((($Count * 2) - ($Metric->"count" * 3)) / 2 = 0 && \
[ :typeof ($HostInfo->"pre-down-hook") ] = "str") do={
$NetwatchNotifyHook $Name $Type "pre-down" ($HostInfo->"pre-down-hook");
} }
:if ($ParentNotified = false && $Metric->"count" >= $Count && $Metric->"notified" != true) do={ :if ($ParentNotified = false && $Metric->"count" >= $Count && \
:local Message ("Host " . $HostName . " (" . $HostVal->"host" . ") is down since " . $HostVal->"since" . "."); $Metric->"notified" != true) do={
:local Message ("The " . $Type . " '" . $Name . "' (" . $HostVal->"host" . \
") is down since " . $HostVal->"since" . ".");
:if ([ :typeof ($HostInfo->"down-hook") ] = "str") do={ :if ([ :typeof ($HostInfo->"down-hook") ] = "str") do={
:set Message ($Message . "\n\n" . [ $NetwatchNotifyHook $HostName "down" ($HostInfo->"down-hook") ]); :set Message ($Message . "\n\n" . [ $NetwatchNotifyHook $Name $Type "down" \
($HostInfo->"down-hook") ]);
} }
:if ($HostInfo->"no-down-notification" != true) do={ :if ($HostInfo->"no-down-notification" != true) do={
$SendNotification2 ({ origin=$0; \ $SendNotification2 ({ origin=$0; \
subject=([ $SymbolForNotification "cross-mark" ] . "Netwatch Notify: " . $HostName . " down"); \ subject=([ $SymbolForNotification "cross-mark" ] . "Netwatch Notify: " . \
$Name . " down"); \
message=$Message }); message=$Message });
} }
:set ($Metric->"notified") true; :set ($Metric->"notified") true;
} }
} }
:set ($NetwatchNotify->$HostName) { :set ($NetwatchNotify->$Name) {
"count"=($Metric->"count"); "count"=($Metric->"count");
"notified"=($Metric->"notified"); "notified"=($Metric->"notified");
"parent"=($Metric->"parent"); "parent"=($Metric->"parent");