Currently AI bots are crawling website all around the world. For a
website hosting git content this adds a lot of extra load and traffic:
The site has lots of sections, repositories have a lot of files,
branches, tags, commit ids, etc...
Multiply that and you have a nearly unlimited number of unique urls. The
bots try to get each and every of these.
To speed up the learing process on their side a swarm of hundreds,
thousands or more ip addresses is active at the same time, ultimately
DDOS'ing the websites, making it inaccessible. 😳🤬
Well, there is one single file all of these AI bots are not interested
in: robots.txt 🤬🤬
On top some use random user agent strings, making filtering impossible.
🤬🤬🤬
For a short term sulution I deploy the repository content as static
files, hopefully making these accessible at least. We will see.
In the beginning of Let's Encrypt their root certificate ISRG Root X1
was not widely trusted, at least some older and/or mobile platforms were
missing that certificate in their root certificate store.
At that time Let's Encrypt was using an alternative chain of trust,
where a certificate was cross-signed with DST Root CA X3.
To make sure a valid chain of trust is available under all circumstances
a set of all certificates had to be supplied: both root vertificates
ISRG Root X1 & DST Root CA X3, and an intermediate certificate.
This was still true after DST Root CA X3 expired, as it could still be
used as a root anchor and was shipped by Let's Encrypt when requested. 🤪
This time is finally over, and we have a clean chain for trust ending in
ISRG Root X1 (or ISRG Root X2).
Well, actually it is the other way round... Let's Encrypt signs with
different tantamount intermediate certificates. There is not only E5, but
also E6 - and we can not know beforehand which one is used on renew.
So let's jetzt drop the intermediate certificates now, and rely on root
certificates only. We are perfectly fine with this these days.
Follow-up commits will do the same for *all* certificates.
The certificate is downloaded with:
curl -d '["ISRG Root X2"]' https://mkcert.org/generate/ | grep -v '^$' > certs/ISRG-Root-X2.pem
This works with something like this:
:global FwAddrLists {
"allow"={
{ url="https://eworm.de/ros/fw-addr-lists/allow";
cert="E1"; timeout=1w };
};
...
}
All urls for one named list should have the same timeout! With different
timeout values and identical addresses the behavior is besically undefined,
depending on order.
old chain: R3 / ISRG Root X1
new chain: E1 / ISRG Root X2
No user interaction or migration is required for existing installations
as we install 'E1' and 'ISRG Root X2' for some time already.
