ServScrape

ServScrape is a small Golang program that scrapes Matrix rooms for Matrix servers, and on a timer, will attempt to check the registration requirements of the servers. Any servers that are found to be vulnerable to bots will be added to a policy list, and humans will be notified to attempt to contact the administrators. Servers that were previously open registration will be automatically removed.

Why do I see this in my logs?

This is a normal part of the operation of ServScrape. It will attempt to scrape the server for registration information, and if it is found to be vulnerable, it will add the server to a policy list. This is done to help prevent abuse of the server, and to assist in locating servers that are vulnerable, to contact their admins.

This operation is entirely safe, and does not cause any damage to your server. Here's how it works:

1. A set of accounts are synced to fetch their joined rooms, and the members of those rooms
2. Each server will be resolved via well-known, falling back to https://<server>.
3. Each server has a /versions call to effectively see if the server is alive.
4. The server then receives one call to /register to see what its registration requirements are.
5. There is then a second request to /register?kind=guest to see if the server has guests enabled.
   1. Guests that register successfully will then try to join #containment:nexy7574.co.uk to see if the guest account can be used to join rooms.
   2. If the guest account can join the room, it then immediately leaves.
6. The server's /.well-known/matrix/support is queried, and a contact (either user ID, room alias, or email address) is saved
7. All of this is saved to a database
8. Servers that are newly found to be vulnerable are added to the policy list, and will be re-checked 24-48 hours later to see if they have changed their registration requirements.
9. Servers that were previously vulnerable and no-longer are, are removed from the policy list.
10. Servers that are not vulnerable will not be checked for a week.

Setup

Install

git clone https://git.nexy7574.co.uk/nex/servscrape.git
cd servscrape
go build ./cmd/servscrape

Then, create the example config file:

./servscrape -config config.json

Edit the config file with the relevant information, then re-run the command to run a scrape.

Performance warning

DNS & TCP Exhaustion

ServScrape contacts a LOT of servers in rapid succession. Small tests have found to have overwhelmed systemd-resolved, and even larger tests have put some immense stress on a typical home router. It is advised that, if you choose to run this program yourself, that the program is run either on a server, or with a connection tunnelled (optimally via WireGuard) to a server that can handle the traffic.

If you cannot run under ideal conditions, or simply want to run this locally for a test, lower the max_concurrency in the config.json:

{
  "max_concurrency": 64
}

When unspecified, max_concurrency defaults to 100. You can, of course, increase this to whatever number you like, however values above 256 cannot be recommended as they reliably cause issues with most DNS resolvers. Please only go above 256 if you know your setup can handle it.

Anti-scraping measures sapping resources

Given this is a web scraping tool, some misbehaving servers may attempt to block the requests. Previously, I have encountered responses of absurd sizes, responses that attempted to simply hog my connection, and even one server that consistently sent me a decompression bomb.

Please be aware that under normal circumstances, ServScrape will only use a fraction of your CPU, and a few hundred Mebibytes of RAM. The RAM usage can increase to a few GiB if you sync with a lot of accounts, or use accounts that are very large (the sync request is intentionally not filtered), but during actual scanning, normal usage should not exceed that of a few hundred MiB. Networking looks to only use a few hundred KiB/s both outgoing and incoming, but may spike to a few MiB/s during the initial sync.

Please ensure that you have sufficient RAM to handle anti-scraping measures attempting to kill the scraper. If the process gets OOM reaped before it can be interrupted, you may encounter minor data loss. Remember, ^C (Ctrl+C) is the abort button.

Policy metadata

Servscrape includes some additional metadata in the policy events it sends:

• uk.timedout.servscrape.reg_enabled - A boolean indicating whether the server responded with a forbidden registration response, meaning it is closed registration.
• uk.timedout.servscrape.guest_enabled - A boolean indicating whether the server allows guest registration and usage.
• uk.timedout.servscrape.vulnerable - A boolean indicating that the server is vulnerable to bots.
• uk.timedout.servscrape.external - A boolean indicating whether the server uses external registration, such as MAS or SSO.
• uk.timedout.servscrape.email - A boolean indicating whether the server requires the user to provide an email address to register. Does not verify that the email verification actually works.
• uk.timedout.servscrape.captcha - A boolean indicating that the server requires the user to complete a captcha to register.
• uk.timedout.servscrape.token - A boolean indicating that the server requires the user to provide a pre-shared secret or token in order to register.

And, alongside my other auto moderation tools, uk.timedout.mod.automatic will be set to true.