In May of 2023 (or maybe earlier, I don't actually know) I did Crowdsec Notes and Stuff.
I'm about to move my homelab to proxmox (well, it's been happening since november last year, on and off) and one of the things that currently run on bare metal, is the Crowdsec LAPI Server.
If I want to put proxmox on the homelab server, I'm going to need to format that guy, which means moving my entire Crowdsec deployment to a new VM.
But, that's cool. I made notes last time!
Lets kick off by creating a new VM. It will be over-spec'd to start with, and if we can, we'll remove resources from it after it's installed and running it's workload.
I usually do this to keep the install times down. How much difference does it make? Not much. I usually alt-tab to something else while the install is happening, and it ends up waiting on me....
After running for a while, the load average on this machine is 0.00, and it's using 430MB of RAM, and zero swap.
Anyway, on with the show. We'll install Debian 12 on this VM. Why Debian 12? Because it's Debian. But why is that good? It's not fucked in the head like Ubuntu (Oh Hi, Snap!) and it's still a stable and slow moving target, which is perfect for being an OS running in a VM.
There's nothing really out of the ordinary for the actual OS install.
When it comes time to select which software to install, un-tick everything, and tick "SSH Server"
The most secure software you have, is software that's not running, and if we're going to do at least the bare minimum in hardening this thing, we can start by not installing an entire GUI that'll never get used.
It also speeds up the OS install significantly, which is also a bonus.
Not long later, we'll be looking at our login screen.
Woo, we installed linux!
Time to log in as root for the first, and only time.
First, some housekeeping.
We want sudo. We also want neovim. Default vi on debian hates the arrow keys. neovim on the other hand knows about the terminal it's attached to, and lets me do things like middle-click-paste without having to press shift. And it has colours.
It's the small things.
Anyway, `apt install neovim sudo qemu-guest-agent curl` and then "usermod -a -G sudo <Your_Username>" and log out. Log in as your admin user, and we'll setup networking.
The qemu-guest-agent was probably installed with the rest of the OS, but just in case, we do it anyway. If proxmox says "Guest Agent not Running" on the VM Summary page, this is the guy you need to install.
We will need curl for the crowdsec installation later.
One of the reasons I like Debian, is a default install like we've done here does not use NetworkManager. There's no need for that bullshit.
Look at this, it's exactly enough config to make it work, and we don't need netplan or any of that noise. Just a config file with the settings we want, as God intended. IPv6 blurred, because my IPv6 stack does work, and while I'm sure the firewall is doing it's job, we don't want to have any accidents.
Add your IPs to local DNS if you want (I do, and did) and reboot. ssh into your new hostname, to ensure everything works as you expect.
This was a lot of steps that were not "Installing Crowdsec" but sometimes tutorials say "Install an operating system" and assume you've done all of these steps a million times and have already internalised some kind of bare-minimum best-practices.
Note: these are not best practices, they're better-than-the-default practices.
Installing the Crowdsec LAPI.
Let's get onto the meat and potatoes of this thing, and Install Crowdsec.
Yes, it's link to the official install documentation. It may change, and you should look at it, at the very least.
It's about to be in charge of a lot of stuff, so you should have at least opened it in a new tab and left it open for a few days.
Crowdsec is in the Debian repos. We're not going to use that, in classic debian-stable fashion, it's a few versions behind, so we're going to use the Crowdsec repos.
Default debian:
Crowdsec repos:
Copy and paste the install command directly from the internet. (Don't do that, copy and paste from websites is bad, type it in, we're about to run this with sudo)
Once installed, you should get something like the following:
Go ahead and enroll your new security engine using the command provided by the crowdsec admin console.
After restarting Crowdsec, rename your new security engine in the console so you know which one it is, and verify that it has a "last actvity" and such.
Here are my new, and old Engines in the admin console.
The old one has a pretty border indicating it's a "Distributed Setup" - soon, our new one will too. This means we have an LAPI server, with Log Parsers connected to it.
So now it's time to move on to the Next Phase: re-configuring everything.
Joy!
Update your /etc/crowdsec/config.yaml to change the listen_uri from localhost to the IP address on your new crowdsec lapi server, or 0.0.0.0 if you want to listen on all interfaces.
If you're using IPv6, it will be a good idea to add your subnet to the whitelist. Use the profile method I've linked, if you want to use the whitelist method, you will need to do it on all of the machines that you're going to be pointing at the LAPI server.
The whitelist rules don't apply on the LAPI server, it's not the one handling bucket overflows, by the time it's being handed an IP address, it's "straight to the shit-list" for that IP.
So throw any IPs (or ranges) you don't want to hit with the ban hammer into the top of /etc/crowdsec/profiles.yaml
If your network is mysterious and important, you probably don't want to do this. Trusting your internal network just makes it easier for bad actors to move about your network once it's been compromised. If this is you, also "cscli parsers remove crowdsecurity/whitelists" on all instances of crowdsec.
If you remove the "debug: true" line, you will see that a rule has triggered on another machine and it has reported to the LAPI server, but you won't see that the LAPI server has ignored the request because of a whitelist rule.
Without the debug: true line, you'd only get the bottom two rows of output, not the initial four.
Once all this has been done, you can reload crowdsec.
Installing Crowdsec Log Processors.
A "log processor" is the thing that looks at logs, and when you're running a distributed setup as we are, I refer to the not-LAPI machines as log processors, because that's what they are.
There's also bouncers, they perform "remediation actions" - in my case, set firewall rules that drop traffic. There are other less intrusive methods, but this is my homelab, and as long as I can access my stuff, I really don't care if you can't.
Installing crowdsec on other machines is basically the same process as above, but you don't need to enroll it in the cloud portal or change any config files. Just do a default install. (and remove the whitelist parser if you want that)
You will need to install some scenarios based on the roll of the server, but for now, we'll enroll it with our LAPI server.
"cscli lapi register --url http://<ip_of_your_lapi>:<port> --machine <name_or_role_of_machine>"
It will tell you that you need to reload the crowdsec service, but I like to leave that until after I've validated it on the LAPI server.
Now, back to your lapi server, you can "cscli machine list"
The top two entries are an example of what you get when you don't provide the --machine argument, it's not required, but it sure makes life a lot easier.
The bottom entry is our newly added example, awaiting validation. validation is nice and easy, just "cscli machines validate <the_name>
You can "watch cscli machines list" on the LAPI server if you want. It gives you something to look at while you go back to server you're installing the log processor on, where we now need to reload the config to have crowdsec starting to report.
You cannot rename a log processor if you enroll it without a name, or typo the name. This is not the drama you think it might be, just "cscli lapi register" / "cscli machine validate" again, and then "cscli machines delete" the one that's wrong. Don't forget to restart things that want restarting.
There is a whole set of instructions for changing it in the database, and frankly, just re-registering the machine is way quicker and easier.
Next we need to add collections that correspond with services running on your virtual machines.
I'm lazy and put all of my acquis rules into /etc/crowdsec/acquis.yaml but you should be motivated and put them into /etc/crowdsec/acquis.d/service_name.yaml, it makes it easier to see at a glance what has configured on a given log processor.
We're pretty much done here. I've not talked about the bouncer config on the firewall, and I sorta-kinda need to.
Using this setup, we cannot use the GUI on the firewall to configure the crowdsec bouncer / log processor. On opnSense at least, it wants to be the LAPI server, so if you hit "save" it'll screw up your config.
You totally can see the decisions in the GUI, and even use the GUI to remove any decisoins that do need to be removed. Just don't press save!
From your LAPI server, you need to "cscli add bouncer bouncer-name"
It will show an API key on the terminal, we need that, so don't loose it!
ssh into your firewall, and edit
/usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml
At the top you need to put the API key from above into the api_key: line, and the url for your LAPI server into api_url:
Use "cscli lapi register" like normal, as once again, this cannot be done from the GUI.
Restart the crowdsec and crowdsec-firewall services, and wait a minute or two, and "cscli bouncers list" from the LAPI server should show a "Last API Pull" time, and "pfctl -t crowdsec_blacklists -T show" should have a whole lot of IPs in it.