As the title says, I want to know the most paranoid security measures you’ve implemented in your homelab. I can think of SDN solutions with firewalls covering every interface, ACLs, locked-down/hardened OSes etc but not much beyond that. I’m wondering how deep this paranoia can go (and maybe even go down my own route too!).
Thanks!
npftables blocks all incoming except a particular set of ips. any connections from those ips hit pubkey authentication.
I’ve never had a problem
deleted by creator
I used to have all VMs in my QEMU/KVM server on their own /30 routed network to prevent spoofing. It essentially guaranteed that a compromised VM couldn’t give itself the IP of say, my web server and start collecting login creds. Managing the IP space got painful quick.
Using SPA firewall knocking (fwknop) to open ports to ssh in. I suppose if I was really paranoid, the most secure would be an air gap, but there’s only so much convenience I’ll give up for security.
I’m going to save your comment because it has opened up a new technique for network security that I had never thought of before. Thanks a bunch
I’m not super paranoid about security, but I do try to have a few good practices to make sure that it takes more than a bot scanning for /admin.php to find a way in.
- Anything with SSH access uses key-based auth with password auth disabled. First thing I do when spinning up a new machine
- Almost nothing is exposed directly to the Internet. I have wireguard set up on all my devices for remote access and also for extra security on public networks
- Anyone who comes to visit gets put on the “guest” network, which is a separate subnet that can’t see or talk to anything on the main network
- For any service that supports creating multiple logins, I make sure I have a separate admin user with elevated permissions, and then create a non-privileged user that I sign in on other devices with
- Every web-based service is only accessible with a FQDN which auto-redirects to HTTPS and has an actual certificate signed by a trusted CA. This is probably the most “paranoid” thing I do, because of the aforementioned not being accessible on the Internet, but it makes me happy to see the little lock symbol on my browser without having to fiddle around with trusting a self-signed cert.
Every web-based service is only accessible with a FQDN which auto-redirects to HTTPS and has an actual certificate signed by a trusted CA
I’m assuming this is in your internal network. The problem with this is that communication from the client to the reverse-proxy (unless you’re running a reverse-proxy sandboxed with each application/are directly decrypting traffic at the base of your application) is encrypted, but the traffic from the server to the reverse-proxy is not.
Definitely a consideration. In my case, the vast majority of my services are running in docker on a single host box, including the reverse proxy itself (Traefik). That unencrypted traffic never goes out over a wire, so for now I’m not concerned.
Bonus points for creating lots and lots of networks grouping the databases together with only their respective containers.
ip ais a huge mess.
For about a year I was running a full out of band IPS on my network. My core switch was set up with port mirroring to spit out a copy of all traffic on one port so that my Suricata server could analyze it. Then, this was fed into ElasticSearch and a bunch of big data crap looked for anomalies.
It was cool. Basically useless because all it did was complain about the same IP crawler bots as my nginx logs. But fun to setup and ultimately good for my career lol.
The problem is, you’d expect your switch to mirror all traffic, including what it is generating (switches with web servers, baseband/backdoors like every big manufacturer), but you can never really be sure.
I have Nginx Proxy Manager set up to let me access services running HTTP on other ports on the machine with a local network only access list just so my traffic even in my own network will use TLS. The likelihood that anyone is sniffing traffic on my own network is extremely small, but I’m paranoid. (Can’t let anyone see that I’m running Ubuntu Server. How embarrassing.)
I’ve got systems that can detect suspicious activities in the net, which result in a shutdown of the router. And not like “could you please shut down” but a hard power off type of shutdown.
Oh, you have a setup that signals to your power source to shut off internet when it detects an anomaly on the internet? That’s quite specific, and I’m having trouble trying to understand the use-case here, but it’s definitely included in the paranoid-list. Thanks!
Now that’s the kind of paranoid I was hoping to see in here. High five, pal.
Neat post and great comments. Saved. Thanks. :)
My personal setup includes:
- non web facing homeserver for the juicy stuff
- vps with stuff I‘d barely miss if it was gone
- far too many backups
- automatic cleanup of backups so my hdds dont fill up
- fail2ban listening on every log, even docker containers with permaban enabled
- scripts are root 700 and so on
I‘m aware that stuff might go horribly wrong but so far it hasnt.
I need to start doing backups but storage costs money.
Losing stuff costs a lot more, depending on what it is. Also the stress and health risks accompanied are too much for me.
You can get backups as low as 3$/tb afaik. But I only backup stuff that actually means something to me. Photos and videos, documents and code. No movies which take up a lot of space if you copy them with all the subtitles and languages.
Hey so uhh… I just formated the wrong drive. It’s recoverable but requires terabytes of network transfers so I’m thinking you may be right.
Thanks
Really all I do is setup fail2ban on my very few external services, and then put all other access behind wireguard.
Logs are clean, I’m happy.
Standard and well-tested setup. Thanks for your reply!
- full disk encryption on everything except the router (no point in encrypting the router)
- the server doesn’t have a display connected for obvious reasons, so I’m manually unlocking it via ssh on each boot
- obviously, the SSH keys are different, so the server has a different IP in initrd. That said, I still don’t have any protection against malicious modification of initrd or UEFI
- the server doesn’t have a display connected for obvious reasons, so I’m manually unlocking it via ssh on each boot
- the server scans all new SSL certificates in realtime using certspotter and notifies me of any new certificates issued for my domains that it doesn’t know about (I use Cloudflare so it triggers relatively often, but I still do checks on who the issuer is)
- firewall blocks outgoing 25 so nobody can impersonate my mailserver
You might be interested in setting up network bound encryption via Clevis and Tang. I use a hidden pi zero in my house acting as a Tang server. It’s great being able to reboot any of my encrypted servers without having to manually unlock disks.
Do you recommend any resources about this? I’d be interested in learning how to implement this.
I’m using the recently merged Clevis module for NixOS. There was a recent talk at FOSDEM about it.
I know about it, but it kinda defeats the purpose (the purpose being police raid protection)
Thanks, I’d like to know more about your public-facing setup using cloudflare
there’s not much to know about it, I use Cloudflare simply because its routing is better than direct IP connections for many places on Earth. I can’t fully use Cloudflare anyway because I host many non-web services.
I use a similar setup, but use a USB for my boot drive that has the lvs partition encryption keyfile. I find it much handier since my computer is not near my server. I can boot and then walk upstairs and it is ready, and remove the USB later.
Then there is no way to brute force the decryption or get a password out of me. Also, when the USB is removed and put in a safe place, there is no way to modify the boot partition or UEFI either.
Then I have a password encryption on my data harddrives that I don’t know the password to, but is on my password manager.
The thing about being paranoid about this stuff is that I probably focused on the wrong thing. A smash & grab is completely protected against, but that is like a 0.1% chance anyway and a 0.1% chance on top of that 0.1% chance that it would be targeted enough that they would even try to decrypt it.
Full disk encryption is really only usefully at all for an unpowered system. Network hardening will probably take care of 99.99% of attack attempts where encryption is 0.01%.
Even for a laptop, if it gets stolen in public, it is still running and can have the keys extracted or break into the running system if someone really wants to hack it. They wouldn’t even try to reboot and break the disk encryption probably…
Too much info, but I guess I am just rambling about how dumb my approach probably is 😅
- full disk encryption on everything except the router (no point in encrypting the router)
Mine’s pretty simple, I have a “don’t open ports until ABSOLUTELY NECESSARY” policy, wireguard works well enough for everything else I need to access remotely. I also keep SSH disabled on any machine that has direct access to the internet.
Do you use a KVM to interact with machines that can access the Internet?
No, as it’s just my main desktop, my laptop and an isolated PiHole VM
I’m somewhat paranoid therefore running several isolated servers. And it’s still not bulletproof and will never be!
- only the isolated server, ie. no internet access, can fetch data from the other servers but not vice versa.
- SSH access key based only
- Firewall dropping all but non-standard ports on dedicated subnets
- Fail2ban drops after 2 attempts
- Password length min 24 characters, 2FA, password rotation every 6 months
- Guest network for friends, can’t access any internal subnet
- Reverse proxy (https;443 port only)
- Any service is accessed by a non-privileged user
- Isolated docker services/databases and dedicated docker networks
- every drive + system Luks-encrypted w/ passphrase only
- Dedicated server for home automation only
- Dedicated server for docker services and reverse proxy only
- Isolated data/backup server sharing data to a tv box and audio system without network access via nfs
- Offsite data/backup server via SSH tunnel hosted by a friend
Why would you rotate passsord though?
Rather choose something random and strong than changing it every 6th moon.Rotating passwords only for web services. Vaultwarden does make it easy. Not all services allow 2FA.
Sounds still excessive but that’s what the thread is here for.
Would probably understand it more if I knew more aspects.Cheers to more cybersec :)
Guessing it is more a habit from back in time when ssl certification wasn’t common. Panic of MITM attacks, friends sharing their trusted access to other friends, etc. all contributed to my actual status of paranoia.
Don’t make me reconsider my cybersec approach ;)You could always add another layer ;D
Would you have to compromise on your security according to your threat model if you ran VMs rather than dedicated devices? I’m no security engineer and I don’t know if KVM/QEMU can fit everyones needs, but AWS uses XCP-ng, and unless they’re using a custom version of it, all changes are pushed upstream. I’d definitely trust AWS’ underlying virtualisation layer for my VMs, but I wonder if I should go with XCP or KVM or bhyve.
This is my personal opinion, but podman’s networking seems less difficult to understand than Docker. Docker was a pain the first time I was reading about the networking in it.
Really like your setup. Do you have any plans to make it more private/secure?
I used VMs some time ago but never managed to look deeper into separation of bare metal vs VMs. Hence I can’t assess this reasonably.
Docker got me interested when it started and after discovering its networking capabilities I never looked back.
Basically I’m trying to minimize the possibility that by intercepting one dockerized service the attacker is able to start interacting with all devices. And I have lots of devices because of a fully automated house. ;) My paranoia will ensure the constant growth of privacy and security :)
Never used it “in anger” but:
I have my firewall plugged into a metered outlet (plugged into a UPS). I have it set up to send me alerts if power draw increases beyond a certain threshold. I’ve tested it and wireguard is measurable (yay) but so are DDOS attacks. If I get that alert, I can choose to turn off that plug and take my whole network offline until I get home and can sort that out.
Gotten a few false positives over the years but mostly that is just texting my partner to ask what they are doing.
Aren’t you just DoSing yourself at that point?
Or you could trigger automation that turns it off for hours, then turns it back on. That way you could get around the need to physically turn it on, in case everyone is away.
With my firewall disabled a lot of my internal network (including home assistant) will fall over sooner than later.
But that is also a recipe for mass stress. Because I know “something happened”. And now I know “in six hours, I need to check in and make sure that ‘something’ is still not happening”. Which is extra shitty if I got the notification late evening local time.
I have friends/neighbors that I trust to swing by and push a button in the event I need to bring it back up before I get home. But if I have reached the point of “it is possible my wireguard credentials were compromised?” then I really don’t need to be able to download the next episode of ATLA NOW.
Well damn that’s clever. I will have to try that.
Care to share what outlet you had success with? I’m comfortable with Home Assistant and ZigBee/Z-Wave. Something this critical probably shouldn’t be wireless, but I digress. I’m also interested in what software you’re using for monitoring and alerts (if you’re willing to share). Cheers!
I just use a pretty generic z-wave plug and home assistant. In the past I did more complex setups that actually determine what process is spiking and so forth. But eventually realized that “this is doing a lot of compute…” is a catch all for a LOT of potential issues.
And I guess I don’t understand what you mean by “shouldn’t be wireless”. It is inherently going to be wireless because you will be on your phone on the other side of the planet. If you genuinely suspect you will be vulnerable to attacks of this scale then you… probably have other things to worry about.
But as a safety blanket?
Gotcha. A ZigBee/Z-wave plug was the first thing that came to mind. I’ll probably go the same route. I was merely saying that wireless-anything is less dependable than wired. But I don’t even know if wired smart plugs exist. I use wireless sensors around my home and I’ve never had an issue with dependability. I definitely wasn’t considering Z-wave as a local attack vector. I was just brainstorming some way to avoid a wireless solution for something where dependability is paramount.
Following for my own edification!
Hope I get a lot of good answers!
