@selfhosted@lemmy.world
Mid 2022, a friend of mine helped me set up a selfhosted Vaultwarden instance. Since then, my “infrastructure” has not stopped growing, and I’ve been learning each and every day about how services work, how they communicate and how I can move data from one place to another. It’s truly incredible, and my favorite hobby by a long shot.
Here’s a map of what I’ve built so far. Right now, I’m mostly done, but surely time will bring more ideas. I’ve also left out a bunch of “technically revelant” connections like DNS resolution through the AdGuard instance, firewalls and CrowdSec on the main VPS.
Looking at the setups that others have posted, I don’t think this is super incredible - but if you have input or questions about the setup, I’ll do my best to explain it all. None of my peers really understand what it takes to construct something like this, so I am in need of people who understand my excitement and proudness :)
Edit: the image was compressed a bit too much, so here’s the full res image for the curious: https://files.catbox.moe/iyq5vx.png And a dark version for the night owls: https://files.catbox.moe/hy713z.png
What did you use to chart this? And nicely done.
Excalidraw. Reading is hard. (Yeah, I missed that it was mentioned in the thread)
Yeah, definitely a concert to Mermaid.
I’ve seen Caddy mentioned a few times recently, what do you like about it over other tools?
I can answer this one, but mainly only in reference to the other popular solutions:
- nginx. Solid, reliable, uncomplicated, but. Reverse proxy semantics have a weird dependency on manually setting up a dns resolver (why??) and you have to restart the instance if your upstream gets replaced.
- traefik. I am literally a cloud software engineer, I’ve been doing Linux networking since 1994 and I’ve made 3 separate attempts to configure traefik to work according to its promises. It has never worked correctly. Traefik’s main selling point to me is its automatic docker proxying via labels, but this doesn’t even help you if you also have multiple VMs. Basically a non-starter due to poor docs and complexity.
- caddy. Solid, reliable, uncomplicated. It will do acme cert provisioning out of the box for you if you want (I don’t use that feature because I have a wildcard cert, but it seems nice). Also doesn’t suffer from the problems I’ve listed above.
I feel so relieved reading that about traefik. I briefly set that up as a k8s ingress controller for educational purposes. It’s unnecessarily confusing, brittle, and the documentation didn’t help. If it’s a pain for people in the industry that makes me feel better. My next attempt at trying out k8s I’ll give Kong a shot.
I really like solid, reliable, and uncomplicated. The fun part is running the containers and VMs, not spending hours on a config to make them accessible.
I have traefik running on my kubernetes cluster as an ingress controller and it works well enough for me after finagling it a bit. Fully automated through ansible and templated manifests.
Heh. I am, as I said, a cloud sw eng, which is why I would never touch any solution that mentioned ansible, outside of the work I am required to do professionally. Too many scars. It’s like owning a pet raccoon, you can maybe get it to do clever things if you give it enough treats, but it will eventually kill your dog.
Care to share some war stories? I have it set up where I can completely destroy and rebuild my bare metal k3s cluster. If I start with configured hosts, it takes about 10 minutes to install k3s and get all my services back up.
Sure, I mean, we could talk about
- dynamic inventory on AWS means the ansible interpreter will end up with three completely separate sets of hostnames for your architecture, not even including the actual DNS name. if you also need dynamic inventory on GCP, that’s three completely different sets of hostnames, i.e. they are derived from different properties of the instances than the AWS names.
- btw, those names are exposed to the ansible runtime graph via different names i.e.
ansible_inventoryvs some other thing, based on who even fuckin knows, but sometimes the way you access the name will completely change from one role to the next. - ansible-vault’s semantics for when things can be decrypted and when they can’t leads to completely nonsense solutions like a yaml file with normal contents where individual strings are encrypted and base64-encoded inline within the yaml, and others are not. This syntax doesn’t work everywhere. The opaque contents of the encrypted strings can sometimes be treated as traversible yaml and sometimes cannot be.
- ansible uses the system python interpreter, so if you need it to do anything that uses a different Python interpreter (because that’s where your apps are installed), you have to force it to switch back and forth between interpreters. Also, the python setting in ansible is global to the interpreter meaning you could end up leaking the wrong interpreter into the role that follows the one you were trying to tweak, causing almost invisible problems.
- ansible output and error reporting is just a goddamn mess. I mean look at this shit. Care to guess which one of those gives you a stream which is parseable as json? Just kidding, none of them do, because ansible always prefixes each line.
- tags are a joke. do you want to run just part of a playbook? --start-at. But oops, because not every single task in your playbook is idempotent, that will not work, ever, because something was supposed to happen earlier on that didn’t. So if you start at a particular tag, or run only the tasks that have a particular tag, your playbook will fail. Or worse, it will work, but it will work completely differently than in production because of some value that leaked into the role you were skipping into.
- Last but not least, using ansible in production means your engineers will keep building onto it, making it more and more complex, “just one more task bro”. The bigger it gets, the more fragile it gets, and the more all of these problems rears its head.
-
Dynamic inventory. I haven’t used it on a cloud api before but I have used it against kube API and it was manageable. Are you saying through kubectl the node names are different depending on which cloud and it’s not uniform? Edit: Oh you’re talking about the VMs doh
-
I’ve tried ansible vault and didn’t make it very far… I agree that thing is a mess.
-
Thank god I haven’t ran into interpreter issues, that sounds like hell.
-
Ansible output is terrible, no argument there.
-
I don’t remember the name for it, but I use parameterized template tasks. That might help with this? Edit: include_tasks.
-
I think this is due to not a very good IDE for including the whole scope of the playbook, which could be a condemnation of ansible or just needing better abstraction layers for this complex thing we are trying to manage the unmanageable with.
-
Fully agree to this summary. traefik also gave me a hard time initially, but once you have the quirks worked out, it works as promised.
Caddy is absolutely on my list as an alternative, but the lack of docker label support is currently the main roadblocker for me.
@oh_gosh_its_osh @xantoxis for #k8s solution though I think traefik has advantage of providing configuration via CRDs, no?
May I present to you: Caddy but for docker and with labels so kind of like traefik but the labels are shorter 👏 https://github.com/lucaslorentz/caddy-docker-proxy
Jokes aside, I did actually use this for a while and it worked great. The concept of having my reverse proxy config at the same place as my docker container config is intriguing. But managing labels is horrible on unraid, so I moved to classic caddy instead.
Nice catch and thanks for sharing. Will definitely check it out.
deleted by creator
In addition to the other commenter and their great points, here’s some more things I like:
- ressource efficient: im running all my stuff on low end servers, and cant afford my reverse proxy to waste gigabytes of RAM (kooking at you, NPM)
- very easy syntax: the Caddyfile uses a very simple, easy to remember syntax. And the documentation is very precise and quickly tells me what to do to achieve something. I tried traefik and couldn’t handle the long, complicated tag names required to set anything up.
- plugin ecosystem: caddy is written in go, and very easy to extend. There’s tons of plugins for different functionalities, that are (mostly) well documented and easy to use. Building a custom caddy executable takes one command.
I think the two of you have convinced me to check it out! It is sounding pretty great, so thank you in advance.
I see everyone else have already chimed in on whats so great about Caddy (because it is!), one thing that has been a thorn in my side though is the lack of integration of fail2ban since Caddy has moved on from the old common log format and moved on to more modern log formats. So if you want to use a IPS/IDS, you’ll have to either find a creative hack to make it work with fail2ban or rely on more modern (and resource heavier) solutions such as crowdsec.
You can install the log transformer plugin for Caddy and have it produce a readable log format for fail2ban: https://github.com/caddyserver/transform-encoder
I had this setup on my VPS before I moved to a k3s setup. I will take a look at how to migrate my fail2ban setup to the new server.
Cool, thanks for this! As a user of Caddy through Docker, I suppose I need to find a way to build a docker image to be able to do this?
Sometimes new simple technologies makes things simple - but only as long as one intends to follow how they are used… 🙃
I think so, but if you check the official image you can definitely find out how to include custom plugins in it. I think the documentation might mention a thing or two about it too.
Sorry if someone already asked this, but do you have any tutorials or guides that you used and found helpful for starting out? I have some small experience with nginx and such, but I would definitely need to follow along with something that tells me what to do and what each part does in a infrastructure like you have haha
I’ve been dabbling in self hosting recently and found that chatgpt can help you setup a lot if you don’t get annoyed and keep fixing your prompts. It even writes out your docker compose files for you and you can ask it questions on what things mean and what’s linked to each other. If you do try it out though, avoid giving personal info like passwords in the chat.
Had to look up what a few of these were myself, check the sidebar resources: https://github.com/awesome-selfhosted/awesome-selfhosted
That’s a tough one. I’ve pieced this all together from countless guides for each app itself, combined with tons of reddit reading.
There are some sources that I can list though:
- https://awesome-selfhosted.net/ is great to find apps you might want to host
- https://docs.ibracorp.io/ mainly aims at Unraid hosting, but the information can oftentimes be transferred
- https://how2host.it has some start-to-finish guides that explain every setup step
- https://github.com/mikeroyal/Self-Hosting-Guide is an incredibly long list of apps and ressources you can use as a launchpad. Note the “Tutorials & Ressources” Section for further links
I am sorry, I am but a worm just starting Docker and I have two questions.
Say I set up pihole in a container. Then say I use Pihole’s web UI to change a setting, like setting the web UI to the midnight theme.
Do changes persist when the container updates?
I am under the impression that a container updating is the old one being deleted and a fresh install taking its place. So all the changes in settings vanish.
I understand that I am supposed to write files to define parameters of the install. How am I supposed to know what to write to define the changes I want?
Sorry to hijack, the question doesn’t seem big enough for its own post.
With containers, most will have a persistent volume that is mapped to the host filesystem. This is where your config data is. When you update a container, just the image is updated(pihole binaries) but it leaves the config files there. Things like your block lists and custom dns settings, theme settings, all of that will remain.
Thank you.
Since nobody else asked about this, why ruTorrent over the other typical download clients?
Pretty sure ruTorrent is a typical download client. The real reason is that it came preinstalled and I never had a reason to change it ¯_(ツ)_/¯
You’re usually stuck with what your seedbox provider gives you.
Ahh I’m not too familiar with seedboxes, thanks 🌻
deleted by creator
I’d love to have everything centralized at home, but my net connection tends to fail a lot and I dont want critical services (AdGuard, Vaultwarden and a bunch of others that arent listed) to be running off of flakey internet, so those will remain in a datacenter. Other stuff might move around, or maybe not. Only time will tell, I’m still at the beginning of my journey after all!
deleted by creator
This is oddly similar to some informal workups I’ve done for our work network.
Nice work 👍.
removed by mod
I have taken a picture and shall study it
What is the proxy in front of crowdsec for?
If you’re referring to the “LabProxy VPS”: So that I don’t have to point a public domain that I (plan to) use more and more in online spaces to my personal IP address, allowing anyone and everyone to pinpoint my location. Also, I really don’t want to mess with the intricacies of DynDNS. This solution is safer and more reliable than DynDNS and open ports on my router thats not at all equipped to fend off cyberspace attacks.
If you’re referring to the caddy reverse proxy on the LabProxy VPS: I’m pointing domains that I want to funnel into my homelab at the external IP of the proxy VPS. The caddy server on that VPS reads these requests and reverse-proxies them onto the caddy-port from the homelab, using the hostname of my homelab inside my tailscale network. That’s how I make use of the tunnel. This also allows me to send the crowdsec ban decisions from the homelab to the Proxy VPS, which then denies all incoming requests from that source IP before they ever hit my homelab. Clean and Safe!
“pinpoint” is a bit hyperbolic. Country, state and maybe city can be pretty good, at least in the US.
It’s fine if that’s important to you to hide, but entirely unnecessary for most people.
Maybe. But I’ve read some crazy stories on the web. Some nutcases go very far to ruin an online strangers day. I want to be able to share links to my infrastructure (think photos or download links), without having to worry that the underlying IP will be abused by someone who doesn’t like me for whatever reason. Maybe that’s just me, but it makes me sleep more sound at night.
Thanks, but I meant the HAProxy in your homelab.
Oh, that! That app proxies the docker socket connections over a TCP channel. Which provides a more granular control over what app gets what access to specific functionalities of the docker socket. Directly mounting the socket into an app technically grants full root access to the host system in case of a breach, so this is the advised way to do it.
What software did you use to make this image? Its very well done
Thank you! It’s done in excalidraw.com. Not the most straightforward for flowcharts, took me some time to figure out the best way to sort it all. But very powerful once you get into the flow.
If you’re feeling funny, you can download the original image from the catbox link and plug it right back into the site like a save file!
Now just gotta understand everything beyond… Jellyfin haha
Draw.io is also pretty good or lucidcharts
This seems like work but from/for home.
You should see some of the literal data centers folks have in their houses. It’s nuts.
Possible for a dark mode version XD? excalidraw can do that.
Of course! here you go: https://files.catbox.moe/hy713z.png. The image has the raw excalidraw data embedded, so you can import it to the website like a save file and play around with the sorting if need be.
Thanks for the dark mode link!!
I was also going to mention draw.io
Thank you 🫶
Very nice setup imho. Quite a bit more complicated than mine - mine is basically just the left box without being behind a VPS or anything. I don’t expose anything through Caddy except Jellyfin. I’m also running fail2ban in front of my services, so that if it gets hit with too many 404s because someone is poking around, they get IP banned for 30d
I’m still on the fence if I want to expose Jellyfin publicly or not. On the one hand, I never really want to stream movies or shows from abroad, so there’s no real need. And in desperate times I can always connect to Tailscale and watch that way. But on the other, it’s really cool to simply have a web accessible Netflix. Idk.
Honestly, I installed Ombi, so friends can request movies - and gave them all jellyfin logins as well. I’m not running any kind of pay-for service, I’m just giving them access to my library. Additionally, my kids will sometimes spend the night at friends, etc - and their friend won’t have an anime, or a crunchyroll subscription, so they’ll pull it up on jellyfin. It’s easy to remember for them because it’s just jellyfin.mydomain.com
They don’t know anything about how the backend gets the movies/tv shows, just that they go to ombi, and it shows up on jellyfin if they want something ;)
Gosh, that’s cute. Probably how I’ll end up too. Right now I’m not ready to let friends use my services. I already have friends and family on adguard and vaultwarden, that’s enough responsibility for now.
Architecture looks dope
Hope you’ve safeguarded your setup by writing a provisoning script in case anything goes south.
I had to reinstall my server from scratch twice and can’t fathom having to reconfigure everything manually anymore
Nope, don’t have that yet. But since all my compose and config files are neatly organized on the file system, by domain and then by service, I tar up that entire docker dir once a week and pull it to the homelab, just in case.
How have you setup your provisioning script? Any special services or just some clever batch scripting?
Old school ansible at first, then I ditched it for Cloudbox (an OSS provisioning script for media server)
Works wonders for me but I believe it’s currently stuck on a deprecated Ubuntu release
