I woke up this morning to a text from my ISP, “There is an outage in your area, we are working to resolve the issue”
I laugh, this is what I live for! Almost all of my services are self hosted, I’m barely going to notice the difference!
Wrong.
When the internet went out, the power also went out for a few seconds. Four small computers host all of my services. Of those, one shutdown, and three rebooted. Of the three that ugly rebooted some services came back online, some didn’t.
30 minutes later, ISP sends out the text that service is back online.
2 hours later I’m still finding down services on my network.
Moral of the story: A UPS has moved to the top of the shopping list! Any suggestions??
In addition to ups, an LTE failover. I’ve had my Comcast crap be offline for hours.
I’d like that, but also a really long-running UPS. multi-hour power outages are surprisingly common in my area.
Thats no longer a UPS.
You could get something like a powerwall, something designed to power things from batteries for a long time.
Or get a generator with an automatic failover. The UPS then covers the downtime between powerfailure and generator taking loadWhy is that no longer a UPS?
Generally, UPS (lead acid) batteries are not designed for long-cycle deep discharge.
They are designed to hold their rated load for a minute or so until the power is restored (generators start, power-uncuts) or the servers have a chance to shut down.
But maybe thats dated information, and modern UPSs are designed to run from batteries for a few hours.That seems like a weirdly and artificially narrow definition of UPS.
Does this require a lot of gear? Or does it simply act as another gateway?
It requires an LTE capable gateway and a data plan. As for the rest you can simply write your routing tables so that if the main gateway doesn’t work, use the secondary gateway with lower prio.
There are devices like the Netgear lm1200 that can do it inline by themselves.
I have that device, but configured as a second gateway. My firewall manages the failover based on primary packet loss and latency.
Yeah if you self host, a UPS is very important.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters CSAM Child Sexual Abuse Material DNS Domain Name Service/System NAS Network-Attached Storage PiHole Network-wide ad-blocker (DNS sinkhole) Plex Brand of media server package RAID Redundant Array of Independent Disks for mass storage SATA Serial AT Attachment interface for mass storage VPS Virtual Private Server (opposed to shared hosting) ZFS Solaris/Linux filesystem focusing on data integrity
8 acronyms in this thread; the most compressed thread commented on today has 11 acronyms.
[Thread #567 for this sub, first seen 3rd Mar 2024, 16:05] [FAQ] [Full list] [Contact] [Source code]
Could also be a good opportunity to add a service monitor like Uptime Kuma. That way you know what services are still down once things come back online with less manual discovery on your part.
Exact same thing happened to me the other day. Like exactly. Maybe we live in the same area.
Not APC. (At least for Windows) trashy software.
It doesn’t need the software, it gets recognized by the system as a battery
Still bad hardware. At least the Back-UPS line.
Had a old APC that worked for many years. Brought a new back-ups 1100va few years ago , wasn’t the quality I expected.
It might work but the software is still trash.
Have a suggestion?
deleted by creator
Not really. I saw multiple mentions of eaton.
Might be worthwile to look online like on reddit or other forums how the perception is there.
Or TrippLite. Owners used it as a massive money laundering front for right-wing donations and bribes.
Laptops
My favorite part about using an old laptop as a 24/7/365 plugged-in server is the anticipation of when the lithium battery will explode from overcharging.
“overcharging” doesn’t exist. There are two circuits preventing the battery from being charged beyond 100%: the usual battery controller, and normally another protection circuit in the battery cell. Sitting at 100% and being warm all the time is enough for a significant hit on the cell’s longetivity though. An easy measure that is possible on many laptops (like thinkpads) is to set a threshold where to stop charging at. Ideal for longetivity is around 60%. Also ensure good cooling.
Sorry for being pedantic, but as an electricial engineer it annoys me that there’s more wrong information about li-po/-ion batteries, chargers and even usb wall warts and usb power delivery than there’s correct information.
For many li-ion laptop batteries, the manufacturer’s configuration of a 100 % charge is pretty much equivalent to overcharging. I’ve seen many laptops over the years with swollen batteries, almost all of them had been plugged in all the time, with the battery kept at 100 % charge.
As an electrical engineer you should know that technically there is no 100 % charge for batteries. A battery can more or less safely be charged up to to a certain voltage. The 100 % charge point is something the manufacturer can choose (of course within limits depending on cell chemistry). A manufacturer can choose a higher cell voltage than another to gain a little more capacity, at the cost of longterm reliability. There are manufacturers that choose a cell voltage of 4250 mV and while that’s possible and works okay if charged only occasionally, if plugged in all the time, this pretty much ensures killing the batteries rather quickly. I would certainly call that overcharging.
Since you already mentioned charging thresholds, I just want to say, anyone considering using a laptop as a server should absolutely make use of this feature and limit the maximum charge.
Isn’t dendrite formation and the shorts they can cause a much bigger concern when dealing with old batteries that are being charged 24/7? Asking a genuine question here, so please don’t shoot me if I’m wrong. 🙂 I’d love to hear more about the most common failure modes and causes for li-po/ion batteries.
Those are symptoms of sitting at that operation point permanently, and they are a of course a concern. What I’m after is that people think that energy gets put in to the battery, i.e. it gets charged, as long as a “charger” is connected to the device (hence terms like “overcharged”). But that is not true, because what is commonly referred to as “charger” is no charger. It is just a power supply and has literally zero say in if, how and when the battery gets charged. It only gets charged if the charge controller in the device decides to do that now, and if the protection circuit allows it. And that is designed to only happen if the battery is not full. When it is full, nothing more happens, no currents flow in+out of the battery anymore. There’s no damage due to being charged all the time, because no device keeps on pumping energy into the cell if it is full.
There is however damage from sitting (!) at 100% charge with medium to high heat. That happens indipendently from a power supply being connected to the device or not. You can just as well damage your cells by charging them to 100% and storing them in a warm place while topping them of once in a while. This is why you want to have them at lower room temperature and at ~60%, no matter if a device/“charger” is connected or not.
(Of course keeping a battery at 60% all the time defeats the purpose of the battery. So just try to keep it cool, charged to >20% and <80% most of the time, and you’re fine)
If you say it quickly enough it may sound plausible to some but this is not how battery technology works, as explained by @skilltheamps@feddit.de
Amen. I appreciate my UPS for sure!
This is why I have about five of these bad boys: CyberPower CP1500PFCLCD.
One is in my utility room for my cable modem and our chest freezer, three back up my homelab and wifi AP, and one is for my office.
They’ve been bulletproof through storms, and when we’ve lost power, but not Internet I can’t keep on working.
The big thing to look for is number of battery+surge outlets vs just surge outlets. Typically they top out at 1500VA - the more overhead for what you’re powering, the longer you can go without mains power.
A screen/display is helpful for at-a-glance information like expected runtime, current output, etc.
Never heard of someone using a UPS on a Fridge/Freezer. Does it make a difference? Seems like the UPS would just died after 10-20 minutes and not really make much difference to your freezer.
I didn’t intend to use it on the chest freezer - it was mostly for the modem, but since I had spare battery capacity and outlets I thought what the heck.
The power load is practically nothing until it cycles, and even then it’s fairly efficient - my current runtime is estimated to be about 18 hours, more than enough to come up with an alternative if we lose power in a storm.
UPS with usb allows you to configure a script to properly shutdown your server when a power outage happens and the UPS battery is about to run out.
deleted by creator
My suggestion just changes your threat model, so may not be a good one based on your wants.
Perhaps consolidate systems? Managing less devices = less points of failure. But adds the risk of any given failure being more severe.
This thought came to me this morning. I have 4 machines both because the BEAST grows organically, and because we’re always trying to avoid that single point of failure. Then a scenario comes along that makes you question your whole way of thinking, diversifying may actually create more problems
Figure out how much power your servers use on average with the help of a wattage meter, then enter that number and how many minutes battery backup you want in Eatons UPS Power Calculator to find a suitable unit. I’m sure other vendors have similar tools too.
When you are bored, backup a VM then hard kill it and see if it manage to restart properly.
Software should be able to recover from that.
If it doesn’t, troubleshoot.That reminds me of Netflix’s Chaos Monkey (basically in office hours this tool will randomly kill stuff).
When I built my home server this is what I did with all VMs. Learned how to change the start up delay time in esxi and ensured everything came back online with no issues from a cold built.
Rip VMware.
While I appreciate the sentiment, most traditional VMs do not like to have their power killed (especially non-journaling file systems).
Even crash consistent applications can be impacted if the underlying host fs is affected by power loss.
I do think that backup are a valid suggestion here, provided that the backup is an interrupted by a power surge or loss.
most traditional VMs do not like to have their power killed (especially non-journaling file systems).
Why are you using a non-journaling file system in 2024 when those were common 10+ years ago?
Or even better use something like ZFS with CoW that can’t corrupt on power loss
I would still consider that generation of filesystem to be effort to use while regular journaling filesystems have been so ubiquitous that you need to invest effort to avoid using one.
It was supported and the default out of the box when I installed my OS
Maybe on some distros that is the case if you install a recent version but to get a non-journaling filesystem you literally have to partition manually to avoid using one on any distro that is still supported today and meant for full sized PCs (as opposed to embedded devices).
Are you talking about Linux distros? What manual partitioning has to occur?
and don’t fuck with sync writes
It’s been a while since a power cut affected my services, is this why?
I remember having to troubleshoot mysql corruption following abrupt power loss, is this no longer a thing?
Databases shouldn’t even need a journaling filesystem, they usually pay attention to when to use fsync and fdatasync.
In fact journaling filesystems basically use the same mechanisms as databases only for filesystem metadata.
Your system should be fine after a hard kill. If its not stop using it as that’s going to be a problem down the road.
I present to you the holy hardware compatibility table:
https://networkupstools.org/stable-hcl.html
Anything not listed there is not worth buying.
removed by mod
Tl;dr apc bad? I have 4 cyberpower so no experience with them
They got bought. They started to suck.
Now they’ve been 'sploited, they’re overpriced, and they still schlepp the same bad software.
Eatons batteries are usually really simple to switch, see
https://www.eaton.com/content/dam/eaton/products/backup-power-ups-surge-it-power-distribution/backup-power-ups/eaton-5s-ups/eaton-5s-120v-user-manual-700-1000-1500-lcd.pdfFor me they are meant for allowing a graceful shutdown in a powerout scenario and to protect the hardware behind them from power surges.
What’s wrong with APC? I have one for 6-7 years. I’ve changed the battery once and I think I’ll have to change it again this year. I didn’t have any problems with it.
Trash software.
At least their service is good.Can confirm. Software is trash. Wanted me to connect it to the internet and setup a cloud access account. Like, dude, you’re a glorified battery pack I’m not adding a backdoor because you want to tell APC when my warrenty is about to expire so I can get marketing emails.
That’s why you integrate with NUT. So you can automate a graceful shutdown when battery levels drop to a set level.
A general tip on buying UPSes: look for second hand ones - people often don’t realise you can just replace the battery in them (or can’t be bothered) so you can get fancier/larger ones very cheap.
Also, a larger capacity one is better, and it’s likely you’ll find a secondhand one with more capacity/features for a similar price.
Why? If the power has gone out there are very few situations (I can’t actually think of any except brownouts or other transient power loss) where it would be useful to power my server for much longer than it takes to shut down safely.
Longer means you’re more likely to be able to ride out a power cut, and gives you more options if you want/need to complete something more involved than saving and shutting down.
