Asking mostly because I have fuckloads of video courses, plus a number of movies, that I have yet to even check if the content is as good as their titles imply and I really feel like I’m mostly hoarding this stuff because I have no fucking clue.
How do you avoid “hoarding”?
Looks at my 28TB storage array that’s 3/4 full…
Time to buy new HDDs.
You’re doing great man, please keep it up i’m not even joking. Maybe someday you’ll be the one guy that still has that old gem everybody lost.
I actually keep a list of works that I’ve shared online that would’ve likely been lost without my intervention. Physical-only Bandcamp releases that I’ve ripped and shared. Sample packs that have been taken down from webstores, etc. The Internet isn’t forever people. Better archive what you can
Yes yes yes ! Keepers of knowledge!
Drive space can be had for less than 10USD a TB, so I’d hardly call hoarding a problem. Unless youre hoarding hundreds of copies of Call of Duty
Where do you get such cheap storage? I’ve seen it closer to $20/TB usually
I buy refirb server drives . With a raid array and warranty, the risk of a failing drive is within an acceptable range for me.
This listing was a bit cheaper last I looked, so just over $10/TB is more accurate to say now haha.
I got a couple 3TBs in my array that just look funny. I don’t like them and I want them gone.
Avoid hoarding? Let’s just say I bring a real “gotta catch em all” energy to the trackers.
My goal is to store everything, that way I have no dependence on remote media
Home datacentre 😋
Well yes, but also actual DC space lol
How do you avoid “hoarding”?
I dont. Hard drives are increasingly cheap/large. I have to really dislike something to delete it. I have a fair amount of content that I don’t really plan on watching again, but someone I know might like it so i just leave it typically.
These are my thoughts exactly; piracy is preservation.
I have to really dislike something to delete it.
The velma tv show was the last item I just deleted.
But for me this is the same story. I’m up to 400TB… I’m just over half full. I’ve got plenty to go, and if I make to to 75-80% full, then I’m going to get me a 45 or 60 bay server and upgrade from my 36 bay one. 6 of the bays are wasted on SSD caching currently… Just finding a chassis that doesn’t waste the 3.5 inch bays on 2.5 drives would allow me to add a full vdev(another 100TB…).
Old chassis can be had on ebay relatively cheaply.
Do you have an offsite backup? Or do you only backup specifics? Like 10-20% of that?
I do not have full proper offsites… yet.
I run proxmox, so if it’s live on a server it’s probably on my ~70TB (really 40*2TB ssd) ceph cluster. Which makes 3 copies across the 5 boxes, so it’s more like 23TB of usable space for all my vms and such. The 400TB of storage is Truenas is really closer to 300TB after all the losses in raidz vdev and hot spares and what have you, there’s 30x 16TB SAS seagates in the box, of which 2 are hot spares and 7 are parity for raidz1… For things that are slow or linear loads (a movie file could be a good example of that type of workload!). Backups of the the proxmox boxes… and mass stored stuff, 99% of it I could easily obtain again if I had to. Although I’d probably be pretty flustered about it.
Truly important stuff gets written to 100GB bluray(s) (specifically m-disc blurays) and put in the safe. I do this probably about once a year or so…
My dad was in the process of setting up his own cluster that’s running 14TB drives rather than my 16TB… When he’s finally done I intend to requisition probably about half of his space for offsite storage (maybe more). I’m figuring about 100TB of space is what I’ll have there. Maybe more. He’s about 65 miles away from me, different electrical grid and all.
So the count as it stands now. Everything running has at least 2 copies on 2 mediums (ceph cluster, and spinning rust). My “linux iso” repositories only live on the spinning rust storage, but is low priority anyway. Super important highly sensitive shit lives on at least 3 copies and 3 mediums, although one of the mediums may be out of date and none is offsite… Though it’s rare I add to this category. There is plans for adding another copy of data, offsite on harddrive storage for most of my dataset as it is now.
Truenas usages:
And here’s Ceph
Your setup: goals.
Where are you sourcing your hardware? Decommissioned enterprise DC stuff or are there options outside of the enterprise space that enable this type of setup?
Ebay and decommission. I got really lucky on my SSDs, those were all from a decommission. Company was going to pay an ITAD for destruction. I picked it all up and wiped it on site. The rest are relatively cheap hardware, supermicros and such… but with enough of them you can build a resilient cluster.
A lot of my stuff is Ebay… I did recently purchase a new rack as probably the only “new” item I have in regards to my setup. The old one had issues… and I didn’t want to deal with thrifting broken racks anymore. And I needed a taller 45U rack rather than a 42U standard rack… Also the more depth means I can accommodate the 60 bay server in the future if it comes to that.
But things like 40gbps networking… ebay. The proxmox servers are decomissioned. the truenas server was ebay. switches was ebay… Oh! The firewalls… That was new purchase. I am stupid lucky to live somewhere with 8gbps fiber. I needed real horsepower to push that with IDS/IPS enabled. So this was a new purchase from supermicro. The SAS spinning rust drives I picked up on Reddit homelabsales or something like that a while back. PDU’s were ebay… UPS were ebay… Expansion batteries were craigslist. Most cables were new from FS
Previous versions of my rack were government liquidation/auctions. My dad has a lot of that equipment now. I found one auction that was 1400$ that was basically a whole racks worth of shit… most of it pretty usable 12 and 13th gen dells. And another auction for 600$ that had a dell m1000e with some 4TB of DDR4 ram…
But you can do a lot of this shit with a cluster of little N100 boxes if you really wanted. I just happened to get my hands on enterprise level equipment… So I joined the Romans…
This is a detailed breakdown so thank you. I guess the tl;dr is that it’s a combo of ebay, luck, and filling in the gaps. Sounds about as complicated as I was expecting if I’m honest but I appreciate you taking the time to write it out.
deleted by creator
All the time. It’s my primary source of entertainment media. And why would I want to avoid hoarding? Hoarding is the goal.
There’s no reason to avoid hoarding!
who avoids hoarding? we are saving it for the future.
Exactly! I assume that my little 4tb external drive full of movies will one day be the only usable relic discovered of our civilization, so I must plan accordingly lol
Avoid hoarding? I don’t understand. My 30 TB file server can be expanded further still.
I do not avoid hoarding.
I’m like a dragon with a media treasure stored in high capacity industrial HDDs.
Someday the age of pirates may come to an end, and I want to be prepared.
Same. I’m a Doctor Who fan. I don’t need to learn lessons twice. My grandchildren will be able to watch the dumb shit I grew up with one way or another!
If i need something, i go look for it, download it and keep it seeding until i have no more space on my hard drive. I rarely download things i don‘t actually need or want at the moment.
Storage is cheap. There’s no reason to delete content.
Only reason I delete content is when I upgrade. Like replacing a low resolution version of show with a higher one. Still, I keep immutable “snapshots” of my entire media folder so even after deleting something, It’ll stick around for at least 6 months in case I need to restore it.
Same deal, got a full 3-2-1 backup of all my data! Easy to recover if I make a mistake but even easier to replace with
higher qualitynewer builds of Linux isos.
depends on what job are you working
And where in the world you live.
I got a friend in Australia with a pretty similar storage setup to me, but he’s paid about 1.5x as much as I did in the UK.
Haha, good question. You’re not alone with that. I suppose you just clean up once per year. Like you’re supposed to do with your wardrobe, or that one drawer in the kitchen…
You’re supposed to clean your wardrobe?
Uh, no. I don’t know what I’m saying. I meant sort through, get rid of old stuff. I’ve never cleaned the insides that way. And I suppose don’t do that to the harddisk either.
Instructions unclear, am currently taking a bath with my SSD
I only pirate TV/movies, and since I never know what I’ll feel like watching it’s pretty easy to just hoard it. Takes a long time to fill up drives so adding a 16TB drive once a year or two is pretty manageable.
But tbh the main reason I hoard them and keep my Plex library full is simply to keep view stats. Prior to Plex I was constantly plagued by “have I seen this” or “what was that movie I liked 10 years ago?”. But not anymore!
Also, when the zombie apocalypse happens I’ll finally have time to rewatch Breaking Bad so I need an offline copy just in case.
Or when streaming services start at $70.00 a month with ads.
Also, when the zombie apocalypse happens I’ll finally have time to rewatch Breaking Bad so I need an offline copy just in case.
Hope you already got solar panels or some other sort of electricity generator for that
What is your drive setup?
Just a Synology NAS with softwarr’s and a Shield TV connected via Ethernet, works great
I only pirate what I’m about to consume? How is this a problem? Lol
I’m concerned that crackdowns on pirating will come sooner or later. At some point it may become too much of a hassle. So I’m hoarding a lifetime of old movies and games to hold me over.
Just keep seeding the torrents. As long as the torrents for stuff is healthy, how could it be cracked down?
Enough countries ban torrenting so you cannot even uses vpn to get around it.
How do they do that?
Ask Germany. Torrenting without VPN is almost imposible, you’ll get sued by copyright lawyers.
Exactly, so you can use a VPN to get around that.
You use a vpn to other country without that ban. Once enough countries apply that ban. Where are you going to VPN to?
Why would any country ban torrenting? Torrenting isn’t illegal, downloading and distributing copyrighted works is
What I mean is what is already happening in some countries, that is having laws that makes extra easy for copyright lawers to sue anyone using torrent for sharing their IP.
