New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Just knowing if there's corruption, getting it auto-fixed and having that comfort that there's no strange issue with a set of file(s) somewhere
except everything else does this as well (RAID5/RAID6, erasure coded, just like RAIDz), but only ZFS magically reports huge numbers on this. I don't know how they arrive on the numbers, perhaps there's low protocol level means to gauge when the drive did a re-read.
Run RAID5/RAID6, does the same thing on multiple drive basis. All drives do this internally too.
Keep moving the goal post, eventually you'll get to the infinite and you win the argument by default.
Do you post everything you try, every time you try?
LET did exist more than a decade ago tho, when we had a lot of massive ZFS arrays, with exactly a nicely round 100% failure rate.
We gave it a trial some months/a year back again since people like you were touting it's now perfect and works flawless. Failed within a week. Same server has ever since been running MD+Ext4 flawlessly.
Bottomline is; There's absolutely no argument you could make which would make us even consider ZFS. Only thing is that if it's fixed, but since it is architecturally and by design bad (activation of all drives for 1 I/O request AND all data is very ephemeral on ZFS), i don't see that ever happening.
For a lot of people this is THE reason to use ZFS (bit rot and the rather vociferous to-ecc-or-not debates of yore).
It's a nice problem to NOT have. Retagging @mailcheap in case he can add some more thoughts from his mail (and maildir?) experiences. I think the original tag had an extra ) thanks to Vanilla.
I think this didn't come across clearly - what I meant is that a differently designed system wouldn't have run into rsync limits (and so maybe one could have managed with rsync on ext4). You interpreted it as ZFS design which is certainly not where I was coming from - though some parts of what I said obviously also apply to ZFS, I think I'm no expert to be able to critique a FS at that level (yet!).
Yes - zfs doesn't have that level of tooling and if things break, the mostly practical approach is to recreate the pool from backups.
I don't quite know what "overhead" you're referring to (other than learning ZFS of course) but fair enough and like I said, it does come down to comfort and use case.
Happy to provide my inputs - it at least opens up possibilities for others (and it is via this sharing that I too eventually came to use ZFS).
but that's not an actual issue
DATALOSS EVENT
How can't you see that you just described a data loss event as routine course of operating ZFS?
ZFS Itself is one magnificent overhead, you add stuff and bloat to it == overhead. This is by design too, 1 I/O request activates all drives on the vdev.
I'm not trying to win anything here.
No, but I don't rant that it is broken either. I try to fix it, learn from it (what would usually be my mistake), find a bug, maybe try to fix it and move forward. Yes, there are times when I give up on whatever but that just means it's not right for me especially if/when many others claim it works and swear by it. I just don't willy-nilly say it stinks without proper justification.
Yes you are, to make people use unstable, by design expensive to operate filesystem.
More goal post moves; This time, you are not allowed to say anything negative about ZFS, only positive!
Yes, we never tried to fix anything. We only throw new hardware at it, and if the installer doesn't work, we just buy a new server. XD
Don't you think, considering we are a provider, didn't try something as obvious as trying to debug and fix the issues, and we always just install new stuff when something breaks, instead of repairing it?
Hell, you even yourself admitted data loss events are normal reoccurence (lack of repair tools, ie. e2fsck counterpart)
Well, most of it is obviously as you suspect complexity (needless complexity and me are basically sworn enemies) but from what i've read ZFS also demands quite a bit of RAM for performant operation (caching? - i'm not sure), which as it's mentioned independently i'd suspect to be actually allocated and not just taken from unused RAM like usual disk caching on Linux. This is something i definitely wouldn't want. Also, while not exactly overhead, the external kernel module approach (i know it's unavoidable but still) is egging me.
(I have no recollection of any such warning, can you see an history somewhere?)
I lost all respect for @angstrom as a moderator on this thread.
He comes in calls me out for making jokes on The Sunny, on provable facts, demanding i silence myself and our business. A company who's built it's reputation around Freedom of Speech and privacy.
But it's completely fine for him to tell people to f##k off permanently from LET, derail threads intentionally and constantly, etc. To the point he is making fun of LET as a whole now, pointing out people make browser plugins to avoid him; His response? To tell people to f##k off from LET.
You are obviously biased and protecting him. You are NOT a qualified moderator.
Moderation is extremely difficult because you only have negative power, to ban and silence people. That ultimately always goes to bullshit like this, The Protector Of The Sunniest of Sunnies (Or The King of The Trolls).
That's why almost every forum ever eventually turns out to be extreme left wing too when moderation grows, just see what a shit a hole that R***** forum of forums is, the most moderated forum on the planet. Such a shit hole that Wokeism originated from there, and it's well documented and known that their for profit model is shady mass manipulation, propaganda, influencing what people are allowed to think. Are you are trying to do the same here.
It takes discipline not to let things derail like that, it's like doing your chores. It's so tempting to abuse your power as a moderator for your own biases and profit. It's like The Ring.
Mind You, this is not even about The Sunniest of Sunny, but selective bias of moderation, essentially saying trolling and harassment by certain individuals of others is OK and FINE, but never ever make fun of their Pro-Authoritarian, Pro-Censorship, Anti-Freedom Of Speech (In Short; CCP Agent!) which is factual (example; https://lowendtalk.com/discussion/comment/3194429/#Comment_3194429 )
It's not the trolls which are the worst here, it's the moderation. If that wasn't clear.
@angstrom you need to step aside for a while and reflect on the what just happened. We'll be doing the same and this might be the last post from us for a while, we'll be sure to reconsider do we want to be involved with LET anymore apart from occasional "here's what we got" thread. So the trolls win, the community loses, but our time is better spent elsewhere than under tyranny of moderators who don't know any better.
@raindog308 you might be interested on this popcorn drama for your amusement (or reporting) XD
That's an option i guess. I hadn't really thought about this as i usually don't have more than 2 drives (did i already mention that i'm boring?), so anything above RAID1 is kind of alien to me.
By the way: Sorry for derailing your thread. I was just curious and didn't think it would take up all that much space.
If you go on your own profile there's a "toggle history" link which I assume shows you expired warnings.
Mine is empty so I can't confirm for sure if it does.
As I'm sure you know, freedom of speech doesn't extend to personal attacks - which "you are an agent of the CCP" is unless he actually is.
You've never described any issue other than that you "lost data".
I never made any such statement or implied any such conclusion. The R in RAID is redundant and that's what ZFS also provides pretty well (with multiple copies of critical metadata as well) and all based on whatever R level you choose. It goes quite far to protect you from your own mistakes but like with many things it is not 100% fool proof and that's why despite ZFS, one still keeps backups. I also have no hesitation to admit that as part of my own learning experiences, I have certainly made (big) mistakes and have had to redo a bunch of things that I would have gladly avoided - but those were my mistakes/typos and also not reading stdout properly.
@PulsedMedia
Now things are starting to get weird
Raid 1 with 4tb x 2 disks (4tb available)
Raid 6 with 1tb x 6 disks (4tb available)
(I'm assuming mdadm is used - Even this configuration is used on the Proxmox master node)
Which one do you prefer in this case? And what are the reasons?
I'm sure you use such configurations. What I understand from what you said is that ZFS cannot detect and prevent silent corruption.
In this type of use, even if the disk does not die due to bad sectors, the raid structure is damaged. There are synchronization errors. There were freezes. Or can you confirm there are other problems?
I'm not a fan of zfs or mdadm. The reason I ask is that you cannot ask for a replacement from companies like Hetzner or Ovh without experiencing strange disk errors. Usually you should see a clear error or wait until at least 1 disk is corrupted.
So before everything breaks (as ZFS claims), how do you detect and prevent some corruption with mdadm?
The disks appear completely intact, but there are read/write errors. Or some Vms on proxmox go into read-only mode. Or the main system gets stuck somewhere. Could this all be some sort of silent degradation that goes undetected? Could some bad sectors be causing this?
What is the difference in error detection between Raid 1 and 5/6?
Just asking. I don't claim. I'm not defending
ZFS' ARC works very well (big improvements in 2.2.x) and with possibilities to only cache metadata also if you so desire. Like with other FSs it works better with more RAM. Running a 10TB pool with 4GB of RAM works quite well but at the end of the day, it all depends on the use case and what your "hot data" is (and how big it is). No matter what FS you use, you're going to run into performance slow downs if you're constantly reading from disk (i.e. hot data > some % of RAM). With proper redundancy, ZFS becomes very effective at dynamic tiered storage (RAM Cache (ARC) -> SSD/NVMe (L2ARC/SLOG/Special) -> HDD pool).
DKMS makes it quite easy now.
Obviously but the main question here is: Does it allocate that RAM or is it dynamically taken from otherwise unused RAM. Besides as far as raw caching goes depending on the use case it might not play a huge role at all or you might still be fine with very little. The idea that's it's used for more than that (aka features i don't really use or wouldn't see as that important) is what's bothering me here.
That's also obvious but it stays an external module that has to be built (every time the kernel changes that is). I know that DKMS is quite robust but there's still an added error margin, which i don't really like in this regard.
I might be somewhat playing captain obvious here but the more dumb a system is the more robust and predictable it behaves. That's pretty much one of my favorite mantras.
It is dynamic (max is 50% of system RAM by default, tuneable via module parameters).
You should read up on ARC to get all your questions answered.
Till the licensing gets resolved (likely never per Linus Torvalds) this is not going to change.
Debian 12 has 2.1.11-1 and 2.2.2-3 on backports now. Do you suggest to go backports?
Unfortunately, you're confusing different issues
I asked you to stop making cheap CCP accusations, and it turns out that you had already received a medium-level warning for this same issue in May 2023. (I didn't issue that first warning, so this shows that another mod/admin had also asked you to stop)
This has absolutely nothing to do with your freedom of speech: your freedom of speech doesn't allow you to make cheap CCP accusations, neither here nor elsewhere. (If you don't understand this -- and you appear not to -- then we have another order of difficulty to contend with, unfortunately)
If you feel that @yoursunny is harassing you, then document the case and give me/us the details, and I'll/we'll investigate. (I've said this several times now)
Otherwise no one is asking you to like @yoursunny or to interact with him. Just ignore him if this makes your life easier
If you have no problems with the pool upgrade (from 2.1 to 2.2), definitely go with 2.2.2 - it has some important fixes and fairly significant performance improvements.
OK but does it actually allocate this? Classic disk caching can per default use more or less 100% of system RAM but it'll shrink down when something else actually needs RAM (aka it's not allocating RAM but rather using it while it's in non-allocated/swapped-out state).
Possibly but at one single upside (which as @PulsedMedia stated also mostly goes away at higher raid levels) it's also hard to justify that.
Yeah, that's basically what i meant by it being unavoidable. It's not really that Linus could make it work anyways. It would need either the (whole - as in each and every contribution of the last ~30 years) kernel to be relicensed or the (whole - as in the original codebase stemming from Solaris + each and every contribution that was made to the open source release) ZFS codebase to be relicensed, none of which is realistically possible. As far as official integration goes ZFS is bound to stay a BSD thing (well at least FreeBSD and to a certain extend NetBSD) or (as far as that's relevant) a Solaris/OpenIndiana/... thing.
Edit: From looking at the known issues listed at the NetBSD wiki (https://wiki.netbsd.org/zfs/) it very much seems that said RAM is actually allocated and by default also doesn't shrink. FreeBSD is said to work around that but how other implementations handle this is obviously a totally different question.
Why is the moderation insisting so much on the issue of the so-called cheap CCP rhetoric/accusations when this is an international forum not property of China or related in any way?
What's the issue or potential issue?
I'm not seeing it and I believe several other users aren't as well.
I don't have much more to add, most certainly @PulsedMedia 's usage for VMs requires different optimizations than what we did for maildir storage. While I haven't run a DB on top of ZFS or Btrfs, it's SOP to disable COW in those cases.
Pulse isn't going to change his mind on ZFS owing to many bad experience, so I'll leave it there.
I'm sure one day but not today a super COW FS would come along that would change his mind.
Pavin.
I have heavy usage of mongodb and mariadb on top of zfs, works pretty well and I can recommend this, if you are ok with the performance drop (which is pretty much ok)
On Linux, it does shrink and the default for the ARC is 50% of RAM. On FreeBSD, the limit is much higher (AFAIK) and it is supposed to shrink as well.
Well, to be honest as long as it does shrink i'm a little bit surprised it isn't allowed to eat all available RAM. I mean, if it frees up as soon as something else needs it there is little obvious reason to net let it be used but i guess that's oversimplifying the process. On one hand at least what FreeBSD does seems pretty racy (a thread running in parallel probably wont guarantee RAM being freed up just-in-time) and on the other hand it very likely does a bit more than stupid caching making the free operation somewhat expensive.
Also the next pressing question would be how much can actually be freed in general. Will it be able to drop to something like 100MB or even plain zero? A dumb cache obviously wouldn't have a problem with that (well beyond any operation started right after that being a 100% miss). I mean who knows? There might as well be some knobs and sliders. The complexity is basically the price for the cleverness after all.
Does nothing.
ZFS doesn't give you any feedback. Now does it?
Hence you win by default, since there is nothing to tell, since nothing is told.
Reboot -> No more Zdevs, vols etc.
Also ZFS keeps "writing" to inexistent drives if you have flaky SATA/SAS connectivity.
Raid 6 with 1tb x 6 disks (4tb available)
(I'm assuming mdadm is used - Even this configuration is used on the Proxmox master node)
2x 4TB for power consumption/3.5" Bays etc. cost reasons
6x 1TB for redundancy and read performance as RAID5.
Wouldn't do RAID6 tho. RAID50 rather.
actually f** this, with these moderators why bother giving value to this forum.
Unless it's a question about our services specifically i will refrain from responding for any future inquiries. Thanks @angstrom for clarifying our goals and showing LET is a waste of effort (for me personally)
He is CCP policy supporter with evidence, and freedom of speech means i am eligible to share my opinions. My opinion is that he is a CCP Agent in the sense he is a supporter of CCP and it's policies (arresting for wrong opinions) as per what he has said himself.
We do not support nor condone violations of privacy or suppression of freedom of speech.
You are asking us to give up our principles and support violations of privacy, freedom of speech and support censorship, authoritarian controls of opinions.
Having an opinion of someone else, like you being a bad moderator on a power trip, is part of freedom of speech. Freedom of Speech is for the speech you do not approve or like. Freedom of Speech is for your worst enemy, not for you. My opinions are my own, and you have no power over them. Perhaps you like and approve of the CCP messaging of authoritarian control of opinions, censorship, mass arrests for wrong opinions; We do not.
This had nothing to do with Yoursunny specifically, everything to do with You @angstrom, and you alone.
I report you @angstrom -- you are harassing us now.
We cannot support a forum which is against freedom of speech.
Found a flag button, i flagged you @angstrom -- now wield your power and strike that ban hammer on us, i dare you. Enjoy that sense of power you cannot attain anywhere else, have that dopamine rush. Our treat.
I bet you came here to harass because your favorite flagged us for calling him out.
We will not silence our opinions on your behalf or anyone else's. We see a dog, we call it a dog. We see anti-freedom of speech of rhetoric, we call it anti-freedom of speech.
Oh boy, OP asked for a solution, and it tuned into 3 pages of ZFS love and hate.
...........
COW file systems VS SW raid VS HW raid is a topic like Desktop Systems VS Server Systems..... black hole
After more than 10 years of happy ZFS use, I can say this:
I will not touch it again, not even with a stick.
I agree that is has it's benefits, but at the cost of enterprise NVME for cache, half of ton of memory for ARC, I would rather just get a decent HW controller, some 3.5 inch shelf's and burn 18TB of disks in raid10 in a 48 drive array, it will cost the same, but the performance will have that..... WOW factor.
Actually we had more problems with software cache than with failed HW controllers.
Caching is great, to some extent, cache solutions that make your raid 5/6 array to behave in synthetic tests ( yabs and other ) like SSD/NMVE usualy suck and fail in high IO load and this is due to the fact that the CPU was not designed to do storage ( where ms of delay can lead to a failed array, ms meaning 1,2,3,4 ).
I know you all are going to bring the sky on me with this, but again, rather use an ASIC to do a specific task, then software to fail, especially on storage, where if you loose the data, you are kinda fucked sincerely. ( this does not mean that a HW raid card will not fail )
@PulsedMedia - did you find the problem? sorry if I missed one of your answers on this thread.
EDIT:
Just bare in mind, that raid 5/50 is not recommended for over 2 decades ( since large drives entered the market )
@host_c no solution found, a roundabout way tho, the kernel settings did something. [rest redacted]
I'd have a lot to say but i'll refrain from bringing any value to something which suppresses other people's opinions and freedom of speech, as stated above. Give thanks to @angstrom
Censorship was sought, censorship is what you get.
@PulsedMedia
If these are working servers for a long time ( 3+ years), you could try:
dry contact spray on the CPU socket, pins/contact surface of the cpu and memory modules and memory modules socket.
clean after that with compressed air, you can use a small brush + air.
Also look for old capacitors that might have failed?
I they were working fine but suddenly started doing shit, might be a physical problem, you never know.
Last time I looked being Chinese doesn't necessarily make one a supporter of the CCP. I imagine he does get a bit more freedom than most Chinese citizens as a result of being an academic and living out of China; but again - that alone doesn't necessarily mean he supports the CCP.
If you have something to show confirming he is an agent or supporter of the CCP do everyone a favour and bring it forward. If you don't have it - I think that says it all.
On a side note if you're suggesting he's a supporter instead of an agent - there's a term for that: tankies.
I did, look above.
Unless angstrom went and modified my message linking to that. Takes less than 1 minute to find with google.
Also all this BS is not about him even, it's about censorship and attempts to control other's opinions by means of suppressing free speech. Fuck that shit.
Supporter or Agent, it's all the same. New term tho. we just call them commies over here.
His goal is to get absolute trackability on a per user basis, so that people could be arrested for wrong opinions more efficiently. It doesn't matter if he's officially an CCP party member or not, end result is the same. Therefore, a bloody CCP Agent.
@angstrom is part of the problem now, he is seeking to suppress people's opinions and freedom of speech. He is just as liable now, and should be considered a supporter of authoritarianism. Despicable.
I looked but all I could find was you calling him a CCP agent both in this thread, one in May last year and a few other comments dotted about, pretty much just from you.
The link you've posted above you can see him clearly acknowledging that China does interfere with people's speech and that a unique IPv4 or IPv6 makes the authorities job easier, especially back in the 00s when he was studying in China -- not that he thinks them doing so was good or necessary.
If you could please present the proof we can both stop wasting time on this. It seems to me that you're connecting dots that aren't there.