New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
I'm giving it to Fran hot and steady
raindog308
Administrator, Veteran
in News
Have some nasty DB processing to do...gigabytes of XBRL (XML) files to import, then extracting, indexing, transforming...
My laptop did not like this, the poor thing. Ran like developmentally challenged molasses.
Then I remember @Francisco offers dedicated-CPU servers. Boom. Now it runs like a raped ape. Been maxing out a couple cores (on an 8GB slice) for the last 12-odd hours with no issues.
So @Francisco , hypothetically if one were to want 1TB of slab storage but the only slabs available are 256GB and one bought several of these...I figure RAID-0 is fine since it's on top of lower redundancy. Well, that's what I did...
Comments
I guess Fran must have been very busy, he never messaged me back
Ahh XBRL. Yeah that reminds me I need to build another tool for that.
I've also been using servers to process these datasets.
Props to @Francisco as usual, top notch service.
@raindog308 however what kind of laptop you have that is not good enough to process a workload that you are eating with only 2 vCores (threads) dedicated of a Ryzen 3900x. Unless is not capped. Not sure. Francisco is usually generous.
"News" eh?
Sounds like IO is the bottleneck, not CPU.
Lord knows that's the case with my laptop.
I mean, you wouldn't be the 1st person to try that. It didn't have great performance at times due to how RDMA worked in our old setup, but the newer setup is a lot smoother.
Francisco
Hopefully busy getting more block storage for LUX
Try LVM instead. Combine them all into one volume group.
I did this a while ago for fun, 8 x 256GB slabs in RAID0. Performance wasn't that much better than just having 1-2 slabs in RAID0 Ended up just cancelling and the remainder of the period was added to account credit so I could spend it on something better.
RAID0 just added more points of failure with little performance gain for me on Frantech
he looks raped ?
XBRL is nasty. There are TAB-delim'd files available: https://www.sec.gov/dera/data/financial-statement-and-notes-data-set.html
You'd think answering a simple question like "hey, I have all the accounting filings for ${company x} so let's calculate their quick ratio as of 2021 Q4" would be easy but nooooooo....you've got calculation arcs, presentation orders, and all this complexity because there are 15,000 possible Gaap entries...
I thought we talked about low-end VPSes here.
Answer:
A 1TB slab just became available so this will be irrelevant moving forward
how do you know its running like a raped ape huh?
That's fair. I've never seen or made one.
But it has a sprightly rhyme.
Goosed moose?
Scared hare?
Spooked fluke? It's a kind of fish.
Just doesn't have the catchiness.
It's such a pain in the ass too. Because yeah and the API just sucks as well.
I mean I'm fortunate that I (for now anyways) only need to focus on the 10-K filings right now but holy shit it's a bitch.
I feel like on the API end they wanted to design it to do everything at once but also wanted to minimize the amount of actual work to do, so they built a "generalized" solution that's just half-baked and terrible for every data collection need.
Blehhh
"Fake news"
I mean, it's called low-end talk. Obviously, it's about the level of conversation.