Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


I'm giving it to Fran hot and steady
New on LowEndTalk? Please Register and read our Community Rules.

I'm giving it to Fran hot and steady

raindog308raindog308 Administrator

Have some nasty DB processing to do...gigabytes of XBRL (XML) files to import, then extracting, indexing, transforming...

My laptop did not like this, the poor thing. Ran like developmentally challenged molasses.

Then I remember @Francisco offers dedicated-CPU servers. Boom. Now it runs like a raped ape. Been maxing out a couple cores (on an 8GB slice) for the last 12-odd hours with no issues.

So @Francisco , hypothetically if one were to want 1TB of slab storage but the only slabs available are 256GB and one bought several of these...I figure RAID-0 is fine since it's on top of lower redundancy. Well, that's what I did...

Comments

  • mhnmhn Member

    @raindog308 said: Now it runs like a raped ape.

    Thanked by 1TimboJones
  • ArkasArkas Member

    I guess Fran must have been very busy, he never messaged me back :neutral:

  • @raindog308 said:
    Have some nasty DB processing to do...gigabytes of XBRL (XML) files to import, then extracting, indexing, transforming...

    My laptop did not like this, the poor thing. Ran like developmentally challenged molasses.

    Then I remember @Francisco offers dedicated-CPU servers. Boom. Now it runs like a raped ape. Been maxing out a couple cores (on an 8GB slice) for the last 12-odd hours with no issues.

    So @Francisco , hypothetically if one were to want 1TB of slab storage but the only slabs available are 256GB and one bought several of these...I figure RAID-0 is fine since it's on top of lower redundancy. Well, that's what I did...

    Ahh XBRL. Yeah that reminds me I need to build another tool for that.

    I've also been using servers to process these datasets.

  • HxxxHxxx Member
    edited July 12

    Props to @Francisco as usual, top notch service.

    @raindog308 however what kind of laptop you have that is not good enough to process a workload that you are eating with only 2 vCores (threads) dedicated of a Ryzen 3900x. Unless is not capped. Not sure. Francisco is usually generous.

  • szarkaszarka Member

    @Hxxx said:
    Props to @Francisco as usual, top notch service.

    @raindog308 however what kind of laptop you have that is not good enough to process a workload that you are eating with only 2 vCores (threads) dedicated of a Ryzen 3900x.

    Sounds like IO is the bottleneck, not CPU.

    Lord knows that's the case with my laptop.

  • FranciscoFrancisco Member, Top Host, Host Rep

    @raindog308 said: So @Francisco , hypothetically if one were to want 1TB of slab storage but the only slabs available are 256GB and one bought several of these...I figure RAID-0 is fine since it's on top of lower redundancy. Well, that's what I did...

    I mean, you wouldn't be the 1st person to try that. It didn't have great performance at times due to how RDMA worked in our old setup, but the newer setup is a lot smoother.

    Francisco

    Thanked by 2Foul szarka
  • corbpiecorbpie Member

    @Arkas said: I guess Fran must have been very busy, he never messaged me back :neutral:

    Hopefully busy getting more block storage for LUX

  • Daniel15Daniel15 Member

    @raindog308 said: , hypothetically if one were to want 1TB of slab storage but the only slabs available are 256GB and one bought several of these...I figure RAID-0 is fine since it's on top of lower redundancy. Well, that's what I did...

    Try LVM instead. Combine them all into one volume group.

    Thanked by 1raindog308
  • AdvinAdvin Member, Host Rep
    edited July 12

    I did this a while ago for fun, 8 x 256GB slabs in RAID0. Performance wasn't that much better than just having 1-2 slabs in RAID0 :) Ended up just cancelling and the remainder of the period was added to account credit so I could spend it on something better.

    RAID0 just added more points of failure with little performance gain for me on Frantech

    Thanked by 1Daniel15
  • jmaxwelljmaxwell Member

    @mhn said:

    @raindog308 said: Now it runs like a raped ape.

    he looks raped ?

  • raindog308raindog308 Administrator

    @HalfEatenPie said: Ahh XBRL. Yeah that reminds me I need to build another tool for that.

    I've also been using servers to process these datasets.

    XBRL is nasty. There are TAB-delim'd files available: https://www.sec.gov/dera/data/financial-statement-and-notes-data-set.html

    You'd think answering a simple question like "hey, I have all the accounting filings for ${company x} so let's calculate their quick ratio as of 2021 Q4" would be easy but nooooooo....you've got calculation arcs, presentation orders, and all this complexity because there are 15,000 possible Gaap entries...

    @dahartigan said: "News" eh?

    I thought we talked about low-end VPSes here.

    @Hxxx said: @raindog308 however what kind of laptop you have that is not good enough to process a workload that you are eating with only 2 vCores (threads) dedicated of a Ryzen 3900x. Unless is not capped. Not sure. Francisco is usually generous.

    Answer:

    @szarka said: Sounds like IO is the bottleneck, not CPU.

    @Advin said: RAID0 just added more points of failure with little performance gain for me on Frantech

    A 1TB slab just became available so this will be irrelevant moving forward :blush:

  • Otus9051Otus9051 Member

    @raindog308 said: raped ape

    how do you know its running like a raped ape huh?

  • raindog308raindog308 Administrator

    @Otus9051 said: how do you know its running like a raped ape huh?

    That's fair. I've never seen or made one.

    But it has a sprightly rhyme.

    Goosed moose?

    Scared hare?

    Spooked fluke? It's a kind of fish.

    Just doesn't have the catchiness.

  • @raindog308 said: XBRL is nasty. There are TAB-delim'd files available: https://www.sec.gov/dera/data/financial-statement-and-notes-data-set.html

    You'd think answering a simple question like "hey, I have all the accounting filings for ${company x} so let's calculate their quick ratio as of 2021 Q4" would be easy but nooooooo....you've got calculation arcs, presentation orders, and all this complexity because there are 15,000 possible Gaap entries...

    It's such a pain in the ass too. Because yeah and the API just sucks as well.

    I mean I'm fortunate that I (for now anyways) only need to focus on the 10-K filings right now but holy shit it's a bitch.

    I feel like on the API end they wanted to design it to do everything at once but also wanted to minimize the amount of actual work to do, so they built a "generalized" solution that's just half-baked and terrible for every data collection need.

    Blehhh

  • @raindog308 said: I thought we talked about low-end VPSes here.

    "Fake news"

    Thanked by 2raindog308 equalz
  • szarkaszarka Member

    @raindog308 said:

    I thought we talked about low-end VPSes here.

    I mean, it's called low-end talk. Obviously, it's about the level of conversation.

    Thanked by 3raindog308 ralf _MS_
Sign In or Register to comment.