Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


(Dedicated) Server for running data analysis +8 cores
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

(Dedicated) Server for running data analysis +8 cores

iburgeriburger Member
edited December 2017 in Help

I'm looking for a cost effective way to run my data analysis scripts in the cloud. Currently, my workload takes 14 hours on my 4770k Intel with 4 cores (8 threads). My script uses multiprocessing, so 7 processes will be spanned. This pretty much maxes out my 4770k CPU.

Ideally I'm looking for:

  • low cost: < $20 / month

  • many cores (something like 8 or 12) and that I can run at 100%

  • something I could run for a few hours and then stop. (pay by the hour)

Comments

  • Shoecoin

    Thanked by 1inthecloudblog
  • It doesn't exist? :D

  • Yet.

    Thanked by 1ChristianDSH
  • vultr has a bunch of hourly dedicated instances nowadays. If you're able to split it over multiple physical servers that might be an option.

  • Scaleway has the "C2M" with 8 dedicated cores (Intel Atoms...) around 18€/Month or the "X64-30G" with 8 dedicated Intel Xeons for 50€/Month. Hourly billed, but the prices are without VAT.

  • Hourly will always be more expensive than monthly if you need it for the whole month, but your workload seems perfect for hourly.

    The standard trustworthy providers for that are DigitalOcean, Vultr, and Linode; others with excellent reviews on LET include LunaNode, iwStack, and VMHaus. Apologies to any others that may have slipped my mind.

  • Nothing like an 8 core Atom for number crunching!

  • @iburger said:
    It doesn't exist? :D

    Unrealistic budget, sorry. 20$ is not even going to get you said 4770k. You could get 3770 at about 30$ but thats probably as close as it gets.

  • You'll probably have better luck if you move to a distributed architecture; lots of little boxes rather then one big box. You're already partially there since you're using multiprocessing with message passing being the missing piece.

    With that being said, AWS Batch (https://aws.amazon.com/batch/) and AWS EC2 Spot Pricing (https://aws.amazon.com/ec2/spot/pricing/). Packet.net, OVH Public Cloud, and Scaleway might be options as well.

  • iburgeriburger Member
    edited December 2017

    Thank you all!

    I have so far really looked into AWS batch and Azure batch (the Microsoft version). These services really have been built with 'number crunching' in mind. It's really cool stuff.

    I think Vultr and Digital Ocean (DA), are nice, but also a bit more complicated to implement. You need to manage the infrastructure with Vultr and DA. So spinning up and shutting down instances, that is your responsibility. With AWS Batch, a lot of the management can be pushed off to Amazon, allowing you to focus on writing production code.

    While reading LET, I also came across this awesome idea: buy some 2nd hand intel server hardware. ("get an X5650", per msg7086).

  • I think your best bet other than buying hardware is get another cheap monthly dedi (e.g. Hetzner auction, or maybe worldstream.nl annual). They cost as much for a whole month as a few days would cost on a cloud host, plus you get a ton of storage and other such goodies.

  • jvnadrjvnadr Member
    edited December 2017

    If you rise a little your budget, you can get a dual Xeon 5420 (benchmark 6495 on passmark) for 30$ per month, plus ssd and /29 ip, at WSI. https://www.wholesaleinternet.net/out-of-stock/?id=281
    Or else, if you can set it up for a few hours to keep costs on ~20$, try a cloud provider like AWS or Linode

  • mkshmksh Member
    edited December 2017

    @jvnadr said:
    If you rise a little your budget, you can get a dual Xeon 5420 (benchmark 6495 on passmark) for 30$ per month, plus ssd and /29 ip, at WSI. https://www.wholesaleinternet.net/out-of-stock/?id=281
    Or else, if you can set it up for a few hours to keep costs on ~20$, try a cloud provider like AWS or Linode

    With $30 he could get an i7-3770 at Hetzner with around 9800 passmark. That's what i was hinting at when i told him about his budget being unrealistic.

    Edit: Sorry, it's just 9445 but there is an 4770 just now at €27.73 (incl. german VAT?) that's really 9840 passmark. Shit, can't justify to grab it. @WSS? :D

    Thanked by 1WSS
  • mksh said: incl. german VAT?

    Excluding. But it is really interesting... Especially if he does not need extra ip (this price is for single ip). I might have a look, also...

  • mkshmksh Member
    edited December 2017

    @jvnadr said:

    mksh said: incl. german VAT?

    Excluding. But it is really interesting... Especially if he does not need extra ip (this price is for single ip). I might have a look, also...

    Yeah their IPs aren't the cheapest with like €0.84/IP for up 6. After that it strangly gets more expensive. Prices are great (Intel Xeon E3-1246V3 at €28.57...) and Hetzner is really trustworthy/reliable considering their dirt cheap pricing. Been with them for almost 10 years now. Zero complaints.

  • mkshmksh Member
    edited December 2017

    Oops

  • jvnadrjvnadr Member
    edited December 2017

    I just bought a Hetzner server, an i7-2600 with 2x3TB disks, for 26€ and 2 extra ips for less than 0.7 each. This is my first Hetzner server (I have several French, a German, an English and some US servers). I thought, "what's the catch"? Well, the catch was that the 2 barracuda disks do have more than 53K hours on them. No errors, just over 6 years of workload... Let's see how this will go...

  • @jvnadr said:
    I just bought a Hetzner server, an i7-2600 with 2x3TB disks, for 26€ and 2 extra ips for less than 0.7 each. This is my first Hetzner server (I have several French, a German, an English and some US servers). I thought, "what's the catch"? Well, the catch was that the 2 barracuda disks do have more than 53K hours on them. No errors, just over 6 years of workload... Let's see how this will go...

    sounds like a good oppertunity to me actually... if you think about it.

  • jvnadrjvnadr Member
    edited December 2017

    needavps said: sounds like a good oppertunity to me actually... if you think about it.

    6 y/o disks are like living on the edge. They maybe last for another 6 years, they maybe start failing along with your valuable data... They are mechanical disks, not ssd... Won't last forever...

    Thanked by 1inthecloudblog
  • mkshmksh Member
    edited December 2017

    @jvnadr said:

    needavps said: sounds like a good oppertunity to me actually... if you think about it.

    6 y/o disks are like living on the edge. They maybe last for another 6 years, they maybe start failing along with your valuable data... They are mechanical disks, not ssd... Won't last forever...

    Are you running RAID 1? If so i wouldn't worry to much. Replacement is quick and painless. Last time i had a failing disk it took around 10 minutes from filing the replacement form to get the notice that my server was ready. You've got a point though. I really would'nt feel good about running those kind of disks in RAID 0 even if i have been quite lucky so far.

    Edit: If you have identical hours on both drives my guess would be that these are probably the disks that came with the server when it was originaly sold. Just think about what those Athlon boxes might contain lol

  • jvnadrjvnadr Member
    edited December 2017

    mksh said: Are you running RAID 1?

    Having so much hours on them? Of course, it's RAID1! Otherwise, it would be what the Italians saying: "Vivere pericolosamente"!

    mksh said: If you have identical hours on both drives

    Yes, this is my feeling, too. They have just an hour difference, so, they are the disks that came with the server when originally installed and they have just 12 power cycles. Well, let's see how this will go!

  • @jvnadr said:

    mksh said: Are you running RAID 1?

    Having so much hours on them? Of course, it's RAID1! Otherwise, it would be what the Italians saying: "Vivere pericolosamente"!

    Had to look this up but yes it's quite fitting. Did this with a server running on a degraded array with just 1 40k hours disk left for about 1 year but then there was nothing of value on it.

    They have just an hour difference, so, they are the disks that came with the server when originally installed and they have just 12 power cycles.

    Not suprising. In almost 10 years i haven't witnessed a single power outage so unless one does a lot of kernel updates this isn't to hard to archive. Also the 2600 + 2x3TB looks suspiciously like their managed line. Maybe you got a server that actualy didn't do a whole lot for the last 6 years.

    Well, let's see how this will go!

    Yep, good luck with your new server!

    Thanked by 1jvnadr
Sign In or Register to comment.