Wanted: Dirt cheap storage for secondary backups [EU]

Falzo

@ferri said:
btw need suggestion from all of you guys,

for storage only and rsync with so many files (mostly images), do you prefer 2 core 512 RAM or 1 core 1024 RAM?

I've tried to running rsync with option like this, but still eat my CPU

/usr/bin/nice -n 19 /usr/bin/ionice -c2 -n7 \
    rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -o Compression=no -x" \
    <user>@<host>:<path> <local-path>

Thanks

first check in top if it is really hogging cpu or more likely running into IO limit, then you'll see IO wait going close to 100% and load rising accordingly. I use rsync a lot and it shouldn't be a problem for cpu at all. I never switch off compression or something like that, as it will only prolongue the process and cause more traffic.

ferri

@Falzo said:

@ferri said:
btw need suggestion from all of you guys,

for storage only and rsync with so many files (mostly images), do you prefer 2 core 512 RAM or 1 core 1024 RAM?

I've tried to running rsync with option like this, but still eat my CPU

/usr/bin/nice -n 19 /usr/bin/ionice -c2 -n7 \
    rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -o Compression=no -x" \
    <user>@<host>:<path> <local-path>

Thanks

first check in top if it is really hogging cpu or more likely running into IO limit, then you'll see IO wait going close to 100% and load rising accordingly. I use rsync a lot and it shouldn't be a problem for cpu at all. I never switch off compression or something like that, as it will only prolongue the process and cause more traffic.

Edited: btw can you tell me how to do exactly about this?

hogging cpu or more likely running into IO limit

Thanks for ur reply

Update:
Btw i'm using nixstat, when rsync is running, my cpu usage sometimes hit 100% and load avg only 1.x.

Update 2:

From my top

%Cpu(s):  1.3 us,  2.0 sy,  0.0 ni,  0.0 id, 96.7 wa,  0.0 hi,  0.0 si,  0.0 st

seems like my I/O wait too high, reached 96.7 while when goes more than 10 considered as bad, right?

Any idea how to reduce that I/O limit?

Thanks

Falzo

@ferri said:

while your rsync process is running (in the background or screen I assume) simply type 'top' in your cli to see live stats on the console. look at the third line in the header where it shows the CPU usage. numbers are percent, the one named 'wa' is what you want to watch, if it is high while running rsync the cpu isn't the limiting factor at all, but the disk/filesystem is...

https://www.lifewire.com/linux-top-command-2201163

Falzo

ferri said: From my top

%Cpu(s): 1.3 us, 2.0 sy, 0.0 ni, 0.0 id, 96.7 wa, 0.0 hi, 0.0 si, 0.0 st

seems like my I/O wait too high, reached 96.7 while when goes more than 10 considered as bad, right?

Any idea how to reduce that I/O limit?

ah sorry, seen this too late ;-)

but as guessed, the CPU isn't your bottleneck here. on a VPS you can't do anything about running into that limit. it is either that the disks are at their limit or the hostnode is limiting IO on purpose to leverage the load on the disk for all neighbours. it shouldn't do much harm other then slowing down your rsync because it needs to wait for IO access.

so for your initial question on the hosthatch storage VM, I'd rather go with more RAM then a second core if all you do is moving files.

ferri

Thanks @Falzo

Yes, my top look like this

%Cpu(s):  1.3 us,  2.0 sy,  0.0 ni,  0.0 id, 96.7 wa,  0.0 hi,  0.0 si,  0.0 st

Any tips for lowering this value (96.7 wa)? with ulimit -Hn and ulimit -Sn I've already get 65536

Thanks

ferri

@Falzo said:

ferri said: From my top

%Cpu(s): 1.3 us, 2.0 sy, 0.0 ni, 0.0 id, 96.7 wa, 0.0 hi, 0.0 si, 0.0 st

seems like my I/O wait too high, reached 96.7 while when goes more than 10 considered as bad, right?

Any idea how to reduce that I/O limit?

ah sorry, seen this too late ;-)

but as guessed, the CPU isn't your bottleneck here. on a VPS you can't do anything about running into that limit. it is either that the disks are at their limit or the hostnode is limiting IO on purpose to leverage the load on the disk for all neighbours. it shouldn't do much harm other then slowing down your rsync because it needs to wait for IO access.

so for your initial question on the hosthatch storage VM, I'd rather go with more RAM then a second core if all you do is moving files.

Thanks a lot for your reply, then i think i dont need to worry about it

Have a nice day

Falzo

@ferri said:
Thanks @Falzo

Yes, my top look like this

%Cpu(s):  1.3 us,  2.0 sy,  0.0 ni,  0.0 id, 96.7 wa,  0.0 hi,  0.0 si,  0.0 st

Any tips for lowering this value (96.7 wa)? with ulimit -Hn and ulimit -Sn I've already get 65536

Thanks

addon: you most likely don't want to limit that for your rsync process anyways, because it'll only slow it down further without any real benefit. using ionice like you already do, to give other processes priority in IO should be the right way, other then that there is nothing wrong to it, if your system maxes out the available ressources/IO. it's just the disturbing look of it and any stats tool eventually freaking out about reaching 100% at some column ;-)

I experimented with limiting IO for single processes using cgroup, start reading here: https://unix.stackexchange.com/questions/48138/how-to-throttle-per-process-i-o-to-a-max-limit

yet this most likely won't do more then slow your rsync down and have a better cosmetical view on the stats ;-)

hosthatch

@ferri said:
Thanks @Falzo

Yes, my top look like this

%Cpu(s):  1.3 us,  2.0 sy,  0.0 ni,  0.0 id, 96.7 wa,  0.0 hi,  0.0 si,  0.0 st

Any tips for lowering this value (96.7 wa)? with ulimit -Hn and ulimit -Sn I've already get 65536

Thanks

As @Falzo mentioned, iowait is generally caused by either the host limiting your IO, or someone else abusing the IO going unnoticed, etc (basically just not enough IO being available vs the in-usage amount). Maybe contact the host to see if they can help.

ferri

@Falzo said:

@ferri said:
Thanks @Falzo

Yes, my top look like this

%Cpu(s):  1.3 us,  2.0 sy,  0.0 ni,  0.0 id, 96.7 wa,  0.0 hi,  0.0 si,  0.0 st

Any tips for lowering this value (96.7 wa)? with ulimit -Hn and ulimit -Sn I've already get 65536

Thanks

addon: you most likely don't want to limit that for your rsync process anyways, because it'll only slow it down further without any real benefit. using ionice like you already do, to give other processes priority in IO should be the right way, other then that there is nothing wrong to it, if your system maxes out the available ressources/IO. it's just the disturbing look of it and any stats tool eventually freaking out about reaching 100% at some column ;-)

I experimented with limiting IO for single processes using cgroup, start reading here: https://unix.stackexchange.com/questions/48138/how-to-throttle-per-process-i-o-to-a-max-limit

yet this most likely won't do more then slow your rsync down and have a better cosmetical view on the stats ;-)

Thanks, definitely will read that

Right now i'm trying to use -z option to minimize bw usage as ur post above

ferri

@Abdullah said:

@ferri said:
Thanks @Falzo

Yes, my top look like this

%Cpu(s):  1.3 us,  2.0 sy,  0.0 ni,  0.0 id, 96.7 wa,  0.0 hi,  0.0 si,  0.0 st

Any tips for lowering this value (96.7 wa)? with ulimit -Hn and ulimit -Sn I've already get 65536

Thanks

As @Falzo mentioned, iowait is generally caused by either the host limiting your IO, or someone else abusing the IO going unnoticed, etc (basically just not enough IO being available vs the in-usage amount). Maybe contact the host to see if they can help.

Thanks for your suggestion, seems like its limit from them

Laxenade

@Abdullah said:

@ferri said:
Thanks @Falzo

Yes, my top look like this

%Cpu(s):  1.3 us,  2.0 sy,  0.0 ni,  0.0 id, 96.7 wa,  0.0 hi,  0.0 si,  0.0 st

Any tips for lowering this value (96.7 wa)? with ulimit -Hn and ulimit -Sn I've already get 65536

Thanks

As @Falzo mentioned, iowait is generally caused by either the host limiting your IO, or someone else abusing the IO going unnoticed, etc (basically just not enough IO being available vs the in-usage amount). Maybe contact the host to see if they can help.

Do you limit the IOPS on the storage vps? Or what IOPS can I expect.

hosthatch

@Laxenade said:
Do you limit the IOPS on the storage vps? Or what IOPS can I expect.

We generally do not need to, unless you are abusing.

https://i.imgur.com/ZhWAwWN.png

This is from our monitoring (past 6 hours) from one of the full storage nodes. There is no abuse. But if any of those servers spiking were consistently like that for a few hours, they would be throttled with an appropriate limit.

I know it's a vague answer, but the basic definition is that you should use a storage VPS for storage related purposes.

uptime

https://i.imgur.com/ZhWAwWN.png

>

This is from our monitoring (past 6 hours) from one of the full storage nodes.

That's a pretty picture. I think I can tell which trace is mine (self-throttled at a touch under 1 Mbit - no hurry since will be loading for a few days - faster speed will be nice for occasional downloads but don't need so much for the upload at the moment, so easy does it - "slow and steady wins the race" as they say ...)

maxwell

@Abdullah

Any LookingGlass Pages for the EU Locations ?

Harambe

@maxwell said:
@Abdullah

Any LookingGlass Pages for the EU Locations ?

https://hosthatch.com/features#datacenters

user123

I'm guessing no one is able to come close to matching some of the current servers I'm looking to migrate?

1) $102/3 years for 4vCPU/3GB RAM/1TB HDD/Unmetered BW on burstable 1Gbit (KVM) [ironically, this CPU is a beast and is the most robust out of all the VPS I've had over the years]

2+3) $20.62/year for 2vCPU/2GB RAM/500GB HDD/Unmetered BW on burstable 1Gbit (KVM)

vimalware

I'm all stocked up on storage from wishosting 1800GB Columbus day ( $6/m)

It's still available at $9/m last I checked

emptyPD

@Falzo said:

@Golbinex said:
@Falzo 750 GB is enough for me, I only hope they won't have problems like ZXHost, I would never host any production stuff on lowend servers, but I still wouldn't want to lose what I have there.

I wouldn't consider them lowend. I do host production stuff there, though storage is going to be used to hold only backups anyway... am a customer for more than 3 years now, haven't had any issues so far, never needed their support.

me too, i have production stuff with them, not a single problem or downtime, 1 and a half year.