Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Compiling on a low end VPS - not fun
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Compiling on a low end VPS - not fun

iburgeriburger Member
edited February 2018 in Help

Compiling the Pandas python package on my Debian 8 KVM VPS (256mb) is taking a long time. More than 30 minutes. I also had to make a ram drive, cus there wasn't enuf memory to run the compiler. All quite annoying.

Is this normal? The package still isn't finished compiling. (30 minutes and counting ...)

https://pandas.pydata.org/

# $ pip3 install pandas

Comments

  • lemonlemon Member
    edited February 2018

    this question is like "guys my car is slow, is that normal?"

    how are we supposed to know your provider/specs?

    but yes, thats probably normal.

  • iburgeriburger Member
    edited February 2018

    Unfortunately, I don't have the exact specs. It's KVM, so they hide it. (virtual cpu) The best info I got is "SuperMicro and Intel E3/E5 serie CPUs."

    I guess I am spoiled with my own system, which is rather fast!

  • AnthonySmithAnthonySmith Member, Patron Provider

    iburger said: Is this normal?

    Well to be fair, you have the same amount of ram I had in about 2003 so yes, that is normal.

  • I'm learning stuff. Cheers.

    Next time Ill compile in a screen session, and just come back x-hours later.

  • You might want to consider to have the same system running at home on a larger VM and to do your compiling there.

  • Yeah these days compiling in such a small vm might result in swapping. Also are you going to run Pandas for anything compute intensive in that vm? It might not go so good.

    I just tried pip3 install pandas and it downloaded and installed a few other packages without compiling anything, so it finished right away. Do you mean you did some kind of manual build and it took a while?

    I can see that the amount of software that it installs is pretty large, so yeah, it's possible that compiling it could be slow. Anyway though, a 256MB KVM isn't likely to give you much sustained CPU, especially these days (think of how many of them are likely to be running in a host node, and divide the number of physical cpus by that). They're intended more for things like monitoring scripts, vpns, low traffic web sites etc.

    Thanked by 1iburger
  • raindog308raindog308 Administrator, Veteran

    iburger said: Compiling the Pandas python package on my Debian 8 KVM VPS (256mb) is taking a long time. More than 30 minutes. I also had to make a ram drive, cus there wasn't enuf memory to run the compiler. All quite annoying.

    I think you need to reread what you wrote. "I also had to make a ram drive". Dude, you don't have enough RAM on your VPS! In order to compile your code, you apparently have to use hard disk to simulate RAM (orders of magnitude slower).

    And by the way, Pandas is a data analysis package. Data analysis usually means "uses lots of CPU and RAM".

    I love LEBs as much as anyone, but...you need a bigger box.

    Ah, waiting for compiling...yeah, I remember that from last decade. My golang codebase compiles in seconds ;-)

  • @iburger said:
    Compiling the Pandas python package on my Debian 8 KVM VPS (256mb) is taking a long time. More than 30 minutes. I also had to make a ram drive, cus there wasn't enuf memory to run the compiler. All quite annoying.

    Is this normal? The package still isn't finished compiling. (30 minutes and counting ...)

    https://pandas.pydata.org/

    # $ pip3 install pandas

    Get a Kimsufi box if you want to do this in proper low end fashion. If you are too poor for that, please abandon the project.

  • williewillie Member
    edited February 2018

    I just did pip3 install pandas on a newly created Debian 9.3 instance after apt-get install python3-pip (which installed a lot of dev stuff). Again the pip3 install didn't compile anything, so the whole install ran pretty fast.

    If you want us to test the compilation speed, can you say exactly what you did to run this compilation? The pandas.pydata.org page leads to a lot of different download files and I don't want to mess with installing them separately. I can say for sure that a full build of everything from scratch will take a while.

    Anyway, as raindog308 says, you need a bigger box. That was obvious even when I missed the part about the ramdisk. A kimsufi is overkill unless you're doing heavy computation but do try to get something with a reasonable amount of ram and cpu. The test I just did was with a 1GB ram, $5/month digitalocean.com instance (effectively free because I spun it down immediately afterwards) but there are some comparable alternatives that would cost a little less on a monthly basis.

    Thanked by 1iburger
  • @raindog308 said:
    I love LEBs as much as anyone, but...you need a bigger box.

    @willie said:
    you need a bigger box.

    Thanked by 2MasonR raindog308
  • seanhoseanho Member
    edited March 2018

    I hope you meant a swap file/partition (using disk to supplement RAM), and not a ramdisk (using a portion of RAM as disk).

    You might start to run afoul of your provider's shared CPU usage expectations -- if not on the compile, then on whatever it is you're planning to do with pandas. Certainly worth looking into a dedi or an hourly provider with lenient CPU rules.

    If installing pandas via pip is doing a lot of compiling (which it shouldn't, on any common distro), you could also try using Anaconda, which gives you pre-compiled numpy, scipy, jupyter, etc.

  • This is what Kimsufi is for. Learning and low $/GB-ram

  • raindog308raindog308 Administrator, Veteran

    vimalware said: This is what Kimsufi is for. Learning

    That's what Virtualbox is for.

    Whatever the OP is connecting from likely has more power than what he's connecting to. They haven't made 256MB PCs in quite a while.

    Thanked by 2netomx iburger
  • @willie said:
    Yeah these days compiling in such a small vm might result in swapping. Also are you going to run Pandas for anything compute intensive in that vm? It might not go so good.

    I just tried pip3 install pandas and it downloaded and installed a few other packages without compiling anything, so it finished right away. Do you mean you did some kind of manual build and it took a while?

    Yes, it did revert to a manual build. And it took 60+ minutes! Hah.

    Today, I installed vagrant and virtualbox and started messing with Ubuntu 16.04. It's sooo much easier to get what I want.
    I learned for instance that on Ubuntu you can install two python versions, the most recent from the deadsnakes PPA.

    $ python3     #-> from ubuntu distro
    $ python3.6   #-> from ppa
    

    ; magic stuff!

    Ubuntu also has a PPA for the last release of neovim. Honestly, I don't understand why Debian doesn't upgrade some of their packages quicker. Nobody wants to run Neovim 1.0.x something. All your plugins will break.

  • iburger said: Honestly, I don't understand why Debian doesn't upgrade some of their packages quicker.

    Stability and security, just to name a few.

    https://debian-handbook.info/browse/stable/sect.release-lifecycle.html

    Thanked by 1ucxo
  • NeoonNeoon Community Contributor, Veteran

    Reminds me of compiling a WoW Private server on a Pentium 4, it was awful.

    Even worse, it was full of shitty bugs, not worth it.

  • raindog308raindog308 Administrator, Veteran

    iburger said: I learned for instance that on Ubuntu you can install two python versions,

    You can do that on any distro. Heck, you can do that on Windows.

    iburger said: ; magic stuff!

    There's really no magic here.

    iburger said: Nobody wants to run Neovim 1.0.x something.

    From what I've seen, nobody wants to run Neovim anything. Fortunately, real vim is still available.

    Thanked by 2ucxo Aidan
  • iburgeriburger Member
    edited March 2018

    @seanho said:

    If installing pandas via pip is doing a lot of compiling (which it shouldn't, on any common distro), you could also try using Anaconda, which gives you pre-compiled numpy, scipy, jupyter, etc.

    That was a really good point seanho.

    I have learned a few lessons now, as they relate to Python and package management on linux.

    1) Go through pip, don't go through apt-get. As the official docs of Pandas say, the 'packages in the linux package managers are often a few versions behind'. -pandas docs

    2) Install the package python3-virtualenv if you have trouble making venvs. Normally, python 3.4.x should allow you to make a venv with python -m venv venv but this did not work for me on Deb 8! (solution apt-get install python3-virtualenv.)

    3) Use a recent distribution. Debian 9 has Python 3.5.2, which should fix any virtual env wrapper problems. As @willie noted, on Debian 9.3 the pandas installation was quite fast. I also saw this with Ubuntu 16.04. Recent is better.

  • iburgeriburger Member
    edited March 2018

    Also, I'm happy to report that everything works now! Thanks for all the good advice.

    -- I will continue my journey as a Ubuntu ppa wizzard now, but to each his own (neo) vim. :)

  • Glad to hear things worked out for you. I've run into similar issues with R packages; the Debian packages are of course going to be fairly old.

  • @iburger said:

    >

    Honestly, I don't understand why Debian doesn't upgrade some of their packages quicker.

    It's very easy to understand. Released distributions almost never upgrade their packages to new feature versions. That's the principle of a distribution release -- only bug fixes goes to a release.

    For example, Debian Jessie comes with Linux 3.16, that package will be 3.16 for its whole life time, and you'd never see 3.18 or 4.x on it. Same applies to RHEL and CentOS, and most part of Ubuntu.

    PPA is sort of, out of distribution packages. And you can install PPAs on Debian as well.

Sign In or Register to comment.