New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Hourly VPS for Google Takeout
Hi,
I have problems downloading Takeout files from Google in bash. I tried curl/wget with different flags for redirecting/authentication but did not succeed.
My plan is to spin up a temporary VPS with xfce so that I can use a full browser with GUI to save the Takeouts. After that I will transfer the Takeouts to multiple storage VPS.
X11 forwarding, rclone or other workarounds are not an option because I like to keep my storage boxes clean.
The specs:
- 2 CPU cores
- 4GB RAM
- 300GB HDD/SSD
- 1Gbit with 1TB traffic
- Location can be anywhere with decent connectivity to Europe
The biggest problem I see is the traffic because I will definitely use that 1 TB. Which provider would be okay to let me spin up an instance, for let's say 24h, and let me use 1 TB of traffic? I don't want to abuse any provider and will happily pay for the traffic if pricing is fair 👍🏼
Thank you.
Comments
Hetzner Cloud is a great option for this - they provide plenty of bandwidth and storage, specifically one of their ARM instances.
this is too much effort, use docker linuxserver/firefox or linuxserver/chromium and save yourself some time for not setting up a full desktop environment along with it's remote desktop.
I do this all the time since I discover that I can actually use dockerized Firefox with webVNC (or is it KasmVNC in this case? Can't recall). Up and running in less than 5 mins.
And agreed with @Advin, Hetzner Cloud is still the best option to do this.
We don't have hourly, but we offer a Linux 'desktop' template with our VMs. It's Debian with a GUI and Apache Guacamole pre-setup so you can use RDP in browser instantly.
Steps to download takeout files on a headless server using wget without additional authentication.
Steps:
Make sure to add the quotes around the pasted_link_address.
Paperspace, save yourself a lot of time..
Why not download it to a PC via your regular Internet provider?
You would better use one of the extensions to save money and headaches:
https://chromewebstore.google.com/detail/curlwget/dgcfkhmmpcmkikfmonjcalnjcmjcjjdn (Chrome)
https://addons.mozilla.org/en-US/firefox/addon/cliget/ (Firefox)
the newer image uses KasmVNC by default, it works great
It sounds like you will download 300GBish from Google and to your machine, if you that in hourly VPS and terminate after that it'd trigger overuse charge for traffic, as hourly VPS's traffic per month will divided into hours too
Get P**NH*B take outs. Be happy and weird.
That remembers me that I never finished downloading my stuff. Add that to my never ending todo list
+1 to that.
Katapult gives £100 free credit that doesn't seem to expire to new customers.
Vultr gives $200 free credit to new customers. The credit expires after 2 weeks.
https://www.vultr.com/promo/try200/?ref=8709382 (no-aff)
Both are billed hourly and both credits are for testing.
Be aware of how they measure bandwidth usage and calculate overage.
debootstrapto make Debian image into directorysystemd-nspawn -bD path-to-bootstrapped-directoryHi dev_vps,
during my research I came along this method on Reddit and I tried it. It works fine for small exports but won't work with big ones somehow. Depending on the size of the export Google has different endpoints involved. I could backup smaller exports with this method but ~250GB of exports, each one bigger than 30GB, couldn't be handled with this method.
Hi valk,
during my research I also came along over this method. I installed curlwget on Chrome. The extension spit out a wget command (somehow curl was not offered) and it threw a 400 - Bad Response error. Will try out the Firefox extension, thank you!
@ScreenReader said:
Hi ScreenReader and ValdikSS,
both methods are new to me, thank you very much for the input!
I think I will use the docker image on a Hetzner Cloud instance to get the job done quickly first. After that I will play around and try to to combine both methods, systemd-container with docker installed inside. I like the idea to have a lightweight systemd-container where I can have docker installed for comfort when I need it and which will not alter my production environment because I only have to delete the folder 👍🏼
Google Cloud gives you $300 credit on first signup, I guess that should give you enough capacity to get your takeout out of there.
One would think that a Google Cloud VPS should have good bandwidth when downloading the takeout.
I don't think it would make much of a difference beyond being a bit lighter on resources as you'd likely just dump a prebuild diskimage onto the VM and not really setup the whole thing time and time again.
I have not looked into Google cloud specifically but usually those $XXX credits on signup offers have a pretty tight time restriction like valid for 1 month or something similar.
I signed up yesterday and today it says "91 days remaining". Kind of an odd number of days so I'm guessing it's 3 months.