New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Where is a good location to host a Usenet Indexer?
yowmamasita
Member
I'm working on a usenet indexer project Ive been building from scratch. Where's a good location to host it? Thanks!
I'm not downloading copyrighted material, I just grab headers from it. Nzbmatrix, a usenet index site, reportedly received massiv DMCA takedowns and decided to close.
Comments
Since you're not downloading copyrighted material, we can host you. We have a few other people running nzb*-esque programs.
@Damian What would you do if DMCA take down notices were issued?
Every now & then you gotta have some balls and tell them to shove it. Not a step you want to take on faith, but when you know you're on solid ground, don't let them bully you.
If they're valid, we're required to act upon them. So far, out of ~50 DMCA notices received in our two years of operation, only two have been valid so far.
How do you know if it's valid? Sorry if this is a dumb question but I really have no idea
Majority (if not all) of the DCMAs are all automated by bots. They pick up a TON of false positives.
I thought about running one, but the legal implications in UK (where my company is), Liechtenstein (where the holding is) and my location (Slovenia) told me it's not good to do it... beware of the attached risks, don't be naive.
AFAIK it should be fine in the US however, if you react to valid DMCA.
I don't think indexing usenet headers is actually illegal anywhere, is it?
That's what i wonder too. But why is it that a number of usenet indexers are shutting down?
Have any pure indexers been shut down? I thought it was just the sites that took submissions from users so quality was higher, as opposed to raw indexers like binsearch.info
They shut down because their payment gateways don't cooperate.
stupid question, what is the usage of usenet indexer?
So rather than having to trawl through all the various usenet groups separately to find what you want, you have a consolidated index of all the post headers (usually searchable) to help you track down things. It's basically a search engine for usenet, but most sites include a few more bells and whistles.
http://www.lowendtalk.com/discussion/6996/newznab-friendly-host
I've run it on a few different providers and all have been accepting of it once confirmed it was an indexer only and no content was being downloaded.
Currently have http://get.nzb.es running with http://getkvm.com
Out of curiousity, what spec are you running that on, and how much are you indexing?
Currently running on a GetKVM node:
RAM: 2560
SWAP: 2560
DISK: 45GB
TRAFFIC: 45GB
PORT: 100Mb
I've run it on a machine with 512Mb in the past and performance was fairly acceptable.
From the testing I've been doing, RAM helps a little but the biggest influence effecting the server load is disk IO.
Before running it on this node I tested on a 512Mb node that was SSD based and load was pretty much nothing. I moved it to a NodeDeploy server with 2GB RAM/1GB SWAP and again load was seldom above .2 when indexing and backfilling groups.
I moved it to the GetKVM server as I wanted a little more diskspace for testing and thought the extra ram wouldn't hurt.
In reality, the ram helps a little at times, but the disk io on this node is significantly lower which really pushes up the load on the server during indexing which is a shame.
The NodeDeploy node gave me about 240MB/s when i tested its io, the GetKVM is at best 70MB/s but typically is nearer 50-60MB/s.
Given the lower IO when indexing it regularly pushes the load over 1 and at times in the past when i was indexing and backfilling it was hitting 2.
I think I will probably move it away from this server when I've finished testing a few other things.
Ideal server would be 1GB ram and 20Gb disk (faster the better). I would say ideal IO would be 150MB/s+ Anything less than 100MB/s will really push your load up if you are indexing a lot of groups.
I was more curious because I'm running it off a KimSufi mKS, and although a tenner a month is a small price to pay really, I do wonder if I couldn't downsize and still keep the same performance...
My NodeDeploy node was $48/ year which was ideal. If it wan't for the extra disk space i wanted for testing it would still be there.
I likely will move it back there or similar host in the near future.
@prae5
One of the restrictions of our OpenVZ based VPS, 2x disk RAID1, which is completely sufficient for the pricing IMHO (And your node is full) but for tasks such as this that require a lot of I/O throughput i can see how it could cause problems such as the ones you have described.
If you like our service and wish to stay we can also work on a KVM deal for you, these servers run HW RAID10 with BBU and offer a lot more I/O to work with. Just let us know
Don't get me wrong - I wasn't knocking the service at all. I think its great for the price - i completely understand the restrictions given the price point.
I'm more than happy with the service - I was just describing the ideal server for indexing.
I've been working on a few optimisations on my dev server, which I'm looking to push in the near future - it may help. If not will be happy to see what else you can offer.
I'm more than happy with the service - I was just describing the ideal server for indexing.
I've been working on a few optimisations on my dev server, which I'm looking to push in the near future - it may help. If not will be happy to see what else you can offer.
Thank you for the kind words, we appreciate it, im glad you're happy
I must admit, running it on a server with hardware RAID is appealing, right now I'm reliant on offsite backups as if I lose the drive in my mKS I'll just get a fresh one.
San Marino
http://www.copyright.gov/circs/circ38a.pdf
http://www.exoticvps.com/
http://www.copyright.gov/circs/circ38a.pdf
http://www.exoticvps.com/
Unnecessarily expensive considering indexing usenet isn't illegal.
In reality, the ram helps a little at times, but the disk io on this node is significantly lower which really pushes up the load on the server during indexing which is a shame.
Given the lower IO when indexing it regularly pushes the load over 1 and at times in the past when i was indexing and backfilling it was hitting 2.
Ideal server would be 1GB ram and 20Gb disk (faster the better). I would say ideal IO would be 150MB/s+ Anything less than 100MB/s will really push your load up if you are indexing a lot of groups.
Thanks, great info!
@Freek IMO you might want more disc than that if you're going to backfill quite a lot - I used one of the community provided nzb packs, and the SD TV pack was nearly 15GB on it's own.
Member
@Freek IMO you might want more disc than that if you're going to backfill quite a lot - I used one of the community provided nzb packs, and the SD TV pack was nearly 15GB on it's own.
That space is only temp required during the backfill process - I used an sshfs temp mount for that.
I'm not sure I get that - the import is a bunch of NZB's that newznab parses and adds to the index, but the nzbs are still required, so the filesize doesn't change surely?
@prae5 We offer disk upgrades you know