New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Disappearing from Google Search Results
slowservers
Member, Host Rep
in Help
This is about SporeStack, not Slow Servers. Slow Servers has always performed terribly in search engine results!
SporeStack does okay on Brave search and is at the top of DuckDuckGo for "monero vps."
However, it seems to be delisted on Google. I believe it used to be at the top for the same query, or very nearly.
Admittedly, this might be my fault because of my robots.txt, attempting to block all sorts of AI-related bots. Now I specifically didn't block the usual "GoogleBot" or whatever it's called. But maybe this is impacting my SEO in another way?
I'd appreciate any advice you can offer.
Thank you!
Comments
Why are you blocking AI bots? These deliver significantly higher quality traffic.
I would probably check Google Search console and see if there are any issues.
In doubt, reverse it
This. The amount of people using AI for research and decision making is off the charts, it’s almost the default.
Google still drives traffic but it’s mostly people confirming their decision or just straight typing your business name to get to your website.
Right, if the search proxy can't return result, the AI will try to curl it through my box. But, if it can't, it will give up, unless I intervened and manually copy paste it.
Not all bots are scrapers. It could be your customer riding the clanker
Google is multipurpose, if you block training you will also block results generally
I've noticed it first sends feelers via GogoleBot then shortly after checks with Extended or others too and if rejected the whole thing is considered rejected
This is a very opinionated provider that would rather explicitly not have AI as customers, which I respect
Yeah, the blocking of bots is going to be a strong challenge. Really the entire way that the internet generates revenue is broken and needs to change... and that is a whole other "loaded" topic haha.
-you need to remove the sitewide Disallow: / (unless you truly want to hide the site from compliant crawlers too... which based on your query, you don't want. The broad block is a problem.
-replace the blanket blocking with narrower rules for truly private paths only
-leave googlebot unblocked, and make sure CSS, JS, and all page assets needed for rendering are crawlable.
-use noindex for pages you do not want indexed rather than the robots.txt file if your goal is search exclusion.
-re-submit the sitemap and request reindexing in search console after the fix.
If you want to block AI scrapers, do it at the firewall/CDN layer if needed instead of a blanket robots.txt block
that's the most fantastic robots.txt i've ever seen
Unfortunate but true. These are not the same bots as the nasty scrapers that ruin the web for training, and they won't even obey robots.txt in the first place.
I can't archive it, so here's posted for archive
User-agent: AddSearchBotUser-agent: AI2BotUser-agent: AI2Bot-DeepResearchEvalUser-agent: Ai2Bot-DolmaUser-agent: aiHitBotUser-agent: amazon-kendraUser-agent: AmazonbotUser-agent: AmazonBuyForMeUser-agent: AndibotUser-agent: AnomuraUser-agent: anthropic-aiUser-agent: ApplebotUser-agent: Applebot-ExtendedUser-agent: atlassian-botUser-agent: AwarioUser-agent: bedrockbotUser-agent: bigsur.aiUser-agent: BravebotUser-agent: Brightbot 1.0User-agent: BuddyBotUser-agent: BytespiderUser-agent: CCBotUser-agent: Channel3BotUser-agent: ChatGLM-SpiderUser-agent: ChatGPT AgentUser-agent: ChatGPT-UserUser-agent: Claude-SearchBotUser-agent: Claude-UserUser-agent: Claude-WebUser-agent: ClaudeBotUser-agent: Cloudflare-AutoRAGUser-agent: CloudVertexBotUser-agent: cohere-aiUser-agent: cohere-training-data-crawlerUser-agent: CotoyogiUser-agent: Crawl4AIUser-agent: CrawlspaceUser-agent: Datenbank CrawlerUser-agent: DeepSeekBotUser-agent: DevinUser-agent: DiffbotUser-agent: DuckAssistBotUser-agent: Echobot BotUser-agent: EchoboxBotUser-agent: FacebookBotUser-agent: facebookexternalhitUser-agent: Factset_spyderbotUser-agent: FirecrawlAgentUser-agent: FriendlyCrawlerUser-agent: Gemini-Deep-ResearchUser-agent: Google-CloudVertexBotUser-agent: Google-ExtendedUser-agent: Google-FirebaseUser-agent: Google-NotebookLMUser-agent: GoogleAgent-MarinerUser-agent: GoogleOtherUser-agent: GoogleOther-ImageUser-agent: GoogleOther-VideoUser-agent: GPTBotUser-agent: iAskBotUser-agent: iaskspiderUser-agent: iaskspider/2.0User-agent: IbouBotUser-agent: ICC-CrawlerUser-agent: ImagesiftBotUser-agent: imageSpiderUser-agent: img2datasetUser-agent: ISSCyberRiskCrawlerUser-agent: Kangaroo BotUser-agent: KlaviyoAIBotUser-agent: KunatoCrawlerUser-agent: laion-huggingface-processorUser-agent: LAIONDownloaderUser-agent: LCCUser-agent: LinerBotUser-agent: Linguee BotUser-agent: LinkupBotUser-agent: Manus-UserUser-agent: meta-externalagentUser-agent: Meta-ExternalAgentUser-agent: meta-externalfetcherUser-agent: Meta-ExternalFetcherUser-agent: meta-webindexerUser-agent: MistralAI-UserUser-agent: MistralAI-User/1.0User-agent: MyCentralAIScraperBotUser-agent: netEstate Imprint CrawlerUser-agent: NotebookLMUser-agent: NovaActUser-agent: OAI-SearchBotUser-agent: omgiliUser-agent: omgilibotUser-agent: OpenAIUser-agent: OperatorUser-agent: PanguBotUser-agent: PanscientUser-agent: panscient.comUser-agent: Perplexity-UserUser-agent: PerplexityBotUser-agent: PetalBotUser-agent: PhindBotUser-agent: Poggio-CitationsUser-agent: Poseidon Research CrawlerUser-agent: QualifiedBotUser-agent: QuillBotUser-agent: quillbot.comUser-agent: SBIntuitionsBotUser-agent: ScrapyUser-agent: SemrushBot-OCOBUser-agent: SemrushBot-SWAUser-agent: ShapBotUser-agent: Sidetrade indexer botUser-agent: SpiderUser-agent: TavilyBotUser-agent: TerraCottaUser-agent: ThinkbotUser-agent: TikTokSpiderUser-agent: TimpibotUser-agent: TwinAgentUser-agent: VelenPublicWebCrawlerUser-agent: WARDBotUser-agent: Webzio-ExtendedUser-agent: webzio-extendedUser-agent: wpbotUser-agent: WRTNBotUser-agent: YaKUser-agent: YandexAdditionalUser-agent: YandexAdditionalBotUser-agent: YouBotUser-agent: ZanistaBotDisallow: /Disallow: /secretscriptsAllow: /scriptsYou can do:
To display it as:
I know, but the usual triple grave accent doesn't work.
Huh, why isn't it working?
(test)
Edit: It's working for me.
Must be encoding/clipboard problem
Oh man, that's what I was afraid of. SporeStack isn't shrinking, but I get the feeling that it may not be doing as well as it could be.
The irony is that when I started SporeStack, I geared it towards ephemeral architecture and I specifically talked about autonomous systems being able to purchase servers. That was back in 2017! At the time, I think the thought was more around DAOs on Ethereum. Now, there's a very large group of capable AI agents that could actually be using it quite easily, which I'm probably blocking the majority of.
I guess at the time, the idea seemed cool. I like it less and less as time goes on.
My pickle is that I've setup SporeStack in a way that makes it very relevant to modern users, except that I have nerfed such usage to some extent. SporeStack, in some ways, makes sense to be as open market as possible.
The irony is that one of my customers (a very nice one) is selling AI agents off SporeStack servers, despite my efforts to deter such activity. I've even been emailed by an OpenClaws agent!
Actually, there's really nothing that shouldn't be crawled -- there's no private paths.
I had a "scripts" and "secretscripts" folder. Both had AI poison code in them. I am not sure if anything actually went into "secretscripts" or not. I figured that the worst bots might.
I don't have that setup. I'm not sure if I want to bother or not, but maybe it'd be worth it. I think I had it going at one point. I hate dealing with the megacorps.
--
Another quandry is that I could allow money from AI sources to come in, while not distributing it back to AI causes. This is tricky for SporeStack, with Vultr and DigitalOcean both being very AI aggressive. It's also not something I want to make money from, but then again I think a lot of SporeStack might fall under that category.
Even with Slow Servers, it's being used for VPNs specifically so people can have access to better AI models. My best efforts at making the least AI compatible infrastructure possible have not worked that well. So it seems that I'm both making some money off of it, but not as much as I could be. I am not a grand profiteer and a possible heretic at the end of the day.
Will I go the way of the Dodo bird? Will I compromise on my values? Am I already?
I do not know...
It's harder to follow a hard line with a family, when there's more consequences for not going with the flow.
I get the anti-AI angle, honestly. But the thing is, blocking bots with robots.txt mostly just hurts your own site traffic and visibility. The bad actor bots usually don’t care about robots.txt anyway and will just ignore it.
The bigger issue is that Disallow: / is blocking the good bots too, so Google and other legit crawlers can’t really see or understand the public site properly. That’s probably doing more damage than good.
If the goal is to keep the AI scrapers out without tanking search traffic, I’d look at a managed bot detection service at the CDN layer instead. That gives you a much better shot at blocking the junk while still letting the legit search bots through.
Lmao, what was the email about?