New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Disappearing from Google Search Results
slowservers
Member, Host Rep
in Help
This is about SporeStack, not Slow Servers. Slow Servers has always performed terribly in search engine results!
SporeStack does okay on Brave search and is at the top of DuckDuckGo for "monero vps."
However, it seems to be delisted on Google. I believe it used to be at the top for the same query, or very nearly.
Admittedly, this might be my fault because of my robots.txt, attempting to block all sorts of AI-related bots. Now I specifically didn't block the usual "GoogleBot" or whatever it's called. But maybe this is impacting my SEO in another way?
I'd appreciate any advice you can offer.
Thank you!
Comments
Why are you blocking AI bots? These deliver significantly higher quality traffic.
I would probably check Google Search console and see if there are any issues.
In doubt, reverse it
This. The amount of people using AI for research and decision making is off the charts, it’s almost the default.
Google still drives traffic but it’s mostly people confirming their decision or just straight typing your business name to get to your website.
Right, if the search proxy can't return result, the AI will try to curl it through my box. But, if it can't, it will give up, unless I intervened and manually copy paste it.
Not all bots are scrapers. It could be your customer riding the clanker
Google is multipurpose, if you block training you will also block results generally
I've noticed it first sends feelers via GogoleBot then shortly after checks with Extended or others too and if rejected the whole thing is considered rejected
This is a very opinionated provider that would rather explicitly not have AI as customers, which I respect
Yeah, the blocking of bots is going to be a strong challenge. Really the entire way that the internet generates revenue is broken and needs to change... and that is a whole other "loaded" topic haha.
-you need to remove the sitewide Disallow: / (unless you truly want to hide the site from compliant crawlers too... which based on your query, you don't want. The broad block is a problem.
-replace the blanket blocking with narrower rules for truly private paths only
-leave googlebot unblocked, and make sure CSS, JS, and all page assets needed for rendering are crawlable.
-use noindex for pages you do not want indexed rather than the robots.txt file if your goal is search exclusion.
-re-submit the sitemap and request reindexing in search console after the fix.
If you want to block AI scrapers, do it at the firewall/CDN layer if needed instead of a blanket robots.txt block
that's the most fantastic robots.txt i've ever seen