Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Quick Links


Categories

In this Discussion

› › Help me with my website and make me happy
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Help me with my website and make me happy

woohoowoohoo Member
edited August 2021 in Help

I've tried everything. No Google tool can access my URL. Even the sitemap returns "Couldn't fetch". The Request Indexing returns "Indexing request rejected. During live testing, indexing issues were detected with the URL".

No error code or anything, just nothing works. It's a totally new domain so it has never been working before and stopped working. DNS is fully propagated though.

URL (trying to index): https:// hundforsakringarna () .se
Sitemap (trying to submit): https:// hundforsakringarna () .se/sitemap_index.xml
Robots.txt: https:// hundforsakringarna () .se/robots.txt

The VPS hosts 5-10 other sites that I’ve set up in exactly the same way and all of them works without any issues. I just can’t seem to figure this one out.

«1

Comments

  • yoursunnyyoursunny Member, IPv6 Advocate

    You can't have spaces in the URI.
    See https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_is_a_URL for how to correctly write an URI.
    Resubmit as https://hundforsakringarna.se and it should work.

  • woohoowoohoo Member

    @yoursunny said:
    You can't have spaces in the URI.
    See https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_is_a_URL for how to correctly write an URI.
    Resubmit as https://hundforsakringarna.se and it should work.

    Nope ;(

    https://search.google.com/test/mobile-friendly?utm_source=gws&utm_medium=onebox&utm_campaign=suit&id=mx8MTL3v3ap3A6j2cY1EQA

  • woohoowoohoo Member
    edited August 2021

    I'm transferring the site from a VPS to a DirectAdmin Shared environment to see if it has anything to do with that. Will update in 5.

  • woohoowoohoo Member

    Now the site is a default WP installation on NEW host and the result is the same, Google can't crawl it.

  • gianggiang Veteran
    edited August 2021

    Check if you have robot.txt file. Delete or rename it.

    NVM, I missed it… Just ignore this comment.

    Try this: https://www.godaddy.com/help/allow-search-engine-indexing-in-wordpress-27809

  • frogfrog Barred
    edited August 2021

    Select Post name on this page: WP-admin->Settings->permalinks

  • woohoowoohoo Member

    @frog said:
    Select Post name on this page WP-admin->Settings->permalinks

    Done, has nothing to do with that though. Google can't even lookup the domain.

  • woohoowoohoo Member

    @giang said:
    Check if you have robot.txt file. Delete or rename it.

    NVM, I missed it… Just ignore this comment.

    Try this: https://www.godaddy.com/help/allow-search-engine-indexing-in-wordpress-27809

    That's just for disabling the noindex tag in WP, it's disabled already. Google can't even lookup the domain really. Thanks for helping bud but that's not it.

  • gianggiang Veteran

    @woohoo said:

    @giang said:
    Check if you have robot.txt file. Delete or rename it.

    NVM, I missed it… Just ignore this comment.

    Try this: https://www.godaddy.com/help/allow-search-engine-indexing-in-wordpress-27809

    That's just for disabling the noindex tag in WP, it's disabled already. Google can't even lookup the domain really. Thanks for helping bud but that's not it.

    Google bot will need some time to re index again.

  • frogfrog Barred

    @woohoo said:

    @frog said:
    Select Post name on this page WP-admin->Settings->permalinks

    Done, has nothing to do with that though. Google can't even lookup the domain.

    Before this change, the robots and sitemap URLs did not work in my browser. They do now.

  • woohoowoohoo Member

    @giang said:

    @woohoo said:

    @giang said:
    Check if you have robot.txt file. Delete or rename it.

    NVM, I missed it… Just ignore this comment.

    Try this: https://www.godaddy.com/help/allow-search-engine-indexing-in-wordpress-27809

    That's just for disabling the noindex tag in WP, it's disabled already. Google can't even lookup the domain really. Thanks for helping bud but that's not it.

    Google bot will need some time to re index again.

    It was never enabled. You can see, live, how Google interprets the site using this tool: https://search.google.com/test/mobile-friendly. Dosen't work.

  • woohoowoohoo Member

    @frog said:

    @woohoo said:

    @frog said:
    Select Post name on this page WP-admin->Settings->permalinks

    Done, has nothing to do with that though. Google can't even lookup the domain.

    Before this change, the robots and sitemap URLs did not work in my browser. They do now.

    No, I transferred the site from the VPS to a web host and I didn't enable any plugins, since there wasn't a SEO plugin, these pages wasn't activated, so I installed Yoast SEO and now they're there again. They wasn't there for maybe 1-2 min.

  • frogfrog Barred
    edited August 2021

    It might be a DNS problem.
    You don't have a DNS record for www.
    Check if your txt record with google-site-verification is correct.

  • gianggiang Veteran

    @woohoo said:

    @giang said:

    @woohoo said:

    @giang said:
    Check if you have robot.txt file. Delete or rename it.

    NVM, I missed it… Just ignore this comment.

    Try this: https://www.godaddy.com/help/allow-search-engine-indexing-in-wordpress-27809

    That's just for disabling the noindex tag in WP, it's disabled already. Google can't even lookup the domain really. Thanks for helping bud but that's not it.

    Google bot will need some time to re index again.

    It was never enabled. You can see, live, how Google interprets the site using this tool: https://search.google.com/test/mobile-friendly. Dosen't work.

    Other tools like: PageSpeed Insights and Web.dev are working fine.

    So I think you should just wait.

    Thanked by 1yoursunny
  • kkrajkkkrajk Member
    edited August 2021

    delete the robots.txt and try again..

    Also make sure the following is unchecked

    Yoast SEO plugin settings -- see if you have settings that might influence bot crawl / indexing

  • woohoowoohoo Member

    Sadly nothing helped so far, and it’s not a matter of waiting. Google can’t aceess the URL at all, nothing is wrong with the web files, Google can’t even access them.

  • nvmenvme Member

    DNS change takes time to penetrate.
    You can flush it manually - https://developers.google.com/speed/public-dns/cache

    If it still doesn't work after sometime, try to disable cloudflare proxy and check.

    Thanked by 1yoursunny
  • Mr_TomMr_Tom Member, Host Rep

    It can't be a DNS/IP/etc issue as I can do a Google PageSpeed test on the site mentioned above, so google can connect to it.

    From looking online the error you're getting would suggest google has previously had issues (search engine visibility off in WP, disallow rules in robots.txt for example) and is still using the old rules.

    If you've not previously had this, check no plugins are blocking googles crawl bots, etc.

    I'm not big up on SEO but might help.

  • woohoowoohoo Member

    @nvme said:
    DNS change takes time to penetrate.
    You can flush it manually - https://developers.google.com/speed/public-dns/cache

    If it still doesn't work after sometime, try to disable cloudflare proxy and check.

    Already did this.

    @Mr_Tom said:
    It can't be a DNS/IP/etc issue as I can do a Google PageSpeed test on the site mentioned above, so google can connect to it.

    From looking online the error you're getting would suggest google has previously had issues (search engine visibility off in WP, disallow rules in robots.txt for example) and is still using the old rules.

    If you've not previously had this, check no plugins are blocking googles crawl bots, etc.

    I'm not big up on SEO but might help.

    Thing is Google can connect, but not using the tools using the same infrastructure as Googlebot (eg. The mobile test).

    No plugins are blocking, removed eveything. This is a default WP installation with Yoast.

  • Mr_TomMr_Tom Member, Host Rep

    Well google can load a website which is using the same IP too.

    Have you tried setting the webserver to force clients to retrieve files rather than use their cached version? This might make google get a new version if it's cached a previous robots.txt that was blocking it?

  • nvmenvme Member

    May be try disabling Cloudflare proxy, turn on Development mode and check? Could be it is serving stale/ cached content. With dev mode on, it should bypass request and hit your origin.

  • frogfrog Barred

    Read this: https://support.google.com/webmasters/answer/9044175

  • woohoowoohoo Member
    edited August 2021

    @frog said:
    Read this: https://support.google.com/webmasters/answer/9044175

    Thanks, no manual action.

    @Mr_Tom said:
    Well google can load a website which is using the same IP too.

    Have you tried setting the webserver to force clients to retrieve files rather than use their cached version? This might make google get a new version if it's cached a previous robots.txt that was blocking it?

    There was never any robots.txt that was blocking access. However,

    @nvme said:

    May be try disabling Cloudflare proxy, turn on Development mode and check? Could be it is serving stale/ cached content. With dev mode on, it should bypass request and hit your origin.

    It might have been a CloudFlare error. CloudFlare blocking Googlebot for some reason. I’ve just now changed nameservers to my registrars default ones instead and once they’ve propogated I’ll see if Google can access the website.

    Thanks everyone for the tips so far!

  • Mr_TomMr_Tom Member, Host Rep

    Ah I checked the IP but it didn't look like a Cloudflare one.

  • woohoowoohoo Member
    edited August 2021

    So I:

    Removed everything, installed Wordpress fresh on a apache web host instead of my Nginx VPS

    Changed nameservers from Cloudflare to my registrars

    Still the same issue. Google is blocking the domain for some reason..

    Btw yes the DNS is done for the Googlebot: https://toolbox.googleapps.com/apps/dig/#NS/

  • yoursunnyyoursunny Member, IPv6 Advocate
    1. Do 7 push-ups every day.
    2. Google will reach your website after you've finished 53 push-ups.
    3. DNS propagation takes as much as 7 days, you know.
  • woohoowoohoo Member
    edited August 2021

    @yoursunny said:
    1. Do 7 push-ups every day.
    2. Google will reach your website after you've finished 53 push-ups.
    3. DNS propagation takes as much as 7 days, you know.

    Google recognizes the new NS records. It still dosen't work though.

  • yoursunnyyoursunny Member, IPv6 Advocate

    @woohoo said:

    @yoursunny said:
    1. Do 7 push-ups every day.
    2. Google will reach your website after you've finished 53 push-ups.
    3. DNS propagation takes as much as 7 days, you know.

    Google recognizes the new NS records. It still dosen't work though.

    Google is waiting on your push-ups.

  • TimboJonesTimboJones Member

    @woohoo said:

    @yoursunny said:
    1. Do 7 push-ups every day.
    2. Google will reach your website after you've finished 53 push-ups.
    3. DNS propagation takes as much as 7 days, you know.

    Google recognizes the new NS records. It still dosen't work though.

    Do you have DKIM keys? If there's a problem (and switching nameservers might do that), Google ignores you with similar symptoms (other providers work, just not Google). You can use 8.8.8.8 to lookup your domain and confirm against 1.1.1.1 response.

  • JabJabJabJab Member
    edited August 2021

    atm you have no A DNS entry for domain.

    $ dig +trace hundforsakringarna.se @8.8.8.8

; <<>> DiG 9.16.1-Ubuntu <<>> +trace hundforsakringarna.se @8.8.8.8
;; global options: +cmd
.                       16834   IN      NS      a.root-servers.net.
.                       16834   IN      NS      b.root-servers.net.
.                       16834   IN      NS      c.root-servers.net.
.                       16834   IN      NS      d.root-servers.net.
.                       16834   IN      NS      e.root-servers.net.
.                       16834   IN      NS      f.root-servers.net.
.                       16834   IN      NS      g.root-servers.net.
.                       16834   IN      NS      h.root-servers.net.
.                       16834   IN      NS      i.root-servers.net.
.                       16834   IN      NS      j.root-servers.net.
.                       16834   IN      NS      k.root-servers.net.
.                       16834   IN      NS      l.root-servers.net.
.                       16834   IN      NS      m.root-servers.net.
.                       16834   IN      RRSIG   NS 8 0 518400 20210911050000 20210829040000 26838 . EDWqxjrHylReZdS9zE7+RJNP5JpnKO7IqhGuApPsZL+ciEcPvF+x6FCX Q0sfKV7jOtSksg+OvoFlnq/5jP0H/vX4Y5C804/Dll9O/B+S7ZXgqlYF LxRIf73lICvH9naml1IBc51Vx4fIALUzX1rP57uSOxJPdz+IYaPOlK70 Uf0a3IOtz+C/RzCW8vNCJ5m+q5zDo4AZG5UG+hv4Pr1+3oSsApzXkWFh GFtkRwLANhBPRc6c6zHMn7XSr1WbwPLdPMAGrScmbzAAjf8Pp9ACzYf/ 5i3cjCHvjLuq1A/liBsh7Ov0LmuNDg7P+F4Iwr6UjLfyGshZxP1JMYkw YTmZWA==
;; Received 525 bytes from 8.8.8.8#53(8.8.8.8) in 12 ms

se.                     172800  IN      NS      y.ns.se.
se.                     172800  IN      NS      a.ns.se.
se.                     172800  IN      NS      z.ns.se.
se.                     172800  IN      NS      x.ns.se.
se.                     172800  IN      NS      m.ns.se.
se.                     172800  IN      NS      f.ns.se.
se.                     172800  IN      NS      i.ns.se.
se.                     172800  IN      NS      g.ns.se.
se.                     172800  IN      NS      b.ns.se.
se.                     172800  IN      NS      c.ns.se.
se.                     86400   IN      DS      59407 8 2 67A8E06FCEFDD9397F77F26C41ADE4EC142F299BCFA1827F0EF8FD87 F2F63022
se.                     86400   IN      RRSIG   DS 8 1 86400 20210911170000 20210829160000 26838 . Z0B/IcFqbgaTerm2XHcIB2KgqFy5VR6YUdwhZ0Sy73Z1GtXSr7j7sIuN jUW0AUcx2d9zbgfRtCKz43emLLpLMiIxU533quRM61XWRlVyqlmt5x8V HJHOgWskQ+FxcG/wIj+lHBu0ijDF7vTVQ1azOfqPD4HHW+6EwKL1TBJM ogGhMJgQkhmKGHEU4jBHXXnDpUkLSeVheu/dEKycvgE5BJurIC0/1F+8 S7G8RweOOURgMA0nFlNoE7HRnQxQO9vtMspkFuvVNrkcOCtHjXBq1hTE wwbFaNhzdTGfT8NURAfJe+9aBuU46PMLeQwhyUlA2L9J4MrzM7jJ2fEn Wom+Gg==
;; Received 1016 bytes from 192.33.4.12#53(c.root-servers.net) in 8 ms

hundforsakringarna.se.  86400   IN      NS      ns1.inleed.net.
hundforsakringarna.se.  86400   IN      NS      ns2.inleed.net.
hundforsakringarna.se.  86400   IN      NS      ns3.inleed.net.
hundforsakringarna.se.  86400   IN      NS      ns4.inleed.net.
hundforsakringarna.se.  86400   IN      NS      ns5.inleed.net.
hundforsakringarna.se.  86400   IN      NS      ns6.inleed.net.
hundforsakringarna.se.  7200    IN      NSEC    hundforsakringen.se. NS RRSIG NSEC
hundforsakringarna.se.  7200    IN      RRSIG   NSEC 8 2 7200 20210912141123 20210829151105 25468 se. IEzdIkEXNanHtiQc4xoA8RTamp7yoA9JbTI24nYry5RY4e863W7yINUT o/1+59QdTHm/rNS8UyPSIC+ZbvGbvIVPy3q4syaEYY8VyNOc+PSo4IDS xJ50tGM25RMlTCXHjpVLocs3grtDENH81wtmO1zXGUIaEXWZSzHxRMEn a56AILXVSvQ4eUqCB57X8jx2nMTagQy5GF9kUXXELZ6FjqUPcXKVTCBp qSIqySu5PQVg9sbwebGHf4BEVCnISij2DnMnGiM5SPdpL4lPe96i8fAT K9hGK8sJbtKXk4F480MrLL6qCsHXoDMqNUcESD6WxUd9+uUsH3GkUaOm vEJwgA==
;; Received 499 bytes from 130.239.5.114#53(g.ns.se) in 40 ms

;; Received 78 bytes from 185.189.49.215#53(ns2.inleed.net) in 24 ms
-- here should be something, there is nothing --
Sign In or Register to comment.