A few months ago I discovered, that MeineKleineFarm.Net is not findable on Google anymore. This is really bugging me, since the purpose of this blog is to share knowledge. But a blog that is not findable doesn’t bring value to anyone. On the other hand, search engines like Bing, DuckDuckGo and Yandex index an rank this blog.
I suspect the Yoast SEO sitemap to be guilty. Let me explain why.
Troubleshooting
To start troubleshooting I registered at the Google Search Console, which allows a webmaster to track down Google Search Index and Crawl issues. First it takes a few days to weeks after the registration and validation for your property to show up results. The results were a disaster but not unexpected:
This was the proof for a 100% exclude, but I already knew that from the test search.
But I still had no clue why, because the reasons presented weren’t true in my personal opinion:
I couldn’t and can’t until this very moment, explain what this duplicates mean. So instead I focused on getting the blog to be crawled again. Because it’s strange, that all Google activity seemed to have stopped in May 2018. On this very day I switch the static IP of the Blog to a new datacenter, but all DNS records were changed accordingly:
(Possible) Solution
The most basic knowledge you need as a webmaster is, if Google doesn’t crawl and index your website by itself, you have to hand in your sitemap.
So I head over to the Yoast Knowledge Base and checked the tutorials here and here, just to make sure what’s happening. Both tutorials explicitly state that the WordPress Yoast SEO plugin generates a sitemap index called sitemap_index.xml and not a sitemap.xml file. Furthermore they state, that you have to turn of possible other sitemaps from Jetpack or other WordPress plugins. And that’s true all true. But for my blog, and I don’t know why, both possible URLs work and both redirect to the sitemap_index.xml:
- https://www.meinekleinefarm.net/sitemap.xml
- https://www.meinekleinefarm.net/sitemap_index.xml
So according to the tutorial I set the sitemap in the Google Search Console to point to sitemap_index.xml. And then I waited for about 6 weeks. Nothing happened. It just started over and over again and presented me “file not found” or “404” errors.
During that time I also handed the sitemap (https://www.meinekleinefarm.net/sitemap_index.xml) to Bing and Yandex and they indexed it after a few days.
This all lead me to the assumption that the URL can’t be culprit.
But yesterday, after weeks of research without any progress I deleted the sitemap_index.xml in the Google Search Console just for fun and replaced it with sitemap.xml, because as I said, both point to the same file. And guess what? Suddenly the red errors disappeared and the file was marked green! And more, it uncovered that the sitemap hasn’t been read since April 2017!
So for whatever strange reason the new Google Search Console does not accept the Yoast SEO sitemap_index.xml file for my personal blog, but I wanted to write that down. Maybe others have the same error with their sitemap sometime in the future, when this blog is findable once again via Google.
And to clarify, I don’t blame Yoast in any way for what happened, but it’s really strange that the tutorials don’t work for me. But maybe I’ve just messed up this Debian machine and it’s WordPress setup.
