Google search console page indexing

P

PeterShene

Active member
HI there anybody else notice a big decline in the rate at which google is indexing new websites and web pages? I'm struggling to get a few sites indexed and even waiting weeks for changes on old existing pages.. cant really find anything on this online except the usual Google takes its own time be patient and make sure you content is good stuff. I understand that just in the many years i've done this its at a all time high for slowness. I wonder if there was some kind of algorithm update, or the spider is taxed because of all the new pages popping from businesses looking to get online during the pandemic.
 
PeterShene said:
HI there anybody else notice a big decline in the rate at which google is indexing new websites and web pages? I'm struggling to get a few sites indexed and even waiting weeks for changes on old existing pages.. cant really find anything on this online except the usual Google takes its own time be patient and make sure you content is good stuff. I understand that just in the many years i've done this its at a all time high for slowness. I wonder if there was some kind of algorithm update, or the spider is taxed because of all the new pages popping from businesses looking to get online during the pandemic.
Click to expand...

The hamster wheel broke and i guess people panic bought these, so Google cant get a replacement
 
So an update on this after much digging around I found that google dedicates crawl index on websites via ip, i have a few websites on shared hosting which has boomed a bit recently and come to think of it the sites that aren't on a shared ip are indexing / updating from crawls much faster. Its a little annoying since in the long run shared ips cant be helped for many people, I also read its a way that google is pushing for more use of its own cloud servers <---- this
 
PeterShene said:
So an update on this after much digging around I found that google dedicates crawl index on websites via ip, i have a few websites on shared hosting which has boomed a bit recently and come to think of it the sites that aren't on a shared ip are indexing / updating from crawls much faster. Its a little annoying since in the long run shared ips cant be helped for many people, I also read its a way that google is pushing for more use of its own cloud servers <---- this
Click to expand...

crawl indexing has always been done using IPs.

why do you think in .htaccess files etc. you allow or deny using IPS
 
For indexing issues, in the search console, click the option to re-submit a sitemap - this will at least get you in the queue.

Also, if you have specific content that you need indexed from a specific page, you can use the Inspect URL, then after processing click the option for Request Indexing (only need to click it once).

We've found that if we use the option of the Inspect URL to request indexing that this usually gives a quicker prod, and often we're seeing a crawl within 24 hours.

I didn't see this particular thread until now, but the pandemic did slow some things at Google, and they are giving more priority to health and wellness sectors - but much of that has returned to normal.

Normal indexing usually can be 7-14 days (we have seen longer in the past) but using the Inspect URL can shorten that down to 24 hours or so.
 
I'm using sitemap to index my all the pages, and you can also metion sitemap path and enable googlebot in robots.txt that help to index all the pages easily by any search engine
 
rules for googlebot (or any bot) in the robots.txt should only be used to EXCLUDE sections, not include. The only exception to that is that you should have a link to your sitemap XML file in the robots.txt

Keep in mind, robots.txt is only used by honest bots. Places like Yandex or aggressive crawlers ignore any suggestions in robots.txt.
 
iwinx said:
There seems to be an issue with it according to google.

https://twitter.com/googlewmc/status/1305433934373806080
Click to expand...

There is a current issue with the Google Search Console reporting system, however this only affects reporting and not actual indexing. If you run a "site:domain.com/link/file.html" from a Google search you should see your page indexed in Google.

Keep in mind that the new issue happening in GSC is only for the past 2 weeks (so, 9/1/2020-9/16/2020). It does not appear to have affected the previous month.

From time to time GSC does have delays in reporting, usually 1-3 days, but this is the longest I've seen in many years (if not the longest ever).

Again, only affects the reporting, not the actual indexing.
 
Hmm there are patent documents talking about crawl limits and these guys are talking about tens of thousands f pages ( i assume per ip ). I know im not close to that and the indexing has sped up so maybe it was a covid thing like red said priority being giving to other sites.

Interesting you mention different ways of submission, i have found that something like yoast indexes slower than google xml sitemaps and if you create you own sitemap its faster. Google says the priority in the sitemap is not important, not sure about that not sure if its the format as yoast it like a html sitemap with a link structure, while what i create with screaming frog is just simple straight up.
 
bigredseo said:
...Also, if you have specific content that you need indexed from a specific page, you can use the Inspect URL, then after processing click the option for Request Indexing (only need to click it once).

We've found that if we use the option of the Inspect URL to request indexing that this usually gives a quicker prod, and often we're seeing a crawl within 24 hours....
Click to expand...

I can +1 this with confidence.

We have been developing content, albeit slowly, of late. Inspect URL and Reqest Indexing button works faster than any other method. Only thing is that you need to do it separately for every URL that you wish Google to index (a real pain if you want 100s or 1000s done at the same time)
 
I am inspecting webs pages just in google search console the way that you guys are suggesting :) It still takes long. Anyway its not an ip issue as some sites index content faster than other on the same shared hosting server and it almost seems to me like the older more authoritative sites ( Note how i avoid the use of words such as DA ).

As with most things in Google I guess we will never have a solid black and white answer.
 
We swapped over to Domain Validated on nearly all our properties, so the Crawl Budget information is no longer available :(

Our sitemaps however look to be "last read" anywhere from 3-8 days ago, depending on the site. Even though there's new content, it's still taking 3+ days to actually crawl and index.

On one of our sites we saw a big drop in the number of pages indexed (went from 423 on 8/1 to 195 on 9/1. So that raised some alarms, but the impressions remained about the same.

On 10/1 we saw the number of pages indexed at 196 (even though there were 4 new posts, older ones were dropped), but our impressions went from 1201 daily impressions to now a solid 2500 impressions.

No changes in the number of pages on the site, only 4 new posts added, but that's a big change for impressions.

We're seeing about a 30% increase on impressions on a number of properties comparing September1 to October 1. Something appears to have changed with indexing and dropping of older content - but the impressions are going higher.

We've always talked about curating content and either removing or updating older posts and pages, but with this shift, we're seeing huge changes. We also saw huge changes when it came to the number of keywords indexed with roughly 50% of the keywords still in the top 100 positions - but upon review, the ones that were dropped were useless and not driving traffic anyway.

So something is up with indexing and rankings for sure - but hard to pinpoint as to what other than Google Algorithm :)
 
I'm using Yoast plugin in wordpress website, My Yoast plugin create XML Sitemap and I show that sitemap to Google Search engine through Google Webmaster tool, that help positive impact in indexing
 
In a REALLY weird twist of fate, Google has "temporarily" removed the ability for people to Request Indexing in the Search Console!

This is effective as of 10/15/2020 - Crazy timing!

Google says that the move is temporary, but did not provide a timeline on when it might return.
 
You must log in or register to reply here.

Latest posts

Forum supporters

Members recently online

Total: 20

Forum statistics

Threads
80,831
Messages
248,323
Members
20,673
Latest member
jean_garcia2
Top