Marketing News – Can Google Crawl Bigger Pages than Bing? What Links Does Google Ignore? How Awesome is Screaming Frog? December 30th, 2021

Today we’ll be talking about Google’s ability to crawl large pages, who uses screaming frog, what backlink types are simply ignored, and was there an algorithm update over the last few days?

Google’s HTML size limit is in the 10’s to 100’s of megabytes. Confirmed via tweet by John Mueller. This was in response to a Bing Webmaster Tools notice. 

Screaming Frog is one of the best pieces of software you can buy. It’s somewhere around $150 – $200 for year and you run it locally on your system. It’s invaluable for ad hoc reports, technical analysis and more. A recent reddit thread got a little more visibility when John Mueller chimed in that he personally pays for a license.

No Follow, User Generated Content, and Sponsored backlinks are all treated the same, basically they’re ignored.

Confirmation – you don’t need to keep a 301 redirect on an old domain for more than a year. 

According to recent SEMRush data, in December of this year was when things started getting really volatile. We saw a lot more movement than usual with Google and their products. In fact, this year is one of the most volatile years according to SEMrush.

Around December 27th and 28th we may have a Google search ranking algorithm unofficial update. It is not unusual for Google’s search results to fluctuate without Google manually pushing an algorithm update. There’s a lot of chatter especially in the affiliate groups..

There are many strategies and tactics in the SEO world that seem completely backwards at first, but if you take them into consideration they may actually work. Aleyda Solis’ recent Twitter thread asking “what’s the most counterintuitive recommendation/implementation I’ve done over my career which paid off as expected?”, while you are at it, sign up for her weekly SEO Fomo Newsletter @


Welcome to the opinionated SEO, where we talk about recent news and updates in the digital marketing world of SEO paid advertising and social media that impact you as a marketer. Also throw some of my opinion into the mix. Okay, let’s go.

Today, we’ll be talking about Google’s ability to crawl large pages who use a screaming frog. What backlink types are simply ignored. And was there an algorithm update over the last few days? So Google’s HTML size limit is actually in the tens to hundreds of megabytes. That’s actually confirmed via tweet by John Mueller.

This was in response to another tweet asking about a Bing webmaster tools file size limit notice. So this was actually a really interesting one. I’ve worked with web pages that were 10, 12, 15 megabytes. And if you are actually including. Beyond just the HTML images and things like that.

Yeah. Sometimes they can get into 20, 30, 40 megabytes and Google’s never had an issue crawling those. I actually couldn’t tell you how being handled that I’ve never had this noticed before. So I think that was just a warning, probably not an issue, but the thing to be concerned about here is if that is truly a 15 megabyte HTML, You may want to start looking at what that page is doing, how much content is on there and maybe pairing that.

Screaming frog is in my opinion, one of the best pieces of SEO software you can buy it’s somewhere around 150 to $200 for a year. You run it locally on your system, and it’s invaluable for things like ad hoc reports, technical analysis extractions from your website. There was a recent thread going on about Getting some buy-in from management for getting this cost covered.

And to be honest, I think that if you’re working in an agency that has a problem with $150 a year tool, you may want to bring that up. That’s a bigger problem. With that said screaming. Frog is truly invaluable because it allows you to access things like your local dev environment or sites behind a VPN.

Especially if you’re working on things like an intranet where you’re not able to get some of those online tools, those hosted cloud-based tools to get access in. Screaming frog, let’s you pull rendered versions of your site as well. And we do this a lot, especially if you have JavaScript running on your site and you’re doing client side rendering to really see how that’s affecting non rendered versus.

It’s a great tool and you can really limit or expand how much you want based on how much system memory you have . Screaming frog is an amazing tool. That thread got a little bit more visibility because John Mueller chimed in saying that he personally pays for a license, even though he doesn’t actually do SEO.

No follow user generated content and sponsored backlinks . Google treats them all the same. And basically they’re ignored. I’ve got two links here in the show notes on the blog page, you can take a look. It’s just a recent tweet that John said. Yeah we treat them all the same. In the video during an office hours, he basically said, it’s something that we might take into consideration the different types to see if it makes a difference. But right now they’re all used the same. And again, my opinion on this is that any kind of link coming in, I think Google looks at though, it’s not factored in quite the same as a standard link, but if thousands of user-generated content links coming in and they all are highly relevant. I think that might have somewhat of an impact though it’s going to be a lot less than having a handful of really great high authority links.

Just a little confirmation about 3 0 1 redirects. This was just confirmed that if you are redirecting from say an old domain to a new domain, you don’t need to have that running for more than a year. John said you really don’t have to pay for registration beyond that year. It’s just an expense that’s not going to do anything for you. In my experience, I’ve seen Google check old redirects as far as two, three, sometimes even four years.

So it’s something to maybe consider keeping that for the long-term, especially if you’re able to track those traffic coming in. And if you have links coming in from sites from a long time ago, that may be highly authoritative, that you can’t get changed. I think it’s very important to keep those coming in.

Even if they’re not bringing in a lot of traffic, it’s still valuable.

SEMrush had a tweet that came out recently basically saying that this was one of the most volatile years in movement for algorithms, for Google and their products. Take a look at this link. It’s interesting. There’s a little bit of data that’s vague, but basically saying that yeah , this was a lot going on. So take a look at that article and definitely look at some of the roundups that have been going around over the last few days.

Around maybe this 27th or 28th of December, 2021 there’s a lot of chatter going on. We may have seen some kind of unofficial search ranking algorithm come up a lot of chatter, especially in the affiliate side of things. Google said they basically rolled out everything allready, so maybe it’s just some things finally settling, but have seen some impacts there on some of those sites.

I haven’t seen it on any of mine but I don’t run any kind of review or affiliate sites.

And the last article from Aleyda Solis who runs the SEO FOMO has some great tweets. So definitely follow her on Twitter. She asked what’s the most counterintuitive recommendation or implementation ever done that paid off unexpected. Some really interesting answers in there and some things that you wouldn’t think would work and they did. And then a couple in there that you knew wouldn’t work and they definitely didn’t. So if you want to spend a few minutes looking at some of that thread, I would definitely do that. And also go to the sign up for her weekly SEO newsletter. Great content in there.

Thank you everybody for listening and we’ll see you tomorrow. .