So if you understand those fundamentals, you can dig into problems, understand their impact, and work with devs to fix the ones that matter.
In this article, we’ll help diagnose some common issues when sites are built on JS frameworks. Plus, we’ll break down the base knowledge every technical SEO needs when it comes to rendering.
Before we get into the more granular stuff, let’s talk big-picture.
The problem is, search engines only have so many resources to use, so they have to be selective about when it’s worth using them. It’s not a given that a page will get rendered, even if the crawler sends it to the rendering queue.
If it chooses not to render the page, or it can’t render the content properly, it could be an issue.
It comes down to how the front end serves HTML in the initial server response.
When a URL is built in the browser, a front end like React, Vue, or Gatsby will generate the HTML for the page. A crawler checks if that HTML is already available from the server (“pre-rendered” HTML), before sending the URL to wait for rendering so it can look at the resulting content.
Whether any pre-rendered HTML is available depends on how the front end is configured. It will either generate the HTML via the server or in the client browser.
The name says it all. In an SSR setup, the crawler is fed a fully rendered HTML page without requiring extra JS execution and rendering.
So even if the page isn’t rendered, the search engine can still crawl any HTML, contextualize the page (metadata, copy, images), and understand its relationship to other pages (breadcrumbs, canonical URL, internal links).
If the rendering service chooses not to render a page sent to the queue, copy, internal URLs, image links, and even metadata remain unavailable to crawlers.
As a result, search engines have little to no context to understand the relevance of a URL to search queries.
Note: There can be a blend of HTML that’s served in the initial HTML response, as well as HTML that requires JS to execute in order to render (appear). It depends on several factors, the most common of which include the framework, how individual site components are built, and the server configuration.
You can do a lot of the investigation using browser tools and Google Search Console. Here’s the shortlist that makes up a solid toolkit:
In addition to the negative implications for crawling and contextualization mentioned earlier, there’s also the issue of the time and resources it could take for a search engine to render a page.
If the crawler chooses to put a URL through the rendering process, it will end up in the rendering queue. This happens because a crawler may sense a disparity between the pre-rendered and rendered HTML structure. (Which makes a lot of sense if there’s no pre-rendered HTML!)
There’s no guarantee of how long a URL waits for the web rendering service. The best way to sway the WRS into timely rendering is to ensure there are key authority signals onsite illustrating the importance of a URL (e.g., linked in the top nav, many internal links, referenced as canonical). That gets a little complicated because the authority signals also need to be crawled.
In Google Search Console, it’s possible to get a sense of whether you’re sending the right authority signals to key pages or causing them to sit in limbo.
Go to Pages > Page indexing > Crawled – currently not indexed and look for the presence of priority pages within the list.
If they’re in the waiting room, it’s because Google can’t ascertain whether they’re important enough to spend resources on.
Most popular front ends come “out of the box” set to client-side rendering, so there’s a fairly good chance default settings are the culprit.
If you’re wondering why most frontends default to CSR, it’s because of the performance benefits. Devs don’t always love SSR, because it can limit the possibilities for speeding up a site and implementing certain interactive elements (e.g., unique transitions between pages).
This has some negative implications, perhaps the most important of which is that pages are potentially undiscoverable.
This isn’t to say that it’s impossible to set a SPA up in a more SEO-friendly way. But chances are, there’s going to be some significant dev work needed to make that happen.
Getting a search engine to render a URL is great, only so long as all of the elements are available to crawl. What if it’s rendering the page, but there are sections of a page that aren’t accessible?
For example, an SEO does an internal link analysis and finds little to no internal links reported for a URL linked on several pages.
To narrow down the culprit, it would be a good idea to look for commonalities in terms of where the missing page content or internal links are on the page across URLs.
For example, if it’s an FAQ link that appears in the same section of every product page, that goes a long way in helping developers narrow down a fix.
So if you go on the hunt for errors, take a long list to your dev, and start the conversation with “What are all these errors?”, they might not receive it all that well.
With that being said, there are syntax errors that could make the rest of the page unparsable (e.g. “render blocking”). When this occurs, the renderer can’t break out the individual HTML elements, structure the content in the DOM, or understand relationships.
Generally, these types of errors are recognizable because they have some sort of effect in the browser view too.
Get the daily newsletter search marketers rely on.
Content requires a user interaction
One of the most important things to remember about rendering is that Google can’t render any content that requires users to interact with the page. Or, to put it more simply, it can’t “click” things.
Why does that matter? Think about our old, trusty friend, the accordion dropdown, and how many sites use it for organizing content like product details and FAQs.
Depending on how the accordion is coded, Google may be unable to render the content in the dropdown if it doesn’t populate until JS executes.
To check, you can “Inspect” a page, and see if the “hidden” content (what shows once you click on an accordion) is in the HTML.
If it’s not, that means that Googlebot and other crawlers do not see this content in the rendered version of the page.
Google may or may not render your page if it crawls it and sends it to the queue. If it doesn’t crawl the page, even that opportunity is off the table.
To understand whether Google is crawling pages, the Crawl Stats report can come in handy Settings > Crawl stats.
Select Crawl requests: OK (200) to see all crawl instances of 200 status pages in the last three months. Then, use filtering to search for individual URLs or entire directories.
If the URLs don’t appear in the crawl logs, there’s a good chance that Google is unable to discover the pages and crawl them (or they aren’t 200 pages, which is a whole different issue).
Internal links are not crawlable
Links are the road signs crawlers follow to new pages. That’s one reason why orphan pages are such a big problem.
If you have a well-linked site and are seeing orphan pages pop up in your site audits, there’s a good chance it’s because the links aren’t available in the pre-rendered HTML.
An easy way to check is to go to a URL that links to the reported orphan page. Right-click on the page and click “view source.”
Then, use CMD + f to search for the URL of the orphan page. If it doesn’t appear in the pre-rendered HTML but appears on the page when rendered in the browser, skip ahead to issue four.
XML sitemap not updated
The XML sitemap is crucial for helping Google discover new pages and understand which URLs to prioritize in a crawl.
Without the XML sitemap, page discovery is only possible by following links.
So for sites without pre-rendered HTML, an outdated or missing sitemap means waiting for Google to render pages, follow internal links to other pages, queue them, render them, follow their links, and so on.
Depending on the front end you’re using, you may have access to plugins that can create dynamic XML sitemaps.
They often need customization, so it’s important that SEOs diligently document any URLs that shouldn’t be in the sitemap and the logic as to why that is.
This should be relatively easy to verify by running the sitemap through your favorite SEO tool.
The unavailability of internal links to crawlers isn’t just a potential discovery problem, it’s also an equity problem. Since links pass SEO equity from the reference URL to the target URL, they’re an important factor in growing both page and domain authority.
Links from the homepage are a great example. It’s generally the most authoritative page on a website, so a link to another page from the homepage holds a lot of weight.
If those links aren’t crawlable, then it’s a little like having a broken lightsaber. One of your most powerful tools is rendered useless (pun intended).
User interaction required to get to the link
The accordion example we used earlier is just one instance where content is hidden behind a user interaction. Another that can have widespread implications is infinite scroll pagination – especially for eCommerce sites with substantial catalogs of products.
In an infinite scroll setup, countless products on a product listing (category) page will not load unless a user scrolls beyond a certain point (lazy loading) or taps the “show more” button.
This is why SEOs generally prefer true pagination in which every page of results has a distinct, crawlable URL.
While there are ways for a site to optimize lazy loading and add all of the products to the pre-rendered HTML, this would lead to differences between the rendered HTML and the pre-rendered HTML.
Effectively, this creates a reason to send more pages to the render queue and make crawlers work harder than they need to – and we know that’s not great for SEO.
At a minimum, follow Google’s recommendations for optimizing infinite scroll.
Links not coded properly
When Google crawls a site or renders a URL in the queue, it’s downloading a stateless version of a page. That’s a big part of why it’s so important to use proper href tags and anchors (the linking structure you see most often). A crawler can’t follow link formats like router, span, or onClick.
For a developer’s purposes, these are all valid ways to code links. SEO implications are an additional layer of context, and it’s not their job to know – it’s the SEO’s.
A huge piece of a good SEO’s job is to provide developers with that context through documentation.
In an HTML page, metadata like the title, description, canonical URL, and meta robots tag are all nested in the head.
For obvious reasons, missing metadata is detrimental for SEO, but even more so for SPAs. Elements like a self-referencing canonical URL are crucial to improving the chances of a JS page making it successfully through the rendering queue.
Of all elements that should be present in the pre-rendered HTML, the head is the most important for indexation.
Luckily, this issue is pretty easy to catch, because it will trigger an abundance of errors for missing metadata in whichever SEO tool a site uses for hygiene reporting. Then, you can confirm by looking for the head in the source code.
Lack of or misconfigured metadata vehicle
In a JS framework, a plugin creates the head and inserts the metadata into the head. (The most popular example is React Helmet.) Even if a plugin is already installed, it usually needs to be configured correctly.
Again, this is an area where all SEOs can do is bring the issue to the developer, explain the why, and work closely toward well-documented acceptance criteria.
Script files and images are essential building blocks in the rendering process.
Since they also have their own URLs, the laws of crawlability apply to them too. If the files are blocked from crawling, Google can’t parse the page to render it.
To see if URLs are getting crawled, you can view past requests in GSC Crawl Stats.
Directory blocked by robots.txt
Both script and image URLs generally nest in their own dedicated subdomain or subfolder, so a disallow expression in the robots.txt will prevent crawling.
Some SEO tools will tell you if any script or image files are blocked, but the issue is pretty easy to spot if you know where your images and script files are nested. You can look for those URL structures in the robots.txt.
You can also see any scripts blocked when rendering a page by using the URL inspection tool in Google Search Console. “Test live URL” then go to View tested page > More info > Page resources.
Here you can see any scripts that fail to load during the rendering process. If a file is blocked by robots.txt, it will be marked as such.
With a solid understanding of the rendering process and some common SEO problems in JS frameworks, you’re well on your way to identifying the issues and being a powerful ally to your developers.
Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.
New on Search Engine Land
About the author
Get the daily newsletter search marketers rely on.
Learn actionable search marketing tactics that can help you drive more traffic, leads, and revenue.
March 15-16, 2023: SMX Munich
Online Aug. 16-17: SMX Master Classes
Online Nov. 14-15: SMX Next
Start training now: SMX Advanced
Discover time-saving technologies and actionable tactics that can help you overcome crucial marketing challenges.
April 15-17, 2020: San Jose
A Revenue-Boosting Blueprint to Ignite Marketplace Sales and Dominate Paid & Organic Campaigns
Ignite Your Marketing Engine with the Power of the Data Cloud
From Chaos to Clarity: Resolving Trust Issues to Optimize Your Tech Stack
Enterprise Account-Based Marketing Platforms: A Marketer’s Guide
Marketing Attribution and Performance Management Platforms: A Marketer’s Guide
Enterprise Marketing Work Management Platforms: A Marketer’s Guide
2023 State of Email Workflows Report
Receive daily search news and analysis.
© 2023 Third Door Media, Inc. All rights reserved.
Third Door Media, Inc. is a publisher and marketing solutions provider incorporated in Delaware, USA, with an address 88 Schoolhouse Road, PO Box 3103, Edgartown, MA 02539. Third Door Media operates business-to-business media properties and produces events. It is the publisher of Search Engine Land the leading Search Engine Optimization digital publication.