Schema App Robots-Only Deployment

This new feature from Schema App lets you choose whether to deploy your markup to all users or select web crawlers. It is designed to improve the performance of sites using our JavaScript deployment for normal web and mobile users while still providing your markup to search engines. This will reduce the impact of Schema App's scripts on your pages' performance for all other users. 

TABLE OF CONTENTS

• How does Robots-Only Deployment work?
• What deployment methods support this?
• How do I set up Robots-Only Deployment?
• What Robots does Schema App Deploy to by Default?
• How can I test my Schema Markup once I've enabled Robots-Only Deployment?
• What if I need to add my own bots?
• How long before my updates to deployment management deploys?
• Should I do anything specific if I use a robot to pre-render my pages?

How does Robots-Only Deployment work?

Robots-Only Deployment is configured in the Deployment Management section of your Schema App projects. No modification to your scripts is needed.

Once you have set your project to deploy to Robots Only, requests from non-bots will be filtered by Schema App's Content Delivery Network (CDN) so that the highlighter is not active on your pages for those users. This means when your page makes the normal async request for highlight.js, it will receive an empty script with no functionality which takes no further actions. For normal web users this is fine since the markup doesn't provide any utility. This means any subsequent work the highlighter would normally take will not impact the user.

Web crawlers like Google and Bing, as well as other customizable user agents, will receive the entire script and the highlighter will work as expected. This filtering is done using the user agent sent by a web browser (or crawler) with each request to the CDN. When Robots Only is enabled, the highlighter will only load and produce markup when the user agent visiting the page matches an agent in the Robots Only list. 

What deployment methods support this?

Robots-Only Deployment is only applicable to Schema App projects using JavaScript deployments (including Crawler deployment), with the exception of JavaScript with Caching. Projects using deployment methods that are not eligible for this feature will not see the option to enable it in the project set up interface.

How do I set up Robots-Only Deployment?

1. Log into Schema App and expand the project settings with the arrow on the right side of your project.
2. Scroll down to the Deployment Management section.
3. Select "Robots Only".
4. Save your changes.

NOTE: If you have more than one project with the same subdomain i.e. https://www.example.com and https://www.example/about then the bots only configuration of one project will apply to all projects that have a website with this subdomain. If this is case you should only configure this on one project. The configuration is based on the subdomain not at the project level so will apply to all. 

What Robots does Schema App Deploy to by Default?

By default Schema App deploys to all major search engine bots. Review the list of default Bots in the table below. 

For a list of other possible Bots to include, review this list.

How can I test my Schema Markup once I've enabled Robots-Only Deployment?

When you use the default bots that Schema App maintains, you should be able to use the Rich Result Test tool or Schema Validator as you normally would to test your markup. If you are using your own set of custom bots be sure to enable Googlebot otherwise these tools will not work. 

If you want to test locally in your browser it is possible to alter the user agent for these purposes. In Google Chrome, open the developer tools (CMD|CTRL+SHIFT+I) and navigate to the Hamburger Menu on the right side of the screen. Select "More Tools" > "Network Conditions". 

Once you click on Network Conditions, panel should appear at the very bottom of your window. Under "Network conditions" you can set the user agent. For most cases choosing Googlebot will show you the results. You can test your own user agents as well by selecting custom. Also make sure that "Disable cache" is selected so that in between user agent selection the browser doesn't cache the empty script or vice versa.

What if I need to add my own bots?

You may have other SEO tools that use bots to report on schema markup. If you'd like to include these bots, Schema App provides an interface to do so, however it is only recommended for advanced users. This configuration is only available by going to https://app.schemaapp.com/home?customBots=true. This URL will show a third Deployment Management option:  "Custom Robots". The default list contains bots Schema App expects to support. You can prevent or allow other bots (non-matches are excluded by default). 

Add your own bot by providing the name of the bot and the user agent Schema App should match with. The name must appear in the user agent or it will not be matched. This is required because some bots have multiple user agents but typically contain the same name, this helps reduce duplication on the Schema App side. 

How long before my updates to deployment management deploys?

This can take about 1 hour to fully deploy so you might still see the CDN deploying to all user agents for a little while. This is normal since the changes need to roll out to all locations within the CDN. After an hour, you should see the markup only deploying to the bots that are registered. 

Should I do anything specific if I use a robot to pre-render my pages?

Yes! If your pre-rendering bot uses a specific user agent, you will want to add that to the list of bots by navigating to https://app.schemaapp.com/home?customBots=true and adding that information to the list of rules. If you are not sure of your specific pre-rendering user agent, you may want to consult with your CSM before implementing Robots Only deployment.

The other import aspect is making sure your pre-rendering solution is aware when highlight.js is done rendering. To do this you can leverage the final event highliight.js emits. The event is called highlightJsOnComplete and it is emitted after the final pagecount call is made. 

This will vary based on your pre-render solution but many will allow you to set a flag or trigger another event to let the pre-render solution it is complete. Below is an example with prerender.io that explains how you could integrate it. Doing it using events as opposed to a timeout is for accuracy, the amount of time will vary based on the complexity of pages since the highlighter relies on other resources and it could take longer in some cases. Responding to the event ensures all the necessary actions have been performed. 

// Outside the event
window.prerenderReady = false;

window.addEventListener('highlightJsOnComplete', (e) => {
        // either set the value to true or notify another process this is complete. 
        window.prerenderReady = true;
});