Sitemap and Robots Generator in Episerver CMS

Sitemap and Robots Generator in Episerver CMS

There are several plugins in the market that can help you with the creation of sitemaps and robots in Episerver, but we decided some time ago to create a new one to solve our own needs in our company and at the same time provide some added value to the current sitemaps plugins. In this post, we are going to talk about the Sitemap and Robots Generator plugin by Verndale, how to install it, configure it and use it. It is important to acknowledge that the plugin only works for Episerver CMS 11 or greater.

To begin, we are going to install the latest package 1.0.0.4 at the time of writing this blog using the Package Management Console and this command

Install-Package Verndale.Sitemap.Robots.Generator -Version 1.0.0.4

Please read carefully the Readme file. Some of the new changes we brought to this version are mentioned in there, but most important, it explains how to configure several parameters of the plugin which we will also explain meticulously later in this post.

After installation, we will copy the following lines to your base page, this is not required, but it is needed if you want more control over the pages that are going to appear in the sitemap, its frequency and priority. Please pay special attention to the comments in the code.

[Display(GroupName = "Navigation",
		Name = "Exclude from Xml Site Map",
		Order = 400)]
public virtual bool ExcludeFromSiteMap { get; set; } // Exclude this page from the sitemap
       
[Display(GroupName = "Navigation",
    Name = "Priority in Site Map",
    Order = 500)]
public virtual string Priority { get; set; } // The priority applied to this page in the sitemap
     
[Display(GroupName = "Navigation",
    Name = "Change Frequency of Page in Site Map",
    Order = 600)]
public virtual string ChangeFrequency { get; set; } // The frequency applied to this page in the sitemap

In addition, if you are using Web Api in your Episerver project, you need to add these lines to your Global.asax.cs.

protected override void RegisterRoutes(System.Web.Routing.RouteCollection routes)
{
    base.RegisterRoutes(routes);
    routes.Ignore("sitemap.xml");
    routes.Ignore("robots.txt");
}

With these changes in your code, it is enough to make the plugin work as expected. Now we are going to explain the app settings configurations which you can modify to fit your needs inside the web.config file.

    <add key="siteMapConfig" value="App_Configsitemap.config" />
    <add key="SiteMap_MAX_URLS" value="50000" />
    <add key="Show_Localized_URLS" value="true" />
    <add key="Display_Localized_URLS_Separated" value="false" />
    <add key="Show_Trailing_Slash" value="false" />
    <add key="SiteMap_OutPutRobots_txt" value="1" />
KeyDescription
siteMapConfigLocation of the sitemap.config file
SiteMap_MAX_URLS (Deprecated)Max number of urls before splitting the sitemap file into smaller files
Show_Localized_URLS (true/false)If the sitemap is going to show localized urls based on google recommendations with href lang attribute
Display_Localized_URLS_Separated (true/false)If the sitemap is going to display the localized urls separated instead of using href lang
Show_Trailing_Slash (true/false)Show a trailing slash at the end of each URL in the sitemap
SiteMap_OutPutRobots_txt (1/0)If the Sitemap URL is going to be added to the robots content after the sitemap is generated

The sitemap.config file has the following configurations

Ke<?xml version="1.0"?>
<sitemapVariables>
  <sitemapVariable name="xmlNsTpl" value="http://www.sitemaps.org/schemas/sitemap/0.9" />
  <sitemapVariable name="xmlNsLangTpl" value="http://www.w3.org/1999/xhtml" />
  <sitemapVariable name="productionEnvironment" value="false" />
</sitemapVariables>
KeyDescription
xmlNsTplDefault namespace for the sitemap xml file
xmlNsLangTplDefault namespace used to allow href lang attributes
productionEnvironment (true/false)If this variable is set to true, after the sitemap is generated, it will notify the sitemap Urls to the engines you have configured in the admin section (Work in progress)

Now we will compile the code and go to the admin interface in the Episerver CMS editor, config tab and then to Sitemap and Robots Manager inside the Tool Settings section.

In this page there are 3 tabs that allows you to configure even more the plugin.

The first tab “Restricted types” allows you to restrict all pages in your site which belong to a specific type instead of doing it one by one. When you finish, click the update restricted types to save your changes.

The second tab “Sites” allows you to restrict sites. If the restricted check is set in here the sitemap will not be generated for that site. In addition, there is a text area field were you can set the information for the robots file also per site. When you finish, click the update sites to save your changes. It is important to mention that the Robots file does not need to be generated. If you saved your changes here, you will see those changes reflected in the robots.txt file immediately.

The last tab “Search engines” allows you to create links to engines that will be notified after sitemap generation. (This is a work in progress)

Finally, to generate the sitemap you must go to the admin interface in the Episerver CMS editor, admin tab and then to Sitemap Generation inside the Schedule Jobs section.

Click the start manually button to run the job and if you want to stop it you can click the stop button any time. Just remember, that before running this job for the first time you must save your configurations as explained above at least once.

To test the results just go your site url and at the end add sitemap.xml or robots.txt

That is it. If you have any questions or suggestions please let me know in the comments. I hope this plugin can help someone and as always keep learning !!!

Written by:

Jorge Cardenas

Developer with several years of experience who is passionate about technology and how to solve problems through it.

View All Posts

8 COMMENTS

comments user
Lars

Hi,
I just tried this sitemapping tool, but when I install Verndale.Sitemap.Robots.Generator.Core, I keep getting an error that the SitemapConfiguration route is already declared and the whole site fails to load ?? Is it not supported anymore ?

C# 4.7.2 MVC
Epi11+
Verndale.Sitemap.Robots.Generator.Core 1
Verndale.Sitemap.Robots.Generator 1.0.0.5

Stacktrace :
[ArgumentException: Der findes allerede en rute med navnet ‘SitemapConfiguration’ i rutesamlingen. Rutenavne skal være entydige.
Parameter name: name]
System.Web.Routing.RouteCollection.Add(String name, RouteBase item) +392
System.Web.Mvc.RouteCollectionExtensions.MapRoute(RouteCollection routes, String name, String url, Object defaults, Object constraints, String[] namespaces) +296
Verndale.Sitemap.Robots.Generator.Initialization.SitemapConfigurationInitializationModule.Initialize(InitializationEngine context) +163
EPiServer.Framework.Initialization.Internal.ModuleNode.Execute(Action a, String key) +58
EPiServer.Framework.Initialization.Internal.ModuleNode.Initialize(InitializationEngine context) +123
EPiServer.Framework.Initialization.InitializationEngine.InitializeModules() +275

[InitializationException: Initialize action failed for Initialize on class Verndale.Sitemap.Robots.Generator.Initialization.SitemapConfigurationInitializationModule, Verndale.Sitemap.Robots.Generator, Version=1.0.0.5, Culture=neutral, PublicKeyToken=null]
EPiServer.Framework.Initialization.InitializationEngine.InitializeModules() +836
EPiServer.Framework.Initialization.InitializationEngine.ExecuteTransition(Boolean continueTransitions) +194
EPiServer.Framework.Initialization.InitializationModule.EngineExecute(HostType hostType, Action`1 engineAction) +885
EPiServer.Framework.Initialization.InitializationModule.FrameworkInitialization(HostType hostType) +226
EPiServer.Global..ctor() +42
Skoda.Web.EPiServerApplication..ctor() +43
ASP.global_asax..ctor() +48

    comments user
    Jorge Cardenas

    Hi Lars.

    Verndale.Sitemap.Robots.Generator.Core is not a dependency package anymore for the version 1.0.0.5.
    If you can please uninstall any package related to the sitemap generator and then again use this command in the nuget package console, checking that you are not using the package in episerver nuget feed which unfortunately is outdated.

    Install-Package Verndale.Sitemap.Robots.Generator -Version 1.0.0.5

    Also, you can check in the modules folder of you project to check if the Verndale.Sitemap.Robots.Generator.Core is there, you can remove it and check if that solves it. If that is not the case. You can always contact me at jorge.cardenas@verndale.com or help you with a quick call whenever you can do it.

comments user
Hariprasath

Hi,
I was trying this latest version 1.0.0.7 and i updated the “Restricted Types” in plugin and ran the scheduler.but still i can see the excluded page templates in the site map.Can you please let me know?

    comments user
    Jorge Cardenas

    HI Hariprasath please check do not forget your changes when choosing the restricted types. If still you cannot make it work. Please write me to jorge.cardenas@verndale.com I may be able to help you or generate a new fix for the package in the worst case scenario.

comments user
KUMAR

Hi,
We have followed the given approach with Verndale.Sitemap.Robots.Generator package with Version 1.0.0.7 but browser showing sitemap.xml file data as plain text. Could you please guide us to show the sitemap.xml file in XML format?

    comments user
    Jorge Cardenas

    Greetings Kumar I had issues accessing my blog platform and I just saw your message, there are newer versions which should fix the issue. Please contact me directly at jorge.cardenas@verndale.com if you still need my help.

Leave a Reply