
It's an all-too-familiar story--you were part of an implementation that created a beautiful search interface based on the latest UX trends and built on top of Sitecore's latest Content Search API. You tested the search interface locally and in your integration environment, and from your view, the results loaded in a timely manner. Heck, maybe you weren't even paying attention to the speed in which the results loaded. But then something terrible happens. The search implementation goes into production and WHAM! The business starts getting reports of how painfully slow the search results are loading. Granted, sometimes it takes months (or even years!) after the implementation for these issues to arise. But the business doesn't care--they want it fixed as soon as possible.

Whether you've been in this situation before, or are currently experiencing performance issues with your search results loading slow, let me offer some advice. First, before you blame the infrastructure or the latest code changes, take a few minutes and evaluate if your search implementation committed one of these common costly mistakes:

### 1.Querying Sitecore databases after retrieving search results

Querying Sitecore databases after retrieving search results is essentially taking each search result item that was found in your search index and using the Item API to retrieve the item from the database. This is typically because you need to output the URL of the item or to filter out the results based on additional criteria not stored in the index.

**What to do instead?**

Instead of querying the Sitecore databases after retrieving your search results, index everything that is needed to support the search interfaces alongside the document--including the relative URL. This favors the query time performance over the index time performance and, unless you have some crazy logic, this is a negligible performance issue that is easily justified.

### 2.Querying without pagination

Querying without pagination puts no limits on the number of records retrieved from the search index. 

Imagine if you went to your favorite search engine and typed in "pizza," and it attempted to show you everything it has relating to pizza. The results would be in the billions. Not only would this be extremely hard to scale for your favorite search engine, you would never be able to review everything that came back. The same principles can be applied to your search interfaces. Querying the search index without pagination will retrieve everything from the index that matches your criteria, and thus has a huge potential for retrieving a large number of records that will not be used by your site visitors.

**What to do instead?**

Use the Content Search LINQ paging methods, _Skip_ and _Take,_ provided by the Content Search API **on every call to the index**. By limiting the amount of data retrieved, you reduce the stress on the network as you have reduced the payload size between systems, as well as the amount of time both Sitecore and your search provider need to process the data. Additionally, I recommend setting the _ContentSearch.SearchMaxResults_ to a number between 100-1000. This setting controls the default number of results that the Content Search API will return if you do not use pagination.

### 3\. Selecting all fields for a search result

Selecting all fields for a search result takes all fields for a document in the index and returns them all for a search result, even though only a handful of the fields will be used.

Take for example Amazon's product catalog. Each product in that catalog could possibly contain 1,000+ dimensions or attributes for every product for sale (i.e. height, weight, width, depth, size, color, brand, rating, price, prime, etc). But when you search for products on Amazon, you do not see all 1,000+ dimensions. You only see a small subset, such as the Image, Title, Description, Rating, and Number of Reviews. 

**What to do instead?**

Instead of selecting all fields for a search result, limit the fields selected to only what is needed to support your search interface by using the _Select_ method from the Content Search API **on every call to the index**. As stated in #2, by limiting the amount of data retrieved for each record, you reduce the stress on the network as you have reduced the payload size between systems, as well as the amount of time both Sitecore and your search provider need to process the data.

### 4\. Enabling Content Search item security

By default, Sitecore Content Search item security is disabled, but can be enabled globally through a configuration setting or for each individual query. If item security is enabled for your query, Sitecore will automatically lookup the item in the database to check the context users permissions--therefore performing #1 for you without you knowing. 

**Note:** Sitecore 9.3 uses additional query filters to use item permissions within a query; this is the preferred approach and the most performant.

**What to do instead?**

For Sitecore versions earlier than 9.3, instead of enabling Content Search item security, leave it disabled. If your solution needs to apply security to restrict access to search results based on the role, my recommendation is to index the roles with the documents and then use queries to filter out the results (this is the approach 9.3 uses.)

### 5\. Querying without relevant filters/filtering in-memory

This is probably the easiest one, but querying without relevant filters is essentially not using the context of the search request with business rules to filter out unnecessary documents and then doing post-processing to filter out documents in-memory. In regards to the context, it can be anything from the template types needed, to the site the visitor is on, or even the language they are searching in (and many, many others).

**What to do instead?**

As mentioned in number #1, always favor the query time performance over the index time performance. All necessary fields to support querying and filtering, along with all fields to show within the search interface, should be indexed with the record. The rules of thumb here are to reduce the available results to the smallest dataset possible and to avoid filtering the search results in-memory and instead use the index to filter.

I hope you enjoyed this post. Please comment below with any questions; I'd love to hear from you!


Top 5 reasons your Sitecore search is slow and how to resolve


If you have been around Sitecore development long enough, you have undoubtedly run into the extension method 'IsDerived'. This extension method allows developers to easily determine whether an item inherits from a specific base template; the template can be anywhere in the inheritance chain. This opens up a lot of possibilities for developers to customize the inheritance chain for their templates. For instance, it allows multiple sites to inherit from a single template, allowing each site to provide their own standard values (in the Helix world, the Feature modules would provide the single template, with each Project inheriting this template).

In recent performance testing for a project I was working on, I noticed that under load, the IsDerived extension method as originally written was taking about 30-60ms per request (again, this is under load). While not extremely high, it was higher than we were comfortable with and any tweaks that we could make would be beneficial in the long-term, especially when code like the IsDerived extension method is so widely used.

Below you can find the updates to the IsDerived extension methods that I came up with that reduced the time taken under load - in our tests, the time per request under load was negligible.

```c#
public static class ItemExtensions
{
    /// <summary>
    /// Determine if the Item inherits a specific template, by ID
    /// </summary>
    /// <param name="item">The item</param>
    /// <param name="templateId">The template id</param>
    /// <returns><c>True</c> if the item inherits from <paramref name="templateId"/></returns>
    public static bool IsDerived(this Item item, ID templateId)
    {
        if (item == null)
        {
            return false;
        }

        return !templateId.IsNull && item.IsDerived(item.Database.GetItem(templateId, item.Language));
    }

    /// <summary>
    /// Determine if the Item inherits a specific template, by TemplateItem
    /// </summary>
    /// <param name="item">The item</param>
    /// <param name="inheritedTemplateItem">the templateItem</param>
    /// <returns><c>True</c> if the item inherits from <paramref name="inheritedTemplateItem"/></returns>
    public static bool IsDerived(this Item item, Item inheritedTemplateItem)
    {
        if (item == null
            || inheritedTemplateItem == null)
        {
            return false;
        }

        var itemTemplate = TemplateManager.GetTemplate(item);

        return itemTemplate != null && (itemTemplate.ID == inheritedTemplateItem.ID || itemTemplate.InheritsFrom(inheritedTemplateItem.ID, item.Database));
    }
}

public static class TemplateExtensions
{
    public static ID StandardTemplateId = ID.Parse("{1930BBEB-7805-471A-A3BE-4858AC7CF696}");

    /// <summary>
    /// 
    /// </summary>
    /// <param name="template"></param>
    /// <param name="inheritedId"></param>
    /// <param name="database"></param>
    /// <returns></returns>
    public static bool InheritsFrom(this Template template, ID inheritedId, Database database)
    {
        if (template == null || database == null)
        {
            return false;
        }

        if (template.ID == inheritedId)
        {
            return true;
        }

        var hasBaseTemplate = template.BaseIDs.Any(baseId => baseId == inheritedId);

        if (hasBaseTemplate)
        {
            return true;
        }

        var baseTemplates = template.BaseIDs
                                    .Where(baseId => baseId != TemplateExtensions.StandardTemplateId)
                                    .Select(baseId => TemplateManager.GetTemplate(baseId, database));

        return baseTemplates.Any(baseTemplate => baseTemplate != null && baseTemplate.InheritsFrom(inheritedId, database));
    }
}
```
This should be a drop-in replacement for your existing extension methods. Give it a try and let me know how it worked for you in the comments!

A faster IsDerived Extension Method


In recent months, our Content Authors for an eCommerce site built with **Sitecore 8.2** running on Azure (IaaS) were complaining about the painstaking process of editing and building pages with Sitecore's Experience Editor. While not unheard for Content Authors to complain that the Experience Editor is a bit sluggish (as in always), the authors were pointing out that _it would take 1-1.5 minutes for a single page to refresh or any action to be taken within Experience Editor_. If you compound this by the numerous edits that a single page would need and having multiple devices to edit, it would take the Content Authors roughly 30-45 minutes for a single page to be finished. This would drive anyone mad...

> it would take 1-1.5 minutes for a single page to refresh or any action to be taken within Experience Editor

So what was causing the issue? As it turns out, the major issue with Experience Editor in the Sitecore 8 series (coming from earlier versions) is that everything is componentized and every component makes one or multiple requests to the backend to build out the UI (images, js, css, data requests, etc), for every page refresh.

More requests generally translates into more latency for the time to edit metric, but more requests shouldn't necessary halt all operations. After digging into the issue for several hours, I came across two major issues:

1. Sitecore Speak was not configured for Production usage
2. Azure Application Gateway v1 was inspecting every request

### Configuring the Speak UI for "Production" use

In general, I would recommend that you first tackle these type of performance issues by reviewing and profiling your custom components and then moving on to another layer in the stack. To quickly rule out the application as the bottleneck, we went ahead and disabled all components on the page and still faced the same latency. A blank layout was still taking roughly 1-1.5 minutes to be editable, that wasn't a good sign.

After reviewing the network requests generated by experience editor and noticing that each request was taking longer than expected, we figured out that by default in the Sitecore 8.2 series, the Sitecore Speak UI is configured for developers instead of production usage. To fix this, there are a few settings that need to be tweaked:

- **Speak.HttpCaching.SetMaxAge** - set to false. The default is true and sets the max age of the HTTP resource to 0 which will force the browser to always request the resources on refresh. Setting to false, sets an appropriate max-age for the resource so the browser can cache it.
- **Speak.Html.MinifyScripts** - set to true. The default is false and does not minify the Speak UI scripts
- **Speak.Html.MinifyStylesheets** - set to true. The default is false and does not minify the Speak UI stylesheets
- **IsProductionMode** - set to true. The default is false but it doesn't seem to be used anywhere - can't hurt right?

Example patch configuration:

```xml
<?xml version="1.0"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <settings>
      <setting name="Speak.Html.MinifyScripts">
        <patch:attribute name="value">true</patch:attribute>
      </setting>
      <setting name="Speak.Html.MinifyStylesheets">
        <patch:attribute name="value">true</patch:attribute>
      </setting>
      <setting name="Speak.HttpCaching.SetMaxAge">
        <patch:attribute name="value">false</patch:attribute>
      </setting>
      <setting name="IsProductionMode">
        <patch:attribute name="value">true</patch:attribute>
      </setting>
    </settings>
  </sitecore>
</configuration>
```

By adding the patch configuration and refreshing the Sitecore page, the time to edit metric went from roughly 1.5 minutes to approx. 40 seconds - note that this only occurs after the page assets were cached by the browser, initial request was still delayed.

### **Configuring the Azure Application Gateway v1**

Unfortunately, 40 seconds until time to edit is still not good enough for a solid editing experience. So after a thorough review of all components and application settings it was time to investigate the network and infrastructure setup. To determine if the network was the issue, we loaded experience editor in three scenarios:

1. On the physical machine, removing all latency
2. Through the clients network, less latency
3. Through VPN

As we hoped, while on the physical machine and working with experience editor, the time to edit metric was roughly 6-8 seconds whereas going through the clients network/vpn was roughly 40 seconds. Whoa... you read that right... Something in the network stack was causing a **huge** performance penalty, almost 30-35 seconds of a latency penalty. After a week or two of toggling switches, speaking with the client's networking teams, and scratching our head of where the bottleneck was, we finally ran out of options and decided to start disabling features in the Azure stack. Luckily for us and this article, the first part of the stack was the Application Gateway which has a setting that enables the Web Application Firewall (WAF) and puts it into detection mode - this mode is diagnostic only and logs all threats, but does not block requests. After disabling the WAF on the Content Management server in the Application Gateway, the time to edit metric inside the clients network and through VPN dropped to roughly 10-12 seconds.

So due to how the Speak UI was modularized, the WAF was inspecting each individual request to determine if the request was a threat or not. This caused each individual request to suddenly take 400ms to 1 second longer per request, compounded by the number of requests the experience editor sends, it was easy to see where the issue was.

**A word of caution.** Disabling the WAF can open your infrastructure to security threats, be sure to review and follow the security hardening guidelines provided by Sitecore and

**Note**: Sitecore 8 series and Sitecore 9.0 supports the WAF in detection mode. Sitecore 9.1 fully supports the WAF and you can read more about that [here](https://doc.sitecore.com/developers/91/sitecore-experience-manager/en/using-azure-application-gateway-to-secure-your-content-delivery-server.html).

**Tl;Dr**

In a recent project, there were two undocumented issues that slowed down the experience editor performance:

- Experience Editor requesting a lot of resources without setting correct HTTP cache headers. To resolve, update Sitecore settings to set the cache headers correctly
- Azure Application Gateway enabled on the Content Management server, set to detection mode. To resolve, disable detection mode for the Content Management server


Kyle KingsburyA Sitecore web development blog

Tag - performance

Top 5 reasons your Sitecore search is slow and how to resolve

A faster IsDerived Extension Method

A tale of improving the Sitecore 8.2 experience editor performance