Tips to Avoid Brittle UI Tests

In the last article I talked about a few ideas and patterns, like the Page Object pattern, that help write maintainable UI tests. In this article we are going to discuss a few advanced topics that could help you write more robust tests, and troubleshoot them when they fail:

• We discuss why adding fixed delays in UI tests is a bad idea and how you can get rid of them.
• Browser automation frameworks target UI elements using selectors and it’s very critical to use good selectors to avoid brittle tests. So I give you some advice on choosing the right selectors and targeting elements directly when possible.
• UI tests fail more frequently then other types of tests, so how can we debug a broken UI test and figure out what caused the failure? In this section I show you how you can capture a screenshot and the page’s HTML source when a UI test fails so you can investigate it easier.

I am going to use Selenium for the browser automation topics discussed in this article.

Much like the previous article the concepts and solutions discussed in this article are applicable regardless of the language and UI framework you use. Before going any further please read the previous article as I am going to refer to it and its sample code a few times. Don’t worry; I’ll wait here.

Don’t Add Delays to Your Tests

Adding Thread.Sleep (or generally delays) feels like an inevitable hack when it comes to UI testing. You have a test that fails intermittently and after some investigation you trace that back to occasional delays in the response; For example, you navigate to a page and look or assert for something before the page is fully loaded and your browser automation framework throws an exception indicating the element doesn’t exist. A lot of things could contribute to this delay. For example:

• The web server, the database and/or the network are overloaded and busy with other requests.
• The page under test is slow because it loads a lot of data and/or queries a lot of tables.
• You’re waiting for some event to happen on the page which takes time.

Or a mix of these and other issues.

Let’s say that you have a page that normally takes less than a second to load but the tests hitting it fail every now and then because of occasional lag in response. You have a few options:

• You don’t add a delay: in this case the tests that hit that page are sometimes going to fail which reduces your trust in the tests.
• You add a one-second delay to the tests hitting that page: in this case all of those tests are always going to take one second longer, even when the page loads fast, but even then the tests are not guaranteed to pass as the page might sometimes take longer then a second to load.
• You might decide to add a few-second delay: this makes sure the page is definitely always loaded, but now your UI tests are taking longer and longer.

You see, there is no winning with arbitrary delays: you either get a slow or a brittle test suite. Here I am going to show you how to avoid inserting fixed delays in your tests. We are going to discuss two types of delays which should cover pretty much all cases you have to deal with: adding a global delay and waiting for something to happen.

Adding a Global Delay

If all of your pages take around the same time to load, which is longer than expected, then most tests are going to fail due to untimely response. In cases like this you can use Implicit Waits:

An implicit wait is to tell WebDriver to poll the DOM for a certain amount of time when trying to find an element or elements if they are not immediately available. The default setting is 0. Once set, the implicit wait is set for the life of the WebDriver object instance.

This is how you set an implicit wait:

WebDriver driver = new FirefoxDriver();driver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(5));

This way you’re telling Selenium to wait up to 5 seconds when it tries to find an element or interact with the page. So now you can write:

driver.Url = "https://somedomain/url_that_delays_loading";IWebElement myDynamicElement = driver.FindElement(By.Id("someDynamicElement"));

instead of:

driver.Url = "https://somedomain/url_that_delays_loading";Thread.Sleep(5000);IWebElement myDynamicElement = driver.FindElement(By.Id("someDynamicElement"));

The benefit of this approach is that FindElement will return as soon as it finds the element and doesn’t wait for the whole 5 seconds when the element is available sooner.

Once implicit wait is set on your WebDriver instance it applies to all actions on the driver; so you can get rid of many Thread.Sleeps in your code.

5 seconds is a wait I made up for this article – you should find the optimal implicit wait for your application and you should make this wait as short as possible. From the API documentations:

Increasing the implicit wait timeout should be used judiciously as it will have an adverse effect on test run time, especially when used with slower location strategies like XPath.

Even if you don’t use XPath, using long implicit waits slows down your tests, particularly when some tests are genuinely failing, because the web driver is going to wait for a long time before it times out and throws an exception.

Waiting for Explicit Events/Changes

Using implicit wait is a great way to get rid of many hardcoded delays in your code; but you are still going to find yourself in a situation where you need to add some fixed delays in your code because you’re waiting for something to happen: a page is slower than all other pages and you have to wait longer, you’re waiting for an AJAX call to finish or for an element to appear on or disappear from the page etc. This is where you need explicit waits.

Explicit Wait

So you have set the implicit wait to 5 seconds and it works for a lot of your tests; but there are still a few pages that sometimes take more than 5 seconds to load and result into failing tests.

As a side note, you should investigate why a page is taking so long first, before trying to fix the broken test by making it wait longer. There might be a performance issue on the page that’s leading to the red test in which case you should fix the page, not the test.

In case of a slow page you can replace fixed delays with Explicit Waits:

An explicit waits is code you define to wait for a certain condition to occur before proceeding further in the code.

You can apply explicit waits using WebDriverWait class. WebDriverWait lives in WebDriver.Support assembly and can be installed using Selenium.Support nuget:

/// <summary>/// Provides the ability to wait for an arbitrary condition during test execution./// </summary>public class WebDriverWait : DefaultWait<IWebDriver>{  /// <summary>  /// Initializes a new instance of the <see cref="T:OpenQA.Selenium.Support.UI.WebDriverWait"/> class.  /// </summary>  /// <param name="driver">The WebDriver instance used to wait.</param><param name="timeout">The timeout value indicating how long to wait for the condition.</param>  public WebDriverWait(IWebDriver driver, TimeSpan timeout);  /// <summary>  /// Initializes a new instance of the <see cref="T:OpenQA.Selenium.Support.UI.WebDriverWait"/> class.  /// </summary>  /// <param name="clock">An object implementing the <see cref="T:OpenQA.Selenium.Support.UI.IClock"/> interface used to determine when time has passed.</param><param name="driver">The WebDriver instance used to wait.</param><param name="timeout">The timeout value indicating how long to wait for the condition.</param><param name="sleepInterval">A <see cref="T:System.TimeSpan"/> value indicating how often to check for the condition to be true.</param>  public WebDriverWait(IClock clock, IWebDriver driver, TimeSpan timeout, TimeSpan sleepInterval);}

Here is an example of how you can use WebDriverWait in your tests:

driver.Url = "https://somedomain/url_that_takes_a_long_time_to_load";WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));var myDynamicElement = wait.Until(d => d.FindElement(By.Id("someElement")));

We are telling Selenium that we want it to wait for this particular page/element for up to 10 seconds.

You are likely to have a few pages that take longer than your default implicit wait and it’s not a good coding practice to keep repeating this code everywhere. After all Test Code Is Code. You could instead extract this into a method and use it from your tests:

public IWebElement FindElementWithWait(By by, int secondsToWait = 10){    var wait = new WebDriverWait(WebDriver, TimeSpan.FromSeconds(secondsToWait));    return wait.Until(d => d.FindElement(by));}

Then you can use this method as:

var slowPage = new SlowPage("https://somedomain/url_that_takes_a_long_time_to_load");var element = slowPage.FindElementWithWait(By.Id("someElement"));

This is a contrived example to show what the method could potentially look like and how it could be used. Ideally you would move all page interactions to your page objects.

Alternative Explicit Wait Example

Let’s see another example of an explicit wait. Sometimes the page is fully loaded but the element isn’t there yet because it is later loaded as the result of an AJAX request. Maybe it’s not an element you’re waiting for but just want to wait for an AJAX interaction to finish before you can make an assertion, say in the database. Again this is where most developers use Thread.Sleep to make sure that, for example, that AJAX call is done and the record is now in the database before they proceed to the next line of the test. This can be easily rectified using JavaScript execution!

Most browser automation frameworks allow you to run JavaScript on the active session, and Selenium is no exception. In Selenium there is an interface called IJavaScriptExecutor with two methods:

/// <summary>/// Defines the interface through which the user can execute JavaScript./// </summary>public interface IJavaScriptExecutor{    /// <summary>    /// Executes JavaScript in the context of the currently selected frame or window.    /// </summary>    /// <param name="script">The JavaScript code to execute.</param<<param name="args"<The arguments to the script.</param>    /// <returns>    /// The value returned by the script.    /// </returns>    object ExecuteScript(string script, params object[] args);    /// <summary>    /// Executes JavaScript asynchronously in the context of the currently selected frame or window.    /// </summary>    /// <param name="script">The JavaScript code to execute.</param<<param name="args"<The arguments to the script.</param>    /// <returns>    /// The value returned by the script.    /// </returns>    object ExecuteAsyncScript(string script, params object[] args);}

This interface is implemented by RemoteWebDriver which is the base class for all web driver implementations. So on your web driver instance you can call ExecuteScript to run a JavaScript script. Here is a method you can use to wait for all AJAX calls to finish (assuming you are using jQuery):

// This is assumed to live in a class that has access to the active `WebDriver` instance through `WebDriver` field/property. public void WaitForAjax(int secondsToWait = 10){    var wait = new WebDriverWait(WebDriver, TimeSpan.FromSeconds(secondsToWait));    wait.Until(d => (bool)((IJavaScriptExecutor)d).ExecuteScript("return jQuery.active == 0"));}

Combine the ExecuteScript with WebDriverWait and you can get rid of Thread.Sleep added for AJAX calls.

jQuery.active returns the number of active AJAX calls initiated by jQuery; so when it is zero there are no AJAX calls in progress. This method obviously only works if all AJAX requests are initiated by jQuery. If you are using other JavaScript libraries for AJAX communications you should consult its API documentations for an equivalent method or keep track of AJAX calls yourself.

ExpectedCondition

With explicit wait you can set a condition and wait until it is met or for the timeout to expire. We saw how we could check for AJAX calls to finish – another example is checking for visibility of an element. Just like the AJAX check, you could write a condition that checks for the visibility of an element; but there is an easier solution for that called ExpectedCondition.

From Selenium documentation:

There are some common conditions that are frequently come across when automating web browsers.

If you are using Java you’re in luck because ExpectedCondition class in Java is quite extensive and has a lot of convenience methods. You can find the documentation here.

.Net developers are not quite as lucky. There is still an ExpectedConditions class in WebDriver.Support assembly (documented here) but it’s very minimal:

public sealed class ExpectedConditions{     /// <summary>     /// An expectation for checking the title of a page.     /// </summary>     /// <param name="title">The expected title, which must be an exact match.</param>     /// <returns>     /// <see langword="true"/> when the title matches; otherwise, <see langword="false"/>.     /// </returns>     public static Func<IWebDriver, bool> TitleIs(string title);     /// <summary>     /// An expectation for checking that the title of a page contains a case-sensitive substring.     /// </summary>     /// <param name="title">The fragment of title expected.</param>     /// <returns>     /// <see langword="true"/> when the title matches; otherwise, <see langword="false"/>.     /// </returns>     public static Func<IWebDriver, bool> TitleContains(string title);     /// <summary>     /// An expectation for checking that an element is present on the DOM of a     /// page. This does not necessarily mean that the element is visible.     /// </summary>     /// <param name="locator">The locator used to find the element.</param>     /// <returns>     /// The <see cref="T:OpenQA.Selenium.IWebElement"/> once it is located.     /// </returns>     public static Func<IWebDriver, IWebElement> ElementExists(By locator);     /// <summary>     /// An expectation for checking that an element is present on the DOM of a page     /// and visible. Visibility means that the element is not only displayed but     /// also has a height and width that is greater than 0.     /// </summary>     /// <param name="locator">The locator used to find the element.</param>     /// <returns>     /// The <see cref="T:OpenQA.Selenium.IWebElement"/> once it is located and visible.     /// </returns>     public static Func<IWebDriver, IWebElement> ElementIsVisible(By locator);}

You can use this class in combination with WebDriverWait:

var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(3))var element = wait.Until(ExpectedConditions.ElementExists(By.Id("foo")));

As you can see from the class signature above you can check for the title or parts of it and for existence and visibility of elements using ExpectedCondition. The out of the box support in .Net might be very minimal; but this class is nothing but a wrapper around some simple conditions. You can just as easily implement other common conditions in a class and use it with WebDriverWait from your test scripts.

FluentWait

Another gem only for Java developers is FluentWait. From the documentation page, FluentWait is

An implementation of the Wait interface that may have its timeout and polling interval configured on the fly. Each FluentWait instance defines the maximum amount of time to wait for a condition, as well as the frequency with which to check the condition. Furthermore, the user may configure the wait to ignore specific types of exceptions whilst waiting, such as NoSuchElementExceptions when searching for an element on the page.

In the following example we’re trying to find an element with id foo on the page polling every five seconds for up to 30 seconds:

// Waiting 30 seconds for an element to be present on the page, checking// for its presence once every five seconds.Wait<WebDriver> wait = new FluentWait<WebDriver>(driver)    .withTimeout(30, SECONDS)    .pollingEvery(5, SECONDS)    .ignoring(NoSuchElementException.class);WebElement foo = wait.until(new Function<WebDriver, WebElement>() {  public WebElement apply(WebDriver driver) {    return driver.findElement(By.id("foo"));  }});

There are two outstanding things about FluentWait: firstly it allows you to specify the polling interval which could improve your test performance and secondly it allows you to ignore the exceptions you are not interested in.

FluentWait is quite awesome and it would be cool if an equivalent existed in .Net too. That said it’s not that hard to implement it using WebDriverWait.

Choose the Right Selectors

You have your Page Objects in place, have a nice DRY maintainable test code, and are also avoiding fixed delays in your tests; but your tests still fail!

The UI is usually the most frequently changed part of a typical application: sometimes you move elements around on a page to change the design of the page and sometimes page structure changes based on requirements. These changes on the page layout and design could lead to a lot of broken tests if you don’t choose your selectors wisely.

Do not use fuzzy selectors and do not rely on the structure of your page.

Many times I have been asked if it’s ok to add an ID to elements on the page only for testing, and the answer is a resounding yes. To make our code unit testable we make a lot of changes to it like adding interfaces and using Dependency Injection. Test Code Is Code. Do what it takes to support your tests.

Let’s say we have a page with the following list:

<ul id="album-list">    <li>        <a href="/Store/Details/6">            <span>The Best Of Men At Work</span>         </a>    </li>    <li>        <a href="/Store/Details/12">            <span>For Those About To Rock We Salute You</span>         </a>    </li>    <li>        <a href="/Store/Details/35">            <span>Let There Be Rock</span>         </a>    </li></ul>

In one of my tests I want to click on the “Let There Be Rock” album. I would be asking for trouble if I used the following selector:

By.XPath("//ul[@id='album-list']/li[3]/a")

When possible you should add ID to elements and target them directly and without relying on their surrounding elements. So I am going to make a small change to the list:

<ul id="album-list">    <li>        <a id="album-6" href="/Store/Details/6">            <span>The Best Of Men At Work</span>         </a>    </li>    <li>        <a id="album-12" href="/Store/Details/12">            <span>For Those About To Rock We Salute You</span>         </a>    </li>    <li>        <a id="album-35" href="/Store/Details/35">            <span>Let There Be Rock</span>         </a>    </li></ul>

I have added id attributes to anchors based on the unique albums’ id so we can target a link directly without having to go through ul and li elements. So now I can replace the brittle selector with By.Id("album-35") which is guaranteed to work as long as that album is on the page, which by the way is a good assertion too. To create that selector I would obviously have to have access to the album id from the test code.

It is not always possible to add unique ids to elements though, like rows in a grid or elements in a list. In cases like this you can use CSS classes and HTML data attributes to attach traceable properties to your elements for easier selection. For example, if you had two lists of albums in your page, one as the result of user search and another one for suggested albums based on user’s previous purchases, you can differentiate them using a CSS class on the ul element, even if that class is not used for styling the list:

<ul class="suggested-albums"></ul>

If you prefer not to have unused CSS classes you could instead use HTML data attributes and change the lists to:

<ul data-albums="suggested"></ul>

and:

<ul data-albums="search-result"></ul>

Debugging UI Tests

One of the main reasons UI tests fail is that an element or text is not found on the page. Sometimes this happens because you land on a wrong page because of navigation errors, or changes to page navigations in your website, or validation errors. Other times it could be because of a missing page or a server error.

Regardless of what causes the error and whether you get this on your CI server log or in your desktop test console, a NoSuchElementException (or the like) is not quite useful for figuring out what went wrong, is it? So when your test fails the only way to troubleshoot the error is to run it again and watch it as it fails. There are a few tricks that could potentially save you from re-running your slow UI tests for troubleshooting. One solution to this is to capture a screenshot whenever a test fails so we can refer back to it later.

There is an interface in Selenium called ITakesScreenshot:

/// <summary>/// Defines the interface used to take screen shot images of the screen./// </summary>public interface ITakesScreenshot{  /// <summary>  /// Gets a <see cref="T:OpenQA.Selenium.Screenshot"/> object representing the image of the page on the screen.  /// </summary>  ///   /// <returns>  /// A <see cref="T:OpenQA.Selenium.Screenshot"/> object containing the image.  /// </returns>  Screenshot GetScreenshot();}

This interface is implemented by web driver classes and can be used like this:

var screenshot = driver.GetScreenshot();screenshot.SaveAsFile("<destination file's full path>", ImageFormat.Png);

This way when a test fails because you’re on a wrong page you can quickly figure it out by checking the captured screenshot.

Even capturing screenshots is not always enough though. For example, you might see the element you expect on the page but the test still fails saying it doesn’t find it, perhaps due to the wrong selector that leads to unsuccessful element lookup. So instead of (or to complement) the screenshot, you could also capture the page source as html. There is a PageSource property on IWebDriver interface (which is implemented by all web drivers):

/// <summary>/// Gets the source of the page last loaded by the browser./// </summary>/// <remarks>/// If the page has been modified after loading (for example, by JavaScript)/// there is no guarantee that the returned text is that of the modified page./// Please consult the documentation of the particular driver being used to/// determine whether the returned text reflects the current state of the page/// or the text last sent by the web server. The page source returned is a/// representation of the underlying DOM: do not expect it to be formatted/// or escaped in the same way as the response sent from the web server./// </remarks>string PageSource { get; }

Just like we did with ITakesScreenshot you could implement a method that grabs the page source and persists it to a file for later inspection:

File.WriteAllText("<Destination file's full path>", driver.PageSource);

You don’t really want to capture screenshots and page sources of all pages you visit and for the passing tests; otherwise you will have to go through thousands of them when something actually goes wrong. Instead you should only capture them when a test fails or otherwise when you need more information for troubleshooting. To avoid polluting the code with too many try-catch blocks and to avoid code duplications you should put all your element lookups and assertions in one class and wrap them with try-catch and then capture the screenshot and/or page source in the catch block. Here is a bit of code you could use for executing actions against an element:

public void Execute(By by, Action<IWebElement> action){    try    {        var element = WebDriver.FindElement(by);        action(element);    }    catch    {        var capturer = new Capturer(WebDriver);        capturer.CaptureScreenshot();        capturer.CapturePageSource();        throw;    }}

The Capturer class can be implemented as:

public class Capturer{    public static string OutputFolder = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "FailedTests");    private readonly RemoteWebDriver _webDriver;    public Capturer(RemoteWebDriver webDriver)    {        _webDriver = webDriver;    }    public void CaptureScreenshot(string fileName = null)    {        var camera = (ITakesScreenshot) _webDriver;        var screenshot = camera.GetScreenshot();        var screenShotPath = GetOutputFilePath(fileName, "png");        screenshot.SaveAsFile(screenShotPath, ImageFormat.Png);    }    public void CapturePageSource(string fileName = null)    {        var filePath = GetOutputFilePath(fileName, "html");        File.WriteAllText(filePath, _webDriver.PageSource);    }    private string GetOutputFilePath(string fileName, string fileExtension)    {        if (!Directory.Exists(OutputFolder))            Directory.CreateDirectory(OutputFolder);        var windowTitle = _webDriver.Title;        fileName = fileName ??                   string.Format("{0}{1}.{2}", windowTitle, DateTime.Now.ToFileTime(), fileExtension).Replace(':', '.');        var outputPath = Path.Combine(OutputFolder, fileName);        var pathChars = Path.GetInvalidPathChars();        var stringBuilder = new StringBuilder(outputPath);        foreach (var item in pathChars)            stringBuilder.Replace(item, '.');        var screenShotPath = stringBuilder.ToString();        return screenShotPath;    }}

This implementation persists the screenshot and HTML source in a folder called FailedTests next to the tests, but you can modify it if you want different behavior.

Although I only showed methods specific to Selenium, similar APIs exist in all automation frameworks I know and can be easily used.

Conclusion

In this article we talked about a few UI testing tips and tricks. We discussed how you can avoid a brittle and slow UI test suite by avoiding fixed delays in your tests. We then discussed how to avoid brittle selectors and tests by choosing selectors wisely and also how to debug your UI tests when they fail.

Most of the code shown in this article can be found in the MvcMusicStore sample repository that we saw in the last article. It’s also worth noting that a lot of code in the MvcMusicStore was borrowed from the Seleno codebase, so if you want to see a lot of cool tricks you might want to check Seleno out. Disclaimer: I am a co-founder of TestStack organization and a contributor on Seleno.

I hope what we’ve discussed in this article helps you in your UI testing endeavors.