Tuesday, August 13, 2013

Implementing HTTP Status Codes in WebDriver, Part 2: Achievement Unlocked


UPDATE (21 August 2013): In response to a comment by Eric Lawrence (author of Fiddler and all around awesome chap), I've updated the code sample for the redirect case. Thanks Eric for taking the time to comment and point out where I could make improvements.

In Part 1 of this series, we looked at the beginnings of implementing HTTP status codes in WebDriver the correct way. That is to say, by using a proxy server to monitor traffic for the information we want. To recap, we're using Fiddler as our proxy, the .NET bindings to execute our WebDriver code, and we're running against Mozilla's website as our test destination. At the end of the last blog post, we successfully had a proxy hooked up, which will log resources to the console as they are requested by the browser. Now it's time to actually extract the HTTP status codes from the information that the proxy is able to collect. As a reminder, here's what our WebDriver execution looks like:
private static void TestStatusCodes(IWebDriver driver)
{
    // Using Mozilla's main page, because it demonstrates some of
    // the potential problems with HTTP status code retrieval, and
    // why there is not a one-size-fits-all approach to it.
    string url = "http://www.mozilla.org/";
    driver.Navigate().GoToUrl(url);

    string elementId = "firefox-promo-link";
    IWebElement element = driver.FindElement(By.Id(elementId));
    element.Click();

    // Demonstrates navigating to a 404 page.
    url = "http://www.mozilla.org/en-US/doesnotexist.html";
    driver.Navigate().GoToUrl(url);
}
So the first thing we are doing in our WebDriver code is navigating to http://www.mozilla.org/. So let's create a method that will perform the navigation, and return us the status code. As we saw last time, Fiddler lets us hook up an event delegate to respond every time a resource is retrieved by the browser, and analyze that response. The nice thing about event delegates in .NET is that we don't need to leave them hooked up any longer than necessary. Here's our first stab at a method that will hook and unhook the delegate for the navigation:
public static int NavigateTo(IWebDriver driver, string targetUrl)
{
    int responseCode = 0;
    SessionStateHandler responseHandler = delegate(Session targetSession)
    {
        responseCode = targetSession.responseCode;
    };

    FiddlerApplication.AfterSessionComplete += responseHandler;
    driver.Url = targetUrl;
    while (responseCode == 0)
    {
        System.Threading.Thread.Sleep(100);
    }

    FiddlerApplication.AfterSessionComplete -= responseHandler;
    return responseCode;
}
Astute readers will see that this has a couple of issues with it. First, how do we know what behavior we want for redirects? Our base URL to which we're navigating has just such a redirect. Do we expect to return a 300-level response, or follow the navigations through until we receive a 200-level or 400-level response? This is a perfect example of why there's no one-size-fits-all approach to HTTP status codes that will work for every WebDriver user, and a reason why, in turn, this feature is out of scope in the WebDriver API. In our case, if the URL redirects for navigation, we're going to return the redirect response code. In your implementation, if you decide on another approach, you'll want to modify the event handler delegate to meet your own needs.

The second issue is that we aren't guaranteed that we're returning the response code for the proper resource. So we want a modification that will validate that. Also, we'll probably want to create a timeout so that we don't inadvertently loop infinitely in the while loop. Making these modifications, you'll get a method that looks something like this:
public static int NavigateTo(IWebDriver driver, string targetUrl)
{
    int responseCode = 0;
    SessionStateHandler responseHandler = delegate(Session targetSession)
    {
        if (targetSession.fullUrl == targetUrl)
        {
            responseCode = targetSession.responseCode;
        }
    };

    FiddlerApplication.AfterSessionComplete += responseHandler;

    // Yes, we're hard-coding a 10 second timeout here. Don't worry, we'll
    // make that configurable before we're done.
    DateTime endTime = DateTime.Now.Add(TimeSpan.FromSeconds(10));
    driver.Navigate().GoToUrl(targetUrl);
    while (responseCode == 0 && DateTime.Now < endTime)
    {
        System.Threading.Thread.Sleep(100);
    }

    FiddlerApplication.AfterSessionComplete -= responseHandler;
    return responseCode;
}
Okay, so now we have a method that will return us the status code on explicit navigation to a URL. What about on a click that navigates to a new location? Clicks are a little trickier, because a click might trigger a navigation, or it might not. In my opinion, you should know what type of click you'll be performing, so I'll create a method that we will explicitly call when we want to perform a click that will navigate, and return the HTTP status code of that navigation. I'll also take this opportunity to demonstrate a way to handle redirects, since the link we're clicking on in our test code also causes a redirect. Again, we'll hook up a delegate for the duration of the time we need it, and unhook it after we're done.
public static int ClickNavigate(IWebElement element)
{
    int responseCode = 0;
    string targetUrl = string.Empty;
    SessionStateHandler responseHandler = delegate(Session targetSession)
    {
        // For the first session of the click, the URL should be the initial 
        // URL requested by the element click.
        if (string.IsNullOrEmpty(targetUrl))
        {
            targetUrl = targetSession.fullUrl;
        }

        // This algorithm could be much more sophisticated based on your
        // needs. In our case, we'll only look for responses where the
        // content type is HTML, and that the URL of the session matches
        // our current target URL. Note that we also only set the response
        // code if it's not already been set.
        if (targetSession.oResponse["Content-Type"].Contains("text/html") && 
            targetSession.fullUrl == targetUrl &&
            responseCode == 0)
        {
            // If the response code is a redirect, get the URL of the
            // redirect, so that we can look for the next response from
            // the session for that URL.
            if (targetSession.responseCode >= 300 &&
                targetSession.responseCode < 400)
            {
                // Use GetRedirectTargetURL rather than examining the
                // "Location" header, as some sites (illegally) might
                // use a relative URL for the header (per Eric Lawrence).
                targetUrl = targetSession.GetRedirectTargetURL();
            }
            else
            {
                responseCode = targetSession.responseCode;
            }
        }
    };

    // Note that we're using the ResponseHeadersAvailable event so
    // as to avoid a race condition with the browser (per Eric
    // Lawrence).
    FiddlerApplication.ResponseHeadersAvailable += responseHandler;

    // Yes, we're hard-coding a 10 second timeout here. Don't worry, we'll
    // make that configurable before we're done.
    DateTime endTime = DateTime.Now.Add(TimeSpan.FromSeconds(10));
    element.Click();
    while (responseCode == 0 && DateTime.Now < endTime)
    {
        System.Threading.Thread.Sleep(100);
    }

    FiddlerApplication.ResponseHeadersAvailable -= responseHandler;
    return responseCode;
}
All that remains is to modify our WebDriver code to call our new methods instead of the standard WebDriver ones, and add some console logging to prove that we get actual status codes returned from our methods. That modifies our TestStatusCodes method to look like this:
private static void TestStatusCodes(IWebDriver driver)
{
    // Using Mozilla's main page, because it demonstrates some of
    // the potential problems with HTTP status code retrieval, and
    // why there is not a one-size-fits-all approach to it.
    string url = "http://www.mozilla.org/";
    int responseCode = NavigateTo(driver, url);
    Console.WriteLine("Navigation to {0} returned response code {1}",
                      url, responseCode);

    string elementId = "firefox-promo-link";
    IWebElement element = driver.FindElement(By.Id(elementId));
    responseCode = ClickNavigate(element);
    Console.WriteLine("Element click returned response code {0}",
                      responseCode);

    // Demonstrates navigating to a 404 page.
    url = "http://www.mozilla.org/en-US/doesnotexist.html";
    responseCode = NavigateTo(driver, url);
    Console.WriteLine("Navigation to {0} returned response code {1}",
                      url, responseCode);
}
Running our console application from last time, we now will receive output that looks like the following:
Starting Fiddler proxy
Fiddler proxy listening on port 62594
Navigating to http://www.mozilla.org/
Navigation to http://www.mozilla.org/ returned response code 301
Clicking on element with ID firefox-promo-link
Element click returned response code 200
Navigating to http://www.mozilla.org/en-US/doesnotexist.html
Navigation to http://www.mozilla.org/en-US/doesnotexist.html returned response code 404
Shutting down Fiddler proxy
Complete! Press <Enter> to exit.
Now we have a fully functioning example for Firefox. Next time, we'll add the code to make it cross-browser aware, and add a few more tricks to make it more elegant for use with WebDriver.

9 comments:

  1. Hiya, Jim! Your ClickNavigate function is at risk of running on the wrong response because it doesn't check that the URL of the session matches the request (or a subsequent LOCATION header received from redirect response). You might also consider checking the X-ProcessInfo flag on the Session to eliminate any noise from other processes.

    ReplyDelete
    Replies
    1. Hi, Eric. Of course, you're right. I'd like to be able to say that it was intentional, and that the comment in the source code listing of the blog post would be an indicator that this is a naïve algorithm, and the user should plan on implementing their own to meet their specific needs. I'd also like to be able to say a future post in the series would refine the algorithm to make it more robust and correct. Alas, I can't say either of those things. The real reason is that most of the time, I just don't know what I'm doing.

      Delete
    2. And, as of today, I've updated the code sample in ClickNavigate to be more robust and in line with your comments. Thanks again for the help, Eric!

      Delete
  2. Hi Jim,
    Could you please suggest how to create the cookies , probably using the cookie API of WebDriver for IE so IE does not use notation Profiles like FF and chrome does.
    Any document about fixing this issue would be helpful for all.

    ReplyDelete
  3. Hi Jim, Does Fiddler have any Java API or what can be used to implement similar in Java? Thanks...

    ReplyDelete
    Replies
    1. There's nothing to prevent you from using Fiddler from a non-.NET language, but the architecture would be somewhat different. You'd essentially need to start Fiddler executable, have Fiddler capture the traffic to a file, and parse the results of the file yourself.

      If you're using Java, the BrowserMob proxy[1] is a choice I've heard many people having success with. It's native Java, and has both a programmatic API and a REST API. There is even a .NET wrapper for the BrowserMob proxy[2], but I chose Fiddler here to remove the dependency on the Java runtime.

      [1] https://github.com/lightbody/browsermob-proxy
      [2] https://github.com/AutomatedTester/AutomatedTester.BrowserMob

      Delete
  4. I want capture https traffic using selenium and fiddler proxy and registered flags as below
    FiddlerCoreStartupFlags flags = FiddlerCoreStartupFlags.DecryptSSL & FiddlerCoreStartupFlags.AllowRemoteClients & FiddlerCoreStartupFlags.CaptureFTP & FiddlerCoreStartupFlags.ChainToUpstreamGateway & FiddlerCoreStartupFlags.MonitorAllConnections & FiddlerCoreStartupFlags.CaptureLocalhostTraffic;

    but failed to capture https traffic getting only http traffic

    ReplyDelete
    Replies
    1. I am having the same issue. Can only capture http traffic, not getting https traffic even with the flags above. I found out that you may need to install the Fiddler Root Certificate onto the local machine but all attempts at doing that have failed. Could someone please expand this to show how this would work for https traffic?

      Delete
    2. CONFIG.bCaptureCONNECT = true;
      CONFIG.IgnoreServerCertErrors = true;

      int iProcCount = Environment.ProcessorCount;
      int iMinWorkerThreads = Math.Max(16, 6 * iProcCount);
      int iMinIOThreads = iProcCount;
      if ((iMinWorkerThreads > 0) && (iMinIOThreads > 0))
      {
      System.Threading.ThreadPool.SetMinThreads(iMinWorkerThreads, iMinIOThreads);
      }

      FiddlerApplication.Startup(0, FiddlerCoreStartupFlags.Default);

      I have used FiddlerCore dll

      Delete