How to create testing infrastructure?

There are many cases in automated testing where you need to write multiple tests which are basically the same, except for some minor changes.

For example, you need to write a test which inserts a value to a filed, and the value should change every time. If you write the whole thing in one test, a simple loop will solve the issue. But if you want a separate test for each value, you’ll need to create many test files that will include the test code and will differ only in the desired value.

Here is an example for a simple python script which takes its “values” from a simple JSON file and creates siesta tests for as many values as there are in the JSON file, a test for each value: (A short explanation regarding the values and flow of the script will follow)

#!/usr/bin/python

import json

import sys

import getopt

import time

import os.path

#######################################

## Declaring default values

#######################################

## This is the default template to generate test files.

testTemplate = """

    StartTest(function(t) {{

       t.chain(

          //here we click on the button which will open a window

          function (next) {{

             var fieldToClick = Ext.ComponentQuery.query('element')[0];

             t.click(fieldToClick,next);

          }},          

          //wait for the window to open

          {{ waitFor : 'componentQuery', args : "windowToOpen" }},

          //insert to the text field in the window a value

          function (next) {{

              var fieldToType = Ext.ComponentQuery.query('textfield')[0];

              t.type (fieldToType, '{0}', next);

          }}

        );

    }})

"""

## Default JSON input file name

inputJSONfile = '/home/user/values.json'

## The default template for test's file names

testFileNameTemplate = "Insert_Value_{0}_To_Textfield.t.js"

#######################################

## This is the script's actual code (The MAIN function)

#######################################

## Printing the final configuration

print ("#######################################\nConfiguration\n#######################################")

print 'Input JSON file is:', inputJSONfile

print 'Template for test''s file names:' , testFileNameTemplate

print 'Test template used for generating test files:', testTemplate

print ("#######################################\n")

## Open the attributes json file

with open(inputJSONfile) as data_file:   

    ## Parsing the JSON data

    data = json.load(data_file)

    # Loop on all the keys in the json

    for k in data.keys():

        ## Extract the attribute name for each record in the json

        valueToInsert = data[k]

                               

        ## Generating the file name

        fileName = testFileNameTemplate.format(valueToInsert)

        ## Creating the test file (This will OVERWRITE EXISTING FILES)

        ## Injecting the values into the test template

        with open(fileName, "w") as fo:

             fo.write(testTemplate.format(valueToInsert))

             
             print (fileName + " created.")

So what exactly happens in this script?

Each test clicks on an element, and insert a value into the textbox which opens.

The python scripts creates all the test files in the same directory from which it runs, and you can supply the JSON file path in the script itself.

The testTemplate is the code which you will be included in each test file.

The inputJSONfile is the path of the JSON file

The testFileNameTemplate is the file name for each test (and can include the value from the JSON, as you can see on this example).

You can manipulate the script and add more code and more parameters to it; the purpose of this post is to give you the main idea for creating such an infrastructure.

Continues Integration – how to run siesta via Jenkins?

As you’ve seen in the previous post (Running siesta from a remote computer), you can run siesta tests remotely. Moreover, you can create a Jenkins job which will run those tests for you, and will indicate for any failed test.

In order to do that, you need first to complete the setup in the “Running siesta from a remote computer” post, so that you’ll have server and client configuration.

After getting that stage done, you’ll need to:

1.  Install siesta on your Jenkins machine.
2.  Install java on your Jenkins machine.
3.  Validate that you can run the webdriver commands from your Jenkins machine. it means that from the path where siesta is installed on your Jenkins, you need to go to the “bin” directory and run from there a test with webdriver. for example:

# webdriver http://IP/siestaTests --include=MytestFile.t.js --host YourHostName --port 4444 --browser=ie --report-format=JUnit --report-file=MytestFile_log_

(If you experience any problems, please refer to the “Running siesta from a remote computer” post).

After you successfully managed to run the webdriver command, you can start to configure the Jenkins job:

1. Access your Jenkins server -> click on “new item” -> choose the “Build a free style software project” option, and give the job a name.

2. In the job configuration window (which will open as you click on the “ok” button), check the “This build is parameterized” option.

3. The parameters are used for your comfort only, and they are NOT a must. I recommend configuring them so that you can run the same build with different options (such as different browsers or different clients) without needing to create a new job for each option.

4. You’ll need to configure 5 parameters, and in order to do so click on “Add parameter” button under the option you checked in step 2.

·         Parameter type:  Text Parameter

Name: CLIENT_NAME

Default value: insert here your client http address, which is the URL of your siesta GUI.

·         Parameter type:  Choice

Name: BROWSER

Choices: insert here the browsers you want to test on, with a separation of a new line between each.

·         Parameter type:  String Parameter

Name: SERVER_NAME

Default value: insert here your server IP address, which is the machine you want the tests to run against.

·         Parameter type:  String Parameter

Name: PORT

Default value: insert here the webderiver server port, which is 4444 by default.

·         Parameter type:  String Parameter

Name: TESTS_TO_INCLUDE

Default value: insert here the path of the tests you want to run. You can remove this parameter and just run all the tests. If you are using this parameter, note that the path you give here is relative to the index.js file.

5. Now add a very simple shell script which will run the webdriver command on your Jenkins. in order to do that you need to click on “Add build step” under the Build section on your job configuration.

6. Choose “execute shell” and in the window that opens, insert the following:

export JAVA_HOME={the path of your java installation, i.e. /opt/java/jdk1.7.0_51}

export PATH=$JAVA_HOME/bin:$PATH

{full location of siesta webdriver, i.e. /var/siesta/bin/webdriver } ${ CLIENT_NAME } --report-format JUnit --report-file-prefix junit- --browser ${BROWSER} --host ${ SERVER_NAME } --port ${ PORT } --no-color --include=${ TESTS_TO_INCLUDE }

7. Now all that’s left is to get the tests results in Jenkins. to do that go to “Post-build Actions” and click on “Add post-build action”. Choose “Publish JUnit test result report” and in the “Test report XMLs” field insert “junit-*”.

8. Click on “save”.

That’s it! All configuration is done, and from now on you can use siesta tests as a part of the continues integration process, by running the siesta-tests build before/after other builds.

Waiting for components in Siesta

“waitFor” is one of the most important methods for testing UI elements, since there are many occasions in which there can be race conditions in UI testing (which are machine-induced and not user induced). For example, if you click on a button which opens a window with a combo-box in it, and you want to click on that combo-box - you must wait until the window is rendered to the DOM in order to click on its combo-box.

How can you make sure that the window is there? That’s what “waitFor” does for you – it waits until the elements you want are visible (or not visible) and only then runs the code after it.

There are a few options to set waiting times in siesta. The simple and basic option is to set the “waitFor” attribute for a certain amount of time:

StartTest(function(t) {

                t.chain(

{ waitFor : 2000 },

functionX ()

                );

};

This will force the test to wait 2 seconds before executing the next function in the chain.

Another option is to start the test only after a certain attribute is rendered to the screen. If you can query this attribute with ComponentQuery, you can also set the “waitFor” method to wait until it appears:

StartTest(function(t) {

                t.chain(

{ waitFor : 'componentQuery', args :  "button[itemId=Foo]" },

functionY()

                );

};

This will force the test to wait until it finds the “Foo” button and only then the test will execute “functionY”.

·         You can also wait for “componentVisible”, which will wait until the element will be rendered to the screen and not only until it can query for it.

Third option is the opposite - to start the test after some attribute is no longer appears on screen. For example, if there’s a progress bar on the page and you want some tests to run after the progress bar disappears, you can use a different form of the “waitFor” method:

StartTest(function(t) {

                t.chain(

{ waitFor : 'componentQueryNotFound', args :  "progressBar" },

functionY()

                );

};

This will make the tests wait until the progress bar (which has an xtype of “progressBar” in ComponentQuery) disappears and only then execute “functionY”. 

How to run some code before each test?

Let’s suppose you need to run some general “code” (a few commands), like a login process or a button click before every test in siesta. You want that each and every test in your test suite will be executed from where the “general code” stopped.

For example, assume that there are two types of tests suites – the first test suite is running on a specific window, and the other is running from a different window. The windows are two different windows entirely, and each of them can be reached from a “main menu” which has two buttons – one button for each window. This means that you need a click on button “x” to be performed before the first type of tests, and a click on button “y” to be performed before the other type of tests. 

In siesta there isn’t yet a solution for this kind of problems, but we can overcome them by adding a new test class (extending the test class) and by overriding the isReady function, so before each test will start to run the code in the new test class will be executed. 

Setting the environment for a new test class

Now it’s time to configure the environment to use a new test class. There are several phases to do this:

1       1. You need to add the test class name to the harness.configure function in index.js file:

Harness.configure({

    title       : ‘This is my test Suite’,

    testClass  : Siesta.Test.Whatever,

});

NOTE: the new test class MUST extend the siesta test class, so its name should be inside the siesta name space. For example, names like “This.Test.Class” will not work, but names like “Siesta.Test.Whatever” will.

         2. You need to create a file which will contain the new test class, and locate it in the webapp folder. Since all the paths are relative to the index.js file, you can create it in the same folder as index.js located in.

3       3. Add this line (which is the new file you created) to the index.html file, right after the “siesta-all.js”  file and under “script” tags:

<script type="text/javascript" src="Pre-Code.js"></script>

          4. In the new class file, you need to override the “isReady” function. You can read more about the function here: http://www.bryntum.com/docs/siesta/#!/api/Siesta.Test-method-isReady

There are a few important things to note when using siesta to test ExtJS and when wanting to use Ext commands or siesta commands (like Ext.ComponentQuery or t.click):

• The “isa” value should be Siesta.Test.ExtJS since we are testing ExtJS application.

• When using Ext commands, instead of the usual “Ext.ComponentQuery” you should use “this.getExt().ComponentQuery”.

• When using variables that are defined in the preload files, you should add “this.global” before their names.

• When using siesta methods (like click or chain), you should add “this” before the method instead on the usual “t”.

• After you finish writing the code you want to do before each test, don’t forget to set the value of “isCustomSetupDone” to true – this will start the test itself.

Callback Method

If you’re a frequent siesta user, you are probably familiar with the “callback” parameter – you can find it in almost all common methods like “click”, “type”, “drag” etc.

Here is the official documentation:

·         callback : Function (optional)

A function to call after click.

In other words, the callback is a method to be called after the current operation is complete.

To better understand that let’s stop for a moment and explain how these common methods behave. When you call the “type” method in your test code some ‘magic’ happens: the browser will start the operation of typing as you requested and running the code that should run as a result of this. At the same time, your test code will continue to run and the next line in your code will be executed. This is best described as an asynchronous method because it is running “in the background” while your next line of code is already being executed.

Now suppose you want a click to be performed after a type operation. That means that you want the test to “wait” and execute the click only after the typing operation is complete. In order to do that you need to provide a function as a callback to the type operation, so when the typing operation is complete it will call this function which in our examples will perform the click.

Let’s take a look at some examples:

if you want to run click after type you can provide the “next” function as a callback in the type method:

t.chain(

       function (next) {

             t.type(typeField,'text to type',next);

       },

       function (next) {

             t.click(button, next);

       }

);      

                    

You can also call the “next” function yourself after a method that does not have a callback option, like the diag method:

t.chain(

          function (next) {

                  t.diag('Im going to click a button!');

                  next();

          },

          function (next) {

                  t.click(button, next);

          }

);

                          

if you want to write a function of your own which uses the callback option – for example, a function that clicks on a certain button, you can write this in your preload file (if you don’t know what a preload file is or why do you need it – go back to the previous posts):

function clickOnFooButton (t, callback) {

var fooBtn = Ext.ComponentQuery.query(‘button[itemId=Foo]’);

    t.chain(

function (next) {

                  t.click(fooBtn[0],next)

            },

            function (next) {

                  callback();

            }

    );

};

And on your test file you can simply call the “clickOnFooButton” function the same way as you would click on a button:

StartTest(function(t) {

     t.chain(

            function (next) {

                  clickOnFooButton (t,  next);
            },
            function (next) {
                   t.type(typeArea,‘hello’); 
            }

     );

};

This will make the test to click on the button which has the itemId of “Foo” and only after that, will type ‘hello’ to the typeArea. 

Running Siesta from a remote computer

How to run Siesta from a remote computer using Selenium Webdriver:

Suppose you have a machine which has linux file system and you write your test on it; But the test you execute need to run on a windows machine (for all browsers support, or from any other reason). 

Now you want to run a command in your linux machine, that will execute remotely all the desired tests on the windows machine and send back to the linux the alerts regarding which test passed and which tests failed.

On the "Server"

1. On the “server” – where you want to run the browser (aka - the windows machine) copy the siesta files and folders (the latest version which you can get from here: http://www.bryntum.com/customerzone/home.php ) to c:\program files\
2.  Install java (if not installed) from: http://www.oracle.com/technetwork/java/javase/downloads/index.html
3. Access the siesta directory from cmd, and run the following commands:

        # cd c:\Program Files\siesta-2.0.4-standard\bin

        # webdriver-server -port 4444

On the "Client"

On the "client" - (in this example, it's the linux machine) navigate to Siesta's installation folder (the webapp directory) and cd to webapp/bin.

From the bin directory, execute the command:

# webdriver [the URL of your siesta GUI] --host ["server" - windows host name or IP address] --port 4444 --browser=[which browser to run on the windows client, i.e: firefox, chrome, ie, safari ]

Additional Flags

If you want to run just one test you can use the “--include” flag. If you want the output to be in JUnit format  you can use the “--report-format” and “--report-file” flags. 

For Example:

# webdriver http://IP/siestaTests --include=MytestFile.t.js --host YourHostName --port 4444 --browser=ie --report-format=JUnit --report-file=MytestFile_log_

For IE

This configuration works fine on firefox and chrome, but for IE there is some adjustments to make:

1. Download the driver from here: https://code.google.com/p/selenium/downloads/list
2. Copy to the windows machine (to C:\installations\selenium-2.37.0).
3. Add this location in the PATH environment variables on your windows machine.
4.  Set the protector mode: open the IE browser, choose "Internet Options..." from the Tools menu, and click on the Security tab. For each zone (4 zones usually), there will be a check box at the bottom of the tab labeled "Enable Protected Mode" – all of them should be the same, all checked or all unchecked.
5. Set the browser to IE9 mode (press F12 and change the browser mode), and place it on top of all other windows. 

In order to make IE run in standard mode, add this line to the index.html file in webapp directory, right after the <head> tag:

Good luck!

Siesta - Getting Started

Siesta - writing automated tests

Since this page will NOT teach you all you need to know about siesta configuration, settings and functions I strongly recommend that you spend some time reading about siesta.
Here are a few useful links:

• Siesta homepage - http://www.bryntum.com/products/siesta/
• Siesta API - http://www.bryntum.com/docs/siesta/#!/api
• Siesta Introduction - http://www.bryntum.com/docs/siesta/#!/guide/siesta_getting_started

Siesta Basics (Briefly)

There are 3 files that needs to be modified when writing a new test in siesta:

1. index.js
2. preload.js – This file is loaded before your tests run.
3. your test file (you can pick any name you want, but keep it meaning-full and in order) for example - Mytestfile.t.js

The files are located in the siesta installation folder:
In your webapp directory you can place your new siesta tests and directories.

index.js

Configuration file for the siesta. In order to view and execute a test, you need an entry for your test file in index.js

preload.js (not mandatory)

In your index.js file you can define which files should be loaded before your tests run. You can name it anyway you want, “preload.js” is a nice name. Though this file isn’t mandatory once you start developing more tests you’ll need a common place to store common functions and variables. This is why it is very useful.  

Writing a Test in Siesta - an Example

Create a test file in your webapp folder – Mytestfile.t.js

Code that you think you will use again (in other tests) should be added to your preload.js file.

For example, write a function that finds the submit button on your page and clicks it. (You’ll probably need to click the submit button in many of your tests).

function clickOnButtonSubmit (t, callback) {

 // Searching for our Submit button
 var button = Ext.ComponentQuery.query('button[action=submit]');
 t.chain( 
  function (next) {
   // Clicking the button 
   t.click(button[0],callback);
  });  
};

Call the function in your test file.

StartTest(function(t) {

 t.chain(
  function (next) {
   clickOnButtonSubmit(t,next);
  }
 );
});

Add the test file to index.js

var Harness = Siesta.Harness.Browser.ExtJS;
Harness.configure({
 title  : 'Awesome Test Suite',
 hostPageUrl : '../yourapp/index.html',
 preload  : [ 'preload.js' ]  // Path is relative to the location of the index.js file
});
Harness.start(
 'Mytestfile.js', // Path is relative to the location of the index.js file
);

Open siesta UI and execute your test!