Author
Joseph Neuhaus

Joe Neuhaus has over twenty years of experience in software development, systems design, and technical management roles.

Date
January 20, 2010

Version
ZK 3.6.3, and ZK 5

Introduction

This article is a quick introduction to an impressive new browser-based testing framework named Sahi. If you have attempted to create browser-based functional tests with Selenium, or load tests using Grinder, then you will marvel at the simplicity of Sahi.

The Problem

Testing ZK Web Applications can be a challenge with Selenium and Grinder because ZK dynamically generates element IDs, and these testing frameworks identify elements within the rendered Web Page using these IDs. Therefore, once a session has been recorded, it cannot be replayed because the element IDs will change the next time the Web Application is launched. To address this, ZK provides a hook so you can generate your own IDs. This is a great feature to be sure; however, now you're not testing the application that will be promoted into production. Also, keeping track of "special" test builds of your application increases work and complexity. To make matters worse, some testing frameworks, such as Selenium, require you to install browser plugins to create the recorded browser sessions used for playback. When browser updates occur, you can't upgrade until the plugins are updated too. More moving parts means more issues maintaining the Test Environment. There must be a better way, and there is. It's called Sahi.

The Solution

Sahi can record and playback sessions directly on your production ZK Web Application without having to use a custom ID generator. It requires no browser plugins to create recorded sessions to be used for playback. Sahi is pure Java, so it integrates nicely with ANT. The Sahi scripting language is simple and elegant, so there's no need to break out the XPATH documentation to get your tests running. You can even run your Sahi tests headless using Firefox in an X-Window virtual frame buffer on Linux. In headless mode you can run real browser-based tests on your continuous integration machines without having to be logged in. If you're testing on windows, you can run your tests on IE, Firefox, Chrome, Safari, and Opera. Despite some idiosyncrasies Sahi seems magical at times, but more importantly it makes testing productive - so productive it's almost fun!

The Test Environment

The following are software components I used to build a dedicated 'Test Environment' in Linux; however, you can start small and experiment with Sahi on your Windows workstation first, and then build a dedicated Test Environment later when and if it's required.

Required Components (for test automation)

Sahi V3 - An open-source Automation Tool for Web Application Testing.
JDK 1.6 - Sun's Java Developer Kit (the JRE works too.)
Firefox 3.X - A full-featured, lightning fast, cross-platform Web browser.

Optional Components (for dedicated Test Environment with batch testing capabilities)

VmWare Server - A free cross-platform virtualization technology.
CentOS 5.X - A RedHat binary-compatible Enterprise-class Linux Distribution.
NX Free Edition for Linux - NoMachine NX is a high-performance solution for remote desktop virtualization.
ANT 1.7 - A pure Java build tool, simpler and easier to use than 'make'.
Xvfb - An X11 server that performs all graphical operations in memory.

Setting up the Test Environment

The steps in this section make quite a few assumptions about the testing environment. All the software I use is either free or open-source; however, instead of running tests with Firefox on Linux, you may elect (or need) to use Chrome on Windows. If that's the case, then your mileage may vary. In general, however, the steps should be roughly the same with similar results, thanks to Sahi.

Installing Required and Optional Components

One very important asset in testing is a dedicated machine. I use VmWare to create dedicated CentOS virtual machines for this purpose; however, you may want to use a physical machine for greater performance. Whatever path you choose, for serious testing you should make sure the machine is dedicated to testing. Using a dedicated machine will remove the complexity that arises from the unintended consequences of having games, development tools, and other applications interacting with your test environment. Remember, you want to test your application, not your testing platform!

Below are "hints and tips" to keep in mind when installing various pieces of the Test Environment using Linux. For full installation and operating instructions please visit the appropriate websites.

VmWare Server
1. Make sure your host system is supported. VmWare Server works great on a CentOS or Windows host.
2. If your host O/S is CentOS be sure to apply updates before installing VmWare Server.
3. You must be 'root' to install, but be sure to create a standard system account to manage VmWare Server as well.
4. When creating your Test Environment virtual machine (the client), choose the following: Processors (1), Memory (512 MB), Hard Disk (8 GB), Network (Bridged), SCSI Controller (LSI)
CentOS 5.X Test Environment
1. Choose a Workstation install and keep it as small as possible since you may want to clone this machine later.
2. This means you should not install Openoffice, developer tools, and GNU Java (we'll use Sun's in the next step.)
3. You should install X-Windows with Gnome (or similar) and make sure Firefox will be installed.
4. Install the 'xorg-x11-server-Xvfb' package if you want to experiment with running your tests in headless mode.
Core Services in the Test Environment
1. Download and install into "/opt" (or similar) the following components: JDK, ANT, and Sahi
2. Download and install the following NoMachine NX RPM files in the Test Environment: client, node, server.
3. Download and install the NoMachine NX Client on your host workstation. We'll use this to access the Test Environment's graphical user interface.
Create the file '/etc/profile.d/sahi_testing.sh' (or similar) to setup the shell environment we'll use for testing in the Test Environment.

#############################################
# File: /etc/profile.d/sahi_testing.sh
#############################################
export JAVA_HOME=/opt/jdk1.6.0_16
export ANT_HOME=/opt/apache-ant-1.7.1
export SAHI_HOME=/opt/sahi
export SAHI_USERDATA_DIR=${HOME}/sahidata
export PATH="${SAHI_HOME}/bin:${JAVA_HOME}/bin:${ANT_HOME}/bin:${PATH}"

Validating the Test Environment

Once you have Firefox, Java, and Sahi installed and configured correctly, you're ready to create your first automated Web Application test. I'll assume in this article that we're operating in our dedicated Test Environment. To confirm everything is working as expected, perform the following steps:

Use the NoMachine NX client from your host (or similar X-Session utility) to connect to your virtual machine Test Envrionment. This is the Test Environment with Sahi installed. You'll use the same credentials you use for logging in with SSH.
- IMPORTANT: The user you login in as should be the owner of the $SAHI_USERDATA_DIR directory. When you make recordings the Sahi proxy application needs to write script files to the $SAHI_USERDATA_DIR directory. Even though it is optional, it's a good idea to set the SAHI_USERDATA_DIR environment variable to manage where your test data will be stored.
- IMPORTANT: Be sure to *copy* the contents of Sahi's *userdata* directory to your own $SAHI_USERDATA_DIR directories before running tests. The contents of the *userdata* folder provided in the Sahi package is a set of metadata and data files used by Sahi during testing. This is the data you should check into source code control.
Once you've established an X-Window session, start your favorite terminal shell application, and type the following command:
- % sh $SAHI_HOME/bin/sahi.sh (you can make this file executable to get rid of the leading 'sh')
- Be sure to read the output generated by the sahi.sh command; it tells you where to connect your browser for testing.
Now the Sahi proxy application is running, which is required for recording and playing back sessions. We will connect to it shortly.
Start Firefox and configure it to use a proxy for 'localhost' on port '9999' (detailed instructions are here.)
You should now be able to navigate to your web application normally through the proxy. Once the page is loaded, perform a Ctrl+Alt+Double-Click anywhere within the Web Application's browser page. This will bring up the Sahi controller.
If you see the controller, congratulations, you're ready to move on and start testing. If not, try the tutorial found here; it is much more detailed.

Recording and Playing Back Manually with Sahi

Assuming the steps above are successful, you have launched the Sahi controller from within the browser. Now you're ready to record and playback tests manually. There are already great tutorials for how to make manual recordings with Sahi, so instead of repeating that content here I'll redirect you to the tutorial here where you will learn how to create Sahi test scripts (.sah files) and play them back in a test suite (.suite file.) Once you can create a suite of Sahi tests you're ready to create an automated daily functional test using ANT.

Playing Back Automatically with Ant and Sahi

One really great feature in the Sahi package is ANT integration. As you can see in the sample ANT script below, it's very easy to run Sahi using the <sahi> ANT task. The sample ANT script will start the Sahi proxy server, pause for a few seconds to allow the proxy to start up, begin playback of your recorded tests (organized into a test suite), pause again for a few seconds, and then stop the Sahi proxy server. The example below makes use of environment variables to locate the Sahi libraries and User Data required for the tests. By using environment variables, we can run our script from any arbitrary location and be ensured it will execute correctly.

<?xml version="1.0" encoding="ISO-8859-1"?>
<project name="bids" default="runfftests">
    <property environment="env"/>
    <property name="sahi.home" value="${env.SAHI_HOME}"/>
    <property name="user.data" value="${env.SAHI_USERDATA_DIR}"/>
    <property name="urlbase" value="http://www.zkoss.org/zkdemo/userguide"/>
    <taskdef name="sahi" classname="net.sf.sahi.ant.RunSahiTask" classpath="${sahi.home}/lib/ant-sahi.jar"/>

    <target name="runfftests">
        <antcall target="startsahi"/>
        <sleep seconds="4"/>

        <sahi suite="${user.data}/scripts/zkdemo/zkdemo.suite"
              browser="/usr/lib/firefox-3.0.16/firefox"
              baseurl="${urlbase}"
              sahihost="localhost"
              sahiport="9999"
              failureproperty="sahi.failed"
              haltonfailure="false"
              browseroption="-profile ${user.data}/browser/ff/profiles/sahi$threadNo -no-remote"
              browserProcessName="firefox"
              threads="1">
            <report type="html"/>
            <report type="junit" logdir="${user.data}/temp/junit"/>
        </sahi>
        <sleep seconds="4"/>

        <antcall target="stopsahi"/>
        <sleep seconds="4"/>

       <antcall target="failsahi"/>
    </target>

    <target name="failsahi" if="sahi.failed">
        <fail message="Sahi tests failed!"/>
    </target>

    <target name="sahireport" description="show report">
        <exec command="/usr/lib/firefox-3.0.16/firefox ${user.data}/logs/playback/"/>
    </target>

    <target name="startsahi" description="start sahi proxy">
        <java classname="net.sf.sahi.Proxy" fork="true" spawn="true" dir="${sahi.home}">
            <!--<env key="MOZ_NO_REMOTE" value="1"/>-->
            <classpath location="${sahi.home}/lib/sahi.jar">
                <fileset dir="${sahi.home}/extlib" includes="**/*.jar"/>
            </classpath>
            <arg value="${sahi.home}" id="basePath"/>
            <arg value="${user.data}" id="userdataPath"/>
        </java>
    </target>

    <target name="stopsahi" description="stop sahi server">
        <sahi stop="true" sahihost="localhost" sahiport="9999"/>
    </target>
</project>

If the ant script above is named zkdemo.xml, then we would execute the script using ant with the following command: ant -f zkdemo.xml. You can use CRON (see the man page for the 'crontab' command) to execute your test every day as part of your continuous integration system (more about this in the next two sections.)

Playing Back Headless with Sahi

In order to run your Sahi tests in headless mode you must do two things:

Provide a virtual graphical environment in which to run the browser. To do this we'll use Xvfb, the X-Windows virtual frame buffer.
Instruct the browser to use the virtual graphical environment. To accomplish this, we'll modify the <sahi> ANT task.

Assuming you've installed Xvfb successfully, you can launch it into a background process with the following command:

$ Xvfb :99 -ac &

Here we have instructed Xvfb to use the DISPLAY number :99 and to allow all connections (-ac). Now, the only modification we must make to our previous ANT script is to instruct Firefox to use this DISPLAY that is being provided by Xvfb. To accomplish this, we only need to modify the browseroption line within the <sahi> ANT task by adding the argument --display=:99. The updated <sahi> ANT task is now:

        <sahi suite="${user.data}/scripts/zkdemo/zkdemo.suite"
              browser="/usr/lib/firefox-3.0.16/firefox"
              baseurl="${urlbase}"
              sahihost="localhost"
              sahiport="9999"
              failureproperty="sahi.failed"
              haltonfailure="false"
              browseroption="--display=:99 -profile ${user.data}/browser/ff/profiles/sahi$threadNo -no-remote"
              browserProcessName="firefox"
              threads="1">
            <report type="html"/>
            <report type="junit" logdir="${user.data}/temp/junit"/>
        </sahi>

Now that you have Xvfb running in the background, and the updated ANT script, you won't see the browser launch during the tests; it's all happening in memory. This is exactly what we need for our continuous integration system.

Adding Web Application Functional Tests to your Continuous Integration System

In general, a continuous integration system will perform the following steps on a daily schedule:

Initialize Deployment and Test Environment - Remove previous deployments, test results, and verify environment (e.g. make sure Xvfb is running, etc.)
Deploy Application - Install the application to be tested.
Configure Application - Configure the application specifically for the test environment if necessary.
Initialize Data for Functional and Regression Tests - Setup the application's database with static data that your tests will utilize.
Execute Functional Tests - Run the tests (e.g. ant -f zkdemo.xml)
Evaluate Results - Scan the output of the tests and log files for failure conditions.

All of the steps listed above will be specific to your product and/or continuous integration (CI) system. Assuming you have a working CI system already implemented in Linux, all that remains is to integrate Sahi into it (Similar steps can be taken under Windows.) You should be able to run your Sahi ANT scripts as part of your CI daily jobs; however, how do you evaluate the results? Sahi does a good job at keeping its information organized, and this makes it almost trivial to look for errors. I use the following perl-script (errorNotify.pl) to look for errors reported by Sahi, but you can use any technique that works, as long as it can send you an email if errors are detected. This perl script takes two arguments, the directory to search, and the email to which we send failures. Therefore, you might likely execute this perl script like so: perl errorNotify.pl ${SAHI_USERDATA_DIR}/logs/playback my.email@mydomain.com

#!/usr/bin/perl -w

use File::Find;
use Net::SMTP;

if ($#ARGV < 1)
{
  print qq(Usage: Must supply a directory containing log files to scan for class="ERROR"\n);
  exit;
}

################################################################################
# emailNotify($msg) - Send $msg to the team.
################################################################################
sub emailNotify
{
  my ($email_addr) = $_[0];
  my ($msg) = $_[1];

  $smtp = Net::SMTP->new("your.smtp.server.here");

  $smtp->mail($ENV{USER});
  $smtp->to( qq($email_addr) );
  $smtp->data();
  $smtp->datasend( qq(To: <$email_addr>) );
  $smtp->datasend("\n");
  $smtp->datasend( qq(Subject: SAHI-CI Failure) );
  $smtp->datasend("\n");
  $smtp->datasend( qq(From: SAHI-CI <noreply>) );
  $smtp->datasend("\n\n");
  $smtp->datasend( qq(Sahi Continuous Integration Failure:) );
  $smtp->datasend("\n\n");
  $smtp->datasend( qq($msg) );
  $smtp->datasend("\n\n");
  $smtp->dataend();

  $smtp->quit;
}

################################################################################
# Loop through each file in/under directory searching for strings in files.
# If the pattern(s) are found notify the team.
################################################################################
@filelist = ();

find( sub { push(@filelist,$File::Find::name) }, $ARGV[0]);

$errorDetected = 0; # Set to '1' when errors are detected.
@errors = ();

NFIL: foreach $filename (@filelist)
{
    open(IN, $filename) || die "Cannot open($filename)";
    @lines = <IN>;
    close(IN);

    foreach $line (@lines)
    {
        if ( ($line =~ /class="ERROR"/i) )
        {
            $errorDetected = 1;
            push @errors, qq(LogFile: $filename);
            push @errors, qq(ErrLine: $line);
            next NFIL;
        }
    }
}

$body = join("\n", @errors);

if ($errorDetected == 1)
{
  emailNotify($ARGV[1], $body);
}

Summary

We've seen in this article a fairly robust methodology for creating simple functional tests for any ZK web application. In addition, we've covered some of the additional steps required to run ZK functional tests in a Continuous Integration system implemented in Linux using ANT, Sahi, Xvfb, and a bit of Perl.

Enjoy Sahi, it's a great testing framework!

Comments

Anonymous

Making ZK Functional Tests With Sahi

Namespaces

More

Page actions

Contents