One file example to run data driven tests using python and nose with XML and HTML reports

I was discussing data driven tests with a colleague and how easy it to create in python just with nose test framework.

The file looks like:

def test_dirs():
    for data in test_data():
        yield check_dir, data[0], data[1]

def check_dir(dir_ref, dir_new_gen):
    assert len(dir_ref) > 1
    assert len(dir_new_gen) > 1

def test_data():
    return [["dir1_ref", "dir1_gen"], ["dir2_ref", "dir2_gen"]]

I created a repository  and shared with him following is current content of the Readme:

https://github.com/mubbashir/py_nosetest_data_driven

One file example to run data driven tests using python and nose tests with XML and HTML reports

Documents

• nose test
• Test-generators
• nose-html-output

Optional VENV (Python virtual environment)

• Create virtual environment python -m venv venv
• Activate virtual env
  • Linux/Mac source venv/bin/activate

$ source venv/bin/activate
(venv) [akhan@seq-akh-fc pipeline_workflow_tests] which python
~/workspace/pipeline_workflow_tests/venv/bin/python

Install and execute

• Install dependencies: pip install -r requirements.txt
• Run tests

$nosetests -v
nose_test_generator.test_dirs('dir1_ref', 'dir1_gen') ... ok
nose_test_generator.test_dirs('dir2_ref', 'dir2_gen') ... ok
nose_test_generator.test_data ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.003s

OK

Run with reports

$nosetests -v --with-xunit --xunit-file=nose_test_results.xml --with-html-output --html-out-file=nose_test_results.html
nose_test_generator.test_dirs('dir1_ref', 'dir1_gen') ... ok
nose_test_generator.test_dirs('dir2_ref', 'dir2_gen') ... ok
nose_test_generator.test_data ... ok

----------------------------------------------------------------------
XML: /home/akhan/workspace/py_nosetest_data_driven/nose_test_results.xml
----------------------------------------------------------------------
Ran 3 tests in 0.002s

OK

• XML Report
• HTML Report

Repository: https://github.com/mubbashir/py_nosetest_data_driven

Gradle task to check files/commands in PATH environment variable before running tests

Recently I was debugging a failing tests and it turn out the when Jenkins runs the tests on a remote node shell environment is not initialized hence one of binary dependency of the application under tests was missing.

Solution:
Add the additional location in the PATH from jenkins
e.g. export PATH=$PATH:/opt/bin/

Force checking in gradle:
In the light of principle like ""treat test code with the same level of care as production code" and To help my fellow tester (and my future self) it's better to add this as check in gradle.

// rest of your build
// Verfied with gradle 4.10
task("checkEnv"){
    doFirst {
        def listOfFileToCheckInPath = ['find', 'grep']
        listOfFileToCheckInPath.each { file ->
            if(!isFoundInPath(file))
                throw new GradleException("${file} was not found in any of the folder in PATH: ${System.getenv('PATH').split(File.pathSeparator)}")
        }
    }
}
/**
 * Static function to verify if a file/command exist in PATH environment 
 * @param file
 * @return true if found, else false
 */
def static isFoundInPath( file){
    def PATH_ENV = System.getenv('PATH')
    def fileFound = PATH_ENV.split(File.pathSeparator).find{ folder ->
        println("Looking for ${file} in ${folder}")
        if (Paths.get( "${folder}${File.separator}${file}").toFile().exists()){
            println("Found ${file} in ${folder}")
            return true
        }
    }
    return fileFound
}
// Making test task to depend on checkEnv
test.dependsOn checkEnv

The output would look something like:

./gradlew checkEnv
> Task :checkEnv
Looking for find in /usr/local/bin
Looking for find in /usr/bin
Found find in /usr/bin
Looking for grep in /usr/local/bin
Looking for grep in /usr/bin
Found grep in /usr/bin

BUILD SUCCESSFUL in 0s

Gist on github

Jenkins Script Console and some Groovy Scripts

I am one of those jenkins users who use Jenkins Script Console  [1] extensively. I have been maintaining some of those scripts in a private repository, lately I came across this gist [2] by Damien Nozay which uses a much simpler way to share these scripts i.e. using gists :D
After forking it I have started adding my scripts in gists as well   https://gist.github.com/mubbashir/484903fda934aeea9f30 [3].

I found it quite useful e.g. lately I need to know the results of all the downstream jobs triggered by a certain job:
https://gist.github.com/mubbashir/#file-jobstatustriggeredbyupstreamcause

/ author : Ahmed Mubbashir Khan
// Licensed under MIT
// ---------------------------------------------------------
// This script prints out information of last dwonstream job of Upstream job
// e.g. printInformationOfDownstreamJobs("ChangeListner", 11, "All Tests")
// will print all the downstream jobs invodked by ChangeListner build 11 in the view "All Tests"
// ---------------------------------------------------------

import hudson.model.*
//printInformationOfDownstreamJobs("ChangeListner", 11, "All Tests")

def printInformationOfDownstreamJobs(jobName, buildnumber, viewName){
  def upStreamBuild = Jenkins.getInstance().getItemByFullName(jobName).getBuildByNumber(buildnumber)
  println "${upStreamBuild.fullDisplayName}" +
   "${upStreamBuild.getCause(hudson.model.Cause.UpstreamCause).upstreamRun}"
  def cause_pattern = /.*${jobName}.*${buildnumber}.*/
  println "Cause pattern: ${cause_pattern}"
  // Upated for any builds of the upstream  job in a view from only the last build
  def view = Hudson.instance.getView(viewName)
  def buildsByCause = []
   // For each item in the view 
  view.getItems().each{ 
    def jobBuilds=it.getBuilds() // get all the builds 
    jobsBuildsByCause = jobBuilds.findAll { build ->    
    build != null &&
    build.getCause(hudson.model.Cause.UpstreamCause)!= null &&
    build.getCause(hudson.model.Cause.UpstreamCause).upstreamRun==~cause_pattern
    }
    buildsByCause.addAll(jobsBuildsByCause)
  }
   // printing information 
  buildsByCause.each{ d_build->
   // def d_build = job.lastBuild
    println("Build: ${d_build.fullDisplayName}->"+
     "result:${d_build.result}->${d_build.buildStatusSummary.message}, " +
     "(was triggered by:${d_build.getCause(hudson.model.Cause.UpstreamCause).upstreamRun})" )
  }
}




Or information about the builds in a new e.g. node on  which they were executed along with the time they took:
https://gist.github.com/mubbashir/#file-jobs-in-view-with-duration-label-groovy 

// author : Ahmed Mubbashir Khan
// ---------------------------------------------------------
// This script goes through all the jobs in a view, filters succesful and failed jobs seprately
// Then prints outs them along with the time they took
// ---------------------------------------------------------
import hudson.model.*
def str_view = "Pipeline Tests"
def view = Hudson.instance.getView(str_view)
def successfulJobs = view.getItems().findAll{job -> job.lastBuild != null && job.lastBuild.result == hudson.model.Result.SUCCESS}
def faildJobs = view.getItems().findAll{job -> job.lastBuild != null && job.lastBuild.result == hudson.model.Result.FAILURE}
def disabledJob = view.getItems().findAll{job -> job.disabled == true}
def enabledJob = view.getItems().findAll{job -> job.disabled != true}
println "Total jobs: " + view.getItems().size +" Successful: " +successfulJobs.size+
  " Failed: " + faildJobs.size + " Enabled jobs: " +enabledJob.size + " Disabled jobs: " +disabledJob.size 
println "Current Successful job:"
successfulJobs.each{job -> printInfo(job)}
println "Current Fail job:"
faildJobs.each{job -> printInfo(job)}
println "Current disabled job:"
disabledJob.each{job -> printInfo(job)}
println "Current enabled job:"
enabledJob.each{job -> printInfo(job)}

def printInfo(job){
  println "Job: ${job.name} build on ${job.getAssignedLabelString()}, "+
    "took ${job.lastBuild.getDurationString()} to build, is disabled : ${job.disabled}"
}

[1] https://wiki.jenkins.io/display/JENKINS/Jenkins+Script+Console
[2] https://gist.github.com/dnozay/e7afcf7a7dd8f73a4e05
[3] https://gist.github.com/mubbashir/484903fda934aeea9f30 

Update:
Updated https://gist.github.com/mubbashir/#file-jobstatustriggeredbyupstreamcause to list any downstream job which matches and upstream cause

Interacitve shell for php-webdriver to debug selenium

I do most of my selenium (actually locator’s) debugging in chrome console, $() for css selectors and and $x() for xpath selectors but when it actually comes to debugging selenium/webdriver api calls I prefer interactive shell of programming language (python, ruby and php and pretty decent built in shells)

Bellow is how you can configure php shell to play around with selenium

Installation

• get composer Composer (If you don’t already use Composer)

curl -sS https://getcomposer.org/installer | php
mv composer.phar /usr/local/bin/composer

• get psysh A runtime developer console, interactive debugger and REPL for PHP

wget psysh.org/psysh
chmod +x psysh
./psysh

Then install the library:

php composer.phar require facebook/webdriver

All you need as the server for this client is the selenium-server-standalone-#.jar file provided here: http://selenium-release.storage.googleapis.com/index.html

Download and run that file, replacing # with the current server version.

java -jar selenium-server-standalone-#.jar

// author : Ahmed Mubbashir Khan
require_once 'vendor/autoload.php'; //composer generated autoload file
use Facebook\WebDriver\Remote\DesiredCapabilities;
use Facebook\WebDriver\Remote\RemoteWebDriver;
use Facebook\WebDriver\WebDriverBy;

$host = 'http://192.168.99.100:4444/wd/hub';
$driver = RemoteWebDriver::create($host, DesiredCapabilities::firefox());
/* 
Or  if you wan't to modify $capabilities
use Facebook\WebDriver\Remote\WebDriverCapabilityType;
$capabilities = array(\WebDriverCapabilityType::BROWSER_NAME => 'firefox');
$driver = RemoteWebDriver::create($host, $capabilities);
*/
//e.g. Mouse over on an element
$element = $driver->findElement(WebDriverBy::id('some_id'));
$driver->getMouse()->mouseMove( $element->getCoordinates() );

Reference:
 - [1]: http://selenium-release.storage.googleapis.com/index.html
 - [2]: https://github.com/facebook/php-webdriver
 - [3]: https://getcomposer.org/
-  [4]: http://psysh.org/

Jenkins: List all the jobs for which SCM is not currently configured

Put the following in  Jenkins Script Console  to  list all the jobs for which SCM is not currently configured:

// Licensed under MIT
// author : Ahmed Mubbashir Khan
// ---------------------------------------------------------
// This script goes through all the jobs and checks if they configured SCM is hudson.scm.NullSCM
// if they are, then prints it's info
// ---------------------------------------------------------
 

counter = 0
jobs = Jenkins.instance.getAllItems()
for (job in jobs) {
  if (job.scm instanceof hudson.scm.NullSCM){
    println "Job= '${counter++}' '${job.name}' scm '${job.scm}'"
  }
}

SeConf 2015 - Portland

Selenium, Jbehave , gradle and Serenity - Perfect combination for automating BDD styled checks in Java

I have just created this demo project using Selenium, gradle, Jbehave and Serenity

https://github.com/mubbashir/jbehave-selenium-bdd

Why add Serenity?

Serenity adds plugins to Jbehave and makes it a lot easier to write acceptance tests as mentioned on it site Serenity BDD helps you write cleaner and more maintainable automated acceptance and regression tests faster.
The other things I loved about Serenity is the it has wonderful reports  for BDD "Serenity also uses the test results to produce illustrated, narrative reports that document and describe what your application does and how it works."
See:

Structure

├── ReadMe.md  ## ReadME file of cource ;)
├── build.gradle ## gradle build file 
├── gradle
│   └── wrapper
│       ├── gradle-wrapper.jar
│       └── gradle-wrapper.properties
├── gradlew ## grdadle warapper
├── src
│   └── test
│       ├── java
│       │   └── test
│       │       ├── WebTest.java ## This is where the tests resides which add all the steps using net.serenitybdd.jbehave.SerenityStepFactory
│       │       └── selenium
│       │           ├── page ## This where all the pages extending net.serenitybdd.core.pages.PageObject
│       │           │   ├── google
│       │           │   │   └── GoogleLanding.java
│       │           │   └── interfaces
│       │           │       └── Landing.java
│       │           └── steps ## Pages are used in steps 
│       │               └── GoogleSteps.java
│       └── resources
│           └── stories
│               └── google
│                   ├── Search2.story
│                   └── search.story

Refrences

• [jbehave]: http://jbehave.org/
• [gradle]: http://gradle.org/
• [serenity]: http://thucydides.info/docs/serenity/#first-steps

Adding pre-commit (package) to git repo of your python projects

Git pre commits  are quite useful to analyze  the code before you make a commit and pre-commit makes it quite easy to mange these hooks and rules to run via these hooks.

• Install pre-commit, pip install pre-commit ## detailed instruction  on it's website pre-commit
• Run pre-commit install ## Usage instruction  on it's website pre-commit
• Create .pre-commit-config.yaml file in your project root
• I have added it to  behave-selenium, check it out

pre-commit install will install pre-commit into your git hooks, now on words it will be run on every commit. 

Every time you clone a  project running pre-commit install should always be the first thing you do.

The first time pre-commit runs on a file it will automatically download, install, and run the hook. Note that running a hook for the first time may be slow. 

• To Run all pre-commit manually on all file type pre-commit run --all-files
• To run specific hook (via the id in .pre-commit-config.yaml) on specific file(s) pre-commit run autopep8-wrapper --files features/steps/google_steps.py

We can also run all the pre commit hook as a CI step e.g. pre-commit run --all-files 

Individual tools can configure in accordance to according to there configuration. For example flake8 can be configured by adding a setup.cfg file 

Parallel execution of automated check via selenium, behave and docker

We were looking for a solution for executing selenium bases scenarios written in behave in parallel to speed up the execution time. Unluckily behave doesn't have this in build like cucumber  fortunately we came across https://github.com/hugeinc/behave-parallel  in which  upstemsync was just couple of weeks old :)

I already had a picture in mind how would the entire step would work so creating  a demo project wasn't too difficult.

Check out https://github.com/mubbashir/behave-selenium 
Following is screen recoding where I had connected to both nodes via VNC and checks were executed in Parallel  on features level

Links:
https://github.com/hugeinc/behave-parallel
https://github.com/behave/behave/
https://github.com/SeleniumHQ/docker-selenium

Following is what the readme says at the moment:

behave-selenium

Selenium's example using behave

A quick example of python, behave, selenium, webdriver and docker to run prallel tests

• Install docker and create a selenium grid with say 2 nodes
  • docker run -d -p 4444:4444 --name selenium-hub selenium/hub:2.47.1 ## To Run the Hub
  • docker run -d -P --link selenium-hub:hub selenium/node-firefox-debug:2.47.1 ## Run it twice to create 2 nodes, Docker will pick a random name
  • Check the running docker containers

$ docker ps
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS                     NAMES
34911106818f        selenium/node-firefox-debug:2.47.1   "/opt/bin/entry_point"   2 hours ago         Up 2 hours          0.0.0.0:32769->5900/tcp   condescending_elion
14971ef95408        selenium/node-firefox-debug:2.47.1   "/opt/bin/entry_point"   2 hours ago         Up 2 hours          0.0.0.0:32768->5900/tcp   trusting_mahavira
10688bf744d4        selenium/hub:2.47.1                  "/opt/bin/entry_point"   3 hours ago         Up 3 hours          0.0.0.0:4444->4444/tcp    selenium-hub

• Selenium gird console to have two nodes http://192.168.99.100:4444/grid/console # if you are on mac or on windows get the ip of the running docker vm e.g. docker-machine ip default
• Get port of VNC if you want to connect to a node

docker port <container-name|container-id> 5900
docker port 34911106818f
#=> 0.0.0.0:49338

• Clone the repo
• Create virtulenv $virtualenv env
• Activate the virtualenv and run the requirements file via pip source env/bin/activate; pip install -r requirements.txt
• Export GRID_HUB_URL export GRID_HUB_URL='http://192.168.99.100:4444/wd/hub'
• Run behave --processes 2 --parallel-element feature

Contributing

• Run the requiremetns file in the virtual environment
• Run pre-commit install ## To install pre-commit

Run pre-commit install to install pre-commit into your git hooks. pre-commit will now run on every commit. Every time you clone this project running pre-commit install should always be the first thing you do.

If you want to manually run all pre-commit hooks on a repository, run pre-commit run --all-files. To run individual hooks use pre-commit run <hook_id>.

The first time pre-commit runs on a file it will automatically download, install, and run the hook. Note that running a hook for the first time may be slow. For example: If the machine does not have node installed, pre-commit will download and build a copy of node.

• To Run all pre-commit manually on all file type pre-commit run --all-files
• To run specific hook (via the id in .pre-commit-config.yaml) on specific file(s) pre-commit run flake8 --files bdd_feature_tests/features/steps/dubizzle_homepage_steps.py
• To run autopep 8 manually (which will run automatically once pre-commit is configured) type pre-commit run 'autopep8-wrapper' --files bdd_feature_tests/features/steps/dubizzle_homepage_steps.py

Flashback: Intro to Selenium and Automation- Slide deck

Flashback -  Today while going through some documents on my google drive I came across this presentation which I have used in the past back to give very basic introduction to selenium and automation.

My Meeting with James Bach

Last Saturday I had the privileged to meet with James Bach (If you are a software tester, I expect you to know him) and as you can see below I was

Can't express how excited I am :) and he played the dice game with me. Thank a lot @jamesmarcusbach for your time. pic.twitter.com/ZPwD0MfCll

— mubbashir (@mubbashir) September 12, 2015

I asked him about Sapient Testing->Non Scripted Testing->Testing vs Checking  and how these 3 are different.

He corrected my understanding of Oracles I was thinking about it too specific from the perspective of checks by focusing on pre defined rules whereas "In testing, an “oracle” is a way to recognise a problem that appears during testing."

 We talk about check automation where he told me about this wonderful analogy that think of automated checks as a web and tester as Spider.

The spider is responsible for creating it, managing it, to find the right place and spot for it, a place where there is a greater chance to catch bugs and so on...

And then comes the best part when he played the Dice game with me 

Just after few minutes in the game I felt myself bit overwhelmed :)

Few of the points I took as a take away after that session:

• Taking clear and correct (Yes correct) is extremely important
• Simplify problem: Start with the very basic try understand the system with the very basic data
• Don't get yourself too attached with the basic data, once you are sure how it is working for simple data - "Explore", "Randomise" and Look "Below the surface"
• Use Focus / Defocus heuristic when in doubt use focus when frustrated Defocus, walk around, change your perspective of viewing 

While we were walking out of the coffee shop he saw me looking at his Cap (those who know me, would know why :-D I wear Cap all the time ) and then he told me he give Cap to tester who work for.

I have been told by @jamesmarcusbach that only @Lalitbhamare @huibschoots @CuriousTester and @testertested have His branded Cap #SoJealous

— mubbashir (@mubbashir) September 15, 2015

That got some responses with respect to who else has one :D

One thing which made me quite happy was that he was aware of what we Software Testers from Pakistan are doing at the moment mostly because of my dear friend @arslan0644 

I told him briefly about STEP (Software Testers Engagement Programme) an initiative which bunch of my fellow testers took to bring closer the community

@arslan0644 I need to tell you all about my meeting with @jamesmarcusbach https://t.co/HbNsSg5hkq

— mubbashir (@mubbashir) September 12, 2015

Deleting facebook test users of your app

- Create a new user (We know how to do it, sending the key is sensitive so not adding here)

- You will get a response like:

{

• id: "137616849955555",
• access_token:"CAAAAJ6uP3yMBAC56lfzUlZB3uwWms0x6xXoriPn3ZAVko185KZCTxQaYADMqkXhpIH1VxAJtXzJBoNR7TNWKpRBoEAC79AY1VaGss93nHiqVGeZBEvHZALouNw55PfZBujt1Up1PbjEryWB4iptNPF3PDAQuZAup4jGGRFUJeEs6JBvIEpoBvCQZCuB467oznCoE41zDbxFaVGnURP1JdV9H",
• login_url: "https://developers.facebook.com/checkpoint/test-user-login/137616849955555/",
• email: "name_1441262612@tfbnw.net",
• password: "2073387096"

}

- Using the information from the above we can delete the user via the following:

https://graph.facebook.com/137616849955555?method=delete&access_token=CAAAAJ6uP3yMBAC56lfzUlZB3uwWms0x6xXoriPn3ZAVko185KZCTxQaYADMqkXhpIH1VxAJtXzJBoNR7TNWKpRBoEAC79AY1VaGss93nHiqVGeZBEvHZALouNw55PfZBujt1Up1PbjEryWB4iptNPF3PDAQuZAup4jGGRFUJeEs6JBvIEpoBvCQZCuB467oznCoE41zDbxFaVGnURP1JdV9H 

Update the above URL by replacing TEST_USER_ID with the value of id from the first response and TEST_USER_ACCESS_TOKEN with the value of access_token from the first response

https://graph.facebook.com/TEST_USER_ID?method=delete&access_token=TEST_USER_ACCESS_TOKEN

The response from it would be true.

Delete the test users once we are done. 

What is the most important skill a software tester should have?

What is the most important skill a software tester should have?

In the below video are the answers from a conversation with Tony Bruce at EuroSTAR 2011 last November.

Following are the answers:

0:33  Communication- Fred Beringer @fredberinger

1:14 Ability to Think

1:30 Analytical and Technical 

1:50 Objective look, think(+1) and present his/her thought - Fiona Charles @FionaCCharles

2:20 Communication (+1) 

2:30 Communication (+2) 

3:20 (depends) Critical and Through (for some domain)

3:32 Passion and Enthusiasm  

3:48 Knowing what battles to pick

4:14 To learn - (Andy Glover @cartoontester)

4:43 Has to have many (Curiosity, Intelligence, desire for better software/world  )

5: 25 Question every thing

5:32 Think out side the box (+2)

5:46 Think (+3) - Nancy Kelln @nkelln

6:00 Communication (+3) - John Stevenson @steveo1967

6:12 Not an agile tester (coz it has become a hype)- Stefaan Luckermans @stefaanl

(please let me know if you can identify anyone else or for incorrect identification(s) :D)

Thinking and Communication are the winner.

I believe testers should have technical skills, and thinking is pretty much required in any technical job (O.K. in almost any technical job) but see how many people are saying "Communication".

Communication is important, it's real important for a good tester and doesn't only include the formal, template filling kind of communication i.e. Test Plan, Test Cases, Session Reports, Bug Reports or any  other mode of sharing your findings, but informal kind of communication is equally important.

The informal chit chat with a developer or any other team member about the product, process bug or bug fix helps us a lot. Similarly how to be team member rather then acting like quality police, how to help other team members and how to seek help from other team members (yes any team member with any specific role, developers, DBAs, System Administrators included). 

Similarly we should be able to communicate with other testers to learn from them and we should be able to share our views and opinions about testing with other testers.

No matter how good we are, we can not work in isolation.

Server side monitoring in JMeter with JMeter Plugins

Like may I needed some server side statistics while running load tests earlier I had couple of Ruby scripts which were gathering my required information on server as CSV file I do have webmin configured on the server but I was still looking for something which is bit more tightly coupled with one of the most famous (easy to use) tool for load testing JMeter.

Here is how to do it:
We just needed one thing JMeter Plugins
 Prerequisite: JMeter 2.4+ or above with a JRE 1.6+
 Download JMeter Plugins 

• Unzip it and (refer) 
• Copy JMeterPlugins.jar to  JMETER_HOME/lib/ext. 
• Copy zip files content to the Server(s) you need to monitor
• On each server run JMeterPlugins-HOME/serverAgent/startAgent.sh PORT_NUMBER (refer)
• Run JMeter (create your test plan as usual)
• You will see bunch of extra items in Add context menue's item beginning with jp@..  these are the extra plugins we just added via JMeterPlugins.jar
• Add jp@gc - PerfMon Metrics Collector (refer)
• Configure your parameters (refer)

• Run your tests and see server side info

I have use it on Mac and Linux, it should be able to work with most of the systems since its server side agent is build on top of SIGAR - System Information Gatherer And Reporter